Tracing the whole chain: data, code, and figures

A figure in a paper is only trustworthy if you can say exactly what produced it: which data, from which simulation, with which parameters, analysed by which plotting script. Fieldnotes records that chain by capturing an immutable snapshot of the code for each step, and linking the steps through the files they share. Nothing needs to go inside your scripts.

The idea: code plus inputs, in and out

Every run you log is the same shape: some input files go in, a script runs, and some output files (data, figures, animations) come out. Fieldnotes snapshots the script and records the paths. It then traces provenance by matching one run's outputs to another run's inputs. You never write a link by hand; it is worked out from the file paths.

Two ways researchers work, both supported

Persona 1: analysis is separate. The simulation is one run; plotting happens later, often with several scripts reading the same data file.

[sim.py]     inputs: params        outputs: data.jld2
                 |
                 +-- data.jld2 --> [plotA.py]  input: data.jld2   figure: figA.pdf
                 +-- data.jld2 --> [plotB.py]  input: data.jld2   figure: figB.pdf
                 +-- data.jld2 --> [plotC.py]  input: data.jld2   animation: anim.mp4

one data file, many plotting scripts, each its own immutable record

Persona 2: one script does everything. A single driver computes the data and draws the figures in the same run, so its outputs are both.

[driver.py]  inputs: params   outputs: data.csv  +  figure: fig.pdf

one run; the data and the figure share the same script snapshot

Logging Persona 1

Log the simulation once. This snapshots sim.py and records the data file it produced:

fieldnotes log --script sim.py --output results/data.jld2 --tag sim

Later, log each plotting run, telling Fieldnotes which data it read with --input:

fieldnotes log --script plotA.py --input results/data.jld2 \
               --figure results/figA.pdf --tag plot

Because plotA.py's input is results/data.jld2, and that is the output of the sim.py run, Fieldnotes links them. Open the figure's run and you can step back to the data, and from the data to the simulation code that made it.

When you change a plotter

Edit plotA.py, run it again on the same, unchanged data file, and log it again:

fieldnotes log --script plotA.py --input results/data.jld2 \
               --figure results/figA_v2.pdf --tag plot

This creates a new run with a snapshot of the edited script. The data file was never touched, so its record is the same as before, and both plotting runs point back to it. Records are append-only, so you keep the full history: every version of every plotting script that ever touched that data, each with the exact code it ran.

Logging Persona 2

When one script does both, log it once with both kinds of output:

fieldnotes log --script driver.py --output results/data.csv \
               --figure results/fig.pdf --tag run

There is no upstream run to link to; the data and the figure share the one script snapshot, which is exactly right for this way of working.

No Fieldnotes code goes in your scripts

Fieldnotes never asks you to import a library or add a logging call to a script. You point the command line at whatever files you ran and produced, in any language, and it snapshots the code and records the paths as they stand. The only thing it cannot guess is which files were inputs and which were outputs, because it deliberately never reads your code, so you name them with --input and --output.

There is one optional exception, and it is worth being precise about what it actually asks of you. The manifest convention has your script add a couple of plain lines that write your own parameters to a params.json file beside its outputs. That JSON is your data, produced by an ordinary couple of lines you write and own, not a Fieldnotes import or a Fieldnotes call. Logging itself still happens as a separate step afterwards, with a single fieldnotes log --manifest command, often on another machine once the results have arrived. See the manifest convention for when this pays for itself and when a plain --param flag on the command line is simpler.

Seeing the chain

In the run detail (both fieldnotes show <id> and the local web dashboard, fieldnotes open), each run shows its script, its inputs with a link to the run that produced each, and its outputs and figures with links to the runs that consumed them. Follow the links and you walk the whole chain, from a figure in your paper back to the raw data run and the exact code that made every step.