The manifest convention

Some runs are best logged one at a time, right after they finish, with a fieldnotes log command you type yourself. Others are not: a hundred-task SLURM array, or a script that finished overnight on a machine you are not sitting at right now. For those, Fieldnotes has a manifest convention: a run drops a small file describing itself, and logging happens later, separately, in one batch.

What a manifest is

A manifest is a directory holding one run's metadata, written beside its outputs as a side effect of the run itself:

• params.json (required): the parameters object. This alone is enough for a simple run, and it is what fieldnotes show --param queries.
• run.json (optional): run metadata such as SLURM ids or tags, mapped onto the record's columns. See the reserved keys below.

Logging from a manifest exists for two reasons. First, bulk: an array job produces dozens or hundreds of these directories, and one command should ingest all of them rather than one command per task. Second, timing: the data often has to sync back from a cluster to a workstation before anyone logs it, so the parameters need somewhere to wait that travels with the data, rather than living only in your terminal history.

The one place you touch your script

A manifest has to come from somewhere, and that somewhere is a couple of plain lines you add to your own script. This is the only place the manifest convention touches your code, and what it adds is not a Fieldnotes call: it is your own parameters, gathered into one object and written out as JSON, the same way you might already write a results file.

grid, seed, method = 64, 7, "baseline"
data = simulate(grid, seed, method)
save("results/data.csv", data)

becomes:

import json

params = {"grid": 64, "seed": 7, "method": "baseline"}   # your parameters
with open("results/params.json", "w") as f:
    json.dump(params, f)

data = simulate(params["grid"], params["seed"], params["method"])
save("results/data.csv", data)

Nothing here imports Fieldnotes or calls it. Fieldnotes reads the resulting params.json afterwards; it never reads the script that wrote it. And this whole convention is optional: for a single run on your own machine, skip the manifest entirely and pass --param grid=64 --param seed=7 ... straight on the command line. The manifest earns its keep once parameters are numerous, hardcoded, or need to survive a trip to another machine.

Logging it: separately, once, single-writer

Once the manifests exist, ingest them with fieldnotes log --manifest, pointing at a directory, a params.json path, or a glob matching many task directories:

fieldnotes log --manifest results/12345/*/ \
  --slurm-job-id 12345 --tag job-12345

This is one call, one writer, one record per manifest, run on the workstation once the results have arrived rather than from each compute node as it finishes. That matters on a shared cluster filesystem: many nodes writing to the same SQLite file at once is exactly the kind of concurrent write that corrupts it. Flags like --slurm-job-id and --tag apply to every record the call produces; a scalar flag wins over the same key in run.json, and tags are combined with any tags already in run.json.

fieldnotes log is append-only and not idempotent, so ingest each batch of manifests once. Running the same --manifest call twice creates a second set of records, not an update to the first.

Reserved run.json keys, at a glance

run.json is optional, and every key inside it is optional too. Fieldnotes recognises:

• slurm_job_id, slurm_array_task_id: cluster ids.
• walltime_used: seconds, as a number.
• timestamp: the original run time, a date or ISO 8601 datetime.
• tags: an array of strings.
• notes: free text.
• outputs: an array of output paths, relative to the manifest directory. Left out, Fieldnotes treats every regular file in the directory as an output, except params.json and run.json themselves.
• inputs: an array of input paths this run consumed, relative to the manifest directory. This is what lets a plotting run's manifest link back to the run that produced its data, the same way --input does on the command line; see tracing the whole chain.

A manifest is just a slower, batchable way of saying the same things you would otherwise type after fieldnotes log. Nothing about how a run is found, filtered, or chained to other runs changes because it was logged this way.