Logging runs

Logging array jobs safely

A SLURM or PBS array can spin up dozens or hundreds of tasks at once, each writing its own slice of a result. Fieldnotes handles this well, but only if the tasks themselves never call fieldnotes log. This guide covers the safe pattern: each task writes a small manifest describing itself, and the whole array is logged afterwards in one call, from one machine.

Do not log from the compute node

Fieldnotes stores its records in one SQLite database file. SQLite tolerates one writer at a time; it does not tolerate dozens of array tasks opening it at once over a shared cluster filesystem. Running fieldnotes log from every task as it finishes is exactly that: concurrent writes to a single file over a network filesystem, which can corrupt the database. Never call fieldnotes log from the compute nodes of an array job.

Step 1: each task writes its own manifest

Instead, have every array task emit a small manifest beside its own outputs, on the compute node, as a side effect of the job itself: a params.json with that task's parameters and, optionally, a run.json with its SLURM ids. Nothing here calls Fieldnotes; it is a couple of plain lines that write out data the task already has. One directory per task keeps everything tidy:

results/12345/1/params.json   {"size": 32, "seed": 1, "method": "baseline"}
results/12345/1/run.json      {"slurm_job_id": "12345", "slurm_array_task_id": "1"}
results/12345/1/data.out
results/12345/2/params.json   {"size": 64, "seed": 2, "method": "baseline"}
results/12345/2/run.json      {"slurm_job_id": "12345", "slurm_array_task_id": "2"}
results/12345/2/data.out
...

This is the same manifest convention used everywhere else in Fieldnotes; see the manifest convention for the full picture of params.json and run.json. The repo's examples/slurm_array.sh is a ready-to-adapt SLURM array template that writes exactly this per-task manifest and then runs the computation; copy it and change the paths, the parameter mapping, and the compute command.

Step 2: sync back, then log the whole array in one call

Once the array finishes and the results have synced back to a single machine (your workstation), log the whole batch in one single-writer call with fieldnotes log --manifest, pointing it at a glob that matches every task directory:

fieldnotes log --manifest results/12345/*/ \
  --slurm-job-id 12345 --tag job-12345

--manifest also accepts the flag repeated, if you would rather list task directories explicitly than glob them:

fieldnotes log --manifest results/12345/1/ --manifest results/12345/2/

Each --manifest PATH becomes one new run record. It does not search for, attach to, or update an existing run: pointing at the same directory twice creates two records, not one updated record. The examples/log_runs.sh template wraps this single-writer call, and keeps a plain per-task loop as a fallback for older Fieldnotes versions without --manifest.

Cluster metadata travels with run.json; scalar flags apply to the whole call

run.json's reserved keys are how the per-task cluster metadata reaches the record: slurm_job_id, slurm_array_task_id, walltime_used, tags, notes, outputs, and inputs. A scalar CLI flag such as --slurm-job-id or --tag is applied to every record the call produces, not just one; it wins over the same key in run.json where both are present, and --tag is combined with any tags already listed in run.json.

Ingest each array exactly once

fieldnotes log is append-only and deliberately not idempotent. Running the same --manifest command a second time does not update the first set of records; it creates a second, duplicate set. If a log_runs.sh run gets interrupted partway, check fieldnotes show --tag job-12345 before re-running it, so you know whether some tasks are already logged.

Use params.json, not a custom-named file

--manifest also accepts a path straight to a parameters file rather than a directory, which is useful if your task writes something like task_params.yaml instead. It works, but that file then gets scanned in as one of the run's own outputs, alongside the real data, since Fieldnotes cannot tell a custom-named parameter file from an output file it should link. The params.json plus directory convention avoids that: Fieldnotes recognises the filename and excludes it from the outputs it records.