Integrations

Logging cluster runs with ClusterPilot

ClusterPilot is a sibling tool that submits and manages your cluster jobs. If you use it, it can also log every finished job to Fieldnotes for you, with no extra command to run. This is the fully automatic end of the logging spectrum: you turn on one setting, and your cluster runs record themselves. The two tools are independent, though. Fieldnotes never requires ClusterPilot, and ClusterPilot never requires Fieldnotes.

How it works

It is the same manifest convention as everywhere else, driven for you. When a job finishes and its results have synced back to your workstation, ClusterPilot runs Fieldnotes' own command-line tool for you, once, single-writer:

fieldnotes log --manifest <results dir> --slurm-job-id <job id> --tag clusterpilot

There is no network call and no account: ClusterPilot shells out to your local fieldnotes install, which writes to the SQLite database in ~/.fieldnotes exactly as if you had run the command by hand. If fieldnotes is not installed, ClusterPilot simply does nothing, it never fails your job over logging.

Turn it on

The integration is off by default. Enable it in ClusterPilot's config with a single setting, and optionally name the Fieldnotes project the runs should belong to:

[fieldnotes]
enabled = true
project = "spin-glass"   # optional; defaults to the working directory name

You also need the fieldnotes command on your PATH (pipx install fieldnotes). From then on, both the ClusterPilot daemon and its manual sync log completed jobs automatically as their results arrive.

Let ClusterPilot's AI write the manifest for you

For the logging to have parameters to record, each task still needs to drop a params.json beside its outputs (see the manifest convention). When the Fieldnotes integration is on, ClusterPilot's AI script generator does this part too: it writes the params.json-emitting lines straight into the SLURM scripts it generates from your plain-language description, so the manifest appears with no hand-editing. If you write your own scripts, add the couple of emitter lines yourself, or let your own assistant do it with the AI logging artefacts.

What gets recorded

ClusterPilot passes the job's results directory, its SLURM job id, and a clusterpilot tag; each task's scientific parameters come from its own params.json, and any per-task cluster metadata from an optional run.json. The result is an ordinary Fieldnotes record per task, queryable like any other: fieldnotes show --tag clusterpilot, or fieldnotes show --param seed=7, or by output file. Nothing about the record depends on ClusterPilot after it is written.

See also

Want to know when Fieldnotes launches?