Recording (Python API)¶

Summarizes SPEC.md §4; SPEC.md is authoritative.

There are three ways to record, and they combine freely in one run:

Way	Code change	Use it when
`@vouch.track` on the function that computes the numbers	one decorator	the default. Each value records the call that produced it (function, every argument, call site), which is the strongest provenance vouch has
`[[track]]` in `vouch.toml` + `python -m vouch.exec`	none	the code can't or shouldn't import vouch: someone else's script, a frozen baseline, a quick look before committing to vouch
`vouch.record` / `vouch.record_all`	one line per value or per dict	a number that isn't a function's return value: an aggregate computed inline, a value read from a results file, a param

Zero-configuration use: an implicit run¶

import vouch

vouch.params(vars(args))                         # optional: parameters become citable values
acc = vouch.record("cifar.resnet.acc", evaluate(model), fmt=".1pct",
                   desc="top-1 test accuracy on CIFAR-10", better="higher")

The first vouch call outside a with vouch.run(...) block starts an implicit run. Its id is the entry script's project-relative path with / replaced by . and without the .py suffix, e.g. experiments.train.
The implicit run is finalized at interpreter exit, unless an uncaught exception occurred. A script that ends with sys.exit(n), n ≠ 0, is not detectable from atexit. Use an explicit run when the exit code matters.
record() returns the value unchanged, so it can be written inline.

Explicit runs¶

with vouch.run("cifar_resnet", params=args) as run:
    run.input("data/cifar10.npz")                       # file or directory; hashed
    accs = [train_and_eval(seed) for seed in range(5)]
    run.record("cifar.resnet.acc", vouch.Stat.of(accs), fmt=".1pct",
               desc="top-1 test accuracy on CIFAR-10, mean ± std over seeds", better="higher")
    run.claim("cifar.resnet.no_divergence", all(math.isfinite(a) for a in accs),
              desc="no seed diverged")
    run.artifact("results/cifar_resnet.csv")            # hashed at finalize, after it is written
    run.table("ablation", rows, row_key="variant", fmt={"acc": ".1pct"},
              highlight={"acc": "max"}, desc="ablation of augmentations")
    plt.savefig("paper/figs/curve.pdf")                 # tracked automatically as a figure artifact

Semantics:

Atomic. Nothing is written until the block exits cleanly. An exception, KeyboardInterrupt or SystemExit(n≠0) discards the run, and the previous record of this id is left intact.
A run id must match the key grammar. Re-running an id replaces its record. Two runs with the same id in one process: the second replaces the first, with a warning.
Runs cannot nest. Several sequential runs in one process are allowed, e.g. a sweep loop with one run per configuration.
Parameters. params= accepts a dict, an argparse.Namespace, a dataclass, or any object with to_dict()/to_container() (OmegaConf, Pydantic). Each is citable as <run>.param.<name>.
Command. sys.argv and the interpreter are recorded, which is what gives vouch status its exact re-run command.

A slow part and a quick one. A run is its whole block, so a quick change to a script re-ran its slow part too. Give each part a run of its own, with its body in a function: freshness is tracked per function (Python 3.12 and later; per file before, so use two scripts there), and each run records only the functions it executed, with whatever ran outside every run, so an edit inside one function leaves the other run fresh, whichever runs first. An edit to the module-level code around them changes the script's <module> unit, which both runs executed, and makes both stale. Then skip the slow run while its record stands:

def search():                     # the slow part: twenty minutes
    vouch.record("examples.search.left", count_left(), desc="...")

def examples():                   # the quick part: milliseconds
    vouch.claim("examples.cascade", table_matches(), desc="...")

if not vouch.fresh("examples_search"):
    with vouch.run("examples_search"):
        search()
with vouch.run("examples"):
    examples()

vouch.fresh compares parameters only when it is given them, vouch.fresh(id, params=args); it is for runs whose result their code, inputs and parameters determine. The keys that splitting a run moves to the new one keep their values and their script, so vouch build acknowledges them for you (moved).

A figure from a slow run's numbers. A figure drawn inside a measurement run is redrawn only by re-running the measurement, so relabelling one axis re-measured every timing. Draw it in a run of its own that reads the numbers with vouch.value(key), which returns what v[key] returns in a definition: a number, a Stat, a claim's bool, or a table's rows as dicts.

def measure():                    # the slow part: timings
    vouch.table("ablation", rows, columns=[...], row_key="id", desc="...")

def figure():                     # the quick part: the plot
    rows = vouch.value("ablation")
    ...
    plt.savefig("paper/figures/stages.pdf")

if not vouch.fresh("ablation"):
    with vouch.run("ablation"):
        measure()
if not vouch.fresh("ablation_figure"):
    with vouch.run("ablation_figure"):
        figure()

The run records what it read (reads in its record). It goes stale when a value it read moves, not when the measurement re-runs and moves only others, and it is upstream-stale while the measurement is stale.

Recording everything at once: `record_all`¶

metrics = evaluate(model, test_set)            # {"acc": 0.932, "loss": 0.211, "f1": {"macro": 0.90, "micro": 0.93}}
vouch.record_all(metrics, prefix=f"{dataset}.{model_name}")
# → cifar.resnet.acc, cifar.resnet.loss, cifar.resnet.f1.macro, cifar.resnet.f1.micro

Metadata comes from project-wide metric conventions defined once in vouch.toml:

[metrics]
"*.acc"    = { fmt = ".1pct", better = "higher", desc = "top-1 test accuracy, {1} on {0}" }
"*.loss"   = { fmt = ".3f",   better = "lower",  desc = "test cross-entropy, {1} on {0}" }
"*.f1.*"   = { fmt = ".3f",   better = "higher", desc = "{-1} F1, {1} on {0}" }
"*.time_s" = { fmt = ".0f",   unit = "s",        better = "lower", desc = "wall-clock training time, {1} on {0}" }

In a desc template, {0}, {1}, … are key segments and {-1} is the last one.

Signature:

vouch.record_all(values, *, prefix=None, fmt=None, desc=None, unit=None, better=None,
                 include=None, exclude=None, row_key=None, stats=False, table=None) -> values

Argument	Meaning
`values`	a dict (nested dicts are flattened with dots), a dataclass, a pandas Series or DataFrame, or a path to a `.json`, `.jsonl` or `.csv` file (read, recorded, and registered as an input of the run)
`prefix`	prepended to every key
`fmt`, `desc`, `unit`, `better`	a single value for all keys, or a dict mapping key or glob → value (`fmt={"*.acc": ".1pct"}`)
`include`, `exclude`	globs selecting which keys to keep (e.g. `exclude=[".runtime", ".epoch"]` for framework noise)
`row_key`	for DataFrames or lists of dicts: the column naming each row, giving keys `<prefix>.<row>.<col>`
`stats`	`True` turns lists of numbers into `Stat.of(list)` (per-seed results); by default a 2-element list is a range and longer lists are tuples
`table`	also register the DataFrame or list of rows as a table under this key, so one call gives both cell keys and a `\vouchtable`

Rules:

Metadata precedence: an exact-key argument, then the longest matching glob argument, then [metrics] in vouch.toml, then the type default.
Key sanitizing. / becomes . (so eval/accuracy → eval.accuracy). Other invalid characters become _. All renames are reported in one summary line.
Scalars are unwrapped. Single-element numpy arrays and torch tensors become scalars via .item().
Missing descriptions after all defaults are applied produce one aggregated warning.
record_all returns its input unchanged, so it can wrap an expression.

Sweeps and multi-seed experiments reduce to one call:

# a learning-rate sweep → sweep.<lr>.<metric> keys plus a citable table
vouch.record_all(sweep_df, prefix="sweep", row_key="lr", table="lr_sweep")

# per-seed lists → Stat values (mean ± std, n)
vouch.record_all({"acc": accs_per_seed, "loss": losses_per_seed}, prefix="cifar.resnet", stats=True)

The same single-step registration is available outside Python: vouch run ingests a program's whole $VOUCH_VALUES JSON, and vouch import registers an existing results file — see Non-Python experiments.

Recording a function's results: `@vouch.track`¶

Decorating an experiment function records what it returns on every call, keyed by the call's arguments:

@vouch.track(over="seed")
def evaluate(dataset: str, model: str, seed: int = 0, lr: float = 1e-3) -> dict:
    ...
    return {"acc": acc, "loss": loss}

for model in ("resnet", "vit"):
    for seed in range(5):
        evaluate("cifar", model, seed=seed)
# -> evaluate.cifar.resnet.lr_0_001.acc, .loss, evaluate.cifar.vit.lr_0_001.acc, ...
#    each a Stat over seeds 0..4

Keys are the function's name followed by every argument, defaults included, in signature order:

A string argument is written as its value (cifar).
A number, bool or None is written as name_value (lr_0_001, seed_3, flag_true).
An enum is written as its member name; a short list of simple values as layers_64-64.
Arguments that can't be written into a key (arrays, models, dicts, self) are left out of it and described in the call metadata instead.
Methods use the class-qualified name (Trainer.fit…).
Keys longer than 100 characters are shortened with a hash suffix, so they stay unique.
key="{dataset}.{model}" replaces the scheme with a template, checked against the signature when the function is decorated.

Results:

a number or Stat → the base key
a tuple → one key per element (.0, .1, … by position, or named with returns=("mean", "std"))
a dict or dataclass → one key per (nested) field
a DataFrame or list of rows → one key per cell
a list of numbers → one value (a series); its elements are citable as key.0, key.1, …
None → nothing (warned once)

@vouch.track(returns=("mean", "std"), desc="bootstrap accuracy ({-1})")
def bootstrap(model: str, n: int = 1000):
    ...
    return mean, std
# -> bootstrap.vit.n_1000.mean  "bootstrap accuracy (mean)"
#    bootstrap.vit.n_1000.std   "bootstrap accuracy (std)"

fmt, desc, unit, better, include and exclude work as in record_all, and so do [metrics] defaults.

over= names arguments whose calls are combined when the run ends. Numbers become a Stat (mean, std, n); a value that is the same on every call is kept as is; anything else is skipped with a note.

Every value records its call — function, args, over list, each seed's result and duration, and call sites — which is what makes vouch trace and vouch explore show the full per-seed breakdown.

Timing is a value too, by default. Every tracked call also records its duration as <key>.time, in seconds. time="walltime" names it differently; time=False leaves the value out.

Bookkeeping never breaks the experiment:

The result passes through unchanged.
An exception inside the function records nothing and propagates.
A recording problem is a warning, and an error only with VOUCH_STRICT=1.
Async functions are supported.

Tracking without touching the code¶

The functions to track can be listed in vouch.toml instead of decorated. The effect is the same as @vouch.track with the same options:

[[track]]
function = "experiments/train.py::evaluate"      # path::qualname; fnmatch patterns on both
over     = "seed"
desc     = "top-1 test accuracy"

[[track]]
function = ["models.py::Trainer.fit", "*::score_*"]   # a list is fine; no "::" means any file
key      = "{dataset}.{model}"                   # the same fields as the decorator: over key
returns  = ["mean", "std"]                       # returns time name fmt desc unit better include exclude
time     = false                                 # no <key>.time value (on by default)

$ python -m vouch.exec experiments/train.py --epochs 90     # the script doesn't import vouch
vouch: recorded run experiments.train: 12 value(s) → .vouch/runs/experiments.train.json

A script that doesn't import vouch runs under python -m vouch.exec script.py args…, which starts tracking before the script's first line. Only first-party functions can be listed, not library code; module bodies, lambdas and comprehensions never match.

API reference¶

Call	Purpose
`vouch.run(id, params=None, prefix=None)`	Context manager for an explicit run. `prefix` is prepended to every key recorded in the run.
`vouch.fresh(id, params=None)` → bool	Whether run `id`'s record still stands: recorded, complete and fresh (its code, inputs and artifacts unchanged; cosmetic or accepted count), with each of `params` equal to its recorded value. Lets a script skip a slow run whose result cannot have changed. Opt in only for a deterministic (seeded) run.
`vouch.record(key, value, *, fmt=None, unit=None, desc=None, better=None)` → value	Record a value. `better ∈ {"higher", "lower", None}`.
`vouch.record_all(values, *, prefix=None, fmt=None, desc=None, …)` → values	Record every value in a dict, DataFrame or results file in one call.
`vouch.claim(key, holds, *, desc=None, values=None)` → holds	Record a boolean claim.
`vouch.input(path, *, mode=None)`	Declare an input file or directory.
`vouch.value(key)` → value	What another run recorded under `key`: a number, a `Stat`, a claim's bool, a table's rows as dicts. The read is recorded, so the run goes stale when that value moves. A key no run records, or one this run records itself, raises `KeyError`.
`vouch.artifact(path, *, kind=None)`	Declare an output file. `kind ∈ {"figure", "data", "model", "file"}`.
`vouch.table(key, data, *, columns=None, row_key=None, fmt=None, highlight=None, desc=None)`	Record a table.
`vouch.params(obj)`	Set the implicit run's parameters.
`vouch.expect(key, *, desc, producer=None, fmt=None, unit=None, better=None)`	Declare a placeholder.
`vouch.Pending`	Raised when a definition in `vouch_values.py` reads a pending key; `keys` names them. Let it propagate, and the derivation becomes pending too.
`vouch.Stat.of(samples)` / `vouch.Stat(mean, std, n, *, min=None, max=None)`	A summary statistic. `std` is the sample std (ddof = 1). `ci95` uses Student-t quantiles.

Module-level functions act on the active run: the explicit one if inside a with, otherwise the implicit one. run.<method> is the same function bound to a specific run.

Value types¶

Python value	Stored `type`	Default rendering
`bool`	`bool`	`true`/`false` (usually cited through claims)
`int` (incl. numpy ints)	`int`	`d`
`float` (incl. numpy floats)	`float`	`format.default_float` (`.3g`)
`str`	`str`	LaTeX-escaped text
`tuple`/`list` of numbers	`tuple`	`a--b` for 2 elements (a range), comma-joined otherwise
`Stat`	`stat`	`mean \pm std`

Citing part of a value. A Stat exposes key.mean, key.std, key.n, key.min, key.max and key.ci95. A tuple value (up to 64 elements) exposes each element by position: vouch.record("ci", (0.91, 0.95)) makes \vouch{ci.0} and \vouch{ci.1} citable.

Run record format (.vouch/runs/<id>.json)

The file is deterministic: keys are sorted, and every map is written one entry per line, so a changed value is a one-line diff.

{
  "schema": "vouch/1",
  "run": "cifar_resnet",
  "status": "complete",
  "entry": "experiments/train.py",
  "command": ["python", "experiments/train.py", "--model", "resnet50", "--seeds", "5"],
  "params": {"model": "resnet50", "optim.lr": 0.1, "seeds": 5},
  "started": "2026-09-12T14:03:11Z",
  "duration_s": 3412.5,
  "git": {"commit": "0fdc530e7c1b9f6a2d8e4b3c5a7f9e1d2c4b6a8f", "dirty": false},
  "env": {"python": "3.13.5", "platform": "win-amd64", "packages": {"numpy": "2.1.0", "torch": "2.5.1"}},
  "code": {
    "granularity": "function",
    "units": {"experiments/train.py::main": "41d0e7c2aa9b1f05"},
    "files": {"experiments/train.py": "e1b2c3d4e5f6a7b8"}
  },
  "inputs": {"data/cifar10.npz": "sha256:5f1c…"},
  "values": {
    "cifar.resnet.acc": {"type": "stat", "value": {"mean": 0.93214, "std": 0.0041, "n": 5},
                         "fmt": ".1pct", "desc": "top-1 test accuracy on CIFAR-10", "better": "higher",
                         "site": "experiments/train.py:88"}
  },
  "claims": {"cifar.resnet.no_divergence": {"holds": true, "desc": "no seed diverged"}},
  "artifacts": {"paper/figs/curve.pdf": {"hash": "sha256:a41e…", "kind": "figure"}},
  "record_hash": "sha256:3c9e…"
}

code.units holds semantic hashes, truncated to 16 hex characters. record_hash is SHA-256 over the canonical JSON of the record without that field; vouch check recomputes it and reports store-edited on a mismatch.