Artifacts¶

Generated data lives under exports/ and is intentionally not versioned. The environment registry supplies the canonical root.

exports/<family>/<environment>/
exports/<family>/<environment>/<n_agents>/   # Gathering, Pursuit, Congestion

Examples:

exports/matrix/chicken/
exports/gridworlds/pursuit/2/
exports/matrix/congestion/6/

Common Layout¶

<root>/
├── wm/
│   ├── D.pkl
│   ├── legal_joint_graph.pkl
│   ├── legal_joint_graph.metadata.pkl
│   ├── world_model.pt
│   ├── env_transition_graph.pkl
│   ├── models.pkl
│   ├── wm_history.pkl
│   └── wm_report.txt
├── om/
│   ├── iop_stack.pt
│   ├── policy_after_imagination.pt
│   └── *_report.{json,txt}
├── shield/
│   ├── transition_shield_bundle.pkl
│   └── *_report.{json,txt}
├── true_shield/
│   ├── true_transition_graph.pkl
│   ├── true_transition_graph.metadata.pkl
│   └── exact_anytime/
│       ├── certificate.pkl
│       └── report.json
├── experiments/
│   ├── histories/*.pkl
│   ├── tables/*.csv.gz
│   ├── metrics/*.{json,csv.gz}
│   ├── diagnostics/*.{json,txt}
│   ├── checkpoints/*.pkl
│   ├── manifests/*.json
│   ├── traces/*.jsonl.gz
│   ├── runs/<tag>/...
│   └── *.png
├── timings.json
└── timings_hardware.json

Not every run creates every file. Baseline-only experiments do not need model or shield artifacts, and exact/learned graph outputs may be absent when a stage has not completed.

Cache Validity¶

Legal and true-transition graph caches have metadata sidecars. Learned shield bundles package metadata with the pickle. Identity checks cover relevant source semantics, environment dimensions, builder parameters, artifact modification metadata, focal agent, and value-iteration settings. Rebuild upstream artifacts after changing agent count, state layout, action semantics, safety labels, or graph construction.

The exact_anytime/ pair is created only by the opt-in arbitrary-opponent diagnostic. report.json is the portable paper-facing summary; certificate.pkl contains the checked state values, exact robust action risks, and worst-case opponent witnesses used by the strict runtime wrapper.

Histories and Figures¶

Canonical algorithm histories live in experiments/histories/ with names such as ippo_histories.pkl and ippo_shielded_histories.pkl. Loaders still accept some older flat history paths, but new exports use the subfolder.

The normal figure set includes cumulative and episodic views, post-WM timeline variants, no-legend copies, shield telemetry, Bayesian opponent-level views, and structured per-metric episode graphs. Separate legend assets live under exports/legends/.

Every completed algorithm condition also writes compressed episode tables, optimizer/update metrics, aggregate JSON/text diagnostics, final checkpoints, and a manifest. The manifest is published last, so report readers do not treat partially written condition artifacts as complete. Raw trace files appear only when verbose_data=true.

Tagged Campaigns and Reports¶

Setting experiment_tag redirects histories, diagnostics, checkpoints, manifests, and timings to experiments/runs/<normalized-tag>/. It does not overwrite the corresponding canonical stage outputs. Notebook-level aggregate plots still use the ordinary experiments/ plot paths, so concurrent tagged jobs for one environment can replace those derived images. Non-default shield action/opponent/reward conditions receive stable filename suffixes in either layout.

scripts/summarize_evaluation.py <tag> discovers tagged runs across export roots and writes the cross-environment summary to exports/reports/<tag>/. Only conditions with their final manifest are counted as complete.

Timing Sidecars¶

timings.json stores stage totals and per-run entries. Each entry references a deduplicated hardware record in timings_hardware.json. Hardware describes the resources available to the process: CPU affinity/scheduler allocation, enforced memory limit, and visible GPUs—not the whole physical cluster node.

Writers use per-file locks, merge stage keys with the latest file, and replace the JSON atomically so concurrent algorithm jobs do not discard one another’s records.