Skip to content

Experiment Export Layout

Experiment notebooks keep pickled histories separate from rendered plots, but plot variants now stay flat within each experiments/ directory.

Layout

Under each environment export root, experiment artifacts live beneath exports/.../experiments/ with this structure:

  • histories/*.pkl for saved training histories
  • tables/*_episodes.csv.gz and tables/*_om_level_events.csv.gz for reviewer-facing episode and Bayesian-level records
  • metrics/*_updates.{json,csv.gz} for optimizer, mask, and policy-drift summaries
  • diagnostics/*.{json,txt} for condition aggregates and calibration reports
  • checkpoints/*.pkl for final policy and optimizer state
  • manifests/*.json for last-published completion markers
  • traces/*.jsonl.gz for bounded raw traces when verbose_data=True
  • *.png for the default plot image with the shared legend included
  • *_no_legend.png for the legend-free companion

Current plot mappings are:

  • cumulative.png and cumulative_no_legend.png
  • cumulative_post.png and cumulative_post_no_legend.png
  • episodic.png and episodic_no_legend.png
  • episodic_post.png and episodic_post_no_legend.png
  • shield_telemetry.png and shield_telemetry_no_legend.png
  • om_bayesian_levels*.png and their no-legend companions

Episodic plots additionally produce structured single-metric images beneath graphs*/. The manifest for a condition is written after its tables, diagnostics, metrics, checkpoints, and histories; readers must require it before considering the condition complete.

Tagged and condition-aware layouts

When experiment_tag is non-empty, run_jax_stage(...) redirects the condition's history, tables, metrics, diagnostics, checkpoints, manifest, and timing sidecar to experiments/runs/<normalized-tag>/. This isolates durable pilot, tuning, and holdout data from the canonical stage layout. seed_offset selects the trainer seed block while preserving local run indices for paired comparisons.

The retained experiment notebooks resolve a separate plot_exp_dir using the same normalized tag. Their just_vis cells load baselines and every complete ippo_shielded*_histories.pkl condition from that directory, use manifests to recover collision-free OmSh labels, and write the aggregate figures alongside the tagged artifacts. This keeps OmSh-Monotone and OmSh-Credible on the same campaign plot instead of rendering one mode into a suffixed figure or silently falling back to untagged histories. The shared world-model prefix is prepended to both OmSh curves when a with-pretraining view is requested.

Timing exports use the same tagged directory. Each OmSh condition has its own setup and policy-training stage name, and timing writes merge under a file lock so concurrently finishing algorithm jobs do not replace sibling conditions.

The default pure-action, monotone-floor, Bayesian-reward, v0 shield is published as OmSh-Monotone and retains the canonical ippo_shielded_* artifact names. Credible-floor conditions are published as OmSh-Credible and include their schedule and delta in suffixed artifact names. Parameterized action modes, non-default opponent or reward modes, credible-floor parameters, and explicit non-v0 shield versions receive deterministic suffixes. They therefore coexist inside the same tagged or untagged directory.

Cross-environment tagged summaries are written separately beneath exports/reports/<normalized-tag>/ by scripts/summarize_evaluation.py. In addition to evaluation.{json,md}, the report includes omsh_runtime_diagnostics.tex and omsh_opponent_diagnostics.tex for compact paper integration.

This keeps the filenames self-describing without introducing extra with_wm, post, legend, or no_legend directories beneath experiments/.

The shared matplotlib save helper creates parent directories on demand before writing plot exports, so just_vis runs can repopulate an environment's experiments/ directory from histories alone.

The deployed cpucluster checkout uses an untracked symlink ~/Projects/omsh/exports -> /vol/gpudata/oja24-omsh/exports. Git worktrees do not inherit untracked files or symlinks. A notebook launched from a newly created worktree will therefore interpret relative exports/... paths inside that worktree unless the operator explicitly installs and validates the same symlink first.

This matters for exact graph construction: a three-agent Pursuit legal graph can exceed the home quota before atomic publication. Before launching a worktree-backed cluster job, verify both readlink -f <worktree>/exports == /vol/gpudata/oja24-omsh/exports and that the tracked worktree is clean. Prefer the clean main checkout when no source-pinned worktree is required. An EDQUOT failure during graph publication is not a valid cache completion; retain the old cache, correct the export target, and rerun so the atomic writer can publish on /vol/gpudata.

The canonical three-agent Pursuit subtree is currently isolated through /vol/gpudata/oja24-omsh/exports/gridworlds/pursuit/3 -> /vol/gpudata/oja24-omsh-pursuit3/exports/gridworlds/pursuit/3. Its legal graph temporarily needs both the previous cache and the atomic staging file, which does not fit in the main omsh workspace once the other retained exports are present. Keep the whole environment subtree behind this link so WM, OM, shield, algorithm, and report jobs resolve one consistent artifact root.

The three-agent enumeration reached 85,184 states and 415,886,464 edges, but its atomic pickle staging file reached 48,704,852,093 bytes before the 205 GiB job was OOM-killed; it is not a completed graph cache. A known-unreadable 10.9 GB cache plus the other Pursuit artifacts can therefore exceed even a fresh 50 GB workspace while the replacement is staged. Before rebuilding a cache that has already failed load_graph, preserve it outside the constrained workspace (the August 2026 recovery uses /vol/gpudata/oja24-optimisations/exports/omsh/recovery/pursuit3) or remove it only with explicit authorisation. Do the transfer on a scheduled compute node when it requires bulk I/O, then verify that the graph path is absent and the workspace has at least the full staging size free before resubmission.

History Loading

run_jax_stage(...) now saves histories to experiments/histories/.

History loads should request the explicit histories/*.pkl path. The shared load_from_file_if_exists(...) helper no longer performs a legacy flat-file fallback from ippo_histories.pkl to histories/ippo_histories.pkl.

Standalone Legend Notebook

notebooks/generic/legend_mo.py exports five standalone legend variants into exports/legends/:

  • the four-condition primary suite
  • all algorithms
  • all algorithms except CPO
  • all algorithms except CPO and true dynamics
  • all algorithms except true dynamics

Each of those also has a longer_ companion export that keeps the legend on a single horizontal row instead of wrapping onto a second line.

Wrapped standalone legends are rendered as one centered legend per row. This keeps shorter final rows, such as the three-item second row in legend_all_algos.png, visually centered beneath the full first row instead of left-aligned within a four-column legend box.

Examples:

  • legend_primary.png
  • longer_legend_primary.png
  • legend_all_algos.png
  • longer_legend_all_algos.png