Skip to content

OmSh Evaluation and Optimisation Campaign

This directory records the staged evaluation that compares OmSh with fresh-policy IPPO, IPPO-Lagrangian, and plain CPO runs after the reward-aware and uncertainty-mode shield changes.

The campaign is deliberately staged:

  1. pilot the existing two-agent environments at 250,000 policy-training steps and three seeds;
  2. reject broken or clearly dominated configurations before spending on longer runs;
  3. tune only from aggregate evidence, never from a particular seed;
  4. scale promising conditions to longer horizons and more seeds;
  5. run expensive action/budget and uncertainty-mode ablations after the main condition is stable.

The primary OmSh condition is the pure-action, egalitarian-budget transition shield with monotone_floor opponent handling and Bayesian reward ranking. Parameterized mixed-action and learned-budget variants are implemented in every standard environment notebook and have entered targeted tuning. The learned-budget/pure Inspection condition completed a paired holdout, while the tested Gathering, Bertrand, Chicken, and Congestion parameterized variants were not promoted. A same-seed standard-OmSh control confirms that the Inspection variant improves late reward while preserving zero observed late failures, though it has a small transient whole-run safety cost. The dated measurements and qualification are recorded in the result log below.

Shield experiment artifacts are condition-aware. The canonical main condition uses ippo_shielded_* names, while non-default opponent/reward modes and explicit non-v0 shield versions receive deterministic filename, timing, checkpoint, diagnostic, and plot suffixes. This prevents sequential ablation jobs from overwriting the main result.

Set experiment_tag for pilot and tuning campaigns. The shared JAX stage runner normalizes the tag and writes histories, checkpoints, diagnostics, manifests, and timings below experiments/runs/<tag>/. The untagged canonical exports remain untouched, and all algorithms in a tagged comparison share the same isolated run directory.

Run uv run python scripts/summarize_evaluation.py <tag> after or during a campaign to generate JSON, Markdown, and two paper-ready LaTeX diagnostic tabulars under exports/reports/<tag>/. The LaTeX outputs separate shield control metrics from opponent-model and credible-floor metrics so the appendix can remain readable. The reader aggregates episodes within each seed before averaging seeds, reports whole-run and final-window reward and observed safety, uses paired seeds for OmSh-minus-baseline comparisons, and enforces each environment manifest's max_risk against both per-step violations and unsafe-episode frequency in the worst OmSh seed and evaluation window. The command-line ceiling is only a fallback for legacy manifests. Shield risk, budget, coverage, intervention, calibration, and action-availability values remain diagnostic fields in the JSON output; they do not replace observed violations.

The report reader recognizes every completed ippo_shielded_*_histories artifact, including mixed-action, learned-budget, and uncertainty/reward-mode variants. A tuning-only tag with no baseline histories is intentionally marked incomplete, but its OmSh metrics, worst-seed safety gate, and runtime diagnostics are still rendered. Paired baseline deltas are emitted only when the same tagged run contains matching baseline seeds.

Tagged diagnostics tables and JSON files are published by atomic same-directory replacement. The report reader additionally requires the per-condition manifest, which is written last, before treating a condition as complete. This prevents monitoring from interpreting a table that is still being exported as a zero-episode run.

The separate frozen-policy campaign reporter keeps every discovered environment in its JSON and Markdown audit, including partial or failed conditions. Its paper-ready LaTeX result and timing tables are stricter: an environment is rendered only when all four primary families have all expected seeds and the same largest evaluation horizon. This prevents completed baseline shards or mismatched evaluations from becoming a misleading partial paper row.

The paper-ready world-model diagnostics similarly require a discovered OmSh condition. Baseline-only partial environments remain in JSON and Markdown but cannot contribute a standalone learned-model row to the paper.

Set seed_offset to reserve disjoint seed blocks for tuning and final evaluation. A run with seed_offset=100 and num_runs=3, for example, uses trainer seeds 100, 101, and 102 while retaining local run indices 0, 1, and 2 for paired comparisons. The default remains zero for existing experiments.

See assumptions.md for decisions that need validation or eventual human review. The impact-ordered implementation register is maintained in changes.md. The active pilot is recorded in eval-250k-v1.md. Its result table and follow-up decisions are recorded in eval-250k-v1-results.md. The consolidated cross-environment outcome is in final-status-2026-08-12.md. The safety-qualified all-algorithm comparison and follow-up diagnoses are in reward-safety-report-2026-08-12.md. The fresh five-seed paper campaign and its frozen-policy interpretation are in paper-final-results-2026-08-14.md.