Analysis and Audits¶

The scripts in scripts/ analyse artifacts that training already produced. None of them trains a policy. Run them from the repository root, because cached bundles record repository-relative input paths. Guarantees and Robustness explains what each certificate and audit measures; this page explains how to run them.

Most audits default to the campaign tag successor_campaign_v1. Pass --tag for any other tagged condition. Every script accepts --help.

Certificates Without Training¶

These scripts read cached shield bundles and exact graphs.

Script	Output	Purpose
`compute_robust_margins.py <env...>`	`exports/robust_margin/<env>.json`	Nominal reset demand, the largest padding radius that still fits the budget, and learned-versus-exact row TV, per role and floor.
`summarize_robust_margins.py --output <tex>`	LaTeX table	Margin table; multi-role environments report the binding role.
`pursuit_rate_radii.py`	`exports/robust_margin/pursuit_rate_radii.json`	Per-row Pursuit world-model radii from a confidence box on its slip rates, and the padded demand they imply.
`compute_mixed_values.py <env...>`	`exports/mixed_values/<env>.json`	Pure versus randomised certificate values and the permissiveness loss of the pure shield.
`summarize_exact_anytime_diagnostics.py`	Cross-environment summary	Collects every `true_shield/exact_anytime/report.json`.

uv run python scripts/compute_robust_margins.py pursuit gathering
uv run python scripts/summarize_robust_margins.py --output exports/robust_margin/margins.tex
uv run python scripts/compute_mixed_values.py bertrand chicken congestion pursuit

The margin and mixed-value solvers bracket every value with sound lower and upper iterates, so a reported feasibility or infeasibility is decided, not estimated. The exact-world-model comparison reads shield/oracle_iop_transition_shield_bundle*.pkl, which the learned-shield setup writes when true_shield/true_transition_graph.pkl already exists. Without it, the margin script reports oracle_status: missing and skips the comparison. On the largest graph (Markov Stag Hunt), allow about 16 GB of memory.

Audits of Frozen Deployments¶

These replay a tagged condition's frozen evaluation exactly: they rebuild its shield from the completion manifest, restore the post-training controller snapshot, and use the same seeds and training-time action masks.

Script	Output	Purpose
`measure_opponent_level_gap.py`	`exports/opponent_level_gap/`	Closest-retained-level opponent gap per step, and how often the floor excluded the closest level.
`audit_certificate_dominance.py`	`exports/certificate_dominance/`	Step excess `eta_t` of the committed certificate under the true next-state law.
`combine_dominance_shards.py`	One combined report	Merges per-tag audits of a seed-sharded campaign.
`summarize_dominance_audit.py --output <tex>`	LaTeX table	Steps audited, steps with excess, and the plug-in bound per role.

uv run python scripts/measure_opponent_level_gap.py pursuit --tag <tag>
uv run python scripts/audit_certificate_dominance.py \
  --environments bertrand,chicken,pursuit,congestion --tag <tag>
uv run python scripts/audit_certificate_dominance.py \
  --environments pursuit --tag <tag> --kernel learned

--kernel learned swaps the learned world model in for the exact one, which separates world-model error from opponent error. For Congestion at a fleet size other than two, pass --n-agents N and the coordinated artifact stem, for example --artifact-stem ippo_shielded_version_agents_0_1_2_histories.

A seed-sharded campaign keeps each seed's checkpoint in its shard tag, so audit each tag separately (the merged destination with --runs 0, then every shard tag) and combine the reports in seed order:

uv run python scripts/combine_dominance_shards.py \
  <dest_report> <shard1_report> <shard2_report> ... --output <combined.json>

Both audits need true_shield/true_transition_graph.pkl; run the TrueShielded stage or the exact-anytime diagnostic first to build it. They default to 200 episodes per run, a prefix of the campaign's own evaluation.

The opponent gap can also be logged during training with log_opponent_level_gap=true; the per-agent values land in metrics/<stem>_updates.json.

World-Model and Ambiguity Studies¶

Script	Purpose
`wm_variant_study.py`	Stage 1: fits every kernel estimator on the same buffer and scores it against the exact graph.
`summarize_wm_variants.py --tag <tag>`	Applies the predeclared adoption gate and writes the comparison table.
`certified_ambiguity_study.py`	Stage 2B: certificate width and feasibility of count-centred ambiguity sets per sample budget and `delta_cov`.
`summarize_certified_ambiguity.py`	Budget-by-coverage table of width and feasibility.
`attribute_certificate_infeasibility.py`	Splits an infeasible certified demand into structure, estimation, and ambiguity.
`summarize_attribution.py`	Cross-environment attribution table.

The cluster launchers wm_variants.sh and certified_ambiguity.sh fan these out over environments and seeds. The Stage 1 harness defaults reproduce the original gate run; pass --selection-metric objective and --min-optimizer-steps 2000 to probe the shipped successor protocol. See the variant study and ambiguity study notes.

Campaign Reports and Paper Tables¶

Script	Purpose
`summarize_evaluation.py <tag>`	Training-window reward and safety across environments, with paired baseline deltas.
`summarize_final_policy_campaign.py --tag <tag>`	Frozen-policy campaign tables; see Final-Policy Evaluation.
`summarize_scalability.py --output <tex>`	Problem size, per-stage wall-clock, and OmSh outcome per agent count, with TrueShielded and best gate-passing baseline columns. Missing cells render as `//`.
`compare_final_policy_campaigns.py`	Two campaigns that differ only in the world model; separates the model effect from reproduction noise.
`summarize_padded_runs.py --output <tex>`	Padded-shield runs against the campaign OmSh runs on the same seeds.
`summarize_patience_screen.py`	Floor-patience arms, first-raise steps, final floors, gaps, and the predeclared promotion rule.
`summarize_replacement_ablation.py`	Parameterized shield variants against pure replacement.
`decide_replacement_screen.py`	Applies a predeclared promotion rule to a paired screen.
`summarize_reward_inference_ablation.py`	Observed versus inferred private-reward conditions.
`export_paper_figures.py --tag <tag>`	Re-renders the paper's learning curves, legends, and floor-progression panels from saved histories.

Table generators take environment names and row order from PAPER_ENVIRONMENT_LABELS in notebooks/utils/evaluation_report.py, so generated .tex files go into the paper unedited.

Predeclared Screens¶

Comparisons that could change the primary condition follow a fixed procedure:

Write the arms, seeds, and decision rule into goal.md and a protocol note before any result exists.
Run the screen on seeds disjoint from the campaign, each arm under its own tag.
Apply the rule with the matching script, without editing it to fit.
Promote only on a pass, and rerun a promoted variant on the campaign seeds. Otherwise report it as an ablation.

The floor-patience screen and sweep, the mixed-replacement screen, and the Stage 1 world-model gate all followed this procedure. A failed screen is a valid result.

Benchmarks¶

benchmark_congestion_scalability.py --n-agents {2,4,6,8} --output <json> measures exact graph size, opponent-model fit and likelihood time, held-out joint metrics, parameter count, and peak memory. The eight-agent run requires --skip-graph, which guards against building the projected 86-million-edge graph by accident.