Skip to content

Paper data audit (15 Oct 2026)

The user asked that "every data point in the paper is accurate". Every table and every number in the prose and captions of ../omsh-paper (at 9148384 onwards) was checked against the AAMAS archive /vol/bitbucket/oja24/_archive/omsh/aamas/ (then aamas_2026_08_10/), the clean timing archive and the code. Fixes are in omsh-paper 40d64dc (timings), 761b77b and 41d3bb7.

Method

  • Strongest check: re-run the repo's own generator on archived inputs copied locally, and diff the data rows against the paper. Header and label wording and \emph markers are ignored.
  • Generators: summarize_final_policy_campaign.py, summarize_evaluation.py, summarize_scalability.py, summarize_certified_ambiguity.py, summarize_dominance_audit.py, summarize_wm_variants.py, compare_final_policy_campaigns.py, summarize_patience_screen.py and summarize_replacement_ablation.py.
  • Otherwise: read the archived JSON or CSV, or the manifests (algorithm_config is exactly what each trainer received), checkpoint configs and executed notebooks.
  • Rounding rule: a value is correct if it equals the source rounded to the displayed precision. Upper bounds must round up.

Sources worth knowing

  • WM-variant table. It comes from bitbucket/reports/wm_variant_stage1_objective (the amended protocol, selecting on the full objective), not wm_variant_stage1 (the original gate run).
  • WM ablation.
  • The product-Bernoulli side is bitbucket/exports-bitmlp/.../bitmlp_as_trained_v1; rebuild its campaign summary from the per-run reports.
  • Its Congestion value (0.9807) is August's run re-evaluated as trained (job 9183). That job overwrote the per-run reports under gridworlds/convention_merge/2/.../convention_merge_gate_confirm_v1_joint. The archived summary in reports/ (0.9816) predates it.
  • Budget ablation. The 5-seed cells are under cpucluster/exports/ablations/<env>/<cell>; count unsafe episodes from tables/*_episodes.csv.gz, column unsafe_episode.
  • Credible-floor paragraph. From reports/credible_vs_monotone_2026_08_v2; the MSH paired interval comes from that tag's per-run reports.
  • Opponent-model hyperparameters. Read cfg from each deployed om/iop_stack.pt with torch.load(..., weights_only=False). Epochs are only in the executed notebooks.
  • IOP example figure. Graph state indices changed between the August and deployed Pursuit graphs. The figure's guard (5,4) and intruder (4,6) state is index 1168 now, not 120. It is rendered with make_level_action_diagram(..., cfg=LevelVizConfig(dpi=407, remove_max_coords_exceeds_per_channel=1)) to match the original's size and summary.

Fixed

  • Timings: from the clean re-run (Congestion OM 0.327, which the double count had made 0.524). The hardware and timing-convention paragraphs were rewritten.
  • Oracle refresh interval: 1 update in Bertrand, Chicken and Pursuit, 64 elsewhere.
  • Baseline U_H: rollout estimates (the text had claimed exact-graph values). Only U_∞ uses the exact graph.
  • Unsafe proposals: 4 of 1.25×10⁶ in Pursuit (two places had said zero).
  • Rounding:
  • radius 0.0069 (was 0.0070);
  • opponent TV 0.64 (was 0.65);
  • ε bounds 0.159 and 0.121 (0.12 understated 0.1203);
  • p + 2.002×10⁻⁴ (2.0 understated N·δ).
  • The VI stopping rule is described as it runs: stop on a per-sweep change of at most 10⁻⁹.
  • Policy settings:
  • CPO has (64,32) layers and a critic with IPPO's epochs, minibatches and learning rate (the table listed code defaults the campaign overrode);
  • the Lagrangian learning rate is per environment;
  • the Congestion cost limit is 6.3;
  • data collection uses learning rate 3×10⁻⁴.
  • Congestion opponent model: N_root is 192k/256k, hidden width 256 and batch 2048 (the paper had the August pilot's values).
  • IOP example: down 0.73 under the deployed model (was 0.49, from the August model and figure).

Not re-derived (documented only)

These match the provenance notes, but no archived file reproduces them:

  • WM-ablation prose: product-Bernoulli bit accuracy ≥ 0.996, TV 0.022/0.113/0.129 and worst rows 0.915/0.859/0.817.
  • These come from August's audit (docs/AI/optimisation/paper-final-results-2026-08-14.md).
  • Pursuit false-safe rates at budget 0.11: 2.45/3.61/3.66% (August audit) and 0.34/0.90/0.37% (a read-only recomputation, goal.md 2026-09-23).
  • Weak-WM decomposition: 0.964/0.973, 0.107/0.152, 0.103/0.152, masses 0.32/0.84/0.85 and one-step unsafe mass 0.021/0.020.
  • These come from the scratch script weakwm_decompose.py, recorded in goal.md 2026-09-30.
  • The 1.79 and 0.72 NLLs are archived (wm/wm_report.json, successor.row_nll.mean).
  • "One of 750,000" MSH transitions: true per seed. Stage 1 seeds 0, 3 and 6 each had 1 of 750,000 (4 of 10⁷ in total).
  • Gridworld gallery diagram: cell positions were not checked.

Later additions (19 Oct 2026)

  • OmSh's U_∞ entries (‡) in Table 1 and the appendix results table. These are the frontier study's audits at the published p (exports/certificate_dominance_frontier/). Their plug-ins equal tab:dominance_audit.
  • The frontier tables. They are generated by scripts/export_frontier_tables.py; see frontier-study-2026-10.md.