Final-Policy Evaluation¶

Training-window reward and cost average a sequence of changing policies. They do not measure the safety of the policy at the final update. Final-policy evaluation freezes each trained policy and measures it separately, on fresh evaluation seeds, without adding episodes to the training history.

There are two distinct measurements:

Neither measurement turns a baseline into a guaranteed method. The exact result is a property of one frozen policy in one exact benchmark model. It says nothing about safety during training or about later opponent drift.

Enable It in Training¶

Set the evaluation keys on the experiment notebook. With final_policy_eval_episodes=0 (the default), no frozen evaluation runs.

uv run python notebooks/matrix/chicken/experiments_mo.py \
  --run-ippo-shielded=true --run-cpo=true \
  --experiment-tag=chicken_final_v1 --seed-offset=2200 --num-runs=5 \
  --timesteps=250000 \
  --final-policy-eval-episodes=1000 \
  --final-policy-eval-seed-offset=2200000

Congestion has a 64-step episode cap, so its campaign uses horizons (1, 2, 10, 64). Keep the evaluation seed block disjoint from every other campaign so live samples are independent.

Per-run reports are written below experiments/runs/<tag>/final_policy/<artifact>_run<N>.json. Evaluation time is recorded separately from training time.

Evaluate as Trained¶

Shielded IPPO samples only within the shield's admissible set, so a frozen OmSh policy is evaluated with the same action masks it was trained under. Every live report records "action_selection": "training_masks", and reports without it are rejected for OmSh.

uv run python scripts/reevaluate_omsh_final_policies.py \
  --environment-name markov_stag_hunt --tag <tag> \
  --artifact-stem ippo_shielded_histories

OmSh is stateful: its opponent posterior, monotone floor, and patience counters persist across environment resets. The training checkpoint stores that post-training controller state, and the evaluator restores the same snapshot before every episode. Replicates are therefore IID (episode_sampling=iid_controller_snapshot) rather than one adaptive sequence.

Exact Post-Processing for Baselines¶

The exact reachability columns come from a separate post-processor. For a complete tag, one job evaluates IPPO, IPPO-Lagrangian, and CPO sequentially, so the exact graph is loaded once:

uv run python scripts/evaluate_final_campaign_baselines.py \
  --environment-name chicken --tag <tag> \
  --horizons 1 10 50 100 200 \
  --evaluation-seed-offset 2200000 --episodes 1000

Congestion uses --horizons 1,2,10,64 --evaluation-seed-offset 4200000 --episodes 2000, and at any fleet size other than two also --n-agents N.

When the evaluation seed block matches the training-time live reports, those reports are reused and only the exact quantities are computed. The output is final_policy/<stem>_safety.json beside the per-run reports. Memory follows the exact graph: about 64 GB for Markov Stag Hunt, 32 GB for Gathering, 16 GB for Pursuit, and 8 GB otherwise.

Executed OmSh has no exact column. Its actions depend on the carried budget, posterior, and floor, which the public-state graph does not represent. The evaluator rejects the bare neural actor rather than labelling it OmSh. For OmSh, report the live frozen risk next to its model-relative certificate and the risk-dominance audit.

Summarize a Campaign¶

uv run python scripts/summarize_final_policy_campaign.py \
  --tag <tag> --exports-root exports --expected-runs 5

It writes final_policy_campaign.{json,md}, final_policy_compact.tex, final_policy_all_conditions.tex, and timing tables below exports/reports/<tag>/. The compact table gives each primary method as reward per step / worst-seed U_H / exact eventual unsafe reach.

A condition is complete only with exactly --expected-runs valid reports. An environment enters the paper tables only when IPPO, IPPO-Lagrangian, CPO, and OmSh-Monotone are all complete at the same largest horizon. Partial campaigns still render, with explicit missing and incomplete status.

Seed Shards¶

Long OmSh conditions can run one seed per job. Give each shard its own one-run tag, trainer seed s + i, and evaluation offset base + i * episodes, so the merged result equals a serial five-seed run:

uv run python scripts/merge_final_policy_shards.py <destination_tag> \
  --shard-tag-template '<shard_prefix>_seed{trainer_seed}' \
  --seed-offset 2200 --expected-runs 5 \
  --evaluation-seed-offset 2200000 --evaluation-episodes 1000

The template may use {run_index} and {trainer_seed}. Run the merge against a scratch exports root if the destination tag also holds baseline results.

If the destination's seed 0 came from a serial job stopped before it wrote its completion manifest, merge with `--recover-missing-manifests --risk-threshold

`. The manifest is rebuilt only if the checkpoint, trainer seed, and controller state match the live report. A merged tag holds only the reports; checkpoints stay in the shard tags. Anything that replays checkpoints, such as the risk-dominance audit, runs once per tag and is then combined; see [Analysis and Audits](analysis-and-audits.md#audits-of-frozen-deployments). If a job wrote its checkpoint but died before publishing the report or manifest, `scripts/recover_omsh_final_policy_checkpoint.py` (OmSh) and `evaluate_final_policy_checkpoints.py --live-only` (baselines) replay the declared evaluation from the checkpoint. Recovery needs a schema-3 checkpoint with the controller state. ## Paired Comparisons Several scripts compare frozen evaluations on the same trainer seeds: | Script | Compares | | --- | --- | | `compare_final_policy_campaigns.py` | Two campaigns that differ only in the world model. | | `summarize_replacement_ablation.py` | Parameterized shield variants against the pure OmSh control. | | `decide_replacement_screen.py` | Applies a predeclared promotion rule to a paired screen. | | `summarize_patience_screen.py` | Floor-patience arms against the patience-64 control. | | `summarize_padded_runs.py` | Padded-shield runs against the campaign OmSh runs. | See [Analysis and Audits](analysis-and-audits.md) for the full catalogue and the maintainer note on [frozen final-policy safety](../AI/reinforcement-learning/final-policy-safety-evaluation.md) for checkpoint schemas and solver details.