Closest-retained-level opponent gap¶

The opponent half of the anytime coverage assumption (A2 in robust-margin.md) needs, at every step t, some retained imagined level l >= f_t within eps_OM of the real opponent:

g_t = min_{l >= f_t} TV(mu*_t(. | s_t), pi_l(. | s_t)) <= eps_OM.

mu*_t is the law the real (joint) opponent samples from at s_t, pi_l is IOP level l, and f_t is the shield's monotone level floor at that step. The worst case of g_t over a run sets eps_OM.

Why the floor matters¶

The shield only certifies against levels >= f_t, so a level below the floor cannot be the one that covers the opponent. The minimum over all levels (floor 0) is therefore optimistic once the floor has risen. The gap between the two is the overshoot failure mode of A2: the floor rose past the level the opponent resembles. floor_excludes_closest_level_fraction in the script output counts the steps where a level below the floor is strictly closer than every retained one.

Definition details (shared by trainer and script)¶

src/rl/opponent_level_gap.py holds the NumPy definitions used by both:

• The live law is each opponent's policy within the action mask it samples under, so a shielded opponent (Congestion shields both players) is measured as it acts. An all-true mask is the plain softmax.
• Opponents are independent given their observations. The joint law is their product, first opponent most significant, matching encode_joint_action and the shield's opponent order (possible_agents without the focal index).
• closest_retained_level_tv(per_level_tv, floors) takes the per-sample minimum over l >= floor. suffix_min_level_tv gives the same for every floor at once.

Training-time logging (src/rl/ippo.py)¶

IPPO computes the live-opponent vs IOP discrepancy on up to diagnostic_state_sample rollout states every diagnostic_interval_updates updates. It used to be gated by verbose_data, but notebooks pass one verbose_data to both the trainer and the shield wrapper. The shield then keeps its per-step data traces, which are heavy, and the diagnostics export writes them out. log_opponent_level_gap (default False) enables only the discrepancy. verbose_data=True still enables it as before.

When enabled, IPPO records level_floor per step for every shield-protected agent. That is the controller's floor read just before env.step, which is the floor the step's action was certified against. The controllers come from _shield_level_controllers: a single-role TransitionShield is its own controller, and the coordinated MultiAgentTransitionShield exposes one per role through .controllers. Each protected agent's metrics then gain:

• opponent_policy_closest_level_tv_sampled_empirical_{mean,p95,max}, using the recorded floor of each sampled step;
• opponent_policy_level_floor_sampled_{min,max}.

The existing opponent_policy_level_{l}_tv_* and opponent_policy_mixture_tv_* keys are unchanged, except that the opponent's mask is now applied. This only matters for a shielded opponent, and that case was never logged before: the coordinated wrapper has no top-level iop. If an environment exposes an iop but no level_floor, the gap is logged for every floor f as opponent_policy_closest_level_tv_floor{f}_sampled_empirical_{mean,p95,max}. ParameterizedIPPO accepts the flag (it shares the config) but does not compute the discrepancy.

Notebook wiring: build_algo_cfg (notebooks/utils/jax_experiments.py) forwards log_opponent_level_gap from the notebook config dict to every trainer whose default config has the key (IPPO and IPPO-Shielded, not IPPO-Lagrangian). A notebook therefore only needs "log_opponent_level_gap": False next to "verbose_data" in the dict it passes to apply_cli_overrides. That makes it a known CLI key, so a launch can pass log_opponent_level_gap=True. The values land in metrics/<stem>_updates.json per agent.

Offline measurement (scripts/measure_opponent_level_gap.py)¶

The campaign did not log the gap, so the script measures it on the frozen final policies. For each run of a tagged OmSh condition it:

1. rebuilds the run's shield from its manifest config and the cached point-kernel bundles. This mirrors build_shielded_env_factory_from_config, including shield_padding_radius, but skips the factory's diagnostic exports, so nothing is written into the export tree. Bundle staleness is checked on content stamps only (mtime and size), because recorded paths are absolute cluster paths, plus the build parameters the notebook cache compares;
2. restores the post-training controller snapshot and the frozen policies, then replays evaluate_frozen_policies. It uses the same masked sampling and per-agent generators, and the first --episodes seeds of the run's final_policy_eval_seed_offset + run * final_policy_eval_episodes block. Its episodes are therefore a prefix of the campaign's own evaluation. The script checks this against the run's final-policy report (evaluation_reproduction);
3. records per step the binarised base state (the IOP's input in the shield), every agent's observation and mask, and every controller's floor, then evaluates policies and IOP levels in batches.

Output: <output-dir>/<env>.json (default exports/opponent_level_gap/). It holds per-run and pooled summaries per shielded role: closest_retained_level_tv and min_level_tv_floor0 (count/max/p99/p95/mean), per_level_tv, floor_distribution, closest_retained_level_distribution, closest_retained_level_tv_by_floor, floor_excludes_closest_level_fraction, and the worst step with both laws. --save-steps also writes <env>_steps.npz with the per-step arrays.

uv run python scripts/measure_opponent_level_gap.py bertrand chicken pursuit congestion
# Cluster CPU runner (forwards key=value as --key value):
submit -p amd48 --mem=16G py_cpu.sh omsh/scripts/measure_opponent_level_gap \
    environments=markov_stag_hunt save-steps=1 venv=omsh

Each step runs batch-one shield, policy and IOP calls, so the script pins torch, BLAS and XLA to one thread. Multi-threaded runtimes were about 2x slower here. To use more cores, run environments as separate processes. Peak RSS is about 1.5 GB plus about 3.4x the bundle pickle size.

Results (successor_campaign_v1, 200 episodes x 5 runs)¶

Pooled g_t at the actual floor, then the floor-0 minimum. All 1,000 replayed episodes per environment reproduce the campaign's own evaluation reward traces exactly. No bundle stamp problems were found.

Reading:

• A2 fails at any useful eps_OM for the campaign's final policies. The gap is mostly misspecification: even the floor-0 minimum is 0.39 to 0.72. The final co-trained opponents are nearly deterministic. At the worst steps they put 97% on one action in Bertrand, 99% in Chicken, 95 to 98% in Congestion and 85% in Pursuit (78% on average in Pursuit run 0). The IOP levels there are near-uniform: about 44/56 in Bertrand, 50/50 in Chicken, and uniform over three actions in Congestion. In Pursuit, levels 1 and 2 put their mass on the actions the final opponent avoids. Level 0 is fit to the opponent data collected before the campaign training (cross-entropy 1.51 nats over five actions in Pursuit, close to uniform), and the higher levels are imagined from it. The opponent then kept learning, so the gap is the drift from OM-time behaviour to the final co-trained policy.
• Overshoot comes on top of that. Every run except Pursuit run 2 ends training with the floor at the top level (2). In Pursuit, level 0 is closer than every retained level in 74% of steps, and excluding it raises the worst gap from 0.72 to 0.92.
• Within an evaluation episode the floor never moves. The controller snapshot already sits at the top level, or patience never completes in Pursuit run 2. So the per-step floor equals the end-of-training floor here. During training it would not.
• The cross-check against executed opponent actions (Pursuit run 0) agrees. The levels' probability of the executed action, through observed_log_probs (the shield's Bayes-update path), matches the dense per-level path exactly. Levels 1 and 2 give the executed action only 5 to 8% on average, against 64% under the live law.

Cost: Bertrand and Chicken took about 28 minutes each (200,000 steps), Pursuit 9.5 minutes, Congestion 16.5 minutes. They ran as four parallel single-thread processes on a shared 22-core machine. Peak RSS was 1.6 to 1.8 GB.

Alternatives not taken¶

• Reading the floor from the shield-augmented observation ([obs, budget, floor]) would avoid a trajectory field, but it depends on the wrapper's layout (the Bayesian mode appends the posterior), so the controller's level_floor is read directly.
• Measuring on the rollout states of training would need retraining with the flag on. The offline script covers the deployed final policies. The flag covers the opponent drift during training.