Skip to content

True-Policy Exact Shield Baseline

IPPO_TrueShielded is a separate baseline from the learned WM/OM shielding path.

It is also distinct from the theorem-facing exact-anytime-adversarial diagnostic. This baseline marginalizes opponent actions using current live policies. The anytime certificate instead preserves every opponent-action row and takes their adversarial maximum; do not use this true-policy bundle as evidence for an arbitrary-opponent guarantee.

Intent

This baseline answers a narrower question than the main architecture:

  • what happens if the shield has exact environment transition probabilities
  • and exact access to the current opponent policies used by IPPO training

It is meant as a reference baseline, not as a replacement for world modelling or opponent modelling.

Artifact flow

The baseline uses a dedicated cache under exports/.../true_shield/:

  1. build or load true_transition_graph.pkl
  2. validate it against true_transition_graph.metadata.pkl
  3. create a TruePolicyTransitionShield around the labelled env
  4. initialize that wrapper with a conservative one-level bundle built from uniform opponent policies
  5. let IPPO push live opponent policies into the wrapper during training

Unlike IPPO_Shielded, this path does not read wm/env_transition_graph.pkl or om/iop_stack.pt.

The true-graph cache is validated against the current public-state shape, a small environment dynamics signature, and source hashes for the true/exact graph builders. If an environment's public Markov contract or cached dynamics-relevant configuration changes, the cached true graph is rebuilt automatically instead of being silently reused.

Graph validation, construction, and publication are serialized by a process lock next to true_transition_graph.pkl, because IPPO, Lagrangian, and CPO post-processors for one environment may become eligible together. The graph and metadata sidecar are each staged in the cache directory and atomically replaced while the lock is held. A truncated legacy graph or metadata pickle is treated as a miss and rebuilt.

true_transition_graph.pkl is intentionally stored as a plain pickled src.joint_graph.common.Graph, because the runtime wrapper reloads that path through src.joint_graph.load_graph(...). Cache validation metadata lives in the sidecar file instead of wrapping the graph object. The loader still accepts the older wrapped cache artifact and migrates it forward on first reuse.

Exact dynamics graph

src.true_shield.exact_graph.build_true_transition_graph(...) dispatches over the same supported env families as the exact joint-graph legality path:

  • matrix envs
  • markov_stag_hunt
  • gathering
  • pursuit
  • ice_duel

For deterministic envs, the true graph is the legal graph annotated with env_prob=1.0. For stochastic envs, the builder reuses the exact successor distribution helpers from src/joint_graph/exact/* and stores those probabilities directly on the graph edges.

Rewards are zero-filled and intentionally ignored by the shield path. The graph exists only to support exact unsafe-reachability analysis.

Live policy refresh semantics

TruePolicyTransitionShield reuses TransitionShield, but with exactly one level and no opponent-mixture updates.

  • there is no iop_stack.pt
  • there is no posterior update after each environment step
  • level_floor is always effectively 0

The wrapper exposes maybe_refresh_true_shield(update_idx, policies). Plain IPPO calls that hook:

  • once immediately after runtime initialization with update_idx=0
  • once after every PPO update cycle with the incremented update index

The refresh interval is configured in PPO-update units via true_policy_refresh_interval_updates. On refresh, the wrapper rebuilds a one-level bundle from the cached exact graph plus the latest live PolicyAdapters for all non-focal agents, but it does not immediately replace the active shield bundle. The rebuilt bundle is stored as pending and installed at the next reset().

This reset-boundary install is intentional. The shield carries a shrinking eventual-risk budget within an episode; replacing the true-policy model under that partially spent budget invalidates the old budget certificate. Deferring the swap keeps each episode certified under one fixed true-policy bundle while still letting PPO updates schedule fresher bundles for subsequent episodes.

Large gridworld exact graphs, especially Markov Stag Hunt, should use a coarse refresh interval. Rebuilding the true-policy bundle requires marginalizing the exact joint graph and rerunning sound value iteration. The MSH experiment defaults to 64 PPO updates between live-policy refreshes, which gives a first live-policy refresh around 131k environment steps with the current rollouts=2048 setting.

The true-policy bundle builder now marginalizes the exact joint graph directly against the supplied opponent policies. It no longer creates a full intermediate joint graph annotated with opp_prob/prob, which reduces peak memory on large graphs while preserving the induced focal-MDP semantics used by the shield.

The notebook-side true-shield factory also probes reset() once during setup. That means impossible initial budgets now fail in the setup phase with the same infeasibility diagnostics as learned shielding, instead of failing only after training has already started.

Protecting several roles (Fri 9 Oct 2026)

Before 9 Oct, IPPO_TrueShielded protected only player_0 everywhere. In Congestion, OmSh protects both roles (shielded_agent_ids=player_0,player_1), so the oracle there shielded one role against an unshielded learner. Its 0.197 was the uncoordinated fallback, not a like-for-like reference.

MultiAgentTruePolicyShield (src/true_shield/multi_agent.py) fixes this. It is a subclass of MultiAgentTransitionShield whose controllers are TruePolicyTransitionShields, one per role, on the shared exact graph. The base class now accepts any class in its controller_types (exact type match), so plain OmSh still rejects true-policy controllers.

  • Reset. reset() first installs each controller's pending bundle, then runs the usual coordinated reset. Bundles still change only at episode boundaries.
  • Refresh. maybe_refresh_true_shield(update_idx, policies) forwards IPPO's hook to every controller. Each rebuilds its bundle from the other roles' live policies at the same cadence (true_policy_refresh_interval_updates).
  • Opponent law (the one approximation). A shielded role's network observes [public state, carried budget, level floor], and its executed action is filtered by its own shield. Both depend on that role's private episode history, which a per-state bundle cannot condition on. The refresh therefore queries each shielded opponent at its reset augmentation (init_safety_bound, floor 0; a true-policy shield never raises its floor) through _ResetAugmentedPolicy, and marginalises its pre-shield policy. With one shielded role the opponent is unshielded, so this is exact. With several roles each certificate is exact only up to the opponent's budget-dependent behaviour and its shield's replacements. Querying the raw 20-dimensional state instead fails with a Flax shape error (22 inputs expected).
  • Notebook. In the Congestion notebook, the TrueShielded stage follows shielded_agent_ids: one role uses make_true_shielded_env_factory, and several use make_multi_agent_true_shielded_env_factory. The learned-versus-oracle diagnostic is exported for the first role only, as before. The other five environments' notebooks are unchanged; there both methods protect player_0.
  • Tests. tests/test_true_shield.py::*two_role* covers the per-role refresh, install at reset and masked execution, controller-type isolation, and IPPO training with live refreshes.

Why this stays separate

Keeping IPPO_TrueShielded separate avoids muddying the main pipeline:

  • WM/OM shielding still measures learned-model performance
  • true shielding measures an oracle-style upper bound with exact dynamics and exact opponent policies

That separation is useful when comparing:

  • learned uncertainty vs exact dynamics
  • learned opponent reasoning vs true live policies
  • architectural gaps in the learned pipeline

It does not guarantee feasibility. If the exact environment plus safety label already makes the reset state violate the requested eventual-risk budget, true shielding will fail too.

One concrete cache failure mode was found in 2-agent Pursuit after switching to terminal-on-collision dynamics. The old true_transition_graph.pkl had been built before that switch and still let collision states transition onward, so the setup probe reported reset risk 1.0 even though the live env passed terminates_after_collision=True. A fresh true graph has collision states as self-looping safe terminal states; under the uniform bootstrap opponent policy, the default reset state's eventual unsafe risk is about 2.07e-4, so the max_risk=0.2 setup is feasible. The expanded cache metadata prevents reusing that stale graph class.

That repeated-game caveat still applies in general, but the current matrix envs have env-specific repairs:

  • Bertrand now treats only price_war as unsafe.
  • Congestion now includes a safe Detour fallback action.

So both exact baselines are feasible again at max_risk=0.2; see repeated-matrix-feasibility.md.

Why TrueShielded earns less than OmSh in Pursuit-Evasion (7 Oct 2026)

The paper's main table at p = 0.16 gives OmSh +0.286 per step at U 0 and TrueShielded -1.666 at worst-seed U 0.146; both pass the gate. The frozen-policy episodes and training histories (successor_campaign_v1 for TrueSh, successor_campaign_p512_v1 for OmSh) explain the gap.

  • The reward is safety-aligned and a breach is not terminal. An intruder in the goal costs the guard -10 at every step it stays there; an interception gives +10 and ends the episode.
  • TrueShielded:
  • breaches in 12–15% of episodes per seed and intercepts in the rest;
  • median episode 5–10 steps;
  • its breach episodes contribute 1,500–7,400 penalty steps per 1,000 episodes;
  • in training, the certified risk of the chosen action is 0.12–0.18, carried budget 0.18–0.20.

Its certificate is tight and calibrated: realised breaches match the certified risk. The learner spends the slack on fast, direct interceptions. - OmSh: - 0 breaches; - interceptions in 97.5–100% of episodes; - median episode 22–31 steps; - in training, the certified risk is 0.04–0.07 and the carried budget 0.04–0.08.

Its certificate is robust over the retained imagined levels, which include goal-seeking best responses. The reset demand of 0.15 against 0.16 (pre-fix bundle) leaves almost no headroom, so the shield admits little beyond risk-minimising guarding moves. Against the co-trained opponents those never breach. - Admissible-action counts are similar (about 1.3–2.2 of 5 for both), so the difference lies in which actions are admitted, not how many.

Conjecture, as stated in the paper (appendix appsec:true_shield_remarks, omsh-paper 9e2f317): robustness to the imagined levels, a cost elsewhere, happens to align with the reward here; OmSh's models are not more informative than the exact ones. The reward-fix rerun changes the OmSh side's numbers (reset demand 0.132), so recheck this paragraph when it lands.