True-Policy Exact Shield Baseline¶

IPPO_TrueShielded is a separate baseline from the learned WM/OM shielding path.

It is also distinct from the theorem-facing exact-anytime-adversarial diagnostic. This baseline marginalizes opponent actions using current live policies. The anytime certificate instead preserves every opponent-action row and takes their adversarial maximum; do not use this true-policy bundle as evidence for an arbitrary-opponent guarantee.

Intent¶

This baseline answers a narrower question than the main architecture:

• what happens if the shield has exact environment transition probabilities
• and exact access to the current opponent policies used by IPPO training

It is meant as a reference baseline, not as a replacement for world modelling or opponent modelling.

Artifact flow¶

The baseline uses a dedicated cache under exports/.../true_shield/:

1. build or load true_transition_graph.pkl
2. validate it against true_transition_graph.metadata.pkl
3. create a TruePolicyTransitionShield around the labelled env
4. initialize that wrapper with a conservative one-level bundle built from uniform opponent policies
5. let IPPO push live opponent policies into the wrapper during training

Unlike IPPO_Shielded, this path does not read wm/env_transition_graph.pkl or om/iop_stack.pt.

The true-graph cache is validated against the current public-state shape, a small environment dynamics signature, and source hashes for the true/exact graph builders. If an environment's public Markov contract or cached dynamics-relevant configuration changes, the cached true graph is rebuilt automatically instead of being silently reused.

Graph validation, construction, and publication are serialized by a process lock next to true_transition_graph.pkl, because IPPO, Lagrangian, and CPO post-processors for one environment may become eligible together. The graph and metadata sidecar are each staged in the cache directory and atomically replaced while the lock is held. A truncated legacy graph or metadata pickle is treated as a miss and rebuilt.

true_transition_graph.pkl is intentionally stored as a plain pickled src.joint_graph.common.Graph, because the runtime wrapper reloads that path through src.joint_graph.load_graph(...). Cache validation metadata lives in the sidecar file instead of wrapping the graph object. The loader still accepts the older wrapped cache artifact and migrates it forward on first reuse.

Exact dynamics graph¶

src.true_shield.exact_graph.build_true_transition_graph(...) dispatches over the same supported env families as the exact joint-graph legality path:

• matrix envs
• markov_stag_hunt
• gathering
• pursuit
• ice_duel

For deterministic envs, the true graph is the legal graph annotated with env_prob=1.0. For stochastic envs, the builder reuses the exact successor distribution helpers from src/joint_graph/exact/* and stores those probabilities directly on the graph edges.

Rewards are zero-filled and intentionally ignored by the shield path. The graph exists only to support exact unsafe-reachability analysis.

Live policy refresh semantics¶

TruePolicyTransitionShield reuses TransitionShield, but with exactly one level and no opponent-mixture updates.

• there is no iop_stack.pt
• there is no posterior update after each environment step
• level_floor is always effectively 0

The wrapper exposes maybe_refresh_true_shield(update_idx, policies). Plain IPPO calls that hook:

• once immediately after runtime initialization with update_idx=0
• once after every PPO update cycle with the incremented update index

The refresh interval is configured in PPO-update units via true_policy_refresh_interval_updates. On refresh, the wrapper rebuilds a one-level bundle from the cached exact graph plus the latest live PolicyAdapters for all non-focal agents, but it does not immediately replace the active shield bundle. The rebuilt bundle is stored as pending and installed at the next reset().

This reset-boundary install is intentional. The shield carries a shrinking eventual-risk budget within an episode; replacing the true-policy model under that partially spent budget invalidates the old budget certificate. Deferring the swap keeps each episode certified under one fixed true-policy bundle while still letting PPO updates schedule fresher bundles for subsequent episodes.

Large gridworld exact graphs, especially Markov Stag Hunt, should use a coarse refresh interval. Rebuilding the true-policy bundle requires marginalizing the exact joint graph and rerunning sound value iteration. The MSH experiment defaults to 64 PPO updates between live-policy refreshes, which gives a first live-policy refresh around 131k environment steps with the current rollouts=2048 setting.

The true-policy bundle builder now marginalizes the exact joint graph directly against the supplied opponent policies. It no longer creates a full intermediate joint graph annotated with opp_prob/prob, which reduces peak memory on large graphs while preserving the induced focal-MDP semantics used by the shield.

The notebook-side true-shield factory also probes reset() once during setup. That means impossible initial budgets now fail in the setup phase with the same infeasibility diagnostics as learned shielding, instead of failing only after training has already started.

Why this stays separate¶

Keeping IPPO_TrueShielded separate avoids muddying the main pipeline:

• WM/OM shielding still measures learned-model performance
• true shielding measures an oracle-style upper bound with exact dynamics and exact opponent policies

That separation is useful when comparing:

• learned uncertainty vs exact dynamics
• learned opponent reasoning vs true live policies
• architectural gaps in the learned pipeline

It does not guarantee feasibility. If the exact environment plus safety label already makes the reset state violate the requested eventual-risk budget, true shielding will fail too.

One concrete cache failure mode was found in 2-agent Pursuit after switching to terminal-on-collision dynamics. The old true_transition_graph.pkl had been built before that switch and still let collision states transition onward, so the setup probe reported reset risk 1.0 even though the live env passed terminates_after_collision=True. A fresh true graph has collision states as self-looping safe terminal states; under the uniform bootstrap opponent policy, the default reset state's eventual unsafe risk is about 2.07e-4, so the max_risk=0.2 setup is feasible. The expanded cache metadata prevents reusing that stale graph class.

That repeated-game caveat still applies in general, but the current matrix envs have env-specific repairs:

• Bertrand now treats only price_war as unsafe.
• Congestion now includes a safe Detour fallback action.

So both exact baselines are feasible again at max_risk=0.2; see repeated-matrix-feasibility.md.