Skip to content

Coverage-aware world-model pretraining

Aggregate held-out bit accuracy can hide transition errors in infrequently visited state-action pairs. This matters for shielding because even a small set of false-safe pairs can invalidate the learned-model certificate in the live environment.

The eval_250k_v1 Ice Duel pilot exposed this failure mode. The old model had 0.9979 held-out bit accuracy, but an exhaustive comparison with the exact transition graph found:

  • 100% state-action-pair coverage;
  • mean transition TV 0.032490;
  • p95 transition TV 0.175119;
  • maximum transition TV 0.947231; and
  • false-safe state-action rates from 0.04% to 0.12% across OM floors.

At runtime, the shield predicted mean late final risk around 0.0017, while the observed late violation probability was 0.0125. Action availability was 0.994 and the carried budget remained near 0.2, so mixed actions or learned budget allocation would not address the binding error.

Ice Duel WM pretraining therefore combines two disjoint live-environment data sources by default:

  1. 750,000 transitions from the existing entropy-regularized IPPO collector;
  2. 750,000 transitions from an independently seeded uniform-random joint policy.

The random collector samples discrete action indices from its own NumPy RNG; it does not call unseeded Gymnasium space.sample(). The held-out random buffer uses another seed. This improves state-action coverage without using the exact graph as training data or hard-coding any transition outcome. The exact graph remains an audit oracle only.

After retraining, promotion requires re-exporting the learned transition graph, retraining the OM/IOP stack, rebuilding the shield bundle, and rerunning the exhaustive TV/oracle comparison before policy evaluation.