Skip to content

Public Markov State Contract

joint_graph, world modelling, and opponent modelling now share one state contract:

  • env.state() and the shared public observation encode the same Markov state.
  • Exact graph nodes are keyed only by those public state bits.
  • OM / WM do not rely on latent graph-only suffix state.
  • Shielding may wrap that public state in a separate training MDP, but the base environment state used by WM / OM / exact graphs does not change.

What changed

Finite-horizon environments no longer expose their step counters publicly.

  • Hidden max_steps episode caps are metadata only.
  • Public observations and env.state() must stay time-free unless time changes the actual environment transition law in a future-relevant way.
  • Truncation is synthesized outside the base env by the hidden-cap helpers in runtime wrappers and trainers.
  • Value learning bootstraps through truncation; truncation is not treated as a public-state terminal event.

Terminal gridworld events now use the terminal observation returned by PettingZoo directly.

  • IceDuel ends on the lethal step itself; the returned observation already shows the player on the edge.
  • Pursuit ends on the same step as a terminating goal/collision event.
  • We no longer add deferred public "death frame" or latch channels purely to simulate a terminal s'.

Pursuit still keeps private episode bookkeeping such as collision_ever for info reporting, but that history is no longer part of the public state contract because it does not affect future transitions, safety labelling, or termination.

Exact graph implications

Exact builders now hash the stored public state bits directly for supported environments.

  • matrix, gathering, pursuit, ice_duel, and markov_stag_hunt all derive node identity from public state.
  • Hidden episode caps are not part of the node identity.
  • Terminal nodes are the terminal observations actually returned by the environment, with absorbing self-loops in exact graphs.

Notes

  • The public state is intentionally observation-shaped so existing WM / OM buffers do not need a parallel richer-state path.
  • If a future environment needs extra future-relevant state, that state must first become public before joint_graph or OM may depend on it.
  • Do not reintroduce public time bits just to model episode truncation; prefer the hidden-cap bootstrapping path documented in docs/AI/environments/hidden-episode-cap.md.
  • Do not add deferred public state solely to invent a terminal s' when the environment already returns the terminal observation from step().
  • The budgeted TransitionShield is the main exception to “state equals public observation”, but only at the wrapper level: it defines a shielded training state flatten(base_env.state()) || current_safety_bound || level_floor while still hashing and learning over the unchanged base public state underneath. The Bayesian-mixture shield ablation also appends the normalized opponent-level posterior because its mask depends on that belief.