Skip to content

Hidden Episode Cap Semantics

The repo is moving away from environment-internal time-limit dynamics and away from an outer "episode-cap wrapper" layer.

Current rule

  • Environments expose the cap through the max_steps constructor argument and attribute.
  • These values are treated as hidden episode-boundary metadata, not as part of the public Markov state and not as environment transition semantics.
  • Runtime code reads env.max_steps; no alternate cap attribute is supported.

Where the cap is enforced

  • LabelledEnv synthesizes truncation when its per-episode step counter reaches the inferred hidden cap. This preserves episode histories and safety logs for the common wrapped-env path.
  • Any wrapper that clears agents when the hidden cap fires must resynchronize its own wrapper-level agents view during reset(...). Otherwise downstream helpers can get stuck seeing a permanently inactive wrapper even though the inner env has already restarted the next episode.
  • IPPO, IPPO_Lagrangian, ICPO, and BufferWrapper also infer the hidden cap and synthesize truncation/reset boundaries when training or collecting on raw envs.
  • There is intentionally no dedicated outer PettingZoo wrapper for cap enforcement.
  • Offline exact-graph enumeration must not advance LabelledEnv bookkeeping while restoring only the inner environment snapshot. The generic matrix builder unwraps bookkeeping-only wrappers before reset/step/state exploration. Otherwise the wrapper counter spans independent BFS branches, eventually triggers the hidden cap, and silently truncates reachability. For the default DPGG dynamics this failure produced 177 states instead of the complete 281-state graph, which OM input validation correctly rejected.

Bootstrap semantics

  • Hidden-cap truncation is an episode boundary for reset/logging purposes.
  • Hidden-cap truncation is not terminal for value bootstrapping.
  • PPO-family trainers therefore keep separate:
  • episode-end flags: termination or truncation
  • bootstrap-stop flags: termination only

This keeps the public state stationary while still letting finite-length training episodes be cut at a configurable boundary.