Algorithms and Baselines¶

Experiment learners operate independently per agent on PettingZoo parallel environments. Baselines use discrete actions; parameterized OmSh additionally uses a categorical/raw-Gaussian focal action. Trainers return a shared TrainResult containing episode history, metrics, policy adapters, and agent-specific training state.

IPPO¶

IPPO is the JAX/Flax independent PPO baseline. Each agent owns a separate actor-critic and optimizer; parameters are not shared. Python collects one live environment rollout, then JAX batches the PPO updates. The trainer supports Gymnasium Discrete and arbitrary-rank MultiDiscrete action spaces, including non-zero starts. It currently supports num_envs == 1 and requires synchronous episode completion.

IPPO-Lagrangian¶

IPPO_Lagrangian adds one cost critic and dual variable per agent. It reads infos[agent_id]["cost"], forms separate reward and cost advantages, and uses the scaled policy advantage (A_reward - lambda * A_cost) / (1 + lambda).

The dual compares cost_limit with completed episodic cost. Partial episodes persist across rollout boundaries, and no dual update occurs until an episode completes. This avoids treating an arbitrary rollout fragment as a complete constraint sample.

ICPO and CPO¶

ICPO is a local PyTorch categorical-action port of constrained policy optimization. Each agent owns an actor, reward critic, cost critic, and learned future-failure predictor. Actor updates use conjugate gradient, KL/Fisher trust regions, constrained case selection, and backtracking line search.

The default ICPO benchmark shapes the raw cost with predicted near-future failure risk. CPO is the plain preset with learned cost shaping disabled. Notebook names retain run_icpo and icpo_histories.pkl.

Learned Shielded IPPO¶

JAX IPPO acts through TransitionShield. Its focal observation contains the environment observation plus current budget and opponent-level floor. The ordinary learner samples only currently admissible pure actions. External policies can still be replaced by the highest-reward admissible pure action using certificates built from the learned environment graph and opponent stack.

ParameterizedIPPO and ParameterizedTransitionShield provide three further conditions: pairwise primitive-action mixtures with egalitarian budgets, pure actions with learned successor budgets, and their combination. The trainer stores the exact categorical and raw-Gaussian augmented action; projection and primitive sampling belong to the environment transition.

True-Policy Shielded IPPO¶

TruePolicyTransitionShield uses an exact transition graph. It starts with uniform opponent policies and can rebuild a pending shield bundle from the live opponent policies at a configured update interval; pending bundles install at reset. This is the reference for separating learned-world error from shielding behavior.

Boundary Semantics¶

All trainers infer hidden environment episode caps. True terminations stop GAE and do not bootstrap. Truncations stop the recursive trace but retain the actual next-state bootstrap. Shared-environment agents must finish together; asynchronous per-agent endings are rejected.