Conceptual Overview¶

A focal agent acts in a public multi-agent state. Its outcome depends on its own action, the other agents' joint action, and the environment transition. OMSH models those three ingredients separately and combines them only where needed.

public state + joint action
          |
          v
exact legal topology --learned successor kernel--> environment graph
          |                                               |
          |                       imagined rollouts ------+
          |                             |
          |                             v
          |                  opponent reasoning levels
          |                             |
          +-----------------------------+
                          |
                          v
              focal MDPs and risk bounds
                          |
                          v
                 shielded RL action

Public State and Legal Structure¶

Every environment exposes a flattened public Markov state. Exact builders enumerate reachable state keys, legal joint actions, and possible successors. This gives the system a known topology without pretending that the learned world model can invent arbitrary legal transitions.

Learned World Dynamics¶

The default world model is a masked categorical successor kernel. For each legal (state, joint_action) group it predicts a distribution over that group's legal successors, plus a reward for each successor. The product- Bernoulli bit model is retained as a baseline. Either way, the annotated environment transition graph is the handoff to opponent modelling and learned shielding. See World Modelling.

Opponent Reasoning¶

Level 0 imitates observed non-focal joint actions. Higher neural levels are warm-started from the preceding level and improved through imagined rollouts on the learned environment graph. At runtime, a smoothed Bayesian mixture tracks which level best explains observed opponent actions. Several opponents are modelled as one joint team, with an exact autoregressive factorization when there is more than one.

Safety and Shielding¶

Each labelled environment owns an explicit safety abstraction that maps public state to the set of agents currently unsafe. The same abstraction drives both runtime cost reporting and unsafe-node labelling for value iteration.

For each opponent level, the learned shield induces a focal-agent MDP and computes conservative upper bounds on eventual unsafe reachability. A runtime wrapper keeps a safety budget and opponent-level assumption. In the default condition, a monotone floor defines the levels that every discrete action must tolerate; IPPO samples only from the currently certified action mask. The parameterized conditions instead let the focal policy propose a pairwise primitive-action mixture, a successor-budget allocation, or both. Those proposals are projected back into the same robust safety constraints before the environment samples a primitive action.

The default condition allocates successor budgets with a robust egalitarian fill. Learned-budget modes make the allocation a policy output, while retaining hard projection and the conservative unsafe sink for missing coverage. Reward ranks only safety-feasible actions or projections; it never trades directly against the reachability budget.

Several agents can be shielded at once. The coordinated shield gives every protected role its own opponent model, posterior, floor, and budget, and resolves all replacements before one environment step.

From Model Guarantees to Real Guarantees¶

The learned certificate is exact over the learned model. Whether it carries over to the real game depends on how close the world model and the opponent levels are to the truth. The repository provides tools to check this offline and to pay for it at runtime:

• an exact arbitrary-opponent certificate on the serialized exact graph;
• count-centred ambiguity sets with a chosen miscoverage budget;
• a padded runtime shield that certifies against a total-variation ball around every level;
• robustness margins, opponent-gap measurements, and a step-by-step risk-dominance audit of frozen deployments.

Guarantees and Robustness explains what each one claims.

Baselines and References¶

The experiments compare OmSh with three model-free learners (IPPO, IPPO-Lagrangian, and CPO) and one oracle reference. The true-policy shield (IPPO-TrueShielded) uses exact environment transitions and the opponents' current live policies. No deployed agent has that information, so it serves as a reference: comparing it with OmSh separates world- and opponent-model error from the shielding rule itself.

Final policies are also evaluated after training, frozen, on fresh seeds; see Final-Policy Evaluation.