Conceptual Overview¶

A focal agent acts in a public multi-agent state. Its outcome depends on its own action, the other agents’ joint action, and the environment transition. OMSH models those three ingredients separately and combines them only where needed.

public state + joint action
          |
          v
exact legal topology --learned environment probabilities--> environment graph
          |                                                   |
          |                         imagined rollouts --------+
          |                               |
          |                               v
          |                    opponent reasoning levels
          |                               |
          +-------------------------------+
                          |
                          v
              focal MDPs and risk bounds
                          |
                          v
                 shielded RL action

Public State and Legal Structure¶

Every environment exposes a flattened public Markov state. Exact builders enumerate reachable state keys, legal joint actions, and possible successors. This gives the system a known topology without pretending that the learned world model can invent arbitrary legal transitions.

Learned World Dynamics¶

The world-model MLP predicts next-state bits and per-agent rewards from a state and joint action. The predictions score the exact candidate successors; scores are normalized within each legal (state, joint_action) group. The resulting environment transition graph is the handoff to opponent modelling and learned shielding.

Opponent Reasoning¶

Level 0 imitates observed non-focal joint actions. Higher neural levels are warm-started from the preceding level and improved through imagined rollouts on the learned environment graph. At runtime, a smoothed Bayesian mixture tracks which level best explains observed opponent actions.

Safety and Shielding¶

Each labelled environment owns an explicit safety abstraction that maps public state to the set of agents currently unsafe. The same abstraction drives both runtime cost reporting and unsafe-node labelling for value iteration.

For each opponent level, the learned shield induces a focal-agent MDP and computes conservative upper bounds on eventual unsafe reachability. A runtime wrapper keeps a safety budget and opponent-level assumption. In the default condition, a monotone floor defines the levels that every discrete action must tolerate; IPPO samples only from the currently certified action mask. The parameterized conditions instead let the focal policy propose a pairwise primitive-action mixture, a successor-budget allocation, or both. Those proposals are projected back into the same robust safety constraints before the environment samples a primitive action.

The default condition allocates successor budgets with a robust egalitarian fill. Learned-budget modes make the allocation a policy output, while retaining hard projection and the conservative unsafe sink for missing coverage. Reward ranks only safety-feasible actions or projections; it never trades directly against the reachability budget.

Learned and Exact Baselines¶

The normal shield uses learned environment probabilities and learned opponent policies. The true-policy shield uses exact environment transitions but still conditions the opponents on their current live policies. Comparing them helps separate world-model error from the shielding rule itself.