Skip to content

Opponent Modelling

Opponent modelling predicts the joint action of every non-focal agent. The runtime object is ImaginedOpponent, a stack of neural policies, one per reasoning level, together with a Bayesian belief over which level best explains the observed behavior.

One Joint Opponent

All non-focal agents are modelled as one labelled joint opponent with one posterior over team reasoning levels. They are not split into independent per-agent models, because correlated team behavior is exactly what the shield must not assume away.

The output representation depends on the number of opponents:

  • One opponent. A flat categorical OpponentMLP over its A actions.
  • Several opponents. AutoregressiveJointOpponent, which factors the joint law exactly by the chain rule, p(a_1, ..., a_N | s) = prod_j p(a_j | s, a_1, ..., a_{j-1}). It replaces the A ** N output head, and scoring an observed tuple is linear in the number of opponents.

MBOMConfig(architecture="auto") makes this choice and records it in the checkpoint. A checkpoint without an architecture field loads as the flat model. The autoregressive head removes the exponential output layer, but exact shield construction still enumerates the legal joint action space, so graph size remains the scaling limit (see Joint Graphs).

Level 0: Observed Behavior

The opponent buffer stores public states and observed opponent-team actions. fit_level0(...) uses cross-entropy and reserves a 10% validation holdout by default. It reports sample count, cross-entropy, accuracy, error rate, multiclass Brier score, expected calibration error, mean confidence, action counts, and the confusion matrix for both splits. Above 512 joint classes the confusion matrix is stored sparsely, so every observed count is kept without a dense table. A one-sample buffer cannot form a holdout and reports unavailable holdout metrics as NaN in memory and null in the JSON export.

Higher Levels: Imagined Responses

Each higher level is warm-started from the preceding policy and trained with soft-rollout policy improvement on GraphWorldModelAdapter. The adapter samples only legal joint actions and learned environment successors from wm/env_transition_graph.pkl. A counted root state sample may expand into several imagined environment steps, so OM progress budgets are not directly comparable to real environment timesteps.

The rewards used during imagination come from the learned graph. Under the default successor world model these are per-successor rewards, so imagined returns reflect what the realized transition pays.

The OM stage writes level0_fit_report.{json,txt}, om_report.{json,txt}, and pi_training_metrics.csv. The staircase report includes training history, level entropies, and pairwise policy discrepancy over sampled diagnostic states.

Runtime Mixture

mix_alpha starts uniform over levels. After observing an opponent action, the model:

  1. evaluates its likelihood under every level,
  2. combines likelihood and the current mixture,
  3. applies temperature smoothing, and
  4. moves toward the posterior with an exponential update.

By default, the learned shield does not simply average all level certificates. It tracks a monotone floor and requires the selected action to be feasible for every level at or above that floor. shield_patience requires sustained posterior evidence before raising the floor. The optional all_levels and bayesian_mixture shield modes provide conservative and posterior-predictive ablations without changing how mix_alpha itself is updated.

The posterior mode is a modelling diagnostic, not a ground-truth level. Whether the real opponent stays close to a level the floor still retains is measured directly by the closest-retained-level gap.

Role-Specific Models

The default model treats player_0 as focal. To shield another role, train a second model for it with focal_idx:

uv run python notebooks/gridworlds/congestion/train_om_mo.py --focal-idx=1

The world-model buffer is stored relative to player_0, and the OM stage reorients it for the requested role without changing action semantics. This is valid only because the stored observation is a shared global state, which holds in Congestion and Markov Stag Hunt. Role models are written below om/agents/player_<i>/; the player_0 paths are unchanged.

The coordinated multi-role shield then uses one model per protected role. On the cluster, om_focal_indices=0,1 trains these models sequentially (see Cluster Execution).

Artifact

The trained stack is saved as:

exports/<family>/<environment>[/<n_agents>]/om/iop_stack.pt
exports/<family>/<environment>[/<n_agents>]/om/agents/player_<i>/iop_stack.pt

It is paired with the same environment configuration and wm/env_transition_graph.pkl used during imagined training. Changing agent count, public-state width, action size, world model, or graph semantics requires compatible artifacts to be rebuilt.

Private-Reward Inference

In the default pipeline the world-model buffer records every agent's reward, so imagined opponents optimize their true payoff. The private-reward ablation removes the opponent's reward from that buffer and infers a pseudo-reward from joint actions alone, using a windowed inverse quantal-response fit. See Ablations for the configuration.