Skip to content

Opponent-Uncertainty Shield Modes

The standard and parameterized transition shields support four runtime interpretations of the opponent-model levels. The configured default remains monotone_floor.

Configuration

Experiment notebooks expose four CLI/config fields:

  • shield_opponent_mode: monotone_floor, credible_floor, all_levels, or bayesian_mixture
  • shield_reward_mode: bayesian (default) or robust
  • shield_credible_delta: total posterior-tail error budget (default 0.05)
  • shield_credible_schedule: summable (default) or fixed

For example, an all-level run can be launched with --shield-opponent-mode all_levels. These fields change runtime aggregation; they do not change the cached per-level bundle.

Monotone floor (default)

At floor f, safety is enforced against every level in the suffix {f, ..., M - 1}. Action risk is the maximum over those levels and successor budgets must satisfy every corresponding transition distribution. The floor-wise robust continuation certificate is used as the lower bound on each successor budget.

The floor only rises. Since the remaining level set only shrinks, its robust continuation value is antitone in f; a budget carried under an earlier floor therefore remains feasible after a raise. This is the mode used by default and the one corresponding to the robust monotonic-level safety argument.

Posterior-credible floor

credible_floor chooses the largest monotone floor whose retained suffix has at least 1 - delta_t posterior mass, then takes the worst-case action risk over every level in that suffix. It can be less conservative than all-level robustness without collapsing safety to a posterior-average transition.

The default summable schedule allocates shield_credible_delta across the continuing observation sequence. The fixed schedule supplies only a pointwise credible set. The guarantee assumptions, additive risk accounting, posterior reversal diagnostics, and telemetry are documented in credible-tail-floor.md.

All levels

all_levels fixes the floor at zero and takes the worst-case action risk over every level forever. Successor budgets satisfy every level-specific transition constraint. It is the conservative control ablation for measuring how much reward the monotone floor recovers by permanently removing lower-level hypotheses.

Bayesian mixture

bayesian_mixture also fixes the floor at zero, but aggregates each action's transition distribution with the current normalized iop.mix_alpha. Both its action risk and successor-budget constraint use that posterior-weighted distribution.

The continuation lower bound deliberately remains the all-level robust certificate. This has two useful properties:

  1. posterior changes do not require online value iteration; and
  2. every carried successor budget remains large enough for a feasible action after any later posterior change.

This mode is a posterior-predictive ablation. It does not provide the worst-case guarantee over every level or every switching sequence. Its safety claim is relative to the learned Bayesian mixture transition kernel. A fully tight Bayesian infinite-horizon shield would require a belief-state dynamic program over future posterior updates.

For parameterized actions, the same posterior mixture is formed before the edge or successor-budget projection, so those projections enforce the single mixture constraint rather than silently retaining the all-level constraints.

The focal observation and public wrapper state append the normalized posterior weights in this mode, after the usual budget and floor fields. This makes the posterior-dependent action mask visible to the learner. The other modes retain the existing two-field augmentation.

Reward aggregation

Safety feasibility is independent of reward aggregation. With the default shield_reward_mode=bayesian, admissible replacements are ranked by expected focal reward under the posterior restricted and renormalized to the levels considered by the safety mode. robust instead uses the minimum predicted reward over those levels.

Reward only ranks already-admissible actions. It never compensates for excess risk. For masked IPPO, the policy normally chooses among admissible actions itself; the replacement ranking mainly governs unsafe external or off-policy proposals.

Use identical WM/OM artifacts, fresh policy initialization, risk budgets, and training seeds for:

  1. monotone_floor + bayesian reward (main/default);
  2. credible_floor + bayesian reward (posterior-credible robust suffix);
  3. all_levels + bayesian reward (cost of permanent robustness);
  4. bayesian_mixture + bayesian reward (posterior-predictive ablation).

Report inferred downward level changes alongside these results. Frequent posterior returns below the retained floor are evidence against the monotonic hypothesis-elimination approximation, even though they do not by themselves identify a ground-truth opponent reasoning level.