Skip to content

Posterior-Credible Monotone Floor

TransitionShield supports shield_opponent_mode=credible_floor as an experimental alternative to the default MAP-based monotone_floor. It uses the opponent-model posterior to discard only a controlled lower tail, while still enforcing safety robustly over every retained level.

Floor rule

Let alpha_t(l) be the normalized posterior over ordered opponent levels l = 0, ..., M - 1. At observation count t, the candidate floor is the largest z satisfying

sum_{l=z}^{M-1} alpha_t(l) >= 1 - delta_t.

The effective floor is monotone: it is at least the previous floor. Runtime action risks and successor-budget constraints take the worst case over the whole retained suffix {z, ..., M - 1}. The posterior is used only to select the suffix and to rank rewards; safety is not averaged inside that suffix.

The existing patience gate also applies. It can delay a candidate raise and therefore retain extra lower levels, which is conservative. Runtime inspection uses that patience-filtered floor exactly.

Fixed and summable schedules

shield_credible_delta is the posterior-tail error budget. The default is 0.05. shield_credible_schedule controls how it is allocated:

  • fixed uses delta_t = delta at every decision. This is a pointwise posterior-credible suffix. Reusing a fixed 5% allowance indefinitely does not itself give a 5% infinite-horizon error bound.
  • summable uses delta_t = 6 * delta / (pi^2 * (t + 1)^2). Since the allocations sum to delta, a union bound limits the probability of ever excluding the true level to at most delta, subject to the modelling assumptions below.

The observation count belongs to the shield instance and survives episode resets, matching the intended continuing-learning interpretation. Per-step info reports the allowance used for that decision. After the opponent update, the wrapper's credible_delta_t property can already reflect the next observation-indexed allowance.

Safety accounting and assumptions

The credible-floor claim is conditional on all of the following:

  1. mix_alpha is a calibrated posterior over the relevant ordered level hypotheses;
  2. the true opponent path respects the monotone-level assumption, so an excluded lower level cannot later become the true active level;
  3. the learned world model, opponent policies, graph coverage, unsafe labels, and numerical reachability bounds satisfy the ordinary shield assumptions.

Under those assumptions, the summable schedule supports the union-bound accounting

P(eventually unsafe) <= initial shield risk budget + credible_delta.

Notebook factories interpret max_risk as the common total risk target. For credible_floor, they reserve the tail error automatically and initialize the reachability budget at max_risk - shield_credible_delta; monotone and other opponent modes receive the full max_risk. This makes a nominal 0.20 OmSh-Credible run with delta=0.05 enforce a conditional model budget of 0.15 and report an additive nominal quantity of 0.20. Direct construction of TransitionShield remains lower-level: its init_safety_bound is the conditional reachability component and its nominal_posterior_tail_risk_quantity adds the tail allowance only for the summable schedule. A credible delta larger than the total notebook budget is rejected.

This is not a proof that the learned posterior is calibrated. It is also not a safe response to arbitrary non-monotonic switching: without a restriction on switching, all levels remain possible and all_levels is the appropriate robust mode.

Posterior reversals and telemetry

Because the floor is irreversible, a later posterior can place more than delta_t mass below it. The implementation does not lower the floor. Instead, it exposes the mismatch so experiments can test the assumption:

  • decision info: total and pointwise delta, schedule, excluded and retained mass, posterior-tail constraint status, observation count, and the nominal additive risk quantity;
  • traces: the same credible-floor fields;
  • EpisodeLog: credible steps, mean pointwise delta, mean/max excluded mass, mean retained mass, and posterior-tail constraint-miss count/rate.

shield_posterior_tail_constraint_held=false means the current posterior no longer assigns the promised mass to the already-retained suffix. It is an empirical warning about the irreversible monotonic assumption, not a repair or a new safety certificate.

Experiment configuration

The standard and parameterized notebook factories accept the same settings:

--shield-opponent-mode credible_floor
--shield-credible-delta 0.05
--shield-credible-schedule summable

The launcher exposes the two primary conditions as distinct algorithms: omsh_monotone selects monotone_floor, while omsh_credible selects credible_floor. Their published labels are OmSh-Monotone and OmSh-Credible, and their histories remain collision-free. Credible-floor runs remain conditional on posterior calibration and monotonicity diagnostics.

Reward-frontier interpretation and primary-suite status

For the current M = 3 experiments, every summable allowance satisfies delta_t <= 0.05 < 1 / M. At a fixed posterior, the credible-tail candidate floor cannot exceed the MAP candidate floor: excluding the MAP level would discard at least 1 / M posterior mass. The one-level patience operator preserves this ordering on a shared posterior history. Whenever the credible rule requests a raise from a common floor, the MAP rule requests the same next one-level raise, so credible evidence cannot accumulate while MAP evidence resets. By induction, the MAP floor is never below the credible floor when both see the same posterior sequence.

The primary comparison also matches the user-facing total risk target. A credible run reserves delta = 0.05 from that target, whereas monotone uses the full target as its model-controlled reachability budget. Consequently, OmSh-Credible has no intrinsic feasible-set or optimal-reward advantage under matched histories: it retains at least the MAP suffix and also has the smaller model-controlled budget. Separate learning runs can still produce different state and posterior histories after their actions diverge, and the extra conservatism can act as an optimisation regulariser. A higher credible return is therefore possible as a finite-training empirical outcome, but it does not follow from the credible-set construction and cannot improve the best attainable reward frontier relative to a monotone controller that is free to imitate the same actions.

For that reason, Credible is retained as an implemented diagnostic/ablation but retired from the primary launcher and final comparison. The default suite contains IPPO, IPPO-Lagrangian, CPO, and OmSh-Monotone. An explicit algorithms=omsh_credible request remains supported so the posterior-tail assumptions and finite-training regularisation effect can still be studied.