Skip to content

Safety Budgets

A safety budget is the maximum unsafe-reachability risk the shield is allowed to spend.

The common user-facing parameter is max_risk.

Budget Intuition

If max_risk is 0.2, the shield should only allow actions whose conservative eventual-unsafe estimate is at most 0.2 under the relevant opponent assumptions.

The value is not a per-step accident probability. It is an eventual reachability bound from the current state.

Budget Updates

After each environment transition, the shield updates the remaining budget from the realized successor state's projected risk.

This makes the budget stateful:

  • safe successor states can leave more room,
  • risky successor states can tighten future choices,
  • missing coverage is treated conservatively.

Level Floors

The opponent model has multiple reasoning levels. The shield tracks a level floor that can increase as evidence supports higher-level opponent behavior.

Once the floor rises, it does not decrease during that shield instance. This conservative rule prevents the shield from relaxing simply because the opponent mixture temporarily moves back toward a lower level.

Patience

The shield can require sustained evidence before raising the level floor.

The shield_patience setting controls how many successful opponent-observation updates must support the next floor candidate before the floor rises.