Conceptual Overview¶
The project is about decision-making in shared environments. A focal agent must choose actions while other agents are also acting, learning, or following strategies.
The core problem is that safety is not just a property of the focal agent's action. It depends on what the world does next and what the other agents might do.
The Three Pieces¶
World model. A model of environment transitions and rewards. In this repo, the active pipeline starts from exact legal structure and then annotates it with learned transition probabilities and rewards.
Opponent model. A model of the other agents' action choices. The current opponent model uses neural policies organized into reasoning levels and a mixture over those levels.
Probabilistic shield. A runtime wrapper that checks the proposed focal action against a risk certificate. If an action exceeds the allowed risk budget, the shield replaces it with a safer pure action.
Why the Order Matters¶
Opponent modelling depends on the world model because the model supplies the imagined consequences of actions.
Shielding depends on both: it needs environment transition probabilities and opponent action probabilities to estimate the chance of eventually reaching an unsafe state.
That gives the project its main flow:
What Counts as Unsafe¶
Each environment defines labels or state properties that mark unsafe states. The shield is concerned with eventual unsafe reachability, not only immediate next-step violations.
That means a safe-looking action can still be blocked if it tends to lead into a region where unsafe outcomes become hard to avoid.