Experiments¶

Experiment notebooks compare reward, safety cost, and shield behavior across unconstrained, constrained-training, and runtime-shielded algorithms.

Select Stages Explicitly¶

Every environment notebook defines these switches:

Key	Stage	Learned WM/OM required?
`run_ippo`	IPPO	No
`run_ippo_lag`	IPPO-Lagrangian	No
`run_icpo`	ICPO	No
`run_ippo_shielded`	Learned transition shield + IPPO	Yes
`run_ippo_true_shielded`	Exact true-policy shield + IPPO	No
`run_exact_anytime_diagnostic`	Exact arbitrary-opponent certificate audit; no RL training	No
`just_vis`	Skip training and rebuild figures from saved histories	Existing histories

All algorithm stages default to disabled. Select only the run_* keys needed for a given invocation.

Shared Training Configuration¶

Common experiment keys include timesteps, num_runs, rollouts, learning_epochs, mini_batches, learning_rate, max_risk, and use_wandb. Use experiment_tag to isolate a campaign and seed_offset to reserve a disjoint seed block. Shield controls include shield_action_mode, shield_opponent_mode, shield_reward_mode, shield_patience, the credible-floor parameters, debug/trace limits, and true_policy_refresh_interval_updates.

Exact Anytime Diagnostic¶

The optional anytime audit is disabled in every environment notebook. Enable it independently of all training stages:

uv run python notebooks/matrix/inspection/experiments_mo.py \
  --run-exact-anytime-diagnostic=true \
  --max-risk=0.2

It builds or reuses the serialized exact transition graph, binds it to the recorded environment contract, solves the arbitrary-opponent reachability game, independently validates the certificate, and writes true_shield/exact_anytime/certificate.pkl plus report.json. The report always records training_environment_steps=0; enabling this switch does not run PPO or repeat experiment seeds. On small matrix games it is a small offline cost. For gridworlds, a cache miss can make exact graph enumeration the dominant cost, but it is still separate from the multi-run training budget.

Recommended reporting order:

Environment	Use in an anytime table?	Reason
Inspection	Yes, first	Tiny exact graph and an interpretable positive integration case.
Congestion	Yes, at 4 agents	Exercises 27 coordinated joint opponent actions per focal action without the 243-action cost of the 6-agent case.
DPGG	Useful secondary case	Cheap enough to audit; check whether its value adds information beyond `0/1`.
Ice Duel	Diagnostic stress case	Exercises multi-successor rows. Include it as guarantee evidence only if bounded value iteration converges; a plateau is exported as `invalid`.
Pursuit (2 agents)	Yes as a negative control	Arbitrary opponents may make the requested reset budget infeasible.
Bertrand, Chicken	Optional positive replications	Very cheap, but add little beyond Inspection when certificate values are only `0/1`.
Gathering, Markov Stag Hunt	Only with a reset-support caveat	Default exact builds use sampled reset states; do not generalize the certificate to unlisted resets.

Only a report with certificate_status="certified" (complete reset support) or "certified_on_declared_reset_support" (an explicitly conditional claim), solver_converged=true, certificate_inductive=true, and reset_feasible=true supports the guarantee. infeasible is a valid audit outcome, not a certified runtime condition. The scope is one focal agent and one episode beginning at reset, and delta_cov=0 is relative to the serialized exact graph, matching environment contract, and declared reset support.

For evaluation-only deployment, reconstruct existing policy adapters and call evaluate_exact_anytime_policies(...). It writes streaming episode aggregates to true_shield/exact_anytime/evaluation_report.json, performs no learning, and stores no per-step trace.

CLI flags use kebab case:

uv run python notebooks/matrix/chicken/experiments_mo.py \
  --run-ippo-shielded=true \
  --max-risk=0.2 \
  --shield-debug-mode=minimal \
  --shield-patience=64

This call needs the learned graph and opponent stack. If the initial robust risk demand exceeds max_risk, setup fails before training rather than running an uncertified shield.

Parameterized OmSh Conditions¶

Every standard environment notebook exposes all four cells of the pure/mixed × egalitarian/learned-budget design through shield_action_mode:

Value	Primitive action	Successor-budget allocation
`pure`	One discrete action	Robust egalitarian fill
`egalitarian_mixed`	Pairwise mixture	Robust egalitarian fill
`learned_budget_pure`	One discrete action	Learned proposal, robustly projected
`learned_budget_mixed`	Pairwise mixture	Learned proposal, robustly projected

Selecting a non-pure mode automatically uses ParameterizedIPPO and a ParameterizedTransitionShield; no custom driver is required. All conditions start PPO from fresh initialization and share the same compatible WM/OM artifacts. Condition-specific names prevent their histories, diagnostics, checkpoints, timings, and plots from overwriting the default pure condition.

For example:

uv run python notebooks/matrix/inspection/experiments_mo.py \
  --run-ippo-shielded=true \
  --shield-action-mode=learned_budget_pure \
  --experiment-tag=inspection_budget_holdout \
  --seed-offset=400 --num-runs=5

The lower-level PARAMETERIZED_OMSH_VARIANTS and make_parameterized_shielded_env_factory(...) helpers remain available for custom experiment drivers.

Tagged Evaluations¶

A non-empty experiment_tag writes each condition's histories, diagnostics, checkpoints, manifest, and timing sidecar below experiments/runs/<normalized-tag>/ and leaves the corresponding canonical stage artifacts untouched. Tags are normalized to lowercase underscore-separated tokens. seed_offset changes trainer seeds without changing the local run indices used for paired comparisons.

Notebook-level aggregate figures are still written to the ordinary experiments/ plot paths. Treat the tagged tables and reports as the durable campaign record; use separate checkouts or retain copied figures if concurrent tagged jobs for one environment also render plots.

Summarize all matching environment runs with:

uv run python scripts/summarize_evaluation.py inspection_budget_holdout

The command writes JSON and Markdown under exports/reports/<normalized-tag>/. It aggregates within each seed before averaging seeds, reports whole-run and final-window reward and observed safety, and only emits paired baseline deltas when matching baseline seeds exist. Incomplete or tuning-only tags remain visible but are marked incomplete.

Repeated Runs and Histories¶

num_runs selects independent seeds. Each stage writes a list of episode histories under experiments/histories/. EpisodeLog rows contain episode boundaries, focal reward/safety totals, level raises, and shield telemetry aggregates where applicable.

Figure exports include cumulative, episodic, post-WM-timeline, no-legend, and shield telemetry variants. Structured single-metric episode graphs are derived from completed episode rows rather than per-update approximations.

Compact episode tables, update metrics, aggregate diagnostics, checkpoints, and completion manifests are always exported. verbose_data=true additionally enables bounded raw shield traces and sampled live-policy/IOP comparisons; it does not add environment rollouts.

Weights & Biases¶

When use_wandb=true, each algorithm run creates a run using the shared config and logs its history. Most notebooks default this off; Markov Stag Hunt currently defaults it on. Pass --use-wandb=false for unattended smoke tests unless credentials and project behavior are deliberately configured.