Experiments¶

Experiment notebooks compare reward, safety cost, and shield behavior across unconstrained, constrained-training, and runtime-shielded algorithms.

Select Stages Explicitly¶

Every environment notebook defines these switches:

All stages default to disabled. Select only the run_* keys needed for a given invocation. IPPO, IPPO-Lagrangian, CPO, and OmSh-Monotone are the four primary methods. TrueShielded is an oracle reference with information no deployed agent has, and it is worth running for every environment and agent count you report.

Shared Training Configuration¶

Common experiment keys include timesteps, num_runs, rollouts, learning_epochs, mini_batches, learning_rate, max_risk, and use_wandb. Use experiment_tag to isolate a campaign and seed_offset to reserve a disjoint seed block. Shield controls include shield_action_mode, shield_opponent_mode, shield_reward_mode, shield_patience, shield_padding_radius, shielded_agent_ids, the credible-floor parameters, debug/trace limits, and true_policy_refresh_interval_updates. The Configuration reference lists every key with its default.

CLI flags use kebab case:

uv run python notebooks/matrix/chicken/experiments_mo.py \
  --run-ippo-shielded=true \
  --max-risk=0.2 \
  --shield-debug-mode=minimal \
  --shield-patience=64

This call needs the learned graph and opponent stack. If the initial robust risk demand exceeds max_risk, setup fails before training rather than running an uncertified shield.

Reproduce the Campaign Protocol¶

Notebook defaults are convenient for local work, but they are not the published protocol. The campaign passes these settings explicitly:

Always set an experiment_tag; the campaign aggregators only discover tagged runs. Before trusting a new launch, diff its manifest config against the reference campaign's, not only the flags you remember passing.

On the cluster, corrected_final_campaign.sh encodes this protocol for the five non-Congestion environments only. Pass the tag explicitly (its default is the August campaign's paper_final_corrected_2026_08_v2), and pass PURSUIT_MAX_RISK=0.16 for a successor-world-model campaign, because it defaults Pursuit to 0.11. Congestion is a separate predeclared confirmation run through pipeline_env.sh or env.sh with its own seeds. The CX3 submitters encode all six. See Cluster Execution.

Final-Policy Evaluation¶

With final_policy_eval_episodes > 0, every algorithm run is frozen after training and evaluated on fresh seeds, without adding episodes to its training history. Shielded policies are evaluated within the same action masks they were trained under. See Final-Policy Evaluation for the keys, the exact post-processing of baselines, and campaign summaries.

Coordinated Multi-Role Shielding¶

In Congestion and Markov Stag Hunt, shielded_agent_ids names every role the shield protects:

uv run python notebooks/gridworlds/congestion/train_om_mo.py --focal-idx=1
uv run python notebooks/gridworlds/congestion/experiments_mo.py \
  --run-ippo-shielded=true --shielded-agent-ids=player_0,player_1

Each extra role needs its own opponent model, trained first with focal_idx. Coordinated shielding requires shield_action_mode=pure. The campaign shields both roles in Congestion but only player_0 in Markov Stag Hunt. When no explicit shield_version is given, the notebook versions the run by its agent set (for example agents_0_1), so it cannot overwrite the single-role condition. Reports label it OmSh-Monotone (both roles). See Probabilistic Shielding.

Parameterized OmSh Conditions¶

Every standard environment notebook exposes all four cells of the pure/mixed × egalitarian/learned-budget design through shield_action_mode:

Selecting a non-pure mode automatically uses ParameterizedIPPO and a ParameterizedTransitionShield; no custom driver is required. All conditions start PPO from fresh initialization and share the same compatible WM/OM artifacts. Condition-specific names prevent their histories, diagnostics, checkpoints, timings, and plots from overwriting the default pure condition.

For example:

uv run python notebooks/matrix/chicken/experiments_mo.py \
  --run-ippo-shielded=true \
  --shield-action-mode=learned_budget_pure \
  --experiment-tag=chicken_budget_holdout \
  --seed-offset=400 --num-runs=5

The lower-level PARAMETERIZED_OMSH_VARIANTS and make_parameterized_shielded_env_factory(...) helpers remain available for custom experiment drivers. Ablations covers this and the other shield variants, including padding and patience.

Robustness Diagnostics in Training¶

Two keys add guarantee-facing diagnostics to a normal training run:

• shield_padding_radius=r trains with the padded certificate (default 0, the deployed shield). Padded runs get their own artifact stem and label, for example OmSh-Monotone-Padded-r0.15.
• log_opponent_level_gap=true records the closest-retained-level opponent gap on sampled rollout states. Unlike verbose_data, it does not also turn on heavy per-step shield traces.

See Guarantees and Robustness for what they measure.

Exact Anytime Diagnostic¶

The optional anytime audit is disabled in every environment notebook. Enable it independently of all training stages:

uv run python notebooks/matrix/chicken/experiments_mo.py \
  --run-exact-anytime-diagnostic=true \
  --max-risk=0.2

It builds or reuses the serialized exact transition graph, binds it to the recorded environment contract, solves the arbitrary-opponent reachability game, independently validates the certificate, and writes true_shield/exact_anytime/certificate.pkl plus report.json. The report always records training_environment_steps=0; enabling this switch does not run PPO or repeat experiment seeds. On the matrix games it is a small offline cost. For gridworlds, a cache miss can make exact graph enumeration the dominant cost, but it is still separate from the multi-run training budget.

Recommended reporting order for the retained suite:

Only a report with certificate_status="certified" (complete reset support) or "certified_on_declared_reset_support" (an explicitly conditional claim), solver_converged=true, certificate_inductive=true, and reset_feasible=true supports the guarantee. infeasible is a valid audit outcome, not a certified runtime condition. The scope is one focal agent and one episode beginning at reset. delta_cov=0 is relative to the serialized exact graph, matching environment contract, and declared reset support. The gridworld exact builders use separate successor functions, so their certificates are relative to the serialized graph rather than the simulator itself.

For evaluation-only deployment, reconstruct existing policy adapters and call evaluate_exact_anytime_policies(...). It writes streaming episode aggregates to true_shield/exact_anytime/evaluation_report.json, performs no learning, and stores no per-step trace. scripts/summarize_exact_anytime_diagnostics.py collects the reports into paper tables.

Tagged Evaluations¶

A non-empty experiment_tag writes each condition's histories, diagnostics, checkpoints, final-policy reports, manifest, and timing sidecar below experiments/runs/<normalized-tag>/ and leaves the corresponding canonical stage artifacts untouched. Tags are normalized to lowercase underscore-separated tokens. seed_offset changes trainer seeds without changing the local run indices used for paired comparisons.

Tagged runs also render their aggregate figures in the tagged directory, so OmSh-Monotone, OmSh-Credible, and other conditions in one tag share a campaign plot.

Summarize all matching environment runs with:

uv run python scripts/summarize_evaluation.py chicken_budget_holdout

The command writes JSON, Markdown, and compact LaTeX diagnostics under exports/reports/<normalized-tag>/. It aggregates within each seed before averaging seeds, reports whole-run and final-window reward and observed safety, and only emits paired baseline deltas when matching baseline seeds exist. Incomplete or tuning-only tags remain visible but are marked incomplete. For frozen-policy results use scripts/summarize_final_policy_campaign.py instead.

Repeated Runs and Histories¶

num_runs selects independent seeds. Each stage writes a list of episode histories under experiments/histories/. EpisodeLog rows contain episode boundaries, focal reward/safety totals, level raises, and shield telemetry aggregates where applicable.

Figure exports include cumulative, episodic, post-WM-timeline, no-legend, and shield telemetry variants. Structured single-metric episode graphs are derived from completed episode rows rather than per-update approximations.

Compact episode tables, update metrics, aggregate diagnostics, checkpoints, and completion manifests are always exported. verbose_data=true additionally enables bounded raw shield traces and sampled live-policy/IOP comparisons; it does not add environment rollouts.

Weights & Biases¶

When use_wandb=true, each algorithm run creates a run using the shared config and logs its history. Congestion and Markov Stag Hunt default it on; the other notebooks default it off. Pass --use-wandb=false for unattended smoke tests unless credentials and project behavior are deliberately configured.