Troubleshooting¶
Zensical or a Python Command Is Missing¶
Install every dependency group and run commands through the project environment:
A Notebook Rejects a CLI Key¶
Notebook keys use snake case internally and kebab case on the command line.
Check Configuration and CLI. Unknown and duplicate
normalized keys are errors by design. In cluster pipelines, remember that
n_steps is WM-only while most training keys are experiment-only.
A Baseline Run Tries to Load Shield Artifacts¶
All standard experiment algorithms default to disabled. A plain IPPO-only call
only needs --run-ippo=true. If it still reaches a shield stage, verify that a
wrapper or saved command is not explicitly enabling that stage.
Learned Shield Artifacts Are Missing¶
Run the matching WM and OM notebooks first. The learned shield requires:
wm/env_transition_graph.pkl
om/iop_stack.pt
om/agents/player_<i>/iop_stack.pt # each extra role in shielded_agent_ids
Use the same environment parameters and agent count at every stage.
Initial Safety Bound Is Infeasible¶
The message reports the best robust eventual-unsafe demand and best immediate
unsafe probability. This is not a missing-file error: no focal action satisfies
the selected max_risk across the accessible opponent levels. Review the
environment safety objective, start state, opponent artifacts, and intended
risk contract before changing the budget.
The demand depends on the world model. Pursuit's notebook default 0.16 sits
just above the default successor kernel's reset demand (about 0.1515); the
earlier product-Bernoulli model's demand was about 0.103, where 0.11 sufficed. A weak world or opponent
model in the strength ablations can also be
infeasible at the campaign budget; report that rather than raising the budget.
A Successor World Model Fails to Resolve Transitions¶
The successor model resolves each buffered transition to a legal successor in
wm/legal_joint_graph.pkl. More than a small fraction of unresolved rows fails
the stage instead of being converted into probability mass. This usually means
the legal graph is stale or was built for a different environment
configuration. Rebuild the graph rather than relaxing the check.
Graph Width, Metadata, or Opponent Dimension Mismatch¶
Cached graph metadata normally triggers an automatic rebuild. If downstream artifacts were produced from older state/action semantics, move the affected environment export root aside and rerun WM → OM → experiments. Keeping the old directory as a backup is safer than deleting it until the rebuilt outputs have been verified.
Missing Coverage Appears in Shield Telemetry¶
The shield sends missing transition mass to an unsafe sink and treats unknown successors conservatively. Frequent missing coverage usually means incompatible state encoding, stale graph/model artifacts, or incomplete legal topology. It should not be suppressed as a cosmetic logging issue.
A Large Exact Graph Runs Out of Memory¶
Graph size grows exponentially in agent count. Three-agent Pursuit and high-agent Congestion can be much larger than two-agent runs. Confirm the intended count, inspect cached metadata, use the configured graph guards, and avoid building several copies concurrently.
A Short JAX Run Still Takes Time¶
The first run compiles JAX functions, and timesteps is rounded up to a full
rollout. Reduce both --timesteps and --rollouts for a smoke test. The
persistent test compilation cache helps repeated pytest runs but does not make
every new notebook shape compile-free.
Weights & Biases Prompts or Fails¶
Pass --use-wandb=false unless remote logging is intended and authenticated.
Congestion and Markov Stag Hunt enable it by default; the other experiment
notebooks do not. On the cluster, Errno 122 Disk quota exceeded at
wandb.init seconds into a job usually means the W&B cache volume is full,
not this project's storage; only notebooks with W&B enabled are affected.
Frozen OmSh Reward Looks Wrong¶
Frozen OmSh policies must be evaluated within the training-time action masks.
Reports written before 2026-09-24 sampled unmasked logits, which leaves safety
correct but under-reports reward. Check that a report records
"action_selection": "training_masks", and rerun older ones with
scripts/reevaluate_omsh_final_policies.py. See
Final-Policy Evaluation.
A Campaign Summary Is Missing Rows or Columns¶
The final-policy summarizer counts a condition only when it has exactly
--expected-runs valid reports with a completion manifest, and puts an
environment in the paper tables only when all four primary methods are
complete. Empty exact-reachability columns mean
scripts/evaluate_final_campaign_baselines.py has not run for that tag. A new
launch that silently used notebook defaults (no tag, other seeds, no frozen
evaluation) will not be discovered at all. Diff its manifest config against
the reference campaign.
A Cluster Job Is Running but Making No Progress¶
A marimo kernel that dies does not end its job: the job can stay RUNNING
while its afterok dependents wait indefinitely. Check the log for
kernel was stopped unexpectedly. On CX3, CPU-only jobs without
JAX_PLATFORMS=cpu and CUDA_VISIBLE_DEVICES="" stall after setup while their
memory climbs; see Cluster Execution.
A Copied Shield Cache Is Rebuilt¶
Shield cache identity includes the modification times of its input artifacts.
Copy wm/, om/, and shield/ with times preserved (tar, rsync -t), or
the bundle is treated as stale and rebuilt on the new machine.
Cluster Dependencies Are Missing¶
Use DRY_RUN=1 and inspect the submission output. Slurm pipeline launchers need
a submit command compatible with sbatch --parsable; PBS launchers must recover
a full numeric job ID before creating afterok children. The launcher aborts
instead of continuing without a dependency.
Notebook Outputs Land Under the Wrong Directory¶
Maintained marimo notebooks call the project-root helper. A custom notebook
should import and call notebooks.project_root.set_cwd_to_project_root() before
creating relative export paths.