Skip to content

Notebook Hyperparameter Pass (2026-04)

This pass tightens the default training budgets in the marimo notebooks for world-model training, opponent-model training, and RL experiments.

Main changes

  • World-model notebooks now scale replay-buffer size and MLP capacity more explicitly with environment complexity instead of relying on one shared train_world_model_mlp(...) default.
  • Gridworld world models use larger replay buffers and wider hidden layers because their public state is much larger:
  • pursuit: 750k steps for 2 agents, 1.0M for 3 agents, hidden=1024
  • gathering: 750k steps, hidden=768
  • ice_duel: 750k PPO-policy steps plus 750k independently seeded uniform-random steps, hidden=768
  • markov_stag_hunt: 750k steps, hidden=512
  • Small matrix world models now use more data than the old 10k default, but still stay lightweight:
  • bertrand, chicken, inspection: 25k steps, hidden=128, batch=256, epochs=80
  • dpgg: 100k steps, hidden=256
  • congestion: 60k for small agent counts, 100k for larger ones, with hidden size increasing to 512 for larger teams

Experiment-budget rationale

  • Gridworld experiment notebooks now default to longer RL runs because they have larger observations and longer effective horizons:
  • pursuit: 3.0M
  • gathering: 3.0M
  • ice_duel: 2.5M
  • markov_stag_hunt: 3.0M
  • Matrix experiment notebooks keep the same PPO-style update structure, but use longer runs and slightly smaller learning rates where the previous defaults looked too aggressive:
  • bertrand, chicken, inspection: 750k, learning_rate=2e-4
  • dpgg: 1.0M, learning_rate=2.5e-4
  • congestion: 750k for smaller agent counts, 1.0M for larger ones, learning_rate=2.5e-4

Opponent-model changes

  • Only the notebooks that looked clearly under-budgeted were changed.
  • dpgg now gives level-0 fitting more capacity and more imagined updates because the public pot state makes the repeated-game dynamics more stateful.
  • inspection now uses a slightly wider level-0 network and more training updates because the role asymmetry makes small underfit errors more noticeable downstream.

What did not change

  • congestion still uses imagination horizon 1 in opponent-model training. That is intentional: the observed state already summarizes the stage-level load outcome, so longer imagined horizons add cost much faster than they add signal.
  • The PPO rollout structure (rollouts=2048, learning_epochs=4, mini_batches=4) was left alone for now. This pass focused on notebook defaults that were most obviously mismatched to environment size and horizon.

Follow-up if we want a more empirical pass

  • Re-run the world-model notebooks and compare wm_report.txt across environments before increasing model width again.
  • Save level-0 holdout metrics from the OM notebooks to disk so future tuning can compare settings across runs instead of relying on notebook output.
  • If matrix IPPO still shows seed instability after the longer budgets and lower learning rates, the next knob to try is a larger rollout size rather than another learning-rate drop.