Skip to content

Clean timing run on cpucluster (10 Oct 2026)

After the AAMAS submission the user asked for the cleanest possible timing measurements on DoC cpucluster, re-measured from scratch and independent of earlier measurements. This page records the protocol, why it looks the way it does, and what an earlier attempt taught us.

Protocol

  • Hardware. Partition amd48: auklet01 and auklet02, identical single-socket AMD EPYC 7402P (24 cores, SMT, one NUMA node, acpi-cpufreq ondemand, 2.8 GHz max, same kernel).
  • One job per whole node. --exclusive keeps every other job off the node, so nothing competes for memory bandwidth, L3 or boost clock.
  • One physical core. notebooks/experiments/scripts/py_cpu_pinned.sh wraps the DoC runner (~/py_cpu.sh) in taskset -c 12,36, the two hardware threads of core 16.
  • The runner, uv, the Marimo kernel and every thread pool inherit that affinity. The timing sidecar's hardware record shows one physical core.
  • Checked live with srun --jobid=<id> --overlap on the first two jobs.
  • Memory. --mem=64000M for every job: a 62.5 GiB cgroup limit, the same everywhere.
  • Isolation. A separate clone, ~/Projects/omsh_timing (PROJECT_PREFIX=omsh_timing, venv=omsh).
  • Its exports points at a fresh root, /vol/bitbucket/oja24/omsh_timing.
  • No earlier sidecar entry can mix into the timings. Two failure modes made that necessary: refresh_component_totals keeps a larger stale component_om, and a re-run in an old tag reverted entries (see docs/AI/notebooks/marimo-experiments.md).
  • Protocol.
  • The published campaign: tag timing_clean_amd48_v1, trainer seeds 2200–2204 (Congestion 5300–5304), the published evaluation streams and horizons, OmSh with patience 512, and both Congestion roles shielded with two OM roles.
  • use_wandb=False.
  • One job per algorithm.
  • IPPO-TrueShielded is not timed, so it is skipped (TRUE_SHIELDED=1 adds it).
  • Launcher. notebooks/experiments/scripts/cpucluster_timing_rerun.sh, run from the clone; its defaults are this protocol.
  • Scheduling is FIFO, because every multifactor priority weight is 0. So the long chains go first: MSH, then Gathering, Pursuit, Chicken, Bertrand and Congestion.
  • With two nodes the run is bound by total work, about 115 node-hours, i.e. about 2.5 days.

Runner edits break running jobs (11 Oct 2026)

  • Mechanism. bash reads a script as it executes it. A job running ~/py_cpu.sh therefore resumes at its old byte offset once the notebook's Python process exits.
  • If the file was edited in place meanwhile (same inode), bash parses the new text from a stale offset and fails.
  • The job exits non-zero after its work is done, and afterok dependents become DependencyNeverSatisfied.
  • What happened. The Gathering OM (job 39861) failed this way after 26.6 h. Its artefacts and timings were complete, so its dependents were released by hand (scontrol update Dependency=).
  • Batch script vs runner. Slurm spools the batch script (py_cpu_pinned.sh) at submission, so edits to the repository copy never reach queued jobs. The runner it execs is read live from $HOME.
  • Jobs submitted as sbatch ~/py_cpu.sh run their spooled copy and are immune; EqEx's are.
  • Only wrappers that exec the live runner, like ours, are exposed.
  • Prevention.
  • Replace ~/py_cpu.sh atomically: write a new file, then mv it into place. Running jobs keep the old inode.
  • For a stage whose dependents can verify their own inputs, use afterany.

Why not shared nodes (the 9 Oct amd96 attempt)

The first fair-timing attempt ran every job at one core (--mem=64000M) on shared amd96 nodes (jobs 38175–38217, 39689–39696). It was cancelled on 10 Oct. Three lessons:

  • amd96 mixes clock classes. auklet03/04 cap the EPYC 7552 at 2.2 GHz while auklet07/08 boost to 3.3 GHz. timings_hardware.json records each node's clock range, which shows this.
  • One core is not isolation. Co-tenant jobs moved OmSh's per-seed time by up to 1.7× within one job, e.g. Bertrand on auklet08 at 15.4 → 26.9 min. Memory-bound stages (OmSh, the large OMs) suffer most: the EPYC 7552 has half the memory bandwidth per core of the 7402P.
  • Memory bandwidth outweighs clock speed. MSH's OM ran only 11% faster at 3.3 GHz than at 2.2 GHz.

Its partial results stay in /vol/bitbucket/oja24/omsh (tag timing_rerun_amd96_v1) and are not used.

Reading the results

  • Read the per-stage times from <exports>/<env>/timings.json (WM, OM) and <exports>/<env>/experiments/runs/timing_clean_amd48_v1/timings.json (setup and per-seed algorithm times).
  • Check each entry's hostname and affinity in the matching timings_hardware.json.
  • Report the per-seed mean with the median and range.
  • Multi-role OM (Congestion) is component_om alone, which already covers both roles. Do not add component_om_player_1.
  • The frozen-policy reports in the same run directories double as a reproduction check of the published rewards and U_H.

Storage gaps and scheduling (12–14 Oct 2026)

  • /vol/bitbucket automount gaps. Three times in about 20 h the automount vanished briefly on a node. The clone's exports symlink dangled, and the notebook raised FileExistsError: [Errno 17] File exists: 'exports'.
  • Mon 05:35 (auklet01): four jobs failed at start.
  • Mon about 22:05 (auklet02): Gathering CPO failed after 2 seeds.
  • Tue 00:34 (auklet01): MSH OmSh failed after all 5 seeds.
  • A job fails the moment it touches exports during a gap. Resubmit it with its sacct SubmitLine minus --dependency, passing the tokens directly, because the horizons argument has parentheses.
  • Per-seed timings land at job end. A job that fails after its last seed loses all per-seed entries, and only *_setup survives, so it must re-run in full.
  • A re-run reuses the shield bundle its failed predecessor built, so its *_setup is a cache hit. Take the fresh-build setup from the failed job: MSH 0.308 h, against 0.042 h on the re-run.
  • A second "100%" progress line for a job's last run, printed minutes later, is the bar closing after post-run work. It is not an I/O stall.
  • Exclusive jobs starve on a busy partition. Backfill starts small jobs whenever they fit, ahead of a job that needs a whole idle node. The two re-runs waited about 26 h until amd48 drained overnight.

Final results (14 Oct 2026)

Wall-clock hours; per-seed mean for policies, end-to-end for WM and OM, shield construction for Setup. Published values are in brackets.

Environment WM OM Setup IPPO CPO IPPO-Lag OmSh
Congestion (2) 0.135 (0.133) 0.327 (0.351) 0.001 (0.008) 0.099 (0.088) 0.095 (0.104) 0.106 (0.092) 0.413 (0.431)
Markov Stag Hunt 2.636 (1.875†) 29.744 (26.545†) 0.308 (0.797†) 0.435 (0.462) 0.437 (0.473) 0.441 (0.461) 1.154 (1.167†)
Pursuit-Evasion (2) 2.467 (2.462) 9.227 (8.597) 0.016 (0.010) 0.101 (0.110) 0.103 (0.114) 0.104 (0.116) 0.352 (0.354)
Gathering (2) 1.187 (1.236) 26.623 (24.284) 0.096 (0.147) 0.101 (0.102) 0.106 (0.114) 0.104 (0.107) 0.297 (0.310)
Bertrand 0.036 (0.033) 2.440 (2.703) 0.001 (0.000) 0.095 (0.099) 0.087 (0.092) 0.102 (0.102) 0.250 (0.250)
Chicken 0.029 (0.031) 2.586 (2.905) 0.008 (0.000) 0.096 (0.095) 0.088 (0.093) 0.102 (0.100) 0.245 (0.252)

† Published on two cores.

  • Spreads. Baselines vary by 1–5% per seed. OmSh varies more where shield work depends on the seed: Pursuit 0.323–0.362, Bertrand 0.245–0.262, MSH 1.134–1.189.
  • Reproduction. IPPO, IPPO-Lag and CPO match the published reward and U_H exactly in all six environments. OmSh matches in Bertrand and Chicken, and 0.979 against 0.980 in Congestion.
  • It differs where the WM and OM were retrained: MSH 0.603 against 0.592, Pursuit 0.294 against 0.251, Gathering 0.036/0.010 against 0.033/0.012.
  • Archive. /vol/bitbucket/oja24/_archive/omsh_extra/timing_clean_amd48_2026_10/ (its README.md). The live root and the clone were deleted on 15 Oct 2026; the paper's timing tables now use these values (omsh-paper 40d64dc). The AAMAS archive's 24 paper-table timing sidecars were moved to aamas/_superseded_timings/ (the AAMAS archive, _archive/omsh/aamas/); see docs/AI/operations/results-archive.md.