Skip to content

Cluster Execution

Dependency-aware launchers live in notebooks/experiments/scripts/. They validate environment names and scheduler targets, route notebook arguments to the correct stage, and pass only the algorithms selected for each job.

Supported Environment Keys

gathering, markov_stag_hunt, pursuit, bertrand, chicken

An explicit notebook directory or .py path is also accepted by the per-environment launchers. A misspelled bare key is rejected.

Slurm: One Complete Environment

bash notebooks/experiments/scripts/pipeline_env.sh \
  gathering training num_runs=3

The launcher submits WM, attaches OM with afterok:<wm-job-id>, then attaches the four primary algorithm bundles with afterok:<om-job-id>: IPPO, IPPO-Lagrangian, CPO, and OmSh-Monotone.

For a configurable environment:

bash notebooks/experiments/scripts/pipeline_env.sh \
  pursuit a100 n_agents=3 n_steps=1000000 num_runs=3

Argument routing is deliberate:

  • n_agents → WM, OM, and experiments,
  • n_steps → WM only,
  • venv and html → cluster runner,
  • other keys such as num_runs → experiments only.

The launcher rejects caller-provided run_*, run_algorithms, and just_vis because it owns the complete algorithm selection.

Slurm: Suites

The curated partition assignment is launched with:

bash notebooks/experiments/scripts/pipeline.sh num_runs=3

The full six-configuration matrix uses one selected partition:

bash notebooks/experiments/scripts/pipeline_all_envs.sh a100 num_runs=5

It includes every retained environment once, with Pursuit at both two and three agents. This creates 36 Slurm jobs: two preparation jobs and four primary algorithm jobs for each configuration.

After connecting to cpucluster, use bundles=4 to give every primary algorithm its own job:

bash notebooks/experiments/scripts/pipeline_all_envs.sh \
  amd48 bundles=4 num_runs=5

OmSh-Credible remains explicitly available for diagnostic ablations through env.sh ... algorithms=omsh_credible; it is not part of the default suite.

Imperial PBS Pro

The full one-environment PBS pipeline is:

bash notebooks/experiments/scripts/pipeline_env_hpc.sh \
  gathering num_runs=3

It submits GPU WM → GPU OM → four CPU algorithm jobs with native qsub afterok dependencies. The defaults are 16 CPUs/64 GB/1 GPU for WM and OM, 8 CPUs/32 GB/no GPU per algorithm, and 71:59:59 walltime. Override them with the OMSH_HPC_WM_*, OMSH_HPC_OM_*, and OMSH_HPC_ALGORITHM_* variables.

Leave the optional PBS queue unset to use Imperial routing, or provide it as the second positional value. The full six-configuration matrix is:

bash notebooks/experiments/scripts/pipeline_all_envs_hpc.sh num_runs=5

It creates 36 PBS jobs.

Existing Artifacts and Visualization

When compatible WM/OM artifacts already exist, env.sh (Slurm) and env_hpc.sh (PBS) submit only algorithm jobs. vis.sh performs CPU visualization reruns with just_vis=True and defaults to amd48.

Use DRY_RUN=1 to inspect commands without submitting. PROJECT_PREFIX changes the checkout path below $HOME/Projects; JOB_SCRIPT, SUBMIT_CMD, DEFAULT_VENV, and SUBMIT_DELAY_SECONDS control deployment details. Shared cluster checkouts must serialize their git pull --ff-only and uv sync updates before sibling jobs execute notebooks.

See the maintainer launcher contract for bundle composition, job-ID parsing, exact resource variables, and validation rules.