Skip to content

Launching MILES runs

This guide covers the current config-driven launcher. It does not require sibling MILES or Core worktrees. The selected runtime image contains the training dependencies and adapter sources. See architecture for building and updating that image.

Prepare a submitting host

Use a checkout of the intended Open Instruct revision.

Python 3.12 is sufficient for CPU-only planning and submission from the checkout; those modules use the standard library. Do not install the full CUDA training environment on a laptop merely to submit a run. Install and authenticate the Beaker CLI using AI2 setup; the account needs access to the configured workspace, budget, secrets, images and WEKA filesystems. Building an image additionally requires Docker and jq.

git clone https://github.com/allenai/open-instruct.git
cd open-instruct
uv python install 3.12
uv venv --python 3.12 "$HOME/.venvs/oi-miles-submit"
source "$HOME/.venvs/oi-miles-submit/bin/activate"
beaker account login
beaker config test
beaker account whoami

The virtual environment directory must be ignored or outside the repository so that it does not make the checkout dirty. Store personal run files in Git-ignored runs/ (also excluded from Docker build contexts), or outside the checkout. All paths below are shell examples; replace checkpoint/output paths, usernames and image placeholders.

mkdir -p runs
cp configs/miles/examples/dev.toml runs/my-grpo.toml
# Edit name/output.root to use your username and a fresh run path.
# Select a tiny checkpoint that fits on one trainer GPU; this starter uses two GPUs.
python -m open_instruct.miles plan runs/my-grpo.toml
python -m open_instruct.miles validate runs/my-grpo.toml

plan and structured validate inspect schema/options without reading model weights or allocating GPUs. They do not verify remote mounts, credentials, memory fit or the installed runtime. Input paths are resolved relative to the TOML location, with no implicit environment substitution. WEKA paths need not exist on the submitting host.

Laptop: choose or build an image

Choose a starter from the three maintained examples and an image built from the intended application revision and runtime lock. Building requires access to the private sources described in architecture. Supply model/data paths and any secrets needed by your run. Beaker resource permissions are required.

For an already built compatible image, obtain its immutable ID and source provenance from its maintainer. Inspect its metadata:

beaker image get IMMUTABLE_IMAGE_ID --format json
export MILES_EXISTING_IMAGE=IMMUTABLE_IMAGE_ID
python -m open_instruct.miles run runs/my-grpo.toml

MILES_EXISTING_IMAGE accepts an immutable Beaker ID, not an alias. The launcher checks the ID, but does not prove source compatibility with your checkout. The submitted config is carried into the job; your local Python changes are not. Use an image built from the intended source revision. The launch receipt records the submitting revision and selected image separately. In particular, the job calls open_instruct.miles.execution.preflight_network and open_instruct.miles.execution.preflight_attention; an image missing these entrypoints must be rebuilt.

The cached B300 preset also requires the FP32 tracked-state snapshot implementation in the current runtime lock. plan and image-ID validation do not check that an existing image contains it. Verify its build records against runtime/miles/runtime.lock.json, or rebuild before allocating a run.

To deploy source changes, commit them and build the source overlay instead. Read the immutable base identity from the lock and pull it into local Docker:

unset MILES_EXISTING_IMAGE
base_id=$(python -c 'import json; print(json.load(open("runtime/miles/runtime.lock.json"))["base_image"]["beaker"])')
beaker image pull "$base_id" miles-core-base
export MILES_BASE_IMAGE=miles-core-base
python -m open_instruct.miles run runs/my-grpo.toml

The builder checks the loaded Docker image ID against the lock, builds the source overlay, uploads it to Beaker, then submits. It reuses a matching image when one already exists for the commit. This is not an automatic blessed-image resolver. Do not use the general Open Instruct/vLLM auto image for MILES.

Both modes require a clean, committed checkout and delegate through scripts/train/build_image_and_launch.sh --miles. run performs launch checks before building, including mount coverage. The MILES image wrapper passes the frozen configuration directly to open_instruct.miles.execution.submit; no debug launch script is involved.

Host memory and allocated GPU slots

MILES replica startup queries nvidia-smi for device identity and memory, including generic runs with an explicit numeric CUDA_VISIBLE_DEVICES mask. An unavailable or malformed inventory, or a query exceeding 15 seconds, stops startup. An explicitly empty mask or -1 exposes no GPUs; an unset mask selects all devices.

launch.memory requests host RAM per Beaker replica (default "256 GiB"); launch.shared_memory separately sizes shared memory. Both GRPO and OPD launchers honor these fields. They do not set SGLang GPU memory pools.

Beaker can assign extra GPU slots to satisfy a host-memory request. During an H100 inference check on Jupiter, one requested GPU with 256 GiB of host RAM received two GPU slots. Inspect the assigned resources in job events and record both requested/active GPUs and allocated GPUs when reporting throughput or cost. For a small inference-only workload, a lower request such as launch.memory = "128 GiB" with launch.shared_memory = "64 GiB" may avoid this extra allocation. Check the host working set, model loading and response-buffer peaks before reducing it; do not copy that limit into a full trainer job without measurement.

Jupiter CUDA compatibility

For MILES GPU runs, selecting launch.cluster="ai2/jupiter" or "ai2/saturn" (both have CUDA 12.8 host drivers) automatically sets the compatibility library path inside the Beaker job. The resolved value appears in plan, the submitted configuration and the launch receipt:

[launch]
cluster = "ai2/jupiter"
priority = "high"
# Retain the run's GPU count, mounts, timeout and preemption/resume settings.

[launch.env]
LD_LIBRARY_PATH = "/usr/local/cuda/compat:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64"

The September 2026 Jupiter H100 checks initially failed CUDA preflight because the host driver was older than the pinned image's CUDA 13 runtime. Retrying with the same image and /usr/local/cuda/compat first in LD_LIBRARY_PATH passed; subsequent paired hero-checkpoint smoke runs completed four optimizer updates. The [launch.env] entry above shows the default and can be omitted. An explicit value replaces the default exactly; it is never prepended or merged. For example, to use the host driver libraries without the compatibility directory:

[launch.env]
LD_LIBRARY_PATH = "/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64"

The CLI also accepts --set 'launch.env.LD_LIBRARY_PATH="/custom/lib"'. Other clusters receive no automatic library-path override, and explicit Beaker secret references retain precedence. Exporting the variable only on the submission host does not configure the workers. Older submission checkouts need the explicit [launch.env] setting shown above.

Keep the pinned MILES runtime. The ordinary launcher's --cuda-version 12|13 switch cannot be combined with --miles. The override uses compatibility libraries already present in the image; it does not upgrade the host driver or establish support for arbitrary image/driver combinations. After changing either, verify CUDA initialization and a short training lifecycle on the target hardware. Size the workload for Jupiter's H100 memory instead of copying B300 memory budgets. FlashAttention 4 (flash_4, the default elsewhere) supports only Blackwell, so Jupiter (H100) runs default core.attention_backend to flash_3 and Saturn (A100) runs to flash_2; the pinned runtime provides both through runtime/miles/Dockerfile. An explicit value takes precedence.

Then follow the image selection and plan/validate/run steps above, using a fresh output path. This setting does not change scheduling or checkpoint recovery.

Distributed scheduling contract

A distributed run must be submitted as one task with replicas: N and leaderSelection: true. Beaker treats that replica group as a scheduling unit. Putting N independent tasks in the same experiment does not give them this guarantee. Shared WEKA rendezvous and failure propagation coordinate processes; they do not create a scheduler replica group.

For example, the scheduling fields for two eight-GPU replicas are:

tasks:
  - name: distributed-run
    replicas: 2
    leaderSelection: true
    hostNetworking: true
    propagateFailure: true
    propagatePreemption: true
    synchronizedStartTimeout: 60m
    resources:
      gpuCount: 8
    context:
      priority: high
      minRuntime: 4h
    timeout: 6h

This is a scheduling excerpt, not a complete submission. It requests 16 GPUs as a group. A positive minRuntime selects allocated scheduling and provides a preemption-protected window; it neither groups independent tasks nor promises an immediate start. Set launch.min_runtime = 0 (without units), "0", "0s", or "" for unallocated scheduling; blank and zero inputs are submitted as "0s". This is allowed for both single-node and distributed runs. Omitting the field also selects unallocated scheduling ("0s"); a protected window must be explicit.

timeout independently bounds execution and may be shorter than min_runtime; zero requests no explicit execution timeout, subject to Beaker's cluster policy. plan and validate check duration syntax and report scheduling warnings without requiring a protected minimum runtime. Beaker enforces its scheduling limits. Jobs that can outlive their protected window warn when periodic checkpoints and auto-resume are not both enabled. Synchronized start bounds the wait for replica startup and is separate from application rendezvous. Beaker supplies BEAKER_REPLICA_RANK and BEAKER_REPLICA_COUNT to the replicas.

Before building/submitting, render the actual launcher output using the submission environment and inspect it against this contract:

python - <<'PY'
import json
from open_instruct.miles.execution import launch
from open_instruct.miles.configuration import run_spec

spec = run_spec.RunSpec.load("runs/my-grpo.toml")
document = launch.specification("IMMUTABLE_IMAGE_ID", spec)
print(json.dumps(document, indent=2))
PY

Verify the replica group, replicas * resources.gpuCount, context, placement, mounts and immutable image. plan and validate alone do not verify Beaker scheduling. After submission, beaker experiment spec EXPERIMENT_ID expands replicas into separate task entries and omits the group fields; do not infer grouping from the exported task count. Verify the newest job attempt for each task and replica rank instead (python -m open_instruct.miles status reports the same per-replica view):

beaker experiment get EXPERIMENT_ID --format json | jq '.[0].jobs
  | group_by([.execution.task, .execution.replicaRank])
  | map(max_by(.status.created) | {
      id, status,
      group: .execution.replicaGroupID,
      rank: .execution.replicaRank,
      leaderSelection: .execution.spec.leaderSelection
    })'

Earlier attempts of a retried replica are superseded and can be ignored. All current replicas must share a nonempty group ID, cover ranks 0..N-1, and have leader selection enabled. Inspect scheduler events for each current job.

Distinct-host placement is an additional requirement of the MILES bootstrap. Multi-node launches currently require launch.gpus_per_replica=8 and eight-GPU physical nodes. Partial-node replicas may share a host and are rejected by the launcher. Separate tasks with disjoint hostname pools are not an acceptable replacement for group scheduling. Other physical node sizes need explicit placement support before launching. Extending rendezvous timeouts or polling for idle nodes does not repair a missing group.

From a Beaker session

To submit a new allocation, use the same existing-image procedure above. Use the session's provisioned BEAKER_TOKEN or authenticate with beaker account login, check beaker account whoami, and select a compatible immutable image; Docker is unnecessary. Session GPUs and mounts are not inherited by the new job: its TOML must request everything it needs. Do not use a shared Docker daemon just to resubmit a known image.

To train inside an existing single-node allocation, the compatible Core RL runtime must already be installed, with all configured GPUs, input paths, output paths and credentials available. From the runtime checkout:

cd /opt/core-rl
python -m open_instruct.miles validate /path/to/run.toml
python -m open_instruct.miles train /path/to/run.toml

train does not allocate nodes, mount WEKA or provision managed judges. Use run for multi-node or managed-judge jobs: its cluster bootstrap coordinates ranks, Ray and services. Do not manually start one independent train on each node. The submitted workflow runs the attention preflight for the supported torch/FA4 selection before training.

Before attention checks or model/data preparation, each submitted container checks NCCL_SOCKET_IFNAME against its visible interface names, honoring NCCL prefix, exact-match and exclusion selectors. An unmatched selector fails immediately, except for exactly NCCL_SOCKET_IFNAME=ib in a single-replica job using bridge networking: that case logs a warning and removes the override so NCCL selects an interface automatically. The updated environment is inherited by attention checks and training. Host-networked jobs (including single-node jobs with judge bindings) and multi-node jobs never use this fallback. Prefer leaving the variable unset for single-node runs. This check verifies interface names, not connectivity or NCCL collective health; local validate cannot inspect the future container's network.

Placement, secrets and results

At AI2, GPU examples use Holmes, ai2/open-instruct-dev, high priority and minimum runtimes of one hour (dev), four hours (node) or eight hours (three-node). CPU-only WEKA jobs are unallocated (no context.minRuntime, keep the timeout and mounts) and run on Saturn, falling back to Jupiter after canceling an unschedulable attempt; not on Holmes. Set every model/template/data/output/cache filesystem in launch.weka_mounts. Multi-node runs and background evaluation require output.root inside one of these shared mounts: replicas coordinate through files there, and evaluation jobs read saved snapshots and write results there. The mount path can be any absolute directory, such as /shared/training; it need not start with /weka/. The Beaker launcher checks mount coverage at submission; plan and validate check configuration and GPU allocation without assuming a storage provider.

The coordination code requires the same shared directory on every replica, cross-node file locking, atomic renames and visibility of other replicas' writes. The Beaker launcher provisions WEKA mounts; other shared filesystems and launchers are untested.

See topology for replica and engine counts and managed judges for preparation and placement.

[launch.secrets]
HF_TOKEN = "your-hf-beaker-secret"
WANDB_API_KEY = "your-wandb-beaker-secret"

These are secret names, not token values. Use launch.env only for nonsecret strings. Reserved Ray/rank/CUDA environment variables belong to the launcher. Offline W&B examples do not require a W&B credential.

python -m open_instruct.miles status runs/my-grpo.toml
beaker experiment get EXPERIMENT_ID --format json
beaker job events JOB_ID
beaker job logs JOB_ID

status uses receipts in ~/.cache/open-instruct/miles/launches; override with MILES_LAUNCH_RECEIPTS. Keep that directory when switching submitter hosts. Each attempt includes a state and reason alongside Beaker's raw status. Cancellation takes precedence over exit codes, so preemption is reported as preempted even when the process exits nonzero. Node unavailability, failed health checks and sibling task retries are reported as interrupted. auto_resume reports the submitted setting (or null for older receipts without it); neither preempted nor interrupted promises a retry. stopped means Beaker reports an exited or finalized attempt without a success/failure outcome. Scheduling blockers remain pending, with their reason. Inspect the latest attempt for each task and replica rank, not the first job of a retried experiment. Scheduler events explain pending jobs. Small reports/logs are copied to Beaker results; checkpoints and rollout tensors remain on WEKA. See operations for completion, recovery and artifact interpretation.