MILES GRPO documentation

Use python -m open_instruct.miles for new GRPO work with supported Olmo MoE models. Open Instruct owns the researcher configuration and data/reward integration; MILES owns rollout orchestration and shared RL machinery; the adapter trains through OLMo-core and serves through SGLang. Check model and workload support below.

The grpo.py Core/vLLM and grpo_fast.py DeepSpeed/vLLM entry points remain current alternatives for dense-model training, existing workflows and historical reproduction. See their GRPO guide for configuration and launch recipes. MILES also provides a dense Olmo 3 adapter, but that path is less extensively validated; consult model support when choosing a backend. Introducing MILES does not deprecate the existing GRPO trainers.

Start here

1. Start with the MILES GRPO guide for setup, the runtime image and a first run.
2. Check support and limits, model support and topology, and read the development defaults.
3. Copy a structured example.
4. Read the workflow, then follow the launch guide for your host. Planning does not require GPUs or mounted checkpoints.
5. Use operations to check completion and inspect run artifacts.

Documentation map

Document	Use it when
MILES GRPO	Setting up MILES GRPO, selecting its runtime image and launching a first run
Support and limits	Checking what the integration supports and its current limits
Development defaults	Choosing starting settings and what to weigh when changing them
Workflow	Editing one TOML from preparation through training and export
Launching	Submitting from a laptop or Beaker session, or executing in an allocation
Configuration reference	Looking up fields, defaults, aliases, restrictions and overrides
Native option appendix	Looking up an advanced MILES/SGLang flag or its original help
Models and checkpoints	Selecting a supported checkpoint, converting, saving, resuming or exporting
Core-compatible serving	Choosing the scoped fused-rounding default, opt-out and numerical reference modes
Throughput	Sizing trainer/inference allocation and engine admission, and reading queue, waste and warmup measurements
Async queues and discard metrics	Sizing producer and completed buffers, tracing waits, and measuring discarded work
Topology and capacity	Choosing GPU placement, async scheduling, batch geometry and serving capacity
Data and evaluation	Selecting tasks, importing mixtures and retaining held-out generations
Background evaluation	Configuring best-effort external olmo-eval jobs, receipts and W&B publishing
Managed judges	Binding named rubrics and placing fixed-weight judge services
Operations	Reading metrics, diagnosing failures and establishing run completion
Architecture and development	Understanding adapter ownership, runtime images and local checks
Long sequences	Choosing prompt/response budgets, admission and memory controls
Packing	Understanding document isolation and loss semantics
Compiler caches	Understanding cache restore/publication and bounded shutdown
Inference records	Recording every scored group's outcome by prompt and checkpoint for later selection and analysis
Experimental Megatron OPD	Running sampled-token on-policy distillation with a Megatron learner and SGLang teacher
Run-control semantics	Comparing Core controls with the Megatron implementation's terminology
Router auxiliary objectives	Selecting grouping, averaging, count source and coefficients
Implementation contracts	Reviewing detailed trainer, replay and publication checks

Authority and maintenance

Current instructions live in this directory. Example TOMLs express starting recipes. Configuration code, the pinned native parser snapshot, and runtime/miles/runtime.lock.json define accepted settings and runtime provenance.

Maintain the generated reference with python -m scripts.miles.generate_docs and check it with python -m scripts.miles.generate_docs --check. See development for validation.

See publication modes for current mixed-policy refresh, barrier controls and independent engine drain.