Architecture and development
The public entrypoint is python -m open_instruct.miles. Its structured RunSpec
compiles into a RunConfig containing CoreConfig and native MILES options.
The Python API is available for integrations; use the structured workflow when
preparation, placement and submission should come from one file.
| Layer | Owns |
|---|---|
| Open Instruct wrapper | Config validation, preparation, verifiers, launch/receipts, run artifacts |
| MILES runtime | Rollout actors, SGLang engines, sample/advantage/loss machinery, shared transport helpers |
| Core adapter (in MILES) | Model construction, document/replay alignment, scoring/training forward, gradient synchronization, optimizer/checkpoints and tensor export |
| OLMo-core | Model and distributed-training primitives, attention/expert execution |
| olmo-sglang | Serving architecture registration and model-specific execution |
The Core actor at miles.backends.core_utils.actor.OLMoCoreTrainRayActor implements
the MILES trainer boundary. Its model/checkpoint code and publication machinery
live in the pinned MILES fork. The driver coordinates collections, optimization and versioned weight
publication. Dense standard-model support is isolated from the specialized MoE
path. Implementation contracts describe the detailed lifecycle and
packing describes document/routing alignment.
Runtime sources and images
runtime/miles/runtime.lock.json is the
source of truth for exact dependency commits and the immutable binary base.
Builds fetch those commits without patches or moving branch tips. MILES uses the
organized adapter hooks; OLMo-core provides the model, objective and checkpoint
APIs required by this adapter; olmo-sglang supplies compatible serving models.
The ordinary Open Instruct environment keeps its own dependency versions.
The MILES pin includes the v0.1.2 async-consumer compatibility fix from miles-olmo-core PR #10. Its refresh and engine-drain dequeues accept the native consumer's optional trainer identity and preserve Core's explicit rejection of multi-policy training.
The binary base contains the compiled dependencies, including FA2, FA3 and FA4.
It excludes the MILES, Core, SGLang and Open Instruct source overlays. Application
builds add locked source repositories as runtime-base, then the committed Open
Instruct checkout as application. They do not compile FlashAttention, even
without a local Docker cache. Source-only lock changes rebuild source layers;
binary dependency changes require a new base. Editable installs still fetch
build-system dependencies such as setuptools and pybind11, and Megatron builds
a small C++ dataset helper. Those build-tool requirements are not fully pinned;
the source and base pins do not make application builds bit-for-bit reproducible.
The selected base is Beaker 01M5ZYQZVF7DDJ1VR619KRSS42
(robertb/miles-binary-base-20261027), Docker
sha256:947e615f93cb88f8739ba270f30adc7f012af8bc0abf43e96f11cd5d3b2d6354.
Its package versions, CPU imports and source isolation were checked, and a pull
by immutable ID reproduced the same Docker ID. The
single-GPU Core RL smoke
passed on application image 01M61Z2XBQB6EP3SJNAX1YN0HV: four optimizer updates,
shared-engine evaluations, checkpoints and HF export on Holmes. This tiny-model
check uses Torch attention; it does not qualify every FlashAttention backend or
a Qwen OPD GPU training run with the public Megatron sources.
The application image (robertb/miles-main-smoke-5a7be0398) was built from
5a7be0398e8244cd5376983427e57d4deeea2e39, Docker
sha256:83cc907a035f98ba69d3b8b7eaaaaa7477d1c06c6efca39da4414ef174df3c67,
and includes the current public source lock and review cleanup.
A CPU-only runtime-base rebuild from commit
3563eb52ac4b02d731786e8f1a4e39a52951f2a2 is published as Beaker image
01M61YFEWE1BATA8KQ4VAV2DY8 (robertb/miles-runtime-base-3563eb52a), Docker
sha256:f97469549508d2af9896161889cc45060c52ece5743ecfcbe937039d3043b784.
Its packaged lock matches the current lock, all six source checkouts match their
pinned revisions without patches, and serving/attention CPU imports passed.
This records CPU build verification of the runtime sources separately from the
application-image GPU smoke above, which subsequently verified the same source lock.
The source layer includes Megatron-LM from the Radix fork, pinned to
235952df607b3820716e5e67728a5ab470ca33ae (the previous runtime's revision,
including its Qwen3.5 FP32 optimizer fix), rather than the current miles-main
tip used by the pinned MILES Dockerfile. Megatron Bridge uses the public Radix
revision selected by that Dockerfile. These dependencies support Qwen OPD.
The image does not install olmo-megatron, the private Olmo Bridge fork, or
Olmo-specific Megatron patches; Olmo training uses the Core backend. Package
metadata and entrypoints are installed with --no-deps to preserve the binary
stack. /src/Megatron-LM remains a compatibility symlink for the OPD launcher.
EOPD metadata transport is implemented in the pinned MILES source, without a
local runtime patch. Open Instruct's opd_hooks stores teacher top-k values in
Sample.train_metadata;
rollout conversion
copies those dictionaries into train_data["metadata"], and the
Megatron training step
requests metadata from get_batch for the custom loss. This training field is
separate from the input dataset metadata selected by --metadata-key.
For ordinary application builds:
python scripts/miles/build_image.py --base-image LOCAL_LOADED_BASE_IMAGE --tag open-instruct:miles-core
Use --target runtime-base to build only the locked dependency sources, without
adding the Open Instruct application checkout. The default target is application.
The image arguments are local Docker tags loaded on the build host; Beaker names
and immutable IDs identify the images to pull before invoking the helper.
All repositories in the runtime lock are public; no GitHub credential is needed
for these source fetches. Standalone source preparation can use
--cache olmo-sglang=/path/to/clone.
Existing published images retain their original sources; rebuild after changing
the lock.
prepare_runtime refuses to replace existing source directories. Its --cache arguments can use local clones as fetch sources; they do not select uncommitted sibling code. build_image checks the immutable base Docker ID. For actual submission use the committed-image launcher, not a one-off Beaker command with different mounts or source provenance.
Rebuilding the binary base
The reproducible maintainer recipe is runtime/miles/Dockerfile.base; its two
immutable inputs are recorded in runtime/miles/base-build.lock.json. Pull both
Beaker images into Docker, then run:
python scripts/miles/build_base_image.py \
--compiled-parent LOCAL_COMPILED_PARENT \
--flash-attention-builder LOCAL_FA3_BUILDER \
--tag open-instruct:miles-binary-base
Both Docker IDs are verified, and the parent-image label comes from the same
lock. The helper passes the compiler image as FLASH_ATTENTION_BUILDER and the
runtime parent as COMPILED_PARENT. Only FA wheels are transferred from the compiler
image to the clean compiled parent. FA2/FA4 wheel downloads are checksum-verified;
the final stage resets inherited application metadata and entrypoints. This
preserves the qualified CUDA/PyTorch stack rather than rebuilding all dependencies
from an upstream CUDA distribution. Build on an amd64 Linux host: FA3 compilation
defaults to two jobs, with --max-jobs available for hosts with sufficient RAM.
The published base's FA3 build took about 31 minutes with eight jobs.
The final environment list was compared with the locked compiled parent. It
preserves its runtime settings while dropping legacy OLMO_MILES_* cache and
verifier paths. Recheck this list against the parent image whenever changing the
parent pin; FROM scratch does not inherit environment variables automatically.
Before changing the pin again, verify CPU imports, an application build and GPU forward/backward and training on the intended hardware. CPU imports alone do not establish GPU correctness.
Startup dependencies
Managed GRPO starts judge loading, Ray startup, and workflow model/data preparation
concurrently after assigning GPUs and discovering judge endpoint addresses. Ray
readiness and judge known-answer canaries run in separate supervised subprocesses.
The driver waits for the exact Ray layout before placement; it waits for judge
qualification before generation or evaluation. Readiness files live in the fresh
attempt directory, with peer-failure propagation. After replica rendezvous, rank
zero writes one startup.json deadline shared by all supervisors, readiness
probes, and driver barriers. Inventory and service launch consume that same
budget; starting a later stage does not reset it. Post-training completion has
its own timeout.
On disaggregated runs with an explicit rollout count and resident inference, trainer initialization/cache restoration overlaps inference initialization. Connecting trainer parallelism metadata and restoring the rollout cursor remains a barrier after both branches finish, before initial weight publication. Colocated, offloaded-inference, and epoch-derived schedules retain sequential initialization. OPD uses its separate launcher and does not pass through this GRPO supervisor.
A failed or cancelled initialization joins its sibling tasks before driver cleanup; the worker manager owns partially initialized remote workers. Cluster supervision also terminates outstanding readiness subprocesses. Judge liveness monitoring starts only after its HTTP endpoint becomes ready.
For measurement, the attempt directory contains process start receipts, readiness
timestamps. Each probe writes its own timeline under ray-readiness/driver_timing.jsonl
or judge-readiness/driver_timing.jsonl, avoiding concurrent appends on WEKA. The run
root has preparation-stage timings; the checkpoint directory has runtime parsing,
module import, readiness waits, serving/trainer initialization, connection and
publication timings. Overlapping stage durations must not be added together as
elapsed startup time. Small-model integration validates ordering and cleanup;
full-model wall-time savings require measurement on the intended hardware.
Local development
The ordinary CPU suite covers configuration, planning, launch specifications,
rewards, data preparation, rendezvous, heartbeat, recovery and topology contracts
without installing MILES or
SGLang. The MILES contracts workflow also checks the maintained scripts and
checks application types without requiring the private runtime.
Recording and summary implementation tests live with their MILES implementation;
Open Instruct keeps record configuration tests and runtime selection/buffer integration tests.
The moved adapter is checked against that Core API in the dedicated runtime
environment; Open Instruct's ordinary type check covers the application integration.
The MoE numerical tests deliberately require the Core revision in the runtime
lock. The ordinary uv.lock Core version is for existing SFT/DPO paths and is
not upgraded to make those runtime-only tests importable.
uv run pytest open_instruct/test_miles*.py
The dedicated runtime suite uses the separate application image. From its
/opt/core-rl working directory, run:
bash scripts/miles/test_runtime.sh -q
This command fails if runtime dependencies are missing. Without the explicit
runtime flag, ordinary CPU collection omits tests/miles/ when MILES or SGLang
is absent. CUDA numerical cases skip when no GPU is visible. A passing CPU run
therefore does not check GPU kernels, distributed gradients or publication.
The ordinary Open Instruct GPU CI image does not include the private MILES
runtime; dedicated runtime checks are a separate maintainer validation step.
For a training lifecycle check, copy dev.toml
to runs/, select a compatible tiny checkpoint, and follow the
committed-image launcher. Verify the rendered replica group and
retain image, config and completion artifacts. Test refresh separately when
changing publication behavior; a tiny mechanics check does not establish learning
quality.
Documentation checks
python -m scripts.miles.generate_docs --check
uv run pytest open_instruct/test_miles_documentation.py
uv run mkdocs build
make style-check
make quality-check
MkDocs generates the four reference pages in memory from configuration sources,
the compact pinned parser schema and reviewed reference-help.json descriptions.
Neither the pages nor native help are tracked. Ordinary documentation builds and
local plan/validate commands need no MILES installation. Without a native help
capture, option tables show names, types, choices, restrictions and reviewed
supplements; other help and value defaults are explicitly marked unavailable.
When runtime pins change, regenerate the compact schema inside the matching runtime and review its diff:
python -m scripts.miles.snapshot_options open_instruct/miles/configuration/options.json
For full upstream help and parser defaults, capture an artifact in the pinned runtime using a writable checkout, then supply it to the documentation build:
python -m scripts.miles.capture_option_help --image IMMUTABLE_IMAGE_ID --output runs/native-help.json
MILES_NATIVE_HELP=runs/native-help.json uv run mkdocs build
The capture checks the installed parser against the expanded compact schema;
the docs build rejects stale hashes or missing actions. Parser defaults are not
model-dependent resolved defaults. To inspect Markdown separately, run
python -m scripts.miles.generate_docs --output-dir runs/miles-docs (optionally
with --native-help runs/native-help.json). --check validates the reference
inputs without writing files. Edit source definitions and reviewed descriptions,
not generated output. Change maintained examples only as a deliberate recipe
change; see development defaults.
Type checking the moved adapter
In a development environment with the pinned MILES and Core sources, run:
ty check /path/to/miles/miles/backends/core_utils --extra-search-path /path/to/OLMo-core/src
Open Instruct's CPU checks do not fetch the private MILES repository. The committed application image and runtime tests check both repositories together.
Adapter package layout
open_instruct/miles/ groups code by responsibility. The public CLI remains
python -m open_instruct.miles; package initializers do not load the GPU runtime.
| Package | Responsibility |
|---|---|
configuration |
RunSpec/CoreConfig, native options, validation, topology and capacity planning |
execution |
Preparation, Beaker submission, cluster bootstrap and training coordination |
MILES backends/core_utils |
Core actor, model backends, packing, optimizer scheduling, checkpoints and trainer diagnostics |
rollout |
Application data-source selection and rollout metrics; managed generation, admission and completed queues live in MILES backends/core_utils/rollout |
MILES backends/core_utils/publication |
Weight delivery, engine drain, policy versions and durable policy state |
rewards |
Verifiers, reward routing and managed judges |
datasets |
Dataset preparation, mixtures, inference records and prompt selection |
evaluation |
Background evaluation submission, workers and result publication |
infrastructure |
Application artifact publication and bounded infrastructure waits; compiler/HF cache lifecycle lives in MILES |
Keep diagnostics next to the component they observe. Keep configuration imports
CPU-only, model backends loaded on demand, and package initializers minimal.
MILES backends/core_utils/data.py adapts samples to the trainer; rollout/data_source.py owns
runtime data-source behavior; datasets/run_data.py prepares researcher inputs.
Python integration imports use these package paths, for example
from open_instruct.miles.configuration.run_spec import RunSpec. The pinned
AllenAI MILES fork must use the matching trainer and rollout hook paths. Rebuild
the application image when updating this layout and its runtime pin together;
existing images retain their original code. Custom verifier factories and saved
native options must reference these package paths.
The CPU CLI checks in open_instruct/test_miles_package_layout.py run without
site-packages. tests/miles/test_package_layout.py resolves hooks supplied by
both repositories inside the pinned runtime.
Application callbacks and CPU tools
The backend trainer and managed rollout modules do not import Open Instruct.
Open Instruct supplies core_records_factory for per-question recording and
core_evaluation_snapshot for requested post-update HF exports. Those callbacks
preserve the existing recording and evaluation schedules. Dataset preparation,
reward callbacks, data-source selection, Beaker launch and evaluation submission
remain application-owned. The native olmo_core argument loader still resolves
Open Instruct's run configuration; that is an explicit configuration integration.
Small CPU planning and artifact-reader helpers remain local so plan, validate
and offline analysis work without the private runtime. Runtime integration tests
compare their algorithms with the MILES counterparts: scoring policy, capacity
reporting, timeout diagnostics, atomic JSON publication, input validation and
throughput arithmetic.
Core runtime compatibility
The olmo-core revision in
runtime.lock.json is the Core source used
by the runtime. The adapter interfaces it relies on were merged in
OLMo-core PR #888. The runtime pins
merge commit 6ceda6530625caa5d23e3be8a455c600f62a7338, including the reviewed
RoPE import fixes, alongside the MILES v0.1.2 integration.
Core's routed-expert match_eager_rounding defaults to false. The pinned MILES
adapter explicitly enables it for policy scoring and accepts legacy MILES
checkpoint manifests with equivalent scoring semantics. Other Core users keep
the default behavior.
Core's PR checks and application validation cover different boundaries. Rebuild runtime/application images from the lock and validate the intended training topology; an existing image retains its original source revisions. Record application validation with the exact image and source pins. The compact options snapshot must be regenerated whenever runtime source pins change.