MILES with an OLMo-core trainer
Use the operating guide for MILES GRPO runs and the support matrix for supported boundaries. The adapter replaces the MILES Megatron trainer actor, while retaining MILES rollout actors, samples, advantages, PPO loss helpers and transport primitives. Open Instruct supplies configuration, data/verifiers, lifecycle coordination and run artifacts.
Trainer hookup
miles/backends/core_utils/actor.py implements the trainer actor contract. The
models.py facade selects moe_models.py for the specialized OLMoDDP MoE path
or standard_models.py for dense Core TransformerTrainModule/FSDP. MILES calls
the actor for initialization, scoring, training, save/restore and weight export.
The Core objective hook accepts MILES' token log-probability loss and performs
backward, distributed gradient synchronization and optimizer stepping through
Core. It does not select the MILES FSDP trainer or construct a Megatron model.
driver.py coordinates collection, optimizer completion, publication, evaluation
and checkpoint boundaries. A policy version advances only after a successful
optimizer step. The async producer attaches behavior versions to samples; the
trainer checks their age again when consuming the batch.
The runtime uses the exact OLMo-core commit recorded in
runtime.lock.json, including the
optional trainable FP32-output head and production MoE support.
Image builds fetch the full commit in the runtime lock without applying a Core patch.
The port retains custom objectives, routing replay/count controls, bounded
checkpoint planning and streaming HF interchange, including the inherited
per-head attention and hybrid configuration export support.
Local checks cover model/configuration roundtrips, checkpoint planning, adapter contracts and a single-GPU hybrid-MoE scoring/backward/optimizer step. Validate other topologies and feature combinations before relying on them. Use the runtime lock and build procedure to reproduce the current source.
Samples, scoring and objectives
Core microbatch size is one. Optional packing combines complete samples in a document-isolated forward without crossing optimizer partitions. Global batch size must divide collections into whole optimizer steps and be divisible by trainer world size. Every rank executes the same number of forwards/backwards. Masks, response boundaries and next-token logits remain attached to their original sample.
A standalone scoring pass supplies an unchanged trainer-policy anchor when needed. With one optimizer step per collection, no reference KL and no dropout, the training forward can supply that anchor. The first update of each process and periodic checks compare the two forwards before permitting the skip. See scoring controls for forcing the pass and its tolerance. Async TIS corrects the behavior-versus-trainer ratio; it does not make stale data on-policy. Allowed lag is in optimizer steps, including multiple steps in a collection. Async requires positive lag and resident disaggregated engines.
Core's router auxiliary/z losses are independent of PPO clipping, advantage normalization and TIS. Replay fixes selected expert IDs; router probabilities and weights still participate in differentiable computation. Packing retains Core pretraining's local-batch auxiliary loss over each packed forward's tokens. It is not a sum of independently balanced response losses.
Router replay and parallelism
Enable use_rollout_routing_replay=true. Core uses the standard Rust router by
default; the pinned router preserves expert IDs and policy-version spans. The
captured IDs travel with each sample through DP distribution and packing. The
adapter aligns replay by document/token/layer and retains it across backward
recomputation. Each sample's final unscored token has an explicit synthetic tail;
that convention is not a claim of Megatron auxiliary-loss equivalence.
Expert-parallel ranks receive the routing information needed by their local
forward; trainer PP/TP/CP greater than one are rejected, not implicitly supported.
A replay match verifies supplied assignments; it does not measure what the trainer
would have selected without replay.
Checkpoints and publication
Refresh publication pauses and resumes inference at the engines without waiting for producer admission acknowledgments. The driver still waits for publication and the rollout executor's weight-version update before consuming the next batch. On a failed refresh run, cleanup cancels outstanding producer work instead of draining requests that may be waiting on paused engines. Shared-engine evaluation and successful shutdown retain their separate generation-drain boundaries.
SGLang starts from HF config/tokenizer/weights. Core imports those weights into
its native training representation. Updated policies are streamed directly as
serving tensors; no HF directory or Megatron conversion is written per update.
MoE export supports fused expert tensors; final HF exports retain standard
per-expert slices. Dense export gathers parameters through its FSDP backend.
The normal disaggregated transport broadcasts flattened NCCL buckets; colocation
uses IPC. Current full-model starters use mixed-policy refresh; low-level
barrier and independent engine drain remain distinct modes.
Publication contracts explain recorded behavior
probabilities, final-forward replay routes and oldest-token lag checks.
Native checkpoints include model/optimizer/scheduler and rank RNG state. Schema-2 manifests record world size, EP degree and the committed rollout cursor. Restore validates these before native loading collectives. The data cursor includes pristine pending prompts; partially generated responses are regenerated after restart. Serving RNG and exact future generations are not restored. Schema-1 multi-rank checkpoints require explicit migration, and topology changes on restore are not supported.
A deliberate debug_exit_after_rollout stop preserves a resumable workflow.
Final HF export is deferred until the configured collection horizon completes;
the first process must not occupy the final export directory prematurely.
See model/checkpoint operations.
Training contract checks
Every real training step records training_contract_rankN.jsonl beneath
miles.save, when supplied, and emits the same records to logs. Records include:
- Actual global sample, active-token and model-token counts; policy and auxiliary denominators; local accumulation count; response-versus-token reduction mode.
- Local normalized policy objective and weighted auxiliary terms separately. Averaging these local objectives across ranks gives the corresponding global objective; they are not already global metrics.
- Consumed/published policy versions, LR used/next, completed step and elapsed time.
- Globally reduced probability-error histograms, exact maximum, upper-bin estimates of p50/p95/p99, response-position thirds and response-length buckets. A null quantile upper bound means the overflow bin (>1.0), not missing observations.
core.diagnostic_interval=N additionally measures local gradient norms before
optimizer intake and sampled model updates every N optimizer steps. These are
explicitly not global optimizer norms: expert gradients may still need Core's
EP-MP rescaling, and FP8 stores outside named_parameters are outside coverage.
Update samples retain at most 256 values per named parameter, rather than cloning
the model. With check_weight_update_equal=true, the driver also checks serving
weights at publication boundaries whose rollout count is divisible by N
(and on initial publication). A fresh run compares its initial
publication against SGLang's original HF snapshot. Periodic and resumed checks
snapshot the current serving state, reset tensors, republish the same trainer
version, and compare exactly when quantization tolerance is disabled, over the
non-skipped tensors. This catches missing or inconsistent transfers;
it is not an independent proof of the export mapping for changed weights.
Logprob checks and the initial HF comparison are separate checks. The
round trip adds a second full transfer, with repeated_version=true in the
publication log; include both transfers when measuring diagnostic overhead.
The async producer remains paused until the check completes.
The interval defaults to zero, leaving these periodic probes disabled. The maintained
examples inherit that default. The dev mechanics example enables
check_weight_update_equal=true: the full weight audit runs at startup, including
resume, and does not run after ordinary training updates. The node and three-node
learning examples disable this optional audit. Reserve a positive
interval for investigating publication correctness. This interval
also controls the extra gradient-norm and sampled-parameter-update diagnostics;
setting it to zero does not disable the separate policy-version, logprob, replay,
or scoring-pass checks. Upstream MILES supplies the optional startup check; the
periodic same-version round trip is implemented by the MILES Core driver.
Runtime failures include non-finite active inputs/rewards, empty effective batches, inconsistent rank schedules, stale policy versions, non-finite losses, scheduler/clock disagreement and skipped optimizer steps. All ranks agree on skip status before any policy clock advances. The mean score-drift guard is configurable; tail distributions are recorded but have no fixed failure cutoff. A skip aborts the run; this is not transactional rollback of ranks that already performed an optimizer update.
Development checks
The independent fixed-batch reference checks next-token slicing, masks, unequal lengths, PPO clipping, entropy, KL and loss reduction without reusing the loss helpers it is testing. Tiny native EP1/EP2 comparisons cover policy-only, auxiliary-only and combined gradients/Adam state, with recomputation on/off. These are native consistency checks, not full-model Megatron equality.
Run the current packaged regression suite inside the pinned application image:
bash scripts/miles/test_runtime.sh -q
The packaged suite does not run distributed experiments; use the structured examples for end-to-end checks.
Router auxiliary objectives
These optional controls change the MoE router's auxiliary losses. They are independent of the RL policy loss's token-versus-response normalization and of whether routing replay is enabled.
Field under [core] |
Default | Alternative and meaning |
|---|---|---|
router_aux_loss_grouping |
"pack" |
"sequence": compute balancing statistics separately for each original prompt-plus-response document. |
router_aux_loss_reduction |
"token" |
"response": give each document equal weight instead of weighting by model-token count. |
router_z_loss_reduction |
"token" |
"response": independently give each document equal weight for z-loss. |
router_aux_count_source |
"executed" |
"router_selected": use the current forward's router top-k choices to count expert selections for balancing. |
router_aux_loss_weight |
0.01 |
Balancing coefficient; 0.0 disables its gradient contribution. |
router_z_loss_weight |
1e-5 |
Z-loss coefficient; 0.0 disables its gradient contribution. |
Grouping, then averaging, then the coefficient
Grouping decides which tokens share an expert-usage histogram. "pack" shares
one histogram across the physical forward; "sequence" makes a separate histogram
for each original document even when several documents occupy one pack. Neither
choice changes attention boundaries or physical packing.
Averaging decides how each document contributes to the update. With sequence
grouping, suppose two documents have 2 and 6 model tokens and their balancing
penalties are 1 and 3. Token averaging gives (2*1 + 6*3) / 8 = 2.5; response
averaging gives (1 + 3) / 2 = 2. A balancing coefficient of 0.01 makes their
contributions to the training objective 0.025 and 0.02, respectively. The same
weights and coefficient scale their gradients.
With pack grouping and response averaging, documents share the pack's expert histogram, but each document's mean router probabilities has equal weight. Response averaging does not implicitly select sequence grouping. Z-loss has no expert histogram; its averaging is selected independently.
Here a document is the prompt plus its response, including the final forwarded token and its synthetic replay assignment. These are model tokens, not only policy-loss tokens. Padding is excluded. The token and document denominators span the optimizer update and data-parallel ranks, accounting for gradient averaging; this is not an average of microbatch means.
Executed versus router-selected counts
"executed" counts the actual selected experts. With replay active, those are the
replayed assignments. "router_selected" instead derives a detached top-k histogram from
this forward's router scores. It changes the balancing statistics only: replayed
expert dispatch, selected expert weights, load metrics and z-loss keep their
existing meanings. Gradients still flow through router probabilities; the discrete
expert counts themselves are not differentiable.
Count-source selection works with both grouping choices and both averaging choices. It does not change rollout publication, discard behavior, or inference routing. Matching this setting alone does not establish equivalence with Megatron.
Configuration examples
The following explicitly spells out the defaults:
[core]
router_aux_loss_grouping = "pack"
router_aux_loss_reduction = "token"
router_z_loss_reduction = "token"
router_aux_count_source = "executed"
router_aux_loss_weight = 0.01
router_z_loss_weight = 1e-5
For equal document weighting and router-selected balancing counts, change the following fields in a copied run configuration:
[core]
compile_model = false
router_aux_loss_grouping = "sequence"
router_aux_loss_reduction = "response"
router_aux_count_source = "router_selected"
# Z-loss stays token-weighted unless router_z_loss_reduction is also changed.
Set either coefficient to 0.0 to disable that auxiliary loss; set both to zero
to disable both. These are experimental objective choices; see
development defaults for the example settings.
Supported scope and runtime
The all-default pack/token/token objective uses Core's native router methods;
router-selected counts with that objective use Core's native count-source option. Other
grouping/averaging combinations use document metadata preserved across backward
recomputation. They require one unpadded instance per forward (which can contain
multiple packed documents), trainer TP=CP=1, compile_model=false, and no global
balancing or router orthogonal loss. Packed and unpacked execution are supported.
Router-selected counts additionally require plain softmax, local balancing, and no routing biases, expert groups, or uniform/random assignment overrides. EMO routing with router-selected balancing counts is not yet supported. Unsupported native routing combinations fail during model construction. Router objective controls require the Core MoE backend.
The Core commit pinned in runtime/miles/runtime.lock.json includes these
controls; images built from an older lock do not. Selecting router-selected counts
with an older Core dependency fails explicitly.
tests/miles/test_adapter_router_objective.py checks losses and gradients against
an independent reference for every grouping/averaging/count-source combination,
with and without activation recomputation. It also checks unchanged replay
outputs and policy gradients. The pinned Core source includes native count-source
regressions. tests/miles/router_objective_contract.py supplies GPU training and
repacking checks; its --count-source option selects either count source. CPU
checks do not cover distributed GPU execution of every combination.
FP32-output vocabulary head
Opt in with trainer.fp32_lm_head=true in a structured run, or
core.fp32_lm_head=true in the low-level configuration. This sets Core's
LMHeadConfig.fp32_output and SGLang's enable_fp32_lm_head together.
An explicit conflicting serving false value is rejected. Default is false.
Images built from an older runtime lock do not include this option.
CUDA BF16/FP16 projections retain FP32 output instead of rounding logits to the operand dtype. Weights remain in their original storage dtype. A custom backward uses ordinary low-precision gradient GEMMs, including rounding the incoming logit gradient to the operand dtype; this is not full-FP32 training. Autocast continues to select operand precision. Checkpoint parameter names are unchanged. The native Core option currently supports the default head and default loss, without tensor-parallel wrapping. Its low-precision CUDA path supports first-order gradients; higher-order derivatives and fused-linear cross entropy are excluded.
The option targets training-versus-serving probability mismatch; its effect on learning is not established. FP32 logits double that output tensor's storage; measure training memory and step cost for the chosen token/vocabulary sizes. See development defaults.
Prepared batch overlap
core.prepared_batch_overlap defaults to true. It applies to fully asynchronous,
disaggregated refresh runs with one optimizer update per collection, publication
every update, and completed-buffer capacity of at least one collection. Other
schedules retain sequential acquisition; set the option to false to disable it.
Startup logs report the first unmet eligibility condition when overlap is disabled.
While update N trains, the executor reserves and prepares one batch for the currently published policy version plus one. It runs the existing sample postprocessing, diagnostic submission, reward normalization, conversion and object-store handoff. GPU packing, current-policy scoring and trainer validation still run on the trainer. Publication must report the predicted version before this batch can be consumed; an unexpected version fails rather than training on an incorrectly selected batch.
Reserved prompt groups remain in the checkpoint ledger until consumption. They also occupy their original completed-buffer slots, so prefetch does not increase the configured pipeline depth. Checkpoints regenerate prepared-but-unused prompts on resume. Cancellation joins collection on the producer loop, requeues pristine prompts and releases prepared object-store references. Shared-engine evaluation keeps sequential acquisition because its generation-drain boundary temporarily changes buffer capacity.
rollout/prepared_batch/* reports preparation duration, executor-side consumer
wait and how long the batch was ready before consumption. Continue using the
driver's generation_wait stage for the full exposed wait: executor metrics do
not include RPC dispatch delay. The completed-queue consumer-wait metric identifies
waiting for eligible generations within preparation. A longer preparation time
can therefore be harmless overlap, or genuine inference wait; inspect these
metrics together.
Policy scoring activation
MILES selects Core's supported fused_no_grad_activation=false setting so regular
SwiGLU policy scoring uses the same eager SiLU-then-multiply implementation as
training. Scoring still runs under no_grad; it does not retain a backward graph.
The consistency check remains enabled on the first update after process startup
(including resume), and at any configured check interval. This requires the Core
revision in the runtime lock; older revisions fail early rather than silently
ignoring the selection. The eager path computes all capacity rows, so its cost
should be measured on the chosen packing and expert-parallel layout.
Score centering
Select the inherited native score-centering objective through [miles]:
[trainer]
fp32_lm_head = true
[miles]
loss_type = "score_centering"
rollout_top_logprobs_num = 20
score_centering_is = "tis"
score_centering_tis_clip = 2.0
Use advantage_estimator="grpo" and omit legacy use_tis; the compiler selects
score centering's own weighting. Explicit async.off_policy_correction="tis"
also selects this native TIS mode. Refresh accepts tis or mis, retaining its
policy-lag, routing-replay and unfiltered-sampling constraints. Unweighted score
centering is available for synchronous runs. This integration currently requires
trainer.fp32_lm_head=true (or core.fp32_lm_head=true in the low-level API);
validation rejects an unset or false Core setting. Core then enables the serving
FP32 head and rejects an explicit false serving setting. For matched comparisons
use the same head precision and candidate recording in both arms.
For score centering, we enforce trainer FP32 output. This is a numerical-precision recommendation enforced as a conservative Core qualification policy, not a mathematical requirement of score centering. Upstream #4031 automatically enables the rollout FP32 head; it does not require the Core trainer setting. Our paired Core study used FP32 heads but did not compare BF16 against FP32, so it does not establish that trainer FP32 output is necessary. We retain the validation guard until other trainer precision configurations are qualified.
Whenever candidate probabilities are collected, preflight requires
rollout_top_logprobs_num >= rollout_top_k, regardless of the selected loss.
Unfiltered sampling (rollout_top_k=-1, rollout_top_p=1) can record a partial
candidate head. With candidate collection disabled, ordinary top-k sampling is
unaffected; score centering itself still requires a positive candidate count.
The original implementation is upstream #3621–#3626. This runtime also explicitly handles empty tails under complete-support replay, addressing #4204; this does not relax the Core refresh sampling restrictions. Candidate probabilities must be recorded when tokens are generated. OPD, speculative generation and custom converters that bypass candidate validation remain unsupported. The existing PPO+TIS baseline and score-centering+TIS use different policy objectives; that comparison does not isolate centering from removal of PPO clipping. Numerical tests and startup checks do not establish task learning.