Skip to content

Data, rewards and evaluation

The structured workflow prepares data in the GPU job where mounts and tokenizer are available. It records preparation under the run root; do not place reference assistant solutions into training prompts. See workflow for paths.

Selecting data

Choose exactly one mode under [data]:

Mode Contract
[[data.sources]] Separate source datasets with explicit allocation fractions summing to 1 (Core GRPO)
[[data.tasks]] Named tasks with train/eval counts and optional prompt wrapper
rl_manifest Adopt an immutable prepared manifest and supported verifier bindings
prompt_data Prepared JSONL; supply reward_config and optional alternating dataset-name/eval-path entries
recipe Rejected: named recipes are not supported

Current named tasks are gsm8k, gsm8k-less-noise, gsm8k_original, math, legacy ifeval, and generated multiplication, and synlogic-sudoku. Dataset IDs/revisions live in open_instruct/miles/datasets/run_data.py.

gsm8k uses the established ai2-adapt-dev/rlvr_gsm8k_zs train split. gsm8k_original remains a compatibility alias for the same pinned dataset, retaining its historical sampling seed convention.

The maintained dev example uses gsm8k-less-noise, which selects techarb/gsm8k-cleaner, an informally cleaned variant that has not been rigorously validated. It has 39 corrected targets and omits 93 problems judged ambiguous or unreliable, leaving 7,380 rows. Each prepared row records metadata.original_row, its index in the original dataset.

For fresh preparations that previously selected the cleaned data as gsm8k, use task = "gsm8k-less-noise". It retains that variant's sampling seed convention; task names and prepared sample IDs identify it explicitly. Equal seeds across the original and cleaned datasets do not select the same problems. Existing prepared runs retain their immutable data on resume; keep their original configuration rather than renaming the task inside an existing preparation contract.

Fresh preparations select training rows first, independently of eval_count, then select held-out rows from the remainder. This can change selections compared with older preparations that sampled both partitions together. Existing prepared runs reuse their recorded data unchanged. For named source datasets, the train component in prepared_sample_id identifies the source split, including when the row is assigned to this run's held-out partition; it is not a training assignment.

The single-node example takes prepared math, instruction-following and code JSONL with a verifier registry, without judge-dependent tasks; its placeholder paths do not identify the complete published Olmo 3 mixture. Use manifest adoption when starting from dataset manifests. Unsupported task/verifier contracts fail rather than silently substituting a different judge or reward.

The trusted verifier registry names factories and configurations; individual samples select registered names, targets and weights. Preparation writes the resolved registry to prepared/data/verifiers.json. Code verification may need an externally provisioned service. GPU judges have an explicit preparation, service and rubric contract in managed judges; launch does not automatically supply every Olmo 3 external service.

prompt_wrapper supports the task-specific contracts in the preparation code. Review rendered prompts and answer format after changing it. A familiar task name or equal split count does not prove identical questions or reward semantics.

Screening instruction-following training data

MILES preparation rejects known unsuitable targets assigned to the built-in IFEvalVerifier or ManifestIFVerifier, including custom registry names for those factories. The screen applies to every training source and rechecks immutable prepared data on resume. It does not remove rows automatically or change held-out datasets. A rejected historical preparation needs an explicitly cleaned input and a new run root.

These training-data safeguards complement the checker corrections in fix-ifevalg-checkers (#2124) and depend on its lowercase-counting semantics: N caps repetitions of each lowercase word, so positive limits, including two and three, are admitted.

The screen rejects malformed constraint bundles; missing, noninteger or nonpositive N for lowercase and total-letter counting; invalid total-letter relations; and total-letter less than limits with N <= 3. Such tiny total-letter limits encourage answers with almost no letters. It also rejects missing or invalid paragraph counts/indices and first words, including indices beyond the declared paragraph count and first words containing whitespace or punctuation that the checker truncates (. , ? ! ' "). These rules apply only to training admission and do not change checkers, held-out data or scoring. They are not a complete quality or relevance check; admitting a row does not certify it. Explicit response-language constraints remain supported.

Training never filters rows in place. Remove rejected rows during dataset preparation: source adapters omit them with recorded dispositions, and the RL Data Workbench provides a standalone audit and filter for existing JSONL files. Core/vLLM and DeepSpeed/vLLM launchers do not apply this MILES admission screen automatically.

The composition checker (count:counting_composition) already accepts the instructed * * * dividers and ignores punctuation-only word tokens; its rows are retained by this screen. English-request filtering alone does not remove these constraint problems, and neither this screen nor constraint scoring checks semantic relevance. Record changed dataset hashes and mixture counts when adopting cleaned inputs for a new experiment; keep historical run inputs intact.

Fixed allocation from separate source datasets

Replace the run's existing [data] section with this example to allocate 90% of fresh draws to a prepared Dolci Think dataset and 10% to Sudoku:

[data]
seed = 17
shuffle = true

[[data.sources]]
name = "dolci"
fraction = 0.90
prompt_data = "/weka/your-path/dolci/train.jsonl"
reward_config = "/weka/your-path/dolci/verifiers.json"

[[data.sources]]
name = "sudoku"
fraction = 0.10
tasks = [{ task = "synlogic-sudoku", train_count = 800, eval_count = 90 }]

Dolci stays intact as one source. The paths must refer to existing rendered data and a trusted verifier registry, with the same named judge bindings and code services the original run needs. This example does not provision those services. The Sudoku source itself requires neither. Remove eval_count if using the background evaluation path; shared-engine held-out evaluation and background evaluation are mutually exclusive.

Each source accepts one existing preparation mode: named tasks, prepared prompt_data with reward_config, or rl_manifest. Arbitrary raw Hugging Face schemas still need an adapter to render prompts and bind verifier targets. Source names are unique, fractions are positive and must sum to 1 (roundoff tolerance 1e-9), and seed/shuffle are shared at the outer data level. Nested mixtures and per-source seed/shuffle overrides are rejected. Changing the mix requires a new run root, as with other immutable preparation changes. The mixture loader requires pre-rendered text: runtime apply_chat_template must be false and multimodal_keys must be unset for consistent source loading.

Preparation writes separate prepared/data/source-<name>.jsonl files, a merged verifiers.json, and mixture.json containing source fractions, row counts and SHA-256 hashes. It checks prompt limits and train/held-out overlap across all sources and rejects conflicting verifier definitions. It does not deduplicate training sources against each other. The runtime retains a loaded dataset per source; it does not create an interleaved training file or stream file cursors.

For every fresh prompt, the provider draws a source from the configured fractions and walks that source's seeded shuffled order without replacement. When a source is exhausted it independently reshuffles and cycles. Small sources can therefore repeat many times during a run. Fractions describe allocation in expectation, not exact batch quotas; reward filtering and backfill can change the accepted training mix. Buffered retries do not make a new draw. A one-source mixture at fraction 1 preserves the shuffle provider's sequence, with added attribution.

Mixtures reject the global frozen exclusion table ([selection] / core.selection_table): replacing excluded prompts with draws from the whole mixture would change the admitted source fractions. Per-source exclusion files that restrict each source's eligible pool are not implemented yet.

The source RNG and every source cursor are checkpointed with pending groups; resume validates the manifest identity. Metadata preserves original provenance and adds mixture_source and mixture_source_epoch. When inference records are enabled, those fields also appear on scored group records. This fixed provider ignores outcome feedback. Recording remains best effort, as with shuffle: an unavailable store or a failed record/attribution build is logged and counted, without stopping sampling or training. ReMind retains its strict feedback checks because its selections depend on outcomes. Combining mixtures with ReMind is not implemented. Existing configurations without data.sources keep the default shuffle path.

SynLogic Sudoku

task = "synlogic-sudoku" selects the 890 Sudoku rows of MiniMaxAI/SynLogic, configuration hard, split train, pinned to bb4297b82b9c28ef39249f48386cefd5e856618a. The pinned dataset card declares MIT. datasets.load_dataset downloads through the normal Hugging Face cache (HF_DATASETS_CACHE / HF_HOME, or the user's default cache); run-specific rendered artifacts live under output.root/prepared/data. The initial download is the full hard training split, roughly 53 MB, before filtering to Sudoku.

This is an explicit rendering adaptation: the original instructions are replaced with English Sudoku rules, unchanged givens, and a request for a JSON 9×9 integer grid inside one <answer>...</answer> block. No reference solution enters the prompt or verifier target. Preparation checks every source reference against the rules and givens, preserves the original row, UUID, difficulty and revision, and records prompt_rendering = "synlogic-sudoku-english-v1". The 800/90 example partitions these 890 training rows deterministically; it does not use or claim the upstream validation split.

SudokuVerifier gives 1 for a complete grid satisfying all givens, rows, columns, and 3×3 boxes, and 0 otherwise. It accepts any valid completion, not only the published reference, and never executes generated code. Missing/multiple answer blocks, invalid JSON, booleans/floats, partial grids and rule violations score 0. It is an independent rule checker, not a port of SynLogic's upstream scorer.

Prompt provider interface

MILES still requests prompt groups through get_samples(n). Open Instruct's DashboardDrainingRolloutDataSource delegates to ProviderRolloutDataSource, which takes buffered retries first and asks provider.next_samples(n) for the remaining fresh prompts. The adapter makes K independent copies of each prompt and assigns the existing group/response IDs. Generation and reward computation keep their existing inputs.

The default ShuffledPromptProvider walks the same loaded dataset with MILES' seeded shuffle and cursor. No configuration or preparation changes are needed. It also handles requests crossing multiple complete epochs. A provider implements next_samples, feedback, state_dict, and load_state_dict, and declares requires_feedback; the provider itself does not require an epoch or source-mixture abstraction. Alternate implementations can be supplied through the data source's Python provider= argument. The built-in choices are core.prompt_provider = "shuffle" (default), "remind", "mixture", "pools", and "priority". Structured data.sources selects "mixture" automatically.

Provider state is saved atomically alongside the existing pending-group ledger. The default provider accepts legacy shuffle checkpoints and preserves their cursor fields. Provider-aware checkpoints also check dataset size, seed, and shuffle mode; prepared artifact identity remains the preparation layer's job. Retries and resumed pending groups do not request a new prompt from the provider.

feedback(records) receives inference records under the same lock as selection and checkpoints. The shuffle provider ignores feedback. With ReMind enabled, the recorder delivers every scored group before its best-effort disk queue, including all-1 and all-0 groups dropped by the unchanged reward filter. A lost disk row does not lose feedback; an unavailable recorder fails the run rather than silently turning off adaptation.

Historical priority groups

Tagliere v6 annotates each row with a model-specific historical reward priority. The opt-in priority provider visits every row in group 1 before group 2, then 3, then 4 in each dataset traversal. Run duration uses the ordinary MILES epoch or rollout budget; priority does not impose a one-pass stop. This allows a short training run to focus first on prompts with mixed historical rewards, while keeping one dataset artifact for different run budgets.

[core]
prompt_provider = "priority"
prompt_priority_field = "reward_priority_olmo35_sft_hero"
Group Historical evidence
1 At least one fully scored response group had differing rewards.
2 Unknown: no matching history or no fully scored response group.
3 All fully scored groups were zero and at least half of scored responses hit the token cap.
4 Other constant rewards, including all-zero groups without frequent truncation.

All fresh draws come from the active group until it is exhausted. Groups have no sampling weights. Each row appears once per sampler epoch; the next epoch starts at group 1 with a new deterministic within-group shuffle. data.seed fixes an independent shuffle within each group, making the selection order deterministic across batch sizes and resume. data.shuffle = false keeps original row order within each group. A request may cross a group boundary; empty groups are skipped. Priorities control prompt selection; asynchronous generation can complete prompts in a different order.

Optionally weight row sources within the active priority group, including sources inside a single Tagliere artifact:

[core]
prompt_provider = "priority"
prompt_priority_field = "reward_priority_olmo35_sft_hero"
prompt_priority_source_field = "workbench.source.candidate_id"
# Illustrative source labels/weights: use the actual values in your export and
# provide a positive weight for every source that has rows.
prompt_priority_source_weights = { C036 = 1.0, C046 = 2.0 }

The selected metadata path identifies the source. Tagliere exports its canonical Workbench source ID at workbench.source.candidate_id; flat fields such as source_name or source_dataset can also be selected. Relative weights control the next source draw while that source has rows in the active group; they are independent of its remaining row count. An exhausted or absent source is skipped and the remaining weights are renormalized. The whole priority group must finish before moving to the next. Rows appear once per sampler epoch: weights affect their order and early-budget exposure, not their total count after a complete pass. Without source weights, each priority group is one ordinary shuffled row list. With weights, data.shuffle = false preserves row order within each source, while the seed still fixes source draws. All observed source labels need positive finite weights; missing labels or weights fail startup. Configured labels absent from the input are harmless. Changing the source field or weights rejects checkpoint resume.

Missing, noninteger, or out-of-range labels fail startup. Logs record the metadata field, group counts, seed, and shuffle mode. The selected field must match the exported history: choosing a different model's history is a run decision. Historical groups stay fixed as training progresses and do not change grades or the online zero-variance filter. A filtered row is not revisited until the next sampler epoch.

Set the ordinary native MILES budget in the run file, for either priority or shuffle:

[training]
num_epoch = 2

Alternatively, set training.num_rollouts for a fixed collection budget. Choose one; structured runs reject both rather than silently ignoring num_epoch. With neither set, the existing default of 100 rollouts remains. MILES resolves num_epoch after loading the prepared data as num_rollout = num_epoch * (dataset_size // rollout_batch_size). Batch rounding, online filtering/refill, and async prefetch mean this native training-budget estimate need not equal exact fresh-prompt traversals. A rollout can contain several optimizer steps if its collection exceeds the global batch. Epoch-budget runs save the final completed update by default; an explicit training.save_interval still enables periodic checkpoints.

Use this with one data.rl_manifest or prepared data.prompt_data input. Named source mixtures/pools, frozen exclusion tables, and replay cannot be combined with this provider. Ordinary shuffle remains the default. Checkpoints save the sampler epoch and cursor alongside pending prompts, and reject changes to the field, source weights, seed, shuffle mode, or group/source assignments on resume. The existing MILES runtime pin is unchanged.

ReMind review provider

The second implementation illustrates outcome-dependent selection without changing dataset preparation, generation, rewards, or training. It wraps the shuffled stream with a FIFO review queue, following the prompt-review mechanism in ReMind. Enable it in a fully async Core run with inference records:

[core]
prompt_provider = "remind"
remind_review_fraction = 0.02
remind_enqueue_probability = 0.25

[records]
enabled = true
root = "/weka/your-workspace/inference-records"
Event Provider action
Fresh group is verified all-1 Enqueue its dataset row with probability 0.25, unless already queued or under review.
Review slot Pop the oldest queued row and generate fresh responses. An empty queue falls back to the shuffled stream.
Review is verified all-1 Remove the row from review tracking.
Review regresses, including all-0 Return the row to the queue tail.
Review outcome is unknown or invalid Keep it eligible for another review; do not infer mastery or regression.

This example uses a continuous allocation budget, one review slot per 50 new prompt admissions at the default fraction. It is independent of how requests are chunked into get_samples(n) calls. It does not reproduce the paper's default 10%-every-fifth-update bursts, or guarantee 2% of accepted training groups, tokens, or wall-clock time. Async backfill and policy lag still apply. Unused review slots are not accumulated. Ordinary prompts retain shuffled traversal; review deliberately repeats a prompt before the next epoch.

Only complete groups with verified binary rewards establish mastery. Graded or unknown rewards cannot add prompts to the queue. Intentional truncation-to-zero counts as a failure when that reward policy is enabled. Aborted groups retain their selection identity for retries. A group acknowledged without a recorded outcome releases its pending selection and returns an unfinished review to the queue. Reviewed rows and their targets come from the same loaded dataset; no response trajectories are replayed.

The checkpoint contains the base cursor, review queue, pending selections, random state, and allocation counter, alongside pristine pending rollout groups. Duplicate feedback is ignored after the first terminal observation. Feedback takes effect on subsequent selections in completion order; identical seeds do not remove async arrival-order differences. Records include selection IDs and a review flag for measuring the realized allocation. Frozen exclusion tables are currently rejected with ReMind to avoid silently consuming queued reviews without outcomes. This is a CPU-tested interface example, not a GPU-validated reproduction of ReMind's results.

Low-success history and length-triggered prompt replay

A group with zero or one successful response and several truncations may become more productive as the policy changes its answer delivery or subject-matter ability. The opt-in prompt-replay layer records that history and can regenerate failed prompts after recent response length falls. It wraps the ordinary shuffle or mixture provider; it reuses the original prompt and verifier targets, never old completions. Existing configurations remain unchanged.

Add these options to an existing fully async run:

[core]
prompt_replay_enabled = true
prompt_replay_fraction = 0.10
prompt_replay_length_drop = 0.20
prompt_replay_window = 256
prompt_replay_max_successes = 1
prompt_replay_truncation = "any"
prompt_replay_answer_probe = true

[records]
enabled = true
root = "/weka/your-workspace/inference-records"
Option Meaning
prompt_replay_fraction Fraction of proposed prompt draws reserved for eligible retries. Zero enables tracking only. An empty eligible pool falls back to fresh draws; unused slots do not accumulate.
prompt_replay_length_drop Relative reduction since the failed attempt: 0.20 requires the recent mean to fall to at most 80% of its failure-time value.
prompt_replay_window Number of valid fresh groups in each source's rolling window. Require this many additional fresh groups after a failure before retrying.
prompt_replay_max_successes Maximum successful responses in a complete, valid binary-reward group. Default 1 includes 0/K and 1/K; set 0 for all-zero only. Graded nonbinary groups are tracked but excluded from replay eligibility.
prompt_replay_truncation any requires at least one capped response; half requires at least half; all requires every response capped; none imposes no truncation requirement. Default any.
prompt_replay_answer_probe Record a bounded literal-reference match for scalar math targets, including unfinished text. Unsupported targets are unknown. This is a diagnostic, not verified correctness, and never changes a reward.

The length statistic is mean generated tokens per response across the recent fresh groups within the same source, including intentional truncation zeroes. The failure baseline is that rolling mean when feedback arrives, rather than the failed prompt's own length. For example, a 20% threshold with a 10,000-token failure baseline permits a retry once a full fresh window has elapsed and the source mean is at most 8,000. Replay responses do not update this baseline. Sparse sources therefore take longer to become eligible. Shortening is a heuristic for when to retry, not evidence that the answer is now within reach.

Complete all-zero groups are counted separately as no truncations, some truncations, or all truncations. A capped response deliberately assigned zero by reward_zero_truncated is valid evidence under that reward policy. Verifier errors, unknown outcomes, incomplete groups and aborted generations do not count as task failures. Historical IF diagnostics with the successful ifeval: scored status are recognized despite older recorder validity bookkeeping. Truncation membership does not establish that truncation caused an otherwise correct answer to fail.

Eligible prompts are visited in first-observation order, without concurrent replays of the same prompt. A valid group outside the selected criteria retires the candidate; later qualifying fresh observations can enroll it again. A one-success group can already train through the normal zero-variance filter and also become a future replay candidate. A qualifying replay waits for another full fresh window and a further relative length drop. An invalid replay retains the candidate but also resets its cooldown. The first scored generation counts; staleness regeneration is not counted twice.

History, length windows, pending reviews and base-provider cursor are checkpointed with the rollout ledger. Resume requires the same replay configuration and prompt catalog. Enabling replay on an old run requires an explicit new continuation; it cannot reconstruct unrecorded history from a legacy cursor. Frozen exclusions, ReMind, synchronous rollouts and nonglobal datasets are currently rejected. Like ReMind, this feature requires working recorder feedback and fails if the recorder cannot construct outcomes; disk queue drops remain separately reported.

rollout/prompt_replay/* metrics report category counts, candidates, replay outcomes and literal-match counts (including zero-group and truncated-response subsets). Group records carry stable prompt identity, a replay flag, outcome classification, and the baseline/current source lengths. Prompt identity hashes all rendered turns and verifier targets, scoped by source. It is not semantic deduplication across sources. Both online feedback order and async policy lag affect future selections.

Replay fractions refer to proposed groups, not accepted groups, tokens, or compute. The base mixture controls fresh draws; replays can change the overall source mix. Inspect source attribution and accepted-group/token costs when comparing runs. Even a fraction of 1 falls back to fresh draws when no candidate is eligible: this is not a finite, frozen-pool replay experiment.

For a compact historical inventory, stream explicitly chosen record attempts:

python -m open_instruct.miles records replay-pool \
  --manifest /records/run/manifest-ATTEMPT.json \
  --manifest /records/resume/manifest-RESUME.json \
  --max-policy-version 400 \
  --max-successes 1 --truncation any \
  --attempt-ranges attempt-ranges.json \
  --output zero-reward-history

The optional ranges file maps attempt IDs to inclusive policy-version bounds, for example {"ATTEMPT": [0, 100], "RESUME": [100, 400]}. Select these from the actual checkpoint ancestry to exclude abandoned recovery branches. The CPU reader exports summary.json, the selected replay-prompts.jsonl, all ever-zero IDs in zero-prompts.jsonl, and low-success-prompts.jsonl for comparing alternate truncation thresholds locally. They contain per-source counts, stable IDs, success and truncation histograms, and each prompt's latest valid outcome. The latter file keeps both 0/K and 1/K observations, irrespective of the chosen pool threshold. A prompt is included if it ever qualified; later outcomes remain visible so you can choose a latest-outcome rule when preparing a historical experiment. These runs can have K=4 or K=8: do not label a 1/4 observation as 1/8. The reader does not copy response text or create a training dataset. Old records without answer probes remain unknown; this command does not retrospectively verify their text. Missing records and verifier errors are reported separately from valid zeroes.

A 0/K-to-mixed transition alone does not show improvement: ordinary resampling can produce it at a frozen policy. Compare replay productivity and downstream evaluations against matched continuation controls before adopting this experimental sampler.

Three-node training mixture

three-node.toml targets the full train split of allenai/Dolci-Think-RL-7B, pinned to revision 0fb6466d31ef3a9dd16985ef635e6429e05a6491. This is the dataset selected by the checked-in Olmo 3 7B Think RL recipe. The dataset card reports these source-group counts:

Source group Rows
Math 30,180
Instruction following 29,813
Code 21,385
General 20,636
Total 102,014

Prepare the full published split as the starting population, preserving source identities, verifier names and targets. Render and tokenize prompts with the selected policy checkpoint; the source's saved token IDs belong to its original tokenizer. The example explicitly sets data.over_length = "exclude", so the training population contains only rows that fit the run's effective prompt budget. At smaller budgets, this can change source-group proportions in the single input file; the published counts above describe the starting population, not a guarantee about retained rows. Preserve the full question for judged examples in metadata.judge_query. The trusted verifier registry must support math, code/code_stdio and instruction following; general-quality and general-quality_ref use the template's named bindings. Code verification requires an externally provisioned service.

The dolci-think-rl-7b paths in the template are prepared-artifact placeholders, not an automatic Hugging Face download or proof of provenance. Before launch, record the source ID/revision and verify the starting row counts, source-group counts and identities against the pinned source. Preparation records the prompt limit and per-source original/excluded/retained counts in its manifest and logs. Review those counts as the run's changed training population before interpreting results. To require the entire prepared input to fit, explicitly set data.over_length = "fail"; this remains the library default when the option is omitted. Use the existing prepared-JSONL or immutable-manifest input contract. Background benchmark evaluation is configured separately and does not change the training population.

Run-specific prompt limits

A release can serve runs with different context and response budgets. Preparation checks rendered prompts using the run's tokenizer against rollout_max_prompt_len, which defaults to inference.max_context_length - inference.max_response_length. Prompts exactly at the limit are accepted. The default policy, fail, stops on any over-limit prompt. To exclude over-limit training prompts, set:

[data]
over_length = "exclude"
# Keep your existing tasks, sources, rl_manifest, or prompt_data settings.

This applies to all training sources; per-source overrides are rejected. It preserves prompt text, ordering, identities and configured source fractions or pool weights. Named-task counts select the initial candidate rows; exclusion does not backfill them. Held-out prompts still fail on excess length. Other validation, including token-ID/template compatibility, verifier constraints and train/held-out overlap, still applies to all candidate rows, including excluded rows.

prepared/data/manifest.json records prompt_length: the effective policy, integer prompt limit, training scope, and original/excluded/retained counts by configured source name. For a single input, it uses the first nonempty string in metadata.source_dataset, then metadata.source_name, falling back to train when neither is available. This also supports older Workbench exports. Pool counts are included when pools are configured. The same counts appear in preparation logs. Each configured data.sources entry and each previously populated pool must retain at least one row to preserve its sampling allocation; otherwise preparation fails before publishing data. In single-input and named-task runs, metadata-based source groups are reporting-only and may lose all their rows. These labels identify provenance, not per-task allocations; named tasks sharing a dataset share its reporting bucket. The overall training input must still retain at least one row. Configured empty pools retain their existing behavior.

The mixture manifest's row counts and hashes describe the retained files, so the runtime sampler continues to enforce its row-count check. The source release remains unchanged. Policy and limit are part of immutable preparation identity; changing either requires a fresh run root. This is preparation-time filtering; launch-time length preflight remains separate work.

Online group filtering

MILES/Core training defaults to online reward-group filtering:

[training]
filter_zero_std_groups = true

The low-level equivalent is core.filter_zero_std_groups. The compiler selects the native reward-variance filter or the matching exclusion-aware filter described below; plan records the effective miles.dynamic_sampling_filter_path. The filter keeps a complete prompt group only when the standard deviation of its selected reward scores exceeds 1e-8. All-0, all-1 and other constant-reward groups are discarded before trainer packing/scoring. Mixed groups retain every response, including zero-reward ones. This requires more than one training response per prompt. Evaluation is unfiltered and may still use one response per prompt.

By default the compiler installs the ungraded-response exclusion (core.exclude_ungraded, see the GRPO guide) and pairs it with open_instruct.miles.rewards.exclusion.filter_graded_reward_variance, which ignores responses whose grade failed and drops groups with fewer than two graded responses. A placeholder zero from a failed grade cannot keep an otherwise constant-reward group in the training batch. With variance filtering disabled, the compiler selects filter_graded_responses instead: it retains constant rewards and single graded survivors but rejects groups with no trainable responses. The corresponding truncation-exclusion guard is filter_finished_responses. These safety guards run before batch admission and replacement, independently of the variance criterion. Custom dynamic filters require opting out of the managed exclusion hook (core.exclude_ungraded=false) or providing a custom post-process and filter that enforce the same invariant.

With custom_reward_post_process_path="open_instruct.miles.rewards.truncation.exclude_truncated", the compiler instead selects the built-in finished-response filter. It ignores truncated and ungraded responses when checking reward variation and drops groups with fewer than two finished, graded responses. A reward earned by unfinished text cannot keep an otherwise constant-reward group in the training batch.

Generation continues until the configured number of accepted groups fills the collection; global batch size counts retained responses. This works with barrier, engine-drain and mixed-policy refresh publication. Async filtering occurs before completed-buffer admission, after checking group provenance; staleness is still checked at consumption. Filtered groups are dropped, even when async_unused_samples_handler="retry", and retired from the checkpoint retry ledger. Their prompts remain eligible on later dataset passes.

Filtering saves trainer work but still pays generation and verification costs. A low acceptance rate increases collection time; if every group is rejected, collection waits for useful data and cannot advance the policy. The tiny dev mechanics example therefore explicitly sets this switch to false. Monitor rollout/dynamic_filter/drop_no_trainable_responses for fully masked or empty groups. Monitor rollout/dynamic_filter/drop_zero_std_0.0, rollout/dynamic_filter/drop_zero_std_1.0 and the corresponding counters for other constant rewards, together with trainer wait time and accepted task proportions. These are per-report drop counts, not accuracy estimates. Use unfiltered held-out evaluation to assess learning.

Zero group-relative advantage removes the reward-driven policy-gradient signal, but the discarded responses would still affect loss normalization and any enabled KL, entropy or router auxiliary objectives. Treat enabling filtering as a recipe change and compare learning at matched wall time or generated-token budgets. Do not change filtering in-place and call the run a resume.

Offline preprocessing remains compatible and can reduce online rejection rates. The existing correctness filter scores saved completions and retains prompts within inclusive average-score bounds. Its default [0, 1] bounds retain all prompts; use bounds strictly inside that range to exclude all-wrong/all-correct prompts for binary rewards. This is a snapshot of an earlier policy, so newly sampled groups can still have constant rewards after preprocessing. Online filtering continues to check the current groups. For graded rewards, average-score filtering and zero-variance filtering are different tests.

Advanced custom native filters require core.filter_zero_std_groups=false and opting out of the managed exclusion post-process to avoid bypassing its safety guard. They remain unsupported with refresh or engine-drain publication; those modes allow only the built-in reward-group filters.

Held-out evaluation and comparison

Specify eval_count per task, or explicit held-out files. The structured defaults use greedy evaluation, one response per question, initial evaluation and periodic evaluation every 20 collections when held-out data exists. Examples can override these values; inspect plan. Eval shares rollout engine admission and cache limits.

Retain question IDs, rendered prompt/token IDs, generations, rewards, lengths, cap hits, policy versions and source/config provenance. Compare update zero on the same checkpoint before interpreting learning differences. For matched studies use the same immutable held-out set, template, sampling, response limit and reward implementation. Examine paired answer changes rather than just aggregate scores.

Rollout dumps are trusted tensor artifacts on WEKA; do not load arbitrary external pickle files. The small Beaker reports are an index into the larger retained artifacts, not a replacement for per-sample auditing. See operations.

Reward-service failure policies

The original Open Instruct GRPO verifiers assign zero reward when judge or code-service requests fail after retries. That keeps training running through transient failures, but a fallback zero is an ungraded response: in a group with successfully graded responses it lowers the failed response's advantage as though it were incorrect, and failures correlated with response content or length bias learning. MILES training therefore treats a reported service_error or judge_error as ungraded rather than zero. By default the response stays in its group with zero advantage and a zeroed loss mask and leaves the group baseline (core.exclude_ungraded, see the GRPO guide); with that exclusion disabled the whole prompt group is omitted and replaced. Evaluation and distillation scoring still request zero for failed grades. The verifier-level policies below decide whether a failure is reported at all or stops the run.

Service-error and timeout metrics cover the consumed collection, after group filtering. Dropped groups are absent from those fractions, so the metrics can understate failures. With zero-variance filtering enabled, a total outage can prevent a collection from filling and produce no new collection metrics at all. Inspect individual failure warnings as well; a low or missing error fraction does not establish service health.

Code-service failure policy

Code rewards default to Open Instruct's zero-reward continuation behavior at the verifier level. After existing HTTP retries are exhausted, a transport/gateway failure or invalid service reply is reported as service_error with a zero component, logs a warning, and leaves the rollout running. In training the reward adapter then treats the response as ungraded (excluded or group-omitted as above); in evaluation the zero stands. This is a fallback, not proof that the generated code failed its tests. HTTP rejections (4xx, or a 500 that persists through its retries) are scored zero as program failures and are not ungraded.

Set OI_MILES_CODE_FAILURE_POLICY = "raise" in [launch.env] for strict service-failure handling, or set failure_policy to "zero" or "raise" in the code verifier's trusted JSON config. An explicit verifier setting takes precedence over the environment. Configuration errors and cancellation still propagate; known-answer preparation canaries use strict mode.

Per-sample diagnostics distinguish service_error, rejected, and successful grading. W&B records rollout/code_verifier/service_errors and rollout/code_verifier/service_error_fraction, with status breakdowns alongside the existing rejection counters. The fraction covers code verifier calls in the consumed collection; it is not the fraction of HTTP attempts or all samples generated. Retain these metrics when comparing learning curves.

General judge failures

Judge transport failures and invalid or incomplete replies default to a zero component after retries, with a warning and per-sample judge_error diagnostics; training treats the response as ungraded, as above. Set OI_MILES_JUDGE_FAILURE_POLICY="raise" in [launch.env] to stop on those failures. Detected context overflow and configuration errors raise regardless of that fallback policy. The three-node example explicitly selects "zero" for judge failures; both node and three-node select it for code-service failures. Change those settings to opt into strict handling. Managed judge health-check failures separately stop training; the fallback applies to grading requests.

General math grading and timeouts

MILES general math grading uses pinned Hugging Face Math-Verify 0.9.0 for answer extraction and comparison. It supports structured mathematical answers, including sets, tuples and multiple boxes. It removes reference math delimiters only when a single pair encloses the whole answer; separate spans such as \(x=2\) or \(x=-2\) reach the parser intact. Predictions must have a box or a Math-Verify answer anchor such as Answer: or the answer is; bare numbers and unanchored LaTeX are not eligible. Paired Markdown bold markers are removed so **Answer:** **24** is recognized, while exponentiation such as 2**3 is preserved. The library recognizes a limited set of answer phrases; some multiline, escaped-currency or prose answers still fail extraction. New MILES preparation writes open_instruct.miles.rewards.math_verifier.MathVerifier; the MILES reward loader also maps existing registries naming open_instruct.ground_truth_utils.MathVerifier to this implementation, without requiring data regeneration or an opt-in flag. Legacy training retains its original math implementation. GSM8KVerifier and StrictMathVerifier retain their separate contracts. The caller's reward_final_answer_only setting still determines whether reasoning is included. This replacement changes rewards; compare runs using their recorded source revision.

Math-Verify is not a prose or proof judge. Ambiguous comma-separated thousands, some function equations and references mixing prose with math can still be misgraded. Multi-part equation answers can also fail when labels are outside separate value boxes, even when every value is correct. Anchored extraction reduces incidental-number credit in unfinished responses, but does not guarantee completion: an unfinished response can still contain an explicit answer anchor or a box. There is no fallback to the previous general math grader.

Each general-math parse and candidate comparison has an internal five-second limit. A parse timeout returns no answer; a comparison timeout allows later Math-Verify candidates, including its literal-string fallback, to match. Strict math retains its existing text-normalization fallback. The final reward can therefore be zero or one. Completed results retain status=ok and carry a symbolic_timeouts count; a timeout also produces a warning in the parent process. The outer subprocess deadline and zero-reward timeout policy are unchanged, and these timeouts do not exclude samples.

W&B logs the following per-collection metrics under rollout/math_verifier/:

Metric Meaning
symbolic_timeouts Total timed-out parsing or comparison operations reported by workers; one result can contain several.
symbolic_timeout_samples Math verifier results with at least one symbolic timeout.
symbolic_timeout_fraction Affected results divided by all math verifier results in the collection.
symbolic_timeout_rewarded_samples Affected results whose final verifier score was positive.

Like the existing verifier metrics, these describe consumed samples, not all generated attempts or groups discarded by filtering. Counts survive worker IPC in sample diagnostics, but internal timeouts preceding a killed worker are not recoverable from that worker's unfinished result.

Symbolic math runs in a bounded subprocess pool. An individual request exceeding 45 seconds kills and replaces that worker and returns reward zero. The sample retains verifier_diagnostics with kind=math, status=timeout and elapsed time; rollout/math_verifier/timeouts and timeout_fraction count these separately from ordinary incorrect answers. Worker timeouts always return zero with diagnostics; there is no timeout policy setting. Configuration errors, unexpected worker failures and cancellation still propagate. A timeout-zero is an ungraded sample, not proof of a wrong mathematical answer; include the timeout rate when interpreting benchmark scores.

Independent background evaluation

For best-effort external olmo-eval jobs and checkpoint-axis W&B curves, see background evaluation. Selecting this mode replaces shared-engine evaluation; failed or skipped jobs leave gaps without blocking training.

Qualified local source catalogs

python -m open_instruct.miles.datasets.source_catalog resolves reviewed source names to ordinary [[data.sources]] blocks. It checks the prepared prompt and verifier-registry SHA-256 digests, rejects unqualified names, and requires the requested fractions to sum to one. When a source carries a machine-readable qualification receipt, the receipt must also bind both artifact hashes. The catalog and reward registries are trusted configuration, just like a run file; do not accept them from unreviewed datasets.

python -m open_instruct.miles.datasets.source_catalog /path/to/qualified/catalog.json \
  --source c094-deepmath-103k-math=1.0 --output /tmp/deepmath-data.toml

Use the resulting [data] section in a run file, or combine its named source blocks with existing prepared sources and set the final fractions. Moving the whole catalog directory preserves its relative bundle paths. This does not change mixture sampling or introduce another run-file data mode. The catalog CLI emits seed 17 and shuffling enabled.

A qualified slice is not a claim that every row or subtask of its parent Hugging Face dataset is ready. Read the catalog's row counts, strata, exclusions, and qualification receipts. Catalogs, qualification controls, GPU coverage runs and their receipts are produced by the RL Data Workbench, which runs these checks against a pinned Open Instruct runtime. Open Instruct only resolves the resulting catalog; training does not rerun qualification.

The optional deterministic_verifiers factories cover explicit MCQA answer lines, formatting-only checks, Countdown arithmetic, named knight/knave assignments, structural numeric answers, exact text, and typed JSON answers. MCQA uses a prepared Answer: LETTER contract; it must not silently replace a source's original extraction rules. The JSON-schema factory grades schema conformance, not factual content. ReasoningGym uses the pinned optional dependency listed in the verifier dependencies, with explicit family coverage. No factory executes code supplied in a dataset or fetches external schema URLs. The schema subset rejects pattern and patternProperties because the schema library uses unbounded Python regex matching. This conservative recursive check also rejects those names inside property maps or instance literals. Such schemas must be deferred during preparation, not silently have constraints removed.

H169's explicit mcqa_final_choice target kind reads the last explicit choice in the final answer. A declaration is Answer: LETTER, a boxed letter, or the row's own requested format when the target carries pattern (a one-capture regex; the mcqa_v1 adapter copies it from template_metadata.output_regex). Bold, parentheses and \text{} around the letter are ignored. The last declaration is the answer and earlier ones are superseded (Answer: A then Answer: B is B). The rest of its line may explain it but may not name another displayed choice letter (Answer: B, C, or D scores zero); later lines are not inspected. Lowercase letters, a prompt echo (Answer: A/B/C/D) and a letter named only in prose ("The answer is B") are not declarations, and a row pattern match counts only for a displayed uppercase choice that does not begin a word. Because "A" and "I" are also words, Answer: I think ... cannot earn credit. With no declaration at all, the final answer must be the bare letter. Preparation must select this contract explicitly; existing mcqa targets retain their answer-line extraction. Sources prepared before pattern existed keep scoring without it until they are prepared again.

Formatting matching shares a one-second monotonic deadline across the response, with each regex call additionally capped at 100 ms or the remaining time, whichever is shorter. The deadline is checked between lines, markers, matches and patterns, including the quoted-pair fast path. These are in-process checks, not process isolation: linear string operations and target validation are not forcibly interrupted. A timeout returns fallback zero with verifier_diagnostics.status="timeout", the original error, and elapsed time. It is an ungraded response, not evidence of an incorrect answer. There is no timeout policy setting. Invalid targets and unexpected failures still raise. Failure capture reserves two slots per worker/save directory: one for a raised registered-call exception and one for a returned status="timeout" diagnostic from any verifier, including math and formatting. Each records the full prompt, response, scored response and target under <save>/verifier_failures/. A fallback timeout cannot consume the exception slot, and absent error text is recorded with a descriptive placeholder. Minimum-line rules return success as soon as enough lines match; irrelevant trailing lines cannot invalidate that predicate. Qualification rejects timeouts even though runtime validation timeouts return zero.

ReasoningGym families inheriting its default substring scorer use stripped exact matching instead. At the pinned version these are base_conversion, leg_counting and simple_equations. Task-specific scorers retain their semantics, and multiple explicit final answers are rejected. A single boxed answer nested inside answer tags is one answer. Diagnostics identify the corrected protocol as exact_default_v2; earlier screen rewards using the default substring scorer need rescoring before comparison with this protocol.

Dataset integration

Use source preparation to convert reviewed dataset exports into named training sources, preserving provenance and reporting unsupported rows. The reward verifier guide describes their runtime contracts. Qualification of a prepared pool against reviewed reward controls happens in the RL Data Workbench before a named source enters a MILES mixture; it does not establish downstream benefit.

Named pools and independent sampling policies

Named pools organize labeled training data independently of how it is sampled. The same prepared inventory can use a finite global shuffle, uniform pool draws, fixed weighted pool draws, or adaptive training-exposure balancing. The default is one global shuffle without replacement. Pool exposure is recorded for every policy; only the explicit exposure policy uses those counts to adjust draws. Selective truncation replay is independently enabled and retains original verifiers.

To treat entire datasets as pools, add pool to each source entry. The pool is created automatically; neither [data.pools] nor row-level pool_id is needed:

[data.sampling]
policy = "uniform"

[[data.sources]]
name = "deepmath"
pool = "deepmath"
prompt_data = "/data/deepmath.jsonl.gz"
reward_config = "/data/verifiers.json"

[[data.sources]]
name = "polaris"
pool = "polaris"
prompt_data = "/data/polaris.jsonl.gz"
reward_config = "/data/verifiers.json"

Use distinct pool names for dataset-uniform draws, or the same pool = "math" on multiple sources to group them. Source identity remains separate in records. The same assignment works with every sampling policy and optional replay. An explicit source pool takes precedence over any existing row pool_id in the prepared training copy; input files, prompts, answers and verifiers are preserved. If the named pool already has a configured label, that label becomes the prepared row's canonical pool_id. Held-out evaluation rows are not reassigned.

Sources without pool can still use their row labels when [data.pools] is provided. Without that label mapping, every source must specify pool; there is no implicit fallback for unassigned sources. Source fractions remain prohibited in pool mode; fixed pool weights belong in data.sampling.weights.

data.sampling.policy Selection
shuffle (default) One global permutation of all fresh rows, regardless of pool sizes. Replay inserts retries without changing the relative order of fresh rows.
uniform Choose uniformly among currently available pools, then draw the next row from that pool.
weighted Choose available pools by the configured positive weights, renormalizing as pools empty.
exposure Adjust proposals toward uniform trained prompt-group exposure, compensating for delivery rate, accumulated deficits and outstanding work.

All policies consume each fresh row once and allow pools to empty. With replay turned off and a complete traversal, they consume the same fresh inventory; they differ in order, not membership. With a shorter budget, their coverage can differ. data.shuffle=false disables the initial row shuffle; static pool selection still uses its seeded RNG. Existing non-pool shuffle/source-mixture epoch behavior is unchanged.

Priority can modify uniform or exposure allocation without becoming a separate pool-selection policy:

[data.sampling]
policy = "exposure" # or uniform

[data.sampling.priority]
field = "reward_priority_olmo35_sft_hero"

Allocation chooses the pool; priority chooses its next fresh prompt. Each pool consumes priority 1, then 2, then 3, then 4 independently. Every retained training row must carry an integer label in that range in the specified metadata field. With data.shuffle=true, order within each stratum is seeded and shuffled; otherwise it follows input order. Labels are fixed for the prepared inventory. Disk-backed inventories scan labels once at startup and retain one byte per row, without retaining prompt or verifier payloads. Checkpoints preserve progress, pending selections and exposure accounting; changing the modifier invalidates sampler resume identity. Committed exposure receipts include sampling_priority_group for per-stratum accounting. This runtime receipt field is stamped only when the modifier is enabled. Startup label reads are reported separately as rollout/prompt_storage/priority_scan_reads; disk_reads continues to count demand and prefetch reads.

This modifier does not support replay, shuffle or weighted allocation. The standalone core.prompt_provider="priority" remains available with its existing semantics. Exhausting a priority stratum advances within the same pool; exhausting an entire pool still redistributes allocation among available pools. Exposure balances cumulative trained groups, not strict per-update quotas. Monitor whole-pool supply and consumed shares when comparing mixtures.

Exposure counts groups, not draws or tokens: a prompt with eight responses counts as one unit only after the optimizer update using it succeeds. If Math delivers a trained group on 25% of draws and IF on 50%, equal exposure needs approximately two Math draws per IF draw. Only exposure tries to compensate in this way; shuffle, uniform and weighted do not react to those outcome counts.

[data.sampling]
policy = "shuffle" # alternatively: uniform, weighted, exposure
# With policy="weighted", supply every configured pool name:
# weights = { math = 3, if = 1, code = 1, science = 1, logic = 1, general = 1 }

[data.pools]
math = { label = "math" }
if = { label = "instruction_following" }
code = { label = "code" }
science = { label = "science_knowledge" }
logic = { label = "logic_reasoning" }
general = { label = "general" }

[[data.sources]]
name = "labeled_data"
prompt_data = "/data/labeled-trainer-rows.jsonl.gz"
reward_config = "/data/verifiers.json"
# Add additional labeled sources in the same way. Omit fraction in pool mode.

[async]
fully_async = true

[records]
enabled = true
root = "/run/inference-records"

[core]
prompt_replay_enabled = true
prompt_replay_max_successes = 0
prompt_replay_truncation = "any"
prompt_replay_fraction = 0.1
prompt_replay_length_drop = 0.1
prompt_replay_window = 256

This is a fragment to apply to an otherwise valid MILES run, not a complete launch configuration. Prepared inputs must already have the standard rendered input, label, and verifier metadata fields. Dolci's prompt-inventory records still require target resolution and rendering. Compressed JSONL is supported. There must be exactly one optimizer update per collection. Sample count is unchanged by the sampler; an eight-response recipe remains eight-response.

Contract Behavior
Membership A source-level pool assigns the whole source; otherwise rows need a configured top-level pool_id. Missing/unknown labels, duplicate label definitions and conflicting runtime annotations fail preparation.
Source fractions Rejected in pool mode, rather than silently ignored. Use data.sampling.weights for fixed pool weights; every configured pool requires a finite positive weight. Weights are rejected for other policies.
Finite fresh data Each row is drawn once; there are no automatic epochs. Empty pools are skipped.
Exposure credit One group per completed optimizer update that used it. Scored, queued, filtered, or stale groups earn no credit.
Exposure reservations only Outstanding draws reserve estimated delivery, reducing over-correction during asynchronous generation.
Exposure controller only Delivery estimate (trained + 1)/(terminal + 2), floored at 0.05; target deficit corrected over a 64-group horizon. Proposal weights are bounded to [0.25, 5], limiting draw ratios to 20:1.
Exposure target only Each trained group's unit target is divided over the pools active when it was drawn. Inactive pools accrue no new target from later draws. Exact equality is not promised after pools empty or for sources with effectively zero delivery.
Replay Within the selected pool, eligible retries may replace a fresh draw. The replay fraction caps that pool's proposed draws. In global shuffle, a retry does not advance the fresh permutation; after it drains, immediately budget-eligible replays are selected uniformly across available pools. Draw counters do not advance for inactive pools, so replay stops at the first non-replay slot after fresh exhaustion; unused earlier slots are not accumulated.
Replay trigger The low-success history described above, with fresh-response length windows scoped to source within pool. Invalid verifier results are separate from valid zeros.
Exhaustion A pool without fresh rows or a budget-eligible replay is inactive. Once all pools and outstanding generation drain, an incomplete final collection is not trained; the last completed checkpoint is saved when configured.
Stale work Pool mode defaults to async_unused_samples_handler="drop"; explicit retry is rejected. Drops release reservations; dropped fresh rows are not redrawn, so complete traversal does not imply complete observed or trained coverage.
Recovery Manifest and selection-policy identity, cursors, RNG, targets, delivery counters, reservations, and replay histories are checkpointed with pending prompts.

Pool mode supports fully asynchronous disaggregated serving, with or without prepared-batch overlap. With overlap, the batch prepared for the next update is staged while the current update trains; exposure is still credited once per completed update, in training order, and a checkpoint taken in between regenerates the prepared prompts on resume.

Keep core.prepared_batch_overlap unchanged when resuming an existing run: switching it from false to true (or back) is rejected by the run-spec identity check. This support does not enable changing the setting on resume.

The application requires the companion MILES runtime's core_commit_exposure callback and finite-pool termination support, included in this branch's runtime source pin. Older images fail explicitly. Build an image from this branch before launching this mode; the existing image alone is insufficient. Full distributed GPU validation remains pending.

Inference records retain pool_id, sampling_pool, mixture_source, and pool_selection_id. Per-update pool-exposure-<attempt>.jsonl receipts live in records.root and contain trained/proposed/pending counts, generated and trained response tokens, fresh/replay training counts, and replay diagnostics. Every policy also reports cumulative pool metrics through the normal tracking backends, including W&B. After each completed optimizer update, the driver publishes pool/<metric>/<pool-name> against pool/update (the completed update count), including the final update. The recorder retains rollout/pool/<pool-name>/<metric> for compatibility; its snapshots can precede training.

Metric Interpretation
proposed, fresh_proposed, replay_proposed Sampler selections, including work still in flight.
attempted, fresh_attempted, replay_attempted Selections with a first returned group record, including invalid or aborted results. Duplicate feedback does not add another attempt.
scored, invalid, aborted Partition of observed attempts by grading validity. Service failures are not evidence of an all-zero reward group.
filtered, filtered_all_zero, filtered_all_one, filtered_constant_nonzero, filtered_other Groups removed by the filter, with reward breakdown; invalid outcomes belong to other.
discarded, discarded_unobserved Explicitly discarded selections, and those discarded before a group record arrived. Observed discarded work also appears in attempts.
trained, fresh_groups, replay_groups Prompt groups committed after successful optimizer updates, with fresh/replay breakdown.
available_rows, coverage Initial fresh-row inventory and fraction observed at least once. Replay does not increase coverage. These are prepared rows, not globally deduplicated questions.
generated_tokens, trained_tokens Response tokens in first observed groups and the subset subsequently used in completed training. Regenerations reuse the first observation; these counters are not complete compute or loss-token accounting. Groups with incomplete token counts are omitted and counted in the corresponding *_unknown_groups.
groups_with_truncation, truncated_responses, returned_responses Counts for assessing truncation pressure, with a response denominator.
proposed_share, attempted_share, trained_share, generated_token_share Each pool's share of the corresponding cumulative count; zero before any observations.

For a first W&B view, put pool/attempted/*, pool/trained/*, and pool/generated_tokens/* in adjacent multi-series panels, using pool/update as the x-axis. Add shares and coverage to compare allocation, training exposure and breadth. These are logged scalars; no W&B workspace layout is created automatically. Counts restore from the checkpoint, including pending observations, so reconcile failed-attempt tails rather than summing them. These measurements do not change sampling weights or impose an exposure target.

Exposure targets are reported only for the exposure policy. Unknown token counts are recorded separately. Receipts identify the resume point; if an attempt fails before checkpointing, reconcile its tail against the restored checkpoint instead of adding counts from different attempts. Captured response tokens are not loss-mask tokens or GPU cost.

An empty pool does not force another epoch or a replay. When fresh data is gone, replay must still satisfy its budget and shortening trigger; otherwise training can finish before the requested update count. Different pool labels do not change verifiers, reward semantics, truncation policy, or the zero-variance filter.

Changing the policy or weights is a new sampling experiment: checkpoint restore rejects a mismatched identity. Checkpoints from earlier draft pool implementations (before the version-4 compact selection receipts) also require a new run; missing history is not silently reset. No pool-mode production runs have been qualified yet.

Compiled collections preserve original dataset identity in metadata.origin_id. Replay uses that field first, then source_dataset, then the configured physical mixture_source. Thus multiple original datasets can share a file and pool without sharing response-length windows or prompt histories. Pool membership and all four sampling policies remain independent of these identities. Replay keys additionally bind prepared/record IDs and the full verifier configuration, including weights, so identical rendered tasks with different reward settings cannot substitute for one another. The native inference-record task key stays unchanged. Earlier replay catalog checkpoints reject restore under this stronger identity.

Producers may supply immutable release_id, record_id, and labeling_version in row metadata, alongside original source_dataset/source_row provenance and existing prepared_sample_id/sample_id. Group records retain those identities; optimizer receipts include the compact row identities and the prepared manifest hash. Input-file hashes bind all supplied fields, membership, and verifier payloads to preparation and resume; changed identities, labels, policy or weights require a new run. The consumer does not infer cleaning or labeling rules from these fields. Pending selection checkpoints retain only accounting identities, not verifier targets.

Finite exhaustion marks the workflow complete, so automatic resume cannot restart it or repeat its final export. A fresh run exhausted before its first optimizer update completes without exporting an untrained final model. A resumed run can still export its previously trained weights. Inline evaluation follows its configured periodic cadence; exhaustion does not add an off-cadence inline evaluation. Background final evaluation follows the existing final-export path.

Online replay assumes each participating verifier uses 0 for failure and 1 for full credit. It does not infer a source-specific maximum reward. Enable it only for a population with that binary contract: a graded source can sometimes return 0 or 1 without those values meaning binary failure/success. Nonbinary observations remain in records and history, with binary_successes = null, and are ineligible for replay. Valid truncation zeroes and unknown verifier failures stay distinct.

A local CPU capacity check used 1,827,000 synthetic rows with distinct 20 KB verifier strings and a 32 MiB outlier, selecting 256 groups with eight responses each. These numbers measure eager payload retention and checkpoint copying, not model memory, generation, grading, or distributed execution. The test used four CPU cores and a 64 GiB container limit; checkpoint files were on tmpfs.

Replay state Peak process RSS Loader/provider startup Select 256 groups Save checkpoint
Disabled 43.77 GiB 53.9 s 0.21 s 0.08 s
Empty history 44.34 GiB 114.4 s 0.33 s 0.08 s
1,827,000 historical observations 48.89 GiB 118.0 s 3.74 s 53.18 s

The full-history checkpoint was 566.7 MB. Replay history still grows with observed prompts, and checkpoint time is material at this scale. These measurements preceded the removal of duplicate pool template objects; pending accounting was already compact. They are not a bound on an arbitrary verifier payload or production memory use. Validate the actual release and concurrency before a long run.

A separate actual-payload check decoded all 1,827,290 retained rows from release 98696203134d98acf760316cff3e5fd19802a5d0f5155d39d992e0aa630e43be into prepared-shaped rows for the native runtime loader. It preserved labels and verifier targets, used the pinned pool assignments, and forced the largest 930.1 MB canonical record into the first 256-group pending batch. This checks runtime payload capacity; it bypasses JSONL export and tokenizer preparation and runs no model, verifier service, or distributed worker.

Before template sharing, provider RSS reached 62.86 GiB and the process was killed with exit 137 during checkpointing under a 64 GiB container limit. With shared template references, the 96 GiB bounded rerun completed: loader/provider startup 408.2 s, provider RSS 59.26 GiB, first selection 56.02 s, checkpoint 0.51 s and 622.8 MB, peak process RSS 60.47 GiB. The first-selection latency needs profiling before claiming sustained sampler throughput. Checkpoint storage was tmpfs, whose pages also count against the container limit. The corrected path has not been qualified at 64 GiB; 96 GiB is the tested ceiling, not a measured minimum or a full-trainer memory budget. Actual-payload replay and GPU execution remain pending.

Pool storage and prefetch

Prepared pools keep prompt text, labels, and verifier targets in their JSONL files. The runtime holds compact row IDs and memory-mapped byte offsets, rather than a Python object for every prompt. CPU preparation writes hash-bound index sidecars; older bundles build these derived indexes on first use. Loading verifies both the source and index hashes, so startup still reads the source bytes once. These sidecars do not change the preparation identity or selected prompt sequence.

Two reader threads prefetch up to 64 upcoming rows per pool into a shared cache, reducing each window when necessary to fit all pools within the shared row budget. Demand reads consume cached rows, leaving room for future candidates. The cache holds at most ten rollout batches of rows (minimum 64) and 256 MiB of raw JSON, whichever limit is reached first. Outstanding read requests have the same row limit. Only selected rows become prompt/verifier objects; generation receives independent copies. Prefetch completion order does not affect selection, rollback, or checkpoint restore. rollout/prompt_storage/* metrics report cache bytes, rows, pending reads, disk reads, and cache hits.

Replay-enabled pools still scan identities at startup and retain identity-to-row maps, but retrieve replay prompt payloads from disk too. This is additional identity bookkeeping, not a second copy of the prompt population.