Skip to content

Best-effort background olmo-eval

Opt in with evaluation.mode="background". Training captures frozen HF weights, then one driver submits independent Beaker jobs in a daemon worker. Evaluation never borrows, drains or pauses rollout engines, and shutdown never joins the submitter or waits for evaluator jobs. Missing points are expected: a busy submitter drops milestones from the next collection, submission failures are not retried, and preemption can interrupt submission. Submitted evaluator jobs automatically restart after preemption. Snapshot export is a synchronous trainer collective; checkpoint/collective failures still fail training.

Ready milestones from one collection are submitted as one batch, so they do not compete with each other for the submission worker. A wall-clock-stop final evaluation joins the last collection's batch, merging periodic and final tasks at the same update. The worker submits accepted receipts sequentially; it does not queue additional collections or wait at shutdown.

The three-node example opts into background evaluation with a pinned sandbox-enabled image and the 25-entry post-training development/text-safety panel. It sets interval=0, initial=true and final=true to evaluate the full panel before and after training only, including when the run is extended. Zero disables periodic evaluation; it does not disable either endpoint. Positive intervals count completed optimizer updates, not rollout collections. Omitting interval retains the default of 20 updates. Task-level intervals override the global value: a task can opt into periodic evaluation under a global zero, or set zero to opt out under a positive global interval. The global initial and final flags still apply to every task. With both flags false and all intervals zero, no background evaluations or evaluation snapshots are scheduled. The node example runs a smaller GSM8K, GSM8K-Platinum and IFBench panel initially, every 50 updates, and finally. Dev retains shared-engine evaluation, which remains the low-level default. The three-node template is provisional; check evaluator compatibility for the full model. Remove explicit shared settings (training.eval_interval, miles.eval_*, skip_eval_before_train, n_samples_per_eval_prompt, data.eval_prompt_data, positive task eval_count) when selecting background mode. Evaluation tasks are independent of training data.tasks; no evaluation-only dataset needs to be added to the training mix.

[evaluation]
mode = "background"
interval = 0                  # Initial/final only; positive values enable periodic evaluation
initial = true
final = true
image = "IMMUTABLE_BEAKER_IMAGE_ID" # Replace with a compatible evaluator image
revision = "FULL_40_CHARACTER_OLMO_EVAL_COMMIT"
gpus = 1                     # Separate allocation; never training GPUs
cluster = "ai2/holmes"
submit_timeout = 30           # Hard wall-clock bound on a submission subprocess
timeout = "3h"                # Evaluator Beaker task lifetime
provider_kwargs = { profile = "olmo3.5-12.5b", context_length = 34816 }

[[evaluation.tasks]]
task = "gsm8k"
generation = { temperature = 1.0, top_p = 0.95, max_tokens = 32768, num_samples = 1, stop_sequences = [] }
scoring = { limit = 128, num_fewshot = 0 }

[[evaluation.tasks]]
task = "gsm8k:platinum"
generation = { temperature = 1.0, top_p = 0.95, max_tokens = 32768, num_samples = 1, stop_sequences = [] }
scoring = { limit = 128, num_fewshot = 0 }

[[evaluation.tasks]]
task = "ifeval_ood" # IFBench; distinct from the IFEval task named "ifeval".
generation = { temperature = 1.0, top_p = 0.95, max_tokens = 32768, num_samples = 1 }
interval = 100
scoring = { limit = 128 }

[launch.secrets]
BEAKER_TOKEN = "your-beaker-token-secret"
WANDB_API_KEY = "your-wandb-secret"

# Optional task-specific credentials. These are secret references, never tokens.
[evaluation.secrets]
OPENAI_API_KEY = "your-judge-secret"

SciCode uses the SCICODE_H5PY_HOST_PATH Beaker secret by default. The secret contains the reference-file path used by the evaluator; it is provisioned in ai2/olmo-instruct and ai2/open-instruct-dev. The standard three-node panel needs no path edit. Evaluator jobs inherit launch.weka_mounts, which must include the filesystem holding the reference file.

To use another workspace or asset, set evaluation.secrets.SCICODE_H5PY_HOST_PATH to your Beaker secret's name, or set arguments.h5py_host_path explicitly in the SciCode task. An explicit path takes precedence over the secret. The evaluator resolves the secret and verifies that the file exists before starting inference. Secret values are not resolved on the submitting workstation or written into launch receipts. Explicit placeholders are rejected before launch, with a planning warning explaining how to use the default or supply an override.

Workspace and budget default to launch.workspace and launch.budget. Evaluator jobs inherit launch.weka_mounts, WANDB_API_KEY and HF_TOKEN secret references. evaluation.secrets can override these or supply task-specific credentials. The submitter needs Beaker credentials in the trainer allocation; evaluator jobs do not inherit its Beaker token. Outputs and snapshots must be on a WEKA mount accessible to both jobs. Inherit the main run's W&B entity, project and actual run ID; never configure a separate evaluation run.

The limited GSM8K tasks above are small mechanics examples. The three-node template uses individual task variants from math:posttrain:dev, general:posttrain:dev, hillclimb:dev and safety_thinking, deduplicated to 25 entries. Individual names make the panel explicit and do not create synthetic cross-benchmark aggregate scores. It retains full task sizes and explicitly sets candidate policy sampling to temperature 1.0, top-p 0.95 and 32,768 response tokens, including AIME/HMMT with 32 draws. These budgets are independent of the served context limit. The pinned IFBench defaults differ from newer upstream revisions; changing the image/revision can change the benchmark protocol.

Use the full panel at the initial and final checkpoints for official comparisons. The example above shows how one task can additionally run every 100 updates; its scoring.limit applies at every scheduled point, including initial and final. The three-node template has no such limits or periodic overrides. Progress subsets are not enabled by default: fewer prompts still incur separate-job and judge startup costs. If adding a periodic development panel, choose fixed example IDs with a documented seed and relevant category coverage, and distinguish its metrics from the full-panel scores. A simple limit is not a seeded or stratified sample, and this configuration does not switch sampling/scoring settings between endpoints and periodic points. Keep release held-out evaluation separate from development metrics used to select recipes or checkpoints.

Generation overrides map to olmo-eval's flat per-task sampling overrides (for example -o max_tokens=2048, not a nested sampling_params table); scoring maps to olmo-eval task overrides (such as limit, scorer settings, or formatting). Tasks with the same generation overrides, harness preset, harness overrides and execution kind and auxiliary GPU count share a job. Empty generation overrides preserve each task's own defaults. Each unfinished task starts its own native SGLang engine; completed tasks remain reusable after preemption. Model loading repeats between tasks. The three-node panel creates six groups: seven 32-draw math tasks, ten other native tasks, LiveCodeBench, SciCode, five WildGuard-judged tasks and StrongREJECT. Each group requests one policy GPU; each judge group also reserves one GPU with task auxiliary_gpus=1. The worker exposes the policy and auxiliary GPUs to olmo-eval, whose planner assigns non-overlapping GPU sets and manages judge startup and shutdown. One HF snapshot serves all groups. A coincident periodic/final point is submitted only once for each task group. plan displays the milestones, task groups, resources and snapshot paths. validate checks the schema; it does not establish image/model/task compatibility.

Code and safety requirements

harness_preset selects an olmo-eval preset (default: default). The optional task harness table passes flat harness overrides before the --task argument, so auxiliary providers and sandboxes are not mistaken for scoring overrides. The frozen checkpoint's provider settings cannot be overridden here. Harness settings are recorded in receipts and checked when reusing completed tasks.

The three-node template selects codex_python for LiveCodeBench and enables sandboxes.0.inject_swerex to preinstall the sandbox server. The pinned runtime's on-demand installation fallback otherwise invokes an unavailable executable. Code jobs receive Beaker's subcontainer environment flags and need an evaluator image containing Podman and the olmo-eval sandbox dependencies. Container startup and reference-data access still require qualification on the target cluster.

SciCode uses kind="external" and arguments, invoking olmo-eval's run-external command against a dedicated frozen SGLang HTTP server. SciCode requires VLLMServerProvider in the pinned olmo-eval revision and is the explicit exception to native sglang routing. Its server resolves the same evaluation.provider_kwargs profile against its assigned policy GPU, recording sglang-external-profile.json beside the server log. A manually configured server can still use evaluation.external_server_args with profile = "generic"; those CLI arguments replace the typed serving settings for SciCode only. Only SciCode currently has a supported external artifact contract. Development runs require arguments.split="validation" and an accessible arguments.h5py_host_path. The evaluator checks per-problem predictions and step counts, rejecting infrastructure errors even when upstream reports success with numeric zero scores. SciCode's sandbox installs its pinned scientific dependencies from the upstream lock; account for startup time and registry/package access when qualifying jobs.

SciCode's SGLang serving boundary translates an exported HF top_k=0 to SGLang's equivalent top_k=-1 using a temporary local checkpoint view. Positive values and every other generation setting are preserved; the frozen checkpoint is never edited. scicode-sampling-normalization.json records the source config hash and the exact translation. Infrastructure errors in SciCode predictions remain fatal and include the original error in the runner diagnostic. The view lives outside the results directory to avoid copying model weights into Beaker artifacts.

Safety tasks configure olmo-eval managed WildGuard and StrongREJECT providers. WildGuard serves allenai/wildguard; StrongREJECT serves google/gemma-2b with the qylu4156/strongreject-15k-v1 adapter and its tokenizer. The sandbox evaluator image includes the pinned vLLM extra in an isolated environment selected through VLLM_PYTHON, preserving the policy serving environment. Model access credentials (including gated Gemma access) belong in evaluation.secrets. Record resolved judge revisions and serving settings with qualification results.

The maintained managed WildGuard and Gemma StrongREJECT configurations default auxiliary_providers.<name>.kwargs.startup_timeout to 1200 seconds. Explicit values win; external endpoints and other model configurations are unchanged. The runner resolves the value when building each evaluator command and records it in the attempt's command.json. It reaches the evaluator's server lifecycle, not the vLLM CLI. Implicit defaults leave saved run specs and evaluation claims unchanged. Readiness still detects exited processes early and retains startup logs. This allowance does not change the overall Beaker job timeout or any sampling/scoring settings. Slow startup alone does not establish whether loading, compilation, or another cause was responsible.

The maintained three-node evaluator also emits live [vllm-startup] messages with elapsed time, deadline, full log path, and the last phase observed in judge logs. It checks process exit independently of HTTP probes and preserves INFO-level child logs. Unknown phases remain explicit; timeout errors are distinct from process exits. Older evaluator images receive the profile timeout but need the startup image below for phase diagnostics. Submitted jobs embed their runner, so regenerate their specifications from the fixed checkout to receive the live relay.

Planning rejects the maintained managed vllm_server judges that download allenai/wildguard (five tasks) or google/gemma-2b (StrongREJECT) unless evaluation.secrets.HF_TOKEN is configured or inherited from launch.secrets. It also rejects GPQA tasks (gpqa_diamond, gpqa_main:cot), whose Idavidrein/gpqa dataset is gated, without that secret. Set the value to a Beaker secret name, not the token itself. The token's account must have accepted access terms for both models and the GPQA dataset. A local shell token is not forwarded to evaluator jobs. This offline check verifies the secret reference, not its contents or model permissions; it covers these explicit model/tokenizer overrides, not arbitrary upstream presets. Local model paths, existing server endpoints and tasks with all evaluation milestones disabled are exempt.

The image and CLI/config checks do not establish full-panel correctness, GPU memory fit or completion within the configured timeout. Run small samples of every task, inspect predictions and judge/code failures, then measure a full panel before setting production resource bounds. For qualification, verify all 25 entries completed at both expected checkpoints and were published; training completion alone does not establish evaluation completion.

The examples require profile = "olmo3.5-12.5b". The evaluator selects the qualified settings for the actual GPU, including H100 80GB and B300: full decode CUDA graphs, prefill graphs disabled, and hardware-specific cache and memory budgets. An incompatible checkpoint, GPU, topology or context fails instead of silently selecting generic settings. Other background configurations default to profile = "auto", which selects a qualified profile when available and records the reason when it falls back. Profiles are model-specific; an H100 or B300 alone does not qualify an arbitrary model.

Bounded H100 and B300 GPU checks each completed 64 GSM8K prompts with eight samples per prompt (512 expanded samples), selected the corresponding optimized profile with decode CUDA graphs, and recorded no failed requests. These checks qualify profile execution; they are not a matched cross-hardware throughput comparison or full-panel validation.

Native tasks use batching.chunk_size = "auto". For a qualified profile, olmo-eval derives a sequence budget from the running engine's capacity (requested 512, reduced if the engine reports a lower limit). This counts expanded samples: pass@32 uses at most 16 prompts for a 512-sequence budget, rather than 512 prompts. With profile = "generic", automatic batching falls back to 64 examples unless a batch-sequence budget is explicitly supplied. Qualified profiles use their capacity recommendation instead. Explicit task harness overrides still win. Inspect the resolved settings in runtime/sglang-profile-*.json in each task attempt.

This removes the generic fallback's 64-example default, but the pinned native SGLang provider still processes batches sequentially and waits for each batch's longest response. It does not support batching.strategy = "streaming". Continuous admission across batch boundaries requires an upstream provider change. The full safety panel's throughput and memory with these settings still need measurement on each GPU family; keep this separate from profile qualification.

The earlier generic, graphs-disabled configuration took about 21–26 minutes per 64-example safety batch on a B300. Its estimated 10–16 hour safety tasks did not fit the three-hour default. Do not assume the optimized profile makes that timeout sufficient: measure the full tasks and set evaluation.timeout accordingly. An interrupted task restarts from its beginning; completed tasks are reusable.

Snapshots and receipts

Snapshots live at OUTPUT/eval-snapshots/update-NNNNNNNN/hf, including model configuration, tokenizer and chat template. A .complete marker gates submission. Update zero reuses the prepared initial HF checkpoint and records a reference under eval-snapshots/update-00000000/reference.json; retain its source weights as well. Snapshot paths are exclusive and never overwritten. An incomplete snapshot from an interrupted export is retained and skipped on resume.

OUTPUT/evaluation/update-UPDATE-GROUP.json records checkpoint/update, task configuration, evaluator image/revision, runner hash, training identity, and the Beaker experiment ID or diagnostic. All recorded attempts suppress automatic resubmission, including ambiguous timeouts and interrupted pending receipts. No backlog or polling service runs in training. Submission failures produce BACKGROUND EVALUATION GAP warnings. Evaluator failures appear in their Beaker job and result directory and leave a gap in the dashboard.

Results are retained in OUTPUT/evaluation/results/update-UPDATE-GROUP/ and copied to the evaluator's Beaker results: aggregate scores.json, olmo-eval metrics.json, predictions/requests, logs, command/configuration and provenance. A failed W&B upload writes publication.json and preserves these files.

Recovery after preemption

Evaluator jobs use high priority, unallocated scheduling, and Beaker auto-resume. On restart, the runner checks the existing output provenance against the checkpoint, update, tasks/sampling/scoring, evaluator revision/image and server configuration. It refuses incompatible results rather than overwriting them. A process-owned filesystem lock prevents two evaluators from writing the same group and is released when its holder dies.

Tasks execute separately while sharing one inference server. Each successful task receives an atomic completion record after predictions and scores validate. A restart skips those completed tasks. An unfinished task runs in a new attempt directory; its previous partial files remain intact. Native olmo-eval response resumption is not assumed, so an interrupted individual task is regenerated. task-results.json selects only successful attempt directories for validation and publication; abandoned attempts cannot introduce duplicate scores.

A fully completed group validates and reuses its results without loading the model. A restart after the last task but before the group completion record also finishes without loading the model. Publication may be retried, so duplicate W&B points remain possible. Existing flat, completed result directories are accepted only after the same artifact/coverage checks. The runner is embedded in each Beaker spec, so retrying a spec runs the runner it was submitted with.

The runner rejects missing prediction files and empty model_output lists before declaring evaluation complete. Some evaluator provider failures can otherwise appear as zero scores with no failed-instance count. A returned EOS-only completion is distinct from a missing provider response and remains a valid scored output.

W&B publication

The publisher attaches to the exact main training run using W&B's documented shared-mode secondary writer with x_primary=False and x_update_finish_state=False. Late uploads may change the dashboard status from finished to running; this accepted status change does not restart training. The evaluator flushes its writer without declaring an active training run finished. It does not restore or rewrite lifecycle state.

All MILES writers in background mode and every evaluator declare the same metric schema: native training/rollout axes are retained, and eval/* uses the explicit eval/checkpoint_update axis. This matters because W&B replaces metric-definition metadata per writer. A common wildcard covers different task groups without one evaluator erasing another group's axes. No internal W&B step is supplied, so results can arrive out of checkpoint order. The publisher sends no training configuration, name or group updates.

After a successful upload, publication.json records the destination and checkpoint update. Resume skips an upload already confirmed for that same destination and update; failed or deferred uploads can be retried. A crash after W&B accepts the upload but before the local receipt is written can still cause a duplicate.

For offline training, sync the training run first, then publish to that existing run with the manual command. The same command repairs a failed online upload:

python -m open_instruct.miles.evaluation.evaluation_runner publish RECEIPT.json \
  --results /path/to/evaluation/results/update-UPDATE-GROUP \
  --wandb-run ENTITY/PROJECT/RUN_ID

Uploads can change the dashboard status as described above. Failed uploads leave publication.json and retain scores, predictions, requests, configuration and provenance. Repeating publication can append duplicate points; inspect publication.json first.

Manual resubmission and cleanup

Inspect the receipt and Beaker experiments before resubmitting: a timeout may mean the API accepted the job but the trainer missed its response. Check the latest job attempt; if queued, inspect beaker job events JOB_ID.

python -m open_instruct.miles.evaluation.evaluation resubmit RECEIPT.json

This creates a new receipt and results group, retaining the original diagnostic. It requires the committed runner source matching runner_sha256, the recorded image, secrets, mounts and completed snapshot. Submission/application failures are not automatically resubmitted; Beaker auto-resume handles preemption within an already submitted experiment.

Snapshots are excluded from ordinary checkpoint retention and result collection. There are no pins, leases, automatic deletion or retention controller. Manually remove selected eval-snapshots/update-NNNNNNNN directories only after verifying that all jobs referencing them have stopped (including manual resubmissions). Do not delete snapshots while jobs are pending or running. Keep small receipts and results as provenance. Initial checkpoint reuse additionally depends on the original HF source remaining available.

Evaluator image and compatibility

runtime/miles/Dockerfile.evaluator builds a separate evaluator image on the digest-pinned CUDA 13.0.2 / Torch 2.13 compiler-equipped olmo-eval base. It installs the pinned sglang extra, including the engine and Olmo model plugin. Regular background evaluations select provider.kind=sglang and provider.kwargs.model_registration=olmo_sglang:register. SciCode retains its explicit HTTP exception described above. The runner checks the installed olmo-eval revision against the receipt before evaluating.

The node example pins olmo-eval revision 3536fe751e40162cc33d5e647085e986375aadce and image 01M5ZGD5K01GNW3ASDPNXXPEJ3. The three-node example pins ba0ff700e8f9b301cfd8ed5fd462df9d83dddfe1 and image 01M6J0KMAD3CXVGT07V1BHR1VH, retaining H169's qualified sandbox and isolated vLLM runtime. Its only evaluator code change from H169's ede61eb9 image is the shared startup lifecycle fix (olmo-eval PR 541); the earlier 3536fe75 to ede61eb9 change fixes explicit CLI null overrides. Neither change alters benchmark prompts, sampling defaults, or scoring. The startup image recipe is runtime/miles/Dockerfile.evaluator-startup.

Replace the old evaluation.server_args with typed evaluation.provider_kwargs using Python option names, for example context_length = 34816. Let the selected profile own graph, radix-cache and memory settings; remove the old explicit disable_cuda_graph and disable_radix_cache overrides when migrating the examples. B300 enables the qualified hybrid-model radix path, while H100 disables radix caching. Use profile = "generic" for deliberate manual tuning or an unqualified model; profile-conflicting overrides are rejected.

Full-attention Olmo MoE models are distinct from the qualified hybrid geometry. The pinned runtime can enter a Mamba-buffer path without an allocated buffer for those models: a tiny full-attention MoE check failed with caching enabled and passed with disable_radix_cache = true. Keep that workaround when manually configuring those models; it is not a reason to disable the hybrid model's profile. For SciCode, existing configs with external_server_args must either remove them to use the same typed profile settings, or explicitly select provider_kwargs.profile = "generic" to retain the manual CLI server. Omitting the profile now fails planning when those CLI arguments are present. The typed SciCode path accepts serving options and the max_model_len/max_concurrency aliases; native-provider-only options add_bos_token, prompt_format and force_download are rejected at planning time. Use a separate native-only evaluation configuration if those provider features are required. The checkpoint, tokenizer, model registration and GPU allocation remain owned by MILES. Task overrides cannot change the policy provider. Native SGLang supports generation, not tools or continuation likelihood; code scoring in a sandbox is separate from tool use.

docker build -f runtime/miles/Dockerfile.evaluator \
  --secret id=github_token,env=GH_TOKEN \
  -t open-instruct-miles-evaluator:sglang runtime/miles
docker build -f runtime/miles/Dockerfile.evaluator-sandbox \
  --build-arg EVALUATOR_IMAGE=open-instruct-miles-evaluator:sglang \
  -t open-instruct-miles-evaluator:sglang-panel runtime/miles

Keep each config's image and revision matched. Changing only the revision does not upgrade an existing image. CPU tests check command parsing, GPU planning, image dependencies and task resolution. A one-GPU tiny MoE check completed native SGLang generation and scoring for two GSM8K examples on Holmes with radix caching disabled. The repaired sandbox image passed a two-example LiveCodeBench check with a frozen Qwen2.5-Coder-0.5B-Instruct policy: all four sandbox monitors started, both predictions were nonempty, and both examples were scored with zero infrastructure failures. The same policy passed execution checks for all five WildGuard tasks and StrongREJECT, each with two scored examples and zero failed instances. These checks used the repaired image on Holmes in ai2/open-instruct-dev, with one B300 for LiveCodeBench and two B300s for each judge group. They verify execution; model quality and full-panel resource bounds still need separate qualification.

The provider contract tests use the installed olmo-eval CLI and GPU planner, without loading a model or downloading datasets. Run them inside the panel image with this checkout mounted and pytest installed:

python -m pytest --confcutdir=tests/miles_eval tests/miles_eval/test_background_provider.py

Check compatibility for the exported architecture, serving backend, task and image; successful provider initialization alone does not prove task correctness. Use generation tasks for an initial tiny MoE check. A successful mechanics check says nothing about model quality. Launch training through the normal committed-image MILES wrapper. Use ignored runs/ configs and fresh output paths; leave existing experiments unchanged.

Development gold labels and scoring

Use scripts/miles/development_eval.py for custom math/GSM8K development scoring. Prepared math labels can be lists of acceptable strings. Preserve that flat list rather than wrapping it again: ['42'] must not become [['42']]. The scorer rejects malformed labels instead of silently recording zero. GSM8K uses the training reward's exact decimal comparison, so 18.00, +18 and 18 are equivalent. Its parity tests run in the normal CPU suite without olmo-eval; math scoring still requires the olmo-eval environment. Regression tests cover scalar/list/multiple gold answers and completion gates:

# Run inside the pinned olmo-eval environment; no GPU is needed.
python -m pytest --confcutdir=tests/miles_eval -o pythonpath=scripts/miles tests/miles_eval

These tests are separate from the MILES trainer suite because they use the evaluator environment. pythonpath mirrors the standalone module layout in custom evaluator bundles; a missing scoring module fails collection when olmo-eval is installed. Without olmo-eval, ordinary CPU collection skips them.

Include this module in the custom worker's content manifest and import its score_one(row, item) for development tasks. Check the actual panel's label schema, not just synthetic scalar-label canaries. Version corrected bundles; never modify a running experiment's immutable dataset in place.

Reasoning-panel response budgets

The three-node panel uses gpqa_diamond:cot and gpqa_main:cot. The plain gpqa_diamond task is a short-answer protocol with a 1,024-token default; it can truncate almost every response from a reasoning policy. These are different protocols: do not compare their scores as interchangeable.

OMEGA sets generation = { max_tokens = "null" }: the CLI parses this as no fixed response cap, so SGLang generates within the remaining model context. This follows the existing GPQA CoT and IF protocols and retains the qualified 34,816-token serving profile. A fixed 32,768-token response request would instead be rejected whenever the prompt exceeds 2,048 tokens. Requests and predictions retain prompt/completion lengths and finish reasons. The full panel allows 12 hours per job, with completed-task reuse after preemption.

A failed task is recorded in task-failures.json; subsequent independent tasks still execute. The collection remains failed until every task validates. Retrying reuses successful task attempts and reruns the failures.

Shared in-loop policy evaluation

Held-out evaluation in the training allocation defaults to temperature 1.0 and top-p 0.95, with an inherited response budget capped at 32,768 tokens. It retains a shorter training response budget in smoke tests. Explicit miles.eval_temperature, miles.eval_top_p and miles.eval_max_response_len (or the structured inference alias) override these defaults. OPD in-loop evaluation uses the same sampling and cap when its evaluation response length is inherited. These settings do not change training rollouts or reward judges.

The candidate background configuration uses per-task overrides rather than changing the evaluator's global defaults. SciCode uses arguments.temperature=1.0 and arguments.max_tokens=32768; the pinned external task has no top-p control. Keep that limitation visible until the evaluator supports it. For benchmarks with their own decoding contract (for example RULER or BFCL), preserve that contract rather than applying these settings globally. Keep pre-change scores labelled by their original protocol when comparing checkpoints.