Skip to content

Choosing sequence lengths and memory controls

Start from a structured example, then change the length and capacity controls together. A checkpoint's advertised maximum positions (for example 65,536) is an architecture limit, not a guarantee that every training topology or serving concurrency fits.

Choose a prompt/response budget first

inference.max_context_length sets the Core, MILES and SGLang total sequence limits. inference.max_response_length limits newly generated tokens. miles.rollout_max_prompt_len bounds the rendered and tokenized prompt, including its chat template. Reserve room for both; changing the response cap without considering the input budget can reject otherwise useful data.

Examples of budget arithmetic (not capacity guarantees):

Context Long-input, short-answer task: prompt + response Short-input reasoning task: prompt + response
16,384 15,872 + 512 2,048 + 14,336
32,768 32,256 + 512 2,048 + 30,720
65,536 65,024 + 512 2,048 + 63,488

The prompt limit defaults to context minus response. Prefer explicit budgets for long-sequence experiments. A dataset's “64K” label may use another tokenizer; measure with the actual model and template. Imported immutable data is checked against the prompt limit, not silently truncated. The packer likewise never splits an individual sample to make it fit.

Set capacity independently of optimizer batch size

Size engine admission and its pools from GPU memory with the procedure in admission sizing. The table below explains what each control does as length grows.

Control What to change or watch as length grows
trainer.micro_batch_size Keep at one on the currently supported Core path.
trainer.sequence_packing, trainer.packing_max_tokens Packing helps combine short samples; it does not shrink one long sample. The pack budget must cover max_context_length. Start with equal budgets.
trainer.activation_recompute Enable for long responses; trades additional compute for activation memory.
miles.log_probs_chunk_size A value such as 1,024 limits temporary log-probability processing. This does not eliminate the model's full token-by-vocabulary logits allocation.
inference.sglang_server_concurrency Controls client requests. Lower it deliberately for initial long-sequence tests.
inference.sglang_max_running_requests Engine admission ceiling. Set alongside client concurrency; neither is an optimizer batch-size control.
inference.sglang_max_total_tokens Explicit full-attention KV token budget. Inspect the allocated pool in startup logs; do not assume the requested ceiling was allocated.
inference.sglang_max_mamba_cache_size KDA recurrent state capacity, separate from KV tokens. With radix off, cover running requests. Retained prefix caching requires substantially more slots; see the configuration validator and topology guide.
miles.sglang_chunked_prefill_size Bounds work in each prefill chunk, for example 2,048; this does not bound the completed request's KV footprint.
inference.sglang_cuda_graph_max_bs_decode Match the admission range being tested; large graph captures consume memory.
inference.sglang_mem_fraction_static Leaves headroom for transient allocations. A value of 0.6 suits disaggregated engines; this is not a colocated recommendation.

An explicit 131,072-token pool has a theoretical upper bound of eight full 16K, four full 32K, or two full 64K sequences. Treat this as budget arithmetic, not an achieved throughput target: allocation rounding, reserved capacity, other requests and transient memory matter. The default pool request can be much larger because it scales with configured admission and context. Set it explicitly for long-sequence runs instead of retaining a short-sequence concurrency blindly.

Keep collection and optimizer batch sizes fixed while studying engine admission. Reducing admission queues requests; reducing samples per prompt changes the GRPO recipe. EP distributes expert parameters/work, not the token sequence itself. The current adapter does not offer trainer TP/PP/CP greater than one as a remedy for a single sequence that does not fit.

Current limits and serving considerations

Bounded long-response RL runs work with expert parallelism, packing and recomputation at 16K and 32K context. 64K backward passes and high-concurrency long-context training are not yet established; check memory before relying on them.

For long-input, short-output serving, reduce client concurrency as context grows so aggregate tokens stay within the KV pool, and consider turning radix caching off when prompts do not share prefixes. Serving-only memory is not a backward-memory estimate: even long-input, short-answer training must run the model over the prompt, and router trace return and replay add work that a standalone serving check does not measure. Synthetic long inputs show execution capacity; their repeated filler and short outputs do not represent long reasoning throughput or task accuracy.

Recognize the failure before changing knobs

  • Prompt validation failure: inspect rendered token counts and the reserved response budget. Changing GPU memory will not fix a prompt-budget mismatch.
  • Serving OOM or KV retractions: inspect cache allocation and the memory trace; lower admission, graph size or prefill chunk size according to the allocation that failed. Repeated retractions can finish successfully with poor throughput.
  • Trainer forward/backward OOM: inspect actual sample length, pack size, recomputation and logits/scoring temporaries. More inference GPUs do not help.
  • Slow first batch: separate compilation/startup from warm steps. Record cache provenance before treating a cold run as steady throughput.
  • Long generation tails/timeouts: inspect cap hits and actual completions. Async buffering cannot guarantee that a queue stays full when long generations are cancelled at publication. Do not infer a scheduling fix from a larger cap.
  • Verifier failure: code timeouts and judge context limits are independent of policy context. A judge must fit prompt, response and rubric together.

Use a small synchronous run to establish length correctness and memory first, then add the production async and service configuration. A useful result records finite rewards/gradients, nonzero advantages, publication, replay mismatches, actual sequence lengths, memory and stage times. Report a lifecycle pass separately when the sampled rewards do not exercise a policy gradient.