Running SFT on TPU: a known-good starting point
Olmo 3.5 tiny (olmo35-tiny, the 4T EMO base, called "small" before 2026-11) on Dolci-Think with H045's recipe, on one TPU v4 4x4x8 slice (128 chips) of
the gke-tpu-v4 cluster, through MaxText + Tunix.
Measured cost and outcome: 11,768 steps of 16 × 65,536 tokens at a median 3.94 s/step, 13.5 h wall clock from submit to the last step. The loss matches H045's GPU runs (parity windows in open-instruct#1933). The GPU recipe takes 5 h 54 m on 2×8 H100, so the TPU path is slower per run. Its advantage is capacity: three of these runs fit at once without touching Beaker.
The recipe is scripts/tpu/maxtext/k8s/olmo35-sft-4x4x8/.
Its kustomization.yaml header records why each setting is what it is; read it before editing. Olmo 3 7B has its
own overlay, k8s/instruct-sft-4x4x8/; its base k8s/olmo3-sft/ targets a 2x2x4 slice, which no longer exists. scripts/tpu/maxtext/README.md lists the MaxText patches.
Example runs
| Run | What it is |
|---|---|
olmo35-sft-h045-tpu-full-r2 |
This recipe, verbatim: the TPU replica of the ledger's H045 anchor (arm anchor-emo-olmo35-tpu). 0 restarts, final CE 0.9245. HF export: /weka/oe-adapt-default/abhishekr/tpu-posttrain/hf/olmo35-sft-h045-tpu-full-r2-step11768. |
| H067, H072 (olmo-post-training-ledger) | The same recipe with other data: arms, seeds, resumes and their checks are recorded in those hypothesis files and in open-instruct#1976 and #1977. |
Before you start
- IAM on the
ai2-olmoGCP project, e.g.roles/container.developer. - A network the cluster accepts. allenai/olmo-gcp-infra's README
lists them: the office IPs, its Tailscale exit nodes, and Beaker sessions on
ai2/hammond. From anywhere else, use the DNS endpoint (--dns-endpoint). Then:
gcloud container clusters get-credentials gke-tpu-v4 --zone us-central2-b --project ai2-olmo
kubectl -n olmo get workloads
kubectl needs gke-gcloud-auth-plugin (gcloud components install gke-gcloud-auth-plugin, or a binary in ~/.local/bin).
- gcloud logins expire about daily, and kubectl then fails quietly. Watchers go blind without an error. Agents
use the service account tpu-posttrain-agent@ai2-olmo.iam.gserviceaccount.com: export
CLOUDSDK_CORE_ACCOUNT to it in every process, background watchers included. Move
~/.kube/gke_gcloud_auth_plugin_cache aside when switching identity, or kubectl keeps the cached one. People log in
with their own account.
- Agents. Claude Code and Codex in this repo load a summary of this guide's rules from the
tpu skill. Update it when you change
a rule here.
- Where things live. Everything is in us-central2. Data, tokenizers and checkpoints are in gs://ai2-olmo-dev/post-training/,
and the nodes mount its data/ and tokenizers/ subfolders, each on its own, under /mnt/post-training through
gcsfuse. Never mount a folder that holds checkpoints: the metadata prefetch lists all of it on every host at every
pod start (k8s/narrow_mounts.py narrows an overlay). Images are in
us-docker.pkg.dev/ai2-olmo/olmo/maxtext-posttrain.
What the cluster is
gke-tpu-v4 holds 4,096 reserved v4 chips at 4 per host. Since 2026-11-03 they are cut into three 4x4x8 slices
(128 chips), one 8x8x8 (512) and hero's 8x8x48 (3,072), with 128 chips held back so GKE can auto-repair failed
nodes. The layout changes as the cluster's admins reslice it, so check it before planning:
kubectl get nodes -L cloud.google.com/gke-tpu-topology --no-headers | awk '{print $NF}' | sort | uniq -c (hosts per
topology, 32 per 4x4x8; the 2026-11-03 layout prints 96 4x4x8, 128 8x8x8 and 768 8x8x48). A slice runs one job at a time. Kueue admits jobs from four queues in one cohort:
| Queue | Who | Nominal quota (chips) |
|---|---|---|
hero |
the main pretraining run | 8x8x48: 3,072 |
dev |
tests | one 4x4x8: 128 |
post-training |
us | the 8x8x8: 512; two 4x4x8s: 256 |
data-adaptation |
none |
Every queue has reclaimWithinCohort: Any, borrowWithinCohort: Never and withinClusterQueue: Never, so a job
running within its own queue's nominal quota is never preempted, and a queue can reclaim only quota that another has
borrowed from it. Dev may borrow an idle post-training 4x4x8, which we reclaim when we need it; a post-training job
on quota borrowed from hero or dev is preempted when they want it back. Read from
kubectl get clusterqueues -o json (spec.resourceGroups[].flavors[].resources[].nominalQuota) on 2026-11-05 at
11:30 PST; re-read it before planning, since the admins change it.
Use post-training. Checkpoint often enough to lose little on a preemption, and never hold a slice in a way that
delays hero.
The recipe needs exactly a 4x4x8: its mesh puts one 64K row on 8 chips, so 128 chips hold its 16 rows. The
8x8x8 ran the same mesh at 64 rows per step, which is a different recipe (H078's 4x batch). Its measured 16-row
configuration used olmo35-cp-ep, splitting each row 32 ways and changing the MoE load-balance term. Measured meshes are
in Changing the recipe.
Cost per slice
From TensorBoard perf/*. Throughput is tokens per chip-second: tokens per device per step divided by the mean
step time over steps 31-100, so a stalled step counts at its full cost.
| slice | config | rows/step | median s/step | tokens/s/chip | steps over 3x median |
|---|---|---|---|---|---|
4x4x8 |
the recipe (A0') | 16 | 3.88 | 2,055 | 0 |
8x8x8 |
H078's config, with the data-order shim (H088 opt-a) | 64 | 4.28 | 1,386 | 4.3% (13% over H078's whole runs) |
8x8x8 |
same, no shim and no batch hashes (H088 opt-b) | 64 | 4.23 | 1,887 | 0 |
8x8x8 |
opt-b + remat_policy=save_dot_except_mlp (H088 opt-c) |
64 | 4.18 | 1,900 | 0; a tie, keep full |
Measurement windows: the 4x4x8 row is from 2026-10-14 and the 8x8x8 rows from 2026-10-18 (H088, the pre-split
slice), each taking tokens/s/chip from the mean step over steps 31-100. The 2026-11-05 sweep figures quoted below
(4.24 s, 1,934 tokens/s/chip for opt-b's config) take the median of steps 3-14 on the recreated slice, so they don't
mix with this table.
Startup before step 1 (pods, restore, an 11-12 min compile) adds 20-35 min on either slice, paid again at every resume.
- Use a
4x4x8for this recipe. Among the measured configurations it used the fewest chip-hours per token. The8x8x8rows above date from before its 2026-10-21 split. On the slice recreated on 2026-11-03, opt-b's config measured 4.24 s and 1,934 tokens/s/chip (median of steps 3-14, 2026-11-05; see Changing the recipe). The8x8x8overlay isk8s/olmo35-sft-8x8x8/: the same token budget at 64 rows and 2,942 steps. Its data order differs from the4x4x8recipe's, so an arm on it needed a control on the same slice and config. Without the shim the8x8x8is projected at ~1.09x the chip-hours of the4x4x8, finishing a run ~3.6x sooner (H088, allenai/open-instruct#2020). This is a steady-state projection, not a measured cost. Both rates come from stall-free 70-step windows on different days (4x4x810-14,8x8x810-18). The8x8x8's stall rate over a whole run without the shim is not measured. The8x8x8could not run the 16-row recipe at that rate (see Changing the recipe). 8x8x8input stalls (H088). H088 removed the data-order shim (MAXTEXT_R2_DATA_EMULATION) andMAXTEXT_LOG_BATCH_HASH=1together, so their separate contributions are not isolated. With both, one step in eight stalled for 20-35 s, at the same steps in every run, and H078's runs cost 1.8-1.9x the chip-hours of a4x4x8. In xprof the stall is one host's input pipeline (grainprocess_prefetchwaiting 29 s) while every TPU waits for its batch injit__device_put_reshard. The HF input pipeline allows at most one grain worker, so more workers are not a fix.- Keep
remat_policy=full. Saving dot outputs except the MLP's gained 0.7%, with the same loss. - Outside stalls a step is ~4.0 s of device compute and ~0.2-0.5 s idle. FSDP all-gather waits are ~16% of the
compute, and
num_vocab_tiling=16costs ~4% (the 205 MB embedding is all-gathered and its gradient all-reduced 96 times a step). - Both slices run at ~4% of v4's 275 TFLOP/s bf16 peak by MaxText's own FLOP count.
jaxmemory_stats()peaks (0.9-2.3 GiB) leave out XLA temporaries: the same8x8x8config needed 32.17 GB at compile withragged_buffer_factor=4.0. Read HBM headroom from the compile error or xprof, not from those lines.
1. Image
us-docker.pkg.dev/ai2-olmo/olmo/maxtext-posttrain:51f8cc4b-p907d86f3-rfb46e215e
This tag is r2's image (51f8cc4b-p907d86f3, built from image-olmo35/) plus image-olmo35-rf/. That patch adds a
fix for resuming from a checkpoint (without it, restore runs out of HBM) and batch-hash logging. Training numerics are
unchanged: a replay produced bit-identical CE and 9,952 of 9,952 equal batch hashes. This is the default image for the
recipe above. The optional pooled resume image includes additional patches;
its parity checks compare against its own base image. The Olmo 3 image is built by image/build.sh <maxtext commit>.
To build a new image, run build.sh from any host with gcloud (a Beaker session works). Cloud Build carries the context base64-encoded inside the build
config. Builds use Cloud Build's default service account. Set BUILD_SERVICE_ACCOUNT to a full
projects/<project>/serviceAccounts/<email> resource only when the caller is authorized to use that account.
2. Base checkpoint
The loader reads an Orbax parameter tree (…/0/items) from GCS.
- Olmo 3.5 (OLMo-core distcp). This runs in two stages, on CPU:
python scripts/tpu/maxtext/olmo35/dump_olmocore_masters.py <step dir> <masters dir> # needs olmo_core
python scripts/tpu/maxtext/olmo35/convert_olmocore_to_maxtext.py <step dir> <masters dir> <out> # needs maxtext
gcloud storage cp -r <out>/0 gs://ai2-olmo-dev/post-training/checkpoints/<name>/
It keeps OLMo-core's padded 100,352 vocabulary, whose padded rows were trained. The 4T EMO base is already at
gs://ai2-olmo-dev/post-training/checkpoints/olmo35-small-4t-emo-step5961/0/items (named before the small→tiny
rename; it is the tiny base).
- Olmo 3.5 small (MaxText pretraining checkpoint). No conversion: point LOAD_PARAMETERS_PATH at the run's
checkpoints/<step>/items with scan_layers=False, since the hero saves unscanned layers. It restores in ~94 s on
the 8x8x8, also with fewer layers (base_num_decoder_layers=16) for a memory probe.
- Olmo 3 (HF). Use MaxText's converter with model_name=olmo3-7b. olmo3-7b-pt has no YaRN, so it is wrong for
the 1025 base.
JAX_PLATFORMS=cpu python -m maxtext.checkpoint_conversion.to_maxtext model_name=olmo3-7b base_output_directory=<out> \
scan_layers=True use_multimodal=false hardware=cpu skip_jax_distributed_system=true \
checkpoint_storage_use_zarr3=1 checkpoint_storage_use_ocdbt=1 --hf_model_path=allenai/Olmo-3-1025-7B --save_dtype=bfloat16
Check any new conversion numerically against the source model before training on it. Key and shape checks pass on broken conversions.
3. Data and tokenizer
MaxText streams parquet files. Its shuffle only permutes files and mixes a small buffer, and hub shards are grouped by
source, so shuffle globally first. olmo35/build_full_dolci_think.py (Olmo 3.5) and data/build_full_dolci.py
(Olmo 3) write N shuffled shards from one dataset revision and use the standard HF cache settings. Pin the input with
--revision <dataset commit>, or use --data-files '<local parquet glob>' for local files; the output directory must be
new or empty. Upload the shards to gs://ai2-olmo-dev/post-training/data/<name>/. Ledger experiments shard each
arm and seed into 512 files with experiments/h067/shard.py in olmo-post-training-ledger.
The tokenizer is a directory on the bucket. For Olmo 3.5, olmo35/tok_oi35.py saves H045's tokenizer and template
(allenai/dolma2-tokenizer-olmo35 @8b971706, olmo35). For Olmo 3, save
PreTrainedTokenizerFast.from_pretrained("allenai/olmo-3-tokenizer-instruct-dev") (see Traps).
Plain-text rows (long-context replay documents) need scripts/tpu/maxtext/image-olmo35-dr/: a conversation of exactly
one {"role": "document"} message is tokenized without the chat template, gets one EOS and is trained on every token
(1e612a76 → 1e23ebbc);
chat rows tokenize as before. Other images do not special-case the role. It is built on the same ...-dc88c6c7ea
base as the pooled image in section 6, not on top of it, so MAXTEXT_SFT_TEMPLATE_WORKERS does nothing there:
us-docker.pkg.dev/ai2-olmo/olmo/maxtext-posttrain:51f8cc4b-p907d86f3-rfb46e215e-bs3ec936c3-dc88c6c7ea-dra35420de
4. Launch
Copy k8s/olmo35-sft-4x4x8/ to a new directory. Then rename the JobSet and both ConfigMaps (in jobset.yaml and
kustomization.yaml) to a new unique name, and edit the env block:
| Variable | Meaning |
|---|---|
RUN_NAME, OUTPUT_DIR |
Checkpoints go to $OUTPUT_DIR/$RUN_NAME/checkpoints/<step>/. |
LOAD_PARAMETERS_PATH, TOKENIZER_PATH, TRAIN_FILES |
Step 2 and step 3 outputs (/mnt/post-training/... = gs://ai2-olmo-dev/post-training/...). |
TOTAL_DEVICES, GLOBAL_BATCH, PER_DEVICE_BATCH |
128 chips × 0.125 = 16 sequences per step, with no gradient accumulation. The launcher derives accumulation and fails if the numbers don't divide. |
SEQ_LEN |
65,536. Tokens per step = GLOBAL_BATCH × SEQ_LEN. |
STEPS, LR_SCHEDULE_STEPS |
Steps trained, and the horizon the LR decays over (warmup WARMUP_FRAC of it). Set STEPS below the horizon to run the start of a longer recipe, e.g. for a loss-parity check. |
NUM_EPOCH |
Passes the stream holds. STEPS decides what is trained; a pass shorter than STEPS repeats data. |
EXTRA_ARGS |
MaxText flags. The mesh is custom_mesh_and_rule=olmo35-ep-as-cp ici_fsdp_parallelism=16 ici_expert_parallelism=8: batch over FSDP, and the expert axis doubles as context parallelism outside the MoE. checkpoint_period and max_num_checkpoints_to_keep set what is saved. |
images: newTag |
The image tag from step 1. |
kubectl --context gke_ai2-olmo_us-central2-b_gke-tpu-v4 apply -k <your overlay dir>
kubectl -n olmo get workloads # ADMITTED True = it has a slice
kubectl -n olmo get jobset <name> # TERMINALSTATE, RESTARTS, SUSPENDED (a hero preemption suspends it)
kubectl -n olmo get pods -l jobset.sigs.k8s.io/jobset-name=<name>
To stop a run, deactivate its Kueue workload:
kubectl -n olmo patch $(kubectl -n olmo get workloads -o name | grep "/jobset-<name>-") --type merge -p '{"spec":{"active":false}}'.
Setting the JobSet's spec.suspend does nothing while Kueue has it admitted: Kueue resets it, and a watcher that
relied on it left a run training on the 8x8x8 for 4.5 h after its stop rule fired.
Startup takes ~35 min: the 32 pods start a few minutes after admission, then the model compiles and the first step runs. k8s/watch_jobset.sh <jobset> <run> [max minutes]
saves every pod's log when the JobSet ends. Pod logs vanish once the JobSet is deleted
or restarted, so save them before either.
5. Read the loss
Read the step loss only from the LOSS-PARTS step=N ce=… moe_lb=… z=… tokens=… line, which one pod logs. Each pod's
completed step: N, … loss: line is that pod's own value: on one step the 32 pods ranged 1.28–1.68. Compare ce=
with H045's train/CE loss, never Tunix's loss, which adds the MoE load-balancing and z-losses.
A healthy Olmo 3.5 run: REQUIRED_RBF (the MoE buffer factor that would have dropped no token) stays below the
configured ragged_buffer_factor=2.5. r2's maximum was 2.24.
REQUIRED_RBF is logged every step by JAX process 0 only, and that is often not pod slice-0-0: find the pod whose log
has it. Kubelet keeps about 700 steps of each pod's log, and the agent service account cannot read Cloud Logging. So
if a rule needs every step's value, stream that pod with kubectl logs -f from step 1. A watcher polling
kubectl logs every 5 minutes lost 14 steps at one rotation (H078 bb4-lr1e4-s0), and the run had to be relaunched.
6. Checkpoints, preemption and resume
enable_checkpointing=trueis required even if you want no checkpoints. Without it,jax.distributedis never initialised on multi-host TPU and Tunix crashes.- Tunix always saves the final step. An Olmo 3.5 checkpoint with optimizer state is ~137 GB. Agents delete nothing on GCS; ask the run's owner.
- After a preemption, JobSet restarts the run from the last checkpoint, on the same overlay. A resume replays the data pipeline to the restored step at ~1.4 s per skipped batch, which is hours late in a run. Resume only on the restore-fix image (step 1), or a derivative that includes that fix, such as the pooled image below.
scripts/tpu/maxtext/image-olmo35-fs/ adds an optional spawned worker pool for the SFT chat-template map. The
published image is:
us-docker.pkg.dev/ai2-olmo/olmo/maxtext-posttrain:51f8cc4b-p907d86f3-rfb46e215e-bs3ec936c3-dc88c6c7ea-fse2555fa0
It includes the restore fix and these MaxText fork changes:
| Image layer | Fork revisions | Changes |
|---|---|---|
bs3ec936c3 |
c469d980 → 736b296c | Big-slice mesh support, KDA sharding constraints, and optional r2 data-order emulation. |
dc88c6c7ea |
736b296c → 1e612a76 | Optional MoE dropped-token logging, including the dropless fallback correction. |
fse2555fa0 |
1e612a76 → e0e1c2a9 | Pooled chat-template map; unset or zero workers keeps the serial map. |
The bs and dc build recipes are not included here; build.sh depends on the published ...-dc88c6c7ea base.
For a late 4x4x8 resume of a run already using that base, set images[].newTag to the part after : above and
add MAXTEXT_SFT_TEMPLATE_WORKERS=32 to your recipe's environment ConfigMap (configMapGenerator[].literals in
your copied kustomization.yaml, named olmo35-sft-env in the 4x4x8 example). Here 32 is the tested worker count
per data-loading process, not a tuned optimum. The pool stays active during training as well as replay; leave it unset for fresh starts and short resumes unless measured
input-pipeline savings justify its ~4-6 min startup cost. Steady-state TPU training gains have not been measured.
Validation against the same ...-dc88c6c7ea base: 10,000 rows identical on CPU, including file and epoch
boundaries; on TPU, 2,144 of 2,144 batch hashes and steps 61-66 CE equal the serial resume. The CPU map was ~10x
faster, but the 60-step TPU replay took 366 s pooled versus 63 s serial because of startup. Long-resume TPU
throughput is still unmeasured. The checked-in examples use older images (...-rfb46e215e or ...-bs3ec936c3);
these checks do not establish parity when migrating them, since that also adds the layers above. Validate batches
and CE against the run's current image before migrating those runs.
7. Export to HF and evaluate
gcloud storage cp -r gs://ai2-olmo-dev/post-training/runs/<run>/checkpoints/<step>/model_params <local dir>
python scripts/tpu/maxtext/olmo35/export_maxtext_to_hf.py <local dir>/model_params <reference HF dir> <out>
The reference directory supplies tensor names, config.json, modeling code, tokenizer and chat template. Use an HF
export of the same model with the matching tokenizer and chat template, and choose <out> under your own output
location. One existing reference export is H045's:
/weka/oe-training-default/ai2-llm/checkpoints/abhishekr/hero-sft-4t/runs/h045-anchor-emo-olmo35-s34521/hf_step11768.
Its config.json is the HF model config, unlike the run directory's OLMo-core training config. The exporter is
bit-exact on all 23,441 tensors. For Olmo 3, use export_hf.sh, which also takes the base model's config.json,
because MaxText's own is wrong.
Evaluate the HF export with olmo-eval on Beaker GPUs, as for a GPU run (see the GPU guide).
HF Olmo 3.5 needs fla, so every pass runs on GPU. Where exports and checkpoints go, and what to keep, is the ledger's
storage skill (olmo-post-training-ledger skills/storage/SKILL.md).
Traps
Each of these trained silently on wrong data or a wrong model before it was found.
- Dropped EOS targets. Olmo tokenizers have
bos == eos, and MaxText's target shift ignored BOS targets, so SFT never learned to stop while the loss looked fine. Fixed in the image. Check P(EOS) after an answer on any new pipeline. - YaRN. Olmo 3 applies YaRN to full-attention layers only. Unpatched MaxText also applies it to sliding-window layers, which costs +0.1 nats/token at the start. Fixed in the image.
- Vocab padding. 100,278 isn't divisible by tensor parallelism. Zero-padding to 100,352 is wrong (+3.2 nats/token).
Replicate the vocab axis instead:
logical_axis_rules=[['vocab',[]],['activation_vocab',[]]]. - Olmo 3 tokenizer.
allenai/olmo-3-tokenizer-instruct-devloads asGPT2Tokenizerunder transformers 5 and drops the pre-tokenizer. Train with aPreTrainedTokenizerFastsave. - KDA miscompile on v4. Slicing a reshape inside the sub-block inverse dropped a column offset on TPU only (CPU was correct). Fixed in maxtext 9f55e7b6 (in the image). Kernel rewrites need a TPU-vs-exact-scan check, not only a CPU one.
- MoE buffer. Fully dropless runs out of HBM at 64K.
ragged_buffer_factor=2.5drops nothing as long asREQUIRED_RBFstays below it. - Restore OOM. r2's image replicates the expert Adam moments on restore and runs out of HBM. Use the restore-fix image.
- Data order depends on slice shape. Batches are packed per data host, so a different host count gives a different, still valid, data order. Keep every arm of a comparison, and its control, on the same slice shape.
- Context parallelism with load balancing gives a wrong forward. With
ici_context_parallelism> 1 and MaxText's defaultcontext_parallel_load_balance=True, dense Olmo 3 7B's step-1 CE on a fixed packed batch was off by +0.39 (cp4) and +1.98 (cp8) against an HF forward; cp4 withcontext_parallel_load_balance=Falsematches it (0.919794 vs 0.919799). The suspected cause, from the code, is that the SFT input path never applies the load-balancing reorder that attention then assumes; upstream confirmation is pending. Turn load balancing off, and check step-1LOSS-PARTSCE against an HF forward of the same batch before trusting any new mesh. The recipe'solmo35-ep-as-cpuses the expert axis rather thanici_context_parallelism; on Olmo 3.5 tiny its step-1 CE matched an HF forward of the same 64 rows (1.077855 vs 1.077850). enable_checkpointing=falsefails on TPU. MaxText initializesjax.distributedon TPU only when checkpointing is on, and Tunix builds its Orbax checkpoint manager regardless, so the run dies at trainer setup withDistributed system is not available. For a short test, keep checkpointing on with acheckpoint_periodit never reaches, setSTEPSabove the test, and deactivate the workload once the steps you need are logged: Tunix saves the full state whentrain()returns (Tunix's README), ~870 GB for Olmo 3.5 small (the pretraining hero).- Keep the first failure of a gate run. A JobSet restart deletes the failed pods and their logs. Render test
JobSets with
maxRestarts: 0(patch/spec/failurePolicy/maxRestartsto 0;k8s/instruct-sft-4x4x8/shows the patch, with value 3) and save the failing pods' logs before cleanup. - Zero steps, exit 0.
train_sftexits 0 with no steps when the data iterator fails. The launcher's step guard catches it; keep that guard. - KDA all-gathers on a new mesh. Under
olmo35-cp-epon the8x8x8, the KDA chunk scan's inputs and states were sharded on the chunk axis, so XLA all-gathered them inside every scan iteration: 97.6% of a 475 s step. Logical constraints in_delta_rule_chunked_subblock(abhishekraok/maxtext c5f1251d) bring the step to 3.33 s. CPU runs never show in-loop gathers, so profile one TPU step with xprof before trusting a new mesh. On Olmo 3.5 small, the pretraining hero (40 layers, f32 KDA state), two more gathers made the compile need 43.7 GB of HBM: the per-chunk readout gathered every chunk's states over heads and value dim, and the reshard of q/k/v from (context, expert) on the sequence gathered the full[B, T, H, dv]. Constraining the readout to the chunk axis and resharding in two steps (abhishekraok/maxtext e43a7ee3, image suffix-ks5932379d) compiles, with 1.62 GiB per chip after init. - Compile-time HBM. At 64 rows on the
8x8x8,ragged_buffer_factor=4.0failed to compile (HLO temporaries 32.17 GB vs 31.75 GB of HBM); 3.0 compiles, and the runs peaked at 2.39-2.84. shard_optimizer_over_datadoes nothing under Tunix. Small at data 8 x context 8 x expert 8 still held 12.69 GiB per chip after trainer init (1.62 GiB at fsdp 16), and its step needed 8 GB more temporaries. Shard parameters and optimizer state with fsdp instead.- An expert group of 128 chips does not compile.
olmo35-cp-epmaps experts to context x expert; at context 16 x expert 8, XLA fails withAssigned barrier is not acceptable for opcode: ragged-all-to-all. Keep context x expert at 64 or below (hero uses expert 64). - An HBM OOM at compile names its buffers only in libtpu's log. The pod log gives the total;
/tmp/tpu_logs/tpu_driver.ERRORin the container lists the largest buffers and the ops that made them. Copy it out in the launcher's EXIT trap, since it dies with the pod. - Narrowed mounts hide
scratch/. Overlays mount onlydata/andtokenizers/, so a launcher reading/mnt/post-training/scratch/...finds no file and its pods exit at startup. Add a read-only gcsfuse volume for that folder alone (only-dir=post-training/scratch/<dir>, mounted at the same path), never all ofpost-training/. - GCS egress quota. Two big slices loading the base checkpoint at once hit
google_egress_bandwidth(us-central2) and failed at startup. Stagger big-slice launches. - The control host is a single point of failure. Watchers and log tailers die with the host they run on (a Beaker session lost its node on 2026-10-16). Keep any record a rule depends on in a source that survives it.
- hostNetwork. The image's libtpu forms a multi-host slice only on the host network. Keep
hostNetwork: trueanddnsPolicy: ClusterFirstWithHostNet. pgrep -f/pkill -finsidebash -cmatch their own command line: loops never end and kills hit the caller. Use PIDs.
Changing the recipe
- Other slice sizes and meshes.
olmo35-ep-as-cpputs one 64K row on 8 chips (KDA's 8 heads). Measured on Olmo 3.5 tiny with 65,536-token rows:
| slice | mesh | rows/step | s/step | tokens/s/chip | status |
|---|---|---|---|---|---|
4x4x8 |
olmo35-ep-as-cp, fsdp 16 x expert 8 |
16 | 3.94 | 2,080 | r2 and A0'; the recipe |
8x8x8 |
olmo35-ep-as-cp, fsdp 64 x expert 8 |
64 | 4.28 | 1,910 | H078's 4x batch; ragged_buffer_factor=3.0; 1,887 tokens/s/chip without the shim, see Cost |
8x8x8 |
olmo35-cp-ep + c5f1251d |
16 | 3.33 | 620 | runs, but numerics differ from the 4x4x8 (#1933) |
8x8x16 |
olmo35-cp-ep, fsdp 64 x context 2 x expert 8 |
64 | - | - | never reached step 1 (egress quota, then a TPU core halt); shape no longer in the cluster |
Olmo 3.5 small (hero step58000, 72.6B total, 4.1B active) on the 8x8x8, 16 rows of 65,536 tokens, olmo35-cp-ep
fsdp 16 x context 4 x expert 8, remat_policy=full, image -ks5932379d: 6.5 s/step, 315 tokens/s/chip. At
ragged_buffer_factor=3.0 it dropped 1.1% of routed assignments (up to 9.2% in one layer); dropping none would need ragged_buffer_factor up to 8.5 in the worst layer.
Each mesh was picked to keep r2's per-device layout. A short sweep on the 8x8x8 on 2026-11-05 (up to 20 steps per
config, profiler off, median of steps from 3) found one faster split, but no tiny config ran reliably finite there:
| tiny, 64 rows, change from the recipe | s/step | tokens/s/chip | result |
|---|---|---|---|
| none (fsdp 64 x expert 8) | 4.24 | 1,934 | ~3.6% MFU (TFLOP/s/device ÷ 275) |
| data 8 x fsdp 8 x expert 8 | 3.93-3.95 | 2,073-2,085 | +7-8%; NaN at step 13 in one run, finite to step 20 in its repeat |
| data 4 x fsdp 16 x expert 8 | - | - | NaN at step 3 |
num_vocab_tiling=4 (from 16) |
4.15 | 1,972 | NaN at step 10 |
async collective-fusion LIBTPU_INIT_ARGS |
4.24 | 1,930 | no gain |
| 128 rows (per-device batch 0.25) | - | - | compile OOM (33.8 of 31.75 GB) |
| data 16 or 32 | - | - | OOM |
- Small's config is the only one that fits in HBM; other meshes, remat policies and vocab tiling ran out of memory. The collective-fusion flags left it at 6.47 s/step; every fourth step takes ~7.4 s.
- libtpu on v4 rejects
--xla_tpu_enable_async_collective_fusion_fuse_all_gather=true("not supported" outside Viperlite); the rest of that flag set loads. - Tiny's loss went NaN in 5 of 8 runs on this slice, between steps 3 and 13. The cause is the libtpu flag
--xla_tpu_enable_async_ragged_all_to_all=true, which the overlays set until 2026-11-06: it hands one MoE router gate's backward an unwritten buffer, giving ~1e33 gradient elements while the forward loss is unchanged, and clipping then zeroes that step's update. Without it, the failing configs ran 10 of 10 steps finite. On a4x4x8it did not fire: H161's hero and both H190 seeds (~99k steps) have no pre-cliptrain/grad_normabove 1e3 after step 4.
Speedups beyond these need code changes. Any other slice or mesh needs its own same-slice control. - Comparing against a GPU result. TPU and GPU runs of the same recipe differ in data order and numerics. Compare TPU arms with a TPU control on the same slice shape.