Skip to content

Compiler-cache lifecycle

Ordinary MILES/Core runs persist Triton artifacts only. Each worker compiles into a private local directory; verified, immutable generations of those directories are restored from and published to shared WEKA storage. Persistence is an optimization: a miss compiles locally, and a publication failure does not fail completed training. CUDA graphs are captured per process and are not saved.

The implementation lives in the pinned MILES fork under miles.utils.compiler_cache. Open Instruct supplies the run configuration and application source root, then calls the MILES lifecycle from its driver.

Configuration

The cache is enabled by default. Structured run files use the [compiler_cache] section; low-level files use the matching core.compiler_cache* fields.

[compiler_cache]
enabled = true                       # core.compiler_cache; false opts out
# shared_root = "/weka/oe-training-default/open-instruct-compiler-cache/tmp-7d"
restore = true                       # false gives a cold start but still publishes
diagnostics = false                  # opt-in cache/compilation diagnostics
tilelang = false                     # experimental TileLang persistence
max_storage_bytes = 8_589_934_592    # 8 GiB per cache key; 0 disables uploads
publish_interval_seconds = 600       # minimum time between publication checks

The default shared root is /weka/oe-training-default/open-instruct-compiler-cache/tmp-7d. A custom root must be absolute and, on WEKA, contain an exact tmp-N[hdwmy] expiry component (the WEKA TTL naming convention, for example tmp-7d); this is validated before launch. Expired artifacts are simply cold misses, so no cleanup job is needed. Without the default WEKA mount, persistence is skipped and logged. Limits and intervals are operational and do not change the cache key.

Compiler-cache settings (the [compiler_cache] table and the core.compiler_cache* options) are not part of a run's resume identity: a resumed run under the same output.root may change the shared root, disable restore or persistence, or enable TileLang without "Run configuration changed". Every other field must still match the recorded specification.

What is cached

Each trainer rank and TP1 serving engine restores into a new /tmp/core-triton-* directory before actor imports; SGLang's spawned children inherit their engine's cache. Serving engines with TP greater than one, driver warmups/preflights and other compiler families (FA4/CuTe DSL, TorchInductor, FlashInfer, DeepEP/DeepGEMM) are not persisted by the training lifecycle.

Triton uses a core-triton-v3 compatibility envelope. It includes the binary base, actual GPU/toolchain, installed Triton compiler source/binaries, native source/extensions in the application and runtime packages, model configuration, parallel geometry and compile-affecting settings/environment. Trainer ranks remain separate; equivalent TP1 serving slots share a role key.

Full application/MILES source hashes and the runtime lock remain in worker receipts as provenance, but a Python-only orchestration edit no longer discards all kernel artifacts. Triton's own kernel keys distinguish source/dependency, signature, specialization and compiler-option changes. Known non-kernel settings including collection batch size, policy lag, admission, prepared-batch overlap, layer profiling and truncated-response reward gating do not change the envelope; unknown settings remain conservative. This compatibility policy applies only to Triton, not to other compiler families.

Legacy v2 archives lack the new compiler/native identity evidence and are not automatically adopted. The first v3 run can be cold; later compatible Python revisions reuse its artifacts. Rebuilding an application image alone does not invalidate the cache: the pinned binary base and compatibility inputs matter.

Triton publication saves independently completed kernel groups while skipping incomplete or concurrently changing groups. Receipts report skipped_live_groups; those groups can be saved on a later interval. A busy compiler therefore need not invalidate the whole snapshot. Manifest, archive and checksum validation remain required.

Experimental TileLang persistence

compiler_cache.tilelang = true (mapped to core.compiler_cache_tilelang) additionally persists TileLang in a separate core-tilelang-v1 envelope. This remains opt-in; qualify reuse on the intended model, hardware and topology. The pinned FLA KDA backward path uses TileLang, so a Triton hit alone cannot eliminate that compilation. The key includes the installed TileLang Python compiler, bundled TVM and native libraries as well as the device/toolchain and compile settings. The native-library stamp is forced on for this mode. TileLang's own TIR/target keys select each kernel; dataset identity is not a compiler input.

Each worker gets a private TileLang directory and separate compiler scratch. Publication rejects active staging exports and snapshots that change during copying; it never publishes a partially exported kernel. Restore verifies the archive before loading executable artifacts. Each TileLang key has its own storage cap. Receipts include a separate tilelang restore/publication result. With tilelang = true, diagnostics = true also enables TileLang's native JIT phase logs.

Publication

After the first completed training collection in each process, the driver publishes in the background. Later completed collections trigger a check only after publish_interval_seconds has elapsed; this is wall time, not a collection count, and does not depend on checkpoint saving or HF export. Workers whose files are unchanged since the last publication skip the upload. After successful shutdown, a final publication captures remaining artifacts and removes the private copies. Failed training performs no final publication, but generations published earlier remain reusable.

Publication is best-effort. Archives are built on local disk and only the finished archive and manifest are uploaded before atomically advancing CURRENT; conflicting bytes are never overwritten. In the v3 Triton scope, a collision rejects the entire incoming kernel group while preserving the published group; unrelated new groups can still be saved. Receipts list rejected_incoming_groups. Rounds run at most two tasks per node within one shared 240-second wait budget, and expired tasks are cancelled. Hashing and compression still use CPU, local disk and shared-storage bandwidth. Restore validates identity, archive and file checksums before installing; corruption is a recorded rejection and cold miss.

Storage bound

max_storage_bytes limits each key directory, SHARED/core-v1/FINGERPRINT/, counting every retained generation, manifest and pointer across runs. It is not an aggregate limit for an experiment or the shared root, since different keys have separate limits. When the next generation would exceed the limit, that worker reports storage_limit and stops publishing for the rest of the run; existing generations are left intact and restores continue. Lowering the limit does not delete stored data, and the cap does not replace expiry through the root's tmp-N component.

Reading the results

Startup logs one summary per worker with its fingerprint, restore status and reason, and restored file count. A miss reports no_published_generation; a corrupt or incompatible archive reports rejected with the reason. This is an archive-level result, not a kernel hit rate, and an archive hit alone does not show that a restart avoided all compilation.

compiler-cache.json in the run's metric directory records each worker's cache outcome, and compiler-cache-progress.json records publication rounds. Per-round logs report lock wait, merge, compression, upload and pointer phases with file counts, bytes and timings. Startup timings are in driver_timing.jsonl and startup_rankN.jsonl; nested intervals overlap and must not be summed.

With diagnostics = true, a hook on Triton's FileCacheManager.get_group and put appends events to local activity-PID.jsonl files, which publication aggregates into group_hit, group_miss and put counts. These count disk-cache operations, not compilations or GPU launches. A supported Triton compilation listener also records compile_hit / compile_miss, kernel name, compiler microseconds and lowering-stage timings. CUDA graph capture, module loading and other compiler families are outside those measurements. The hook writes one local file append per operation, so leave it off for ordinary runs.

The separate python -m miles.utils.compiler_cache.run command is an experimental single-node wrapper for controlled probes of additional compiler families. Do not wrap an existing remote Ray cluster with it.

With diagnostics enabled, worker receipts also contain compatibility_explanation: changed identity field names, the previous observed fingerprint, and whether source provenance changed. It compares one bounded previous-startup record per model/worker; it does not select a cache or imply that the previous run published successfully. A same-key miss can still mean no published generation, expiry or rejected publication; the restore/publication receipts distinguish these cases. Diagnostics failures do not block restoration.

The runtime's tools/probe_triton_cache.py runs a bounded single-GPU cold/warm probe in fresh processes, including numerical checks, changed kernel/dependency and shape cases, KDA training kernels, SGLang expert inference and experimental FA4 persistence. It does not download a model or qualify distributed training.

For a small end-to-end trainer qualification inside the pinned image, add tests/miles to PYTHONPATH and run python tests/miles/cache_startup_probe.py --output /output/trainer-cache-probe. It creates a tiny random hybrid-MoE fixture, runs one production Core optimizer step in each of two fresh EP1 processes, and compares scores, gradients, optimizer state and final parameters exactly. The warm phase must restore an archive and perform no Triton compilation. This uses the fixture's Torch attention backend; the separate kernel probe covers FA4. Neither probe downloads model weights or reads the production cache.

The pinned Core KDA setup also removes FLA 0.5.2's unused NB convolution compiler specialization. NB still selects the autotuning bucket and all candidate launch configurations remain available; changing the bucket alone no longer compiles identical candidates again. A guarded source adapter requires NB to remain unused in both kernel bodies. The runtime's tools/probe_fla_conv_cache.py checks numerical identity and restored reuse across sequence lengths and convolution options.