Measured on Qwen3.8-Flash-Next UD-Q4_K_XL (512 experts, 10 routed, 28 expert
layers pinned to host RAM by -ot): expert routing has strong temporal locality
(LRU-64 ~67% hit rate over a mixed workload) even though the long-run expert
distribution is near-uniform, so a per-layer LRU cache of expert slices in VRAM
removes most of the per-token host-RAM streaming that bounds decode.
Mechanism (no custom CUDA kernels):
- companion tensors [ne0, ne1, K+1] per cached layer in the device buffer of
that layer's router; slot K stays all-zero (dummy)
- I32 id->slot tables, one device copy (get_rows remaps ids for a second
mul_mat_id chain over the cache) and one host copy (src[3] of the CPU
mul_mat_id, which skips cached ids and zeroes their dst rows)
- the two down outputs are summed; the split is exact by construction
- decode-only (n_tokens == 1); batch/prefill builds the stock graph
- throttled async uploads: evictions are published at a decode-boundary sync
point, slices are copied by a worker thread, and the new mapping is only
published after the upload completed, so a running graph never reads a torn
slot
Enable with LLAMA_MOE_CACHE_SLOTS=<K> (+ LLAMA_MOE_CACHE_INSERTS, _DEBUG).
Inert without the env var.
Measured decode, this box (2x3090, 1 RAM channel/socket, with numactl
--interleave=all): 15.5 -> 19-20 tok/s warm at K=48-64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A7KRyLzuBiGfVn3bdczjka
csantiago78andClaude Fable 5
7391a178e6