MoE expert cache: GPU-resident LRU cache for host-offloaded expert weights
Measured on Qwen3.8-Flash-Next UD-Q4_K_XL (512 experts, 10 routed, 28 expert layers pinned to host RAM by -ot): expert routing has strong temporal locality (LRU-64 ~67% hit rate over a mixed workload) even though the long-run expert distribution is near-uniform, so a per-layer LRU cache of expert slices in VRAM removes most of the per-token host-RAM streaming that bounds decode. Mechanism (no custom CUDA kernels): - companion tensors [ne0, ne1, K+1] per cached layer in the device buffer of that layer's router; slot K stays all-zero (dummy) - I32 id->slot tables, one device copy (get_rows remaps ids for a second mul_mat_id chain over the cache) and one host copy (src[3] of the CPU mul_mat_id, which skips cached ids and zeroes their dst rows) - the two down outputs are summed; the split is exact by construction - decode-only (n_tokens == 1); batch/prefill builds the stock graph - throttled async uploads: evictions are published at a decode-boundary sync point, slices are copied by a worker thread, and the new mapping is only published after the upload completed, so a running graph never reads a torn slot Enable with LLAMA_MOE_CACHE_SLOTS=<K> (+ LLAMA_MOE_CACHE_INSERTS, _DEBUG). Inert without the env var. Measured decode, this box (2x3090, 1 RAM channel/socket, with numactl --interleave=all): 15.5 -> 19-20 tok/s warm at K=48-64. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A7KRyLzuBiGfVn3bdczjka
This commit is contained in:
1 parent
302bd55b5f
commit
7391a178e6
12 files changed
+645
No files matched your search
@@ -1725,6 +1725,8 @@ struct llama_context_params common_context_params_to_llama(const common_params &
|
||||
cparams.n_outputs_max_per_seq = std::max(params.n_outputs_max_per_seq, 0);
|
||||
cparams.n_batch = params.n_batch;
|
||||
cparams.n_ubatch = params.n_ubatch;
|
||||
cparams.n_moe_cache_slots = params.n_moe_cache_slots;
|
||||
cparams.n_moe_cache_inserts = params.n_moe_cache_inserts;
|
||||
cparams.n_threads = params.cpuparams.n_threads;
|
||||
cparams.n_threads_batch = params.cpuparams_batch.n_threads == -1 ?
|
||||
params.cpuparams.n_threads : params.cpuparams_batch.n_threads;
|
||||
|
||||
Reference in new issue
Block a user