wmantly/llama.cpp-xrip
1
0 Fork 0
Code Issues Pull Requests Actions 31 Packages Projects Releases Wiki Activity
Files
09412af38a9fe328da2ed452a40f310eea350290
llama.cpp-xrip/src/models
T
History
Pascal 09412af38a qwen4exp: sum the indexer heads by slices (#28023) 
* qwen4exp: sum the indexer heads by slices

The head reduction went through a transpose and a sum_rows over ne[1],
which left sum_rows with ne0 = 4, one block per row for a four element
reduction, and the transpose copied the whole block by token surface
twice on the way in.

The heads are adjacent on ne[1], so each one is a strided view and the
sum is a short chain of adds.

RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, 55k context, warm
runs on top of #28011:

  prompt processing   2170 -> 2366 t/s

Generation is unaffected. The removed work scales with n_blocks by
n_tokens, so the gain grows with context and with ubatch size.

* qwen4exp: drop the redundant cont on the indexer query

rope returns a freshly allocated, contiguous tensor, so the reshape that
feeds the matmul does not need a copy. ggml_reshape_3d asserts
contiguity, so a layout that would need the cont cannot slip through
silently.

Greedy output is unchanged token for token.

Address review from @ggerganov
2026-09-01 06:23:59 +02:00
..
afmoe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
apertus.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
arcee.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
arctic.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
arwkv7.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
baichuan.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
bailingmoe2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
bailingmoe3.cpp
model : support DSpark for bailingmoe3 (#27508)
2026-08-22 12:19:48 +03:00
bailingmoe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
bert.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
bitnet.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
bloom.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
chameleon.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
chatglm.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
clip.cpp
llama: Restore quantization of mmprojs (#26818)
2026-08-10 11:58:32 +02:00
codeshell.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
cogvlm.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
cohere2.cpp
Add arch support for cohere2-MoE (#24260)
2026-06-13 19:49:00 +02:00
cohere2moe.cpp
llama : load MTP tensors only if they are really used (#26296)
2026-07-31 14:57:02 +02:00
command-r.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
dbrx.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
deci.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
deepseek2.cpp
model: use ggml_rope_set_offset() (#27382)
2026-08-21 18:54:29 +02:00
deepseek2ocr.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
deepseek4.cpp
DeepseekV4: fix rollback with multi-seq (#26756)
2026-08-23 13:57:49 +03:00
deepseek32.cpp
models : remove duplicate metadata load (#27378)
2026-08-19 16:05:50 +02:00
deepseek.cpp
model : NvFP4 quantized LM head support (#23046)
2026-05-16 11:09:27 +02:00
delta-net-base.cpp
llama: refactor fused ops (#24646)
2026-07-08 18:18:09 +08:00
dflash.cpp
spec : fuse the DFlash encoder into the KV cache injection (#27310)
2026-08-31 11:19:20 +02:00
dots1.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
dots3note.cpp
model: add dots3-note (#27060)
2026-08-21 19:52:34 +02:00
dream.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
eagle3.cpp
spec: add eagle3-v3 support for gpt-oss model (#25794)
2026-07-28 09:58:16 +03:00
ernie4-5-moe.cpp
model : NvFP4 quantized LM head support (#23046)
2026-05-16 11:09:27 +02:00
ernie4-5.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
eurobert.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
exaone4.cpp
model : fix SWA not being enabled for EXAONE 4.5 (#26848)
2026-08-11 07:20:17 +03:00
exaone-moe.cpp
minor : fix lint issues (#24165)
2026-06-05 11:17:54 +03:00
exaone.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
falcon-h1.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
falcon.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
gemma2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
gemma3.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
gemma3n.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
gemma4-assistant.cpp
mtp: support for gemma-4 E2B and E4B assistants (#24282)
2026-06-08 13:48:52 -07:00
gemma4.cpp
llama: model_loader: add TENSOR_READ_LAZY (#27794)
2026-08-27 15:14:34 +02:00
gemma-embedding.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
gemma.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
glm4-moe.cpp
model : support MTP in GLM-4.5-Air (#26534)
2026-08-23 21:20:44 +03:00
glm4.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
glm-dsa.cpp
models : remove duplicate metadata load (#27378)
2026-08-19 16:05:50 +02:00
gpt2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
gptneox.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
granite-hybrid.cpp
model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
2026-08-19 16:53:31 +02:00
granite-moe.cpp
model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
2026-08-19 16:53:31 +02:00
granite-swa.cpp
model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
2026-08-19 16:53:31 +02:00
granite-switch.cpp
model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
2026-08-19 16:53:31 +02:00
granite.cpp
model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
2026-08-19 16:53:31 +02:00
grok.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
grovemoe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
hunyuan-dense.cpp
model: move load_hparams and load_tensors to per-model definition (#22004)
2026-05-04 12:36:59 +02:00
hunyuan-moe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
hunyuan-vl.cpp
model : NvFP4 quantized LM head support (#23046)
2026-05-16 11:09:27 +02:00
hy-v3.cpp
llama : load MTP tensors only if they are really used (#26296)
2026-07-31 14:57:02 +02:00
internlm2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
jais2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
jais.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
jamba.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
jina-bert-v2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
jina-bert-v3.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
kimi-k3.cpp
model: add Kimi-K3 text model (#26185)
2026-08-15 17:11:05 +02:00
kimi-linear.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
laguna.cpp
model: Add Laguna-S-2.1 LLM_TYPE (#26233)
2026-07-28 21:02:33 +02:00
lfm2.cpp
model : support DSpark for LFM2 models (#27383)
2026-08-20 16:36:57 +02:00
lfm2moe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
llada-moe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
llada.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
llama4.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
llama-embed.cpp
model: move load_hparams and load_tensors to per-model definition (#22004)
2026-05-04 12:36:59 +02:00
llama.cpp
spec: add EAGLE3 speculative decoding support (#18039)
2026-06-12 10:21:06 +03:00
maincoder.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
mamba2.cpp
mamba2: remove hardcoded 2x expansion factor and invalid d_inner % d_state check (#23082)
2026-06-26 08:50:54 +03:00
mamba-base.cpp
mamba2 : Flatten in/out projections to dispatch GEMM instead of GEMV (#27513)
2026-08-24 09:25:11 +03:00
mamba.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
mellum.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
mimo2.cpp
model : load MiMo V2 MTP tensors only if used (#26412)
2026-08-02 09:03:05 +02:00
minicpm3.cpp
model: use ggml_rope_set_offset() (#27382)
2026-08-21 18:54:29 +02:00
minicpm.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
minimax-01.cpp
tests : run test-save-load-state across all architectures (#27755)
2026-08-28 09:45:19 +03:00
minimax-m2.cpp
spec: add Minimax2 eagle3 support
2026-07-13 15:22:37 +02:00
minimax-m3.cpp
mtmd, common: various fixes (#27071)
2026-08-14 23:34:56 +02:00
mistral3.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
mistral4.cpp
model: move load_hparams and load_tensors to per-model definition (#22004)
2026-05-04 12:36:59 +02:00
models.h
model: qwen4exp: reduce number of graph splits (#27880)
2026-08-28 19:24:46 +02:00
modern-bert.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
mpt.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
muse-glimmer.cpp
model: Muse Glimmer Support (#26841)
2026-08-10 13:07:27 +02:00
nanbeige.cpp
models : support nanbeige4.2-3B (#27730)
2026-08-27 07:55:31 +03:00
nemotron-h-moe.cpp
model: add MTP support for Nemotron model (#26725)
2026-08-10 11:25:24 +03:00
nemotron-h.cpp
Dflash support for nemotron-3.5 (#26905)
2026-08-11 18:46:26 +05:30
nemotron.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
neo-bert.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
nomic-bert-moe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
nomic-bert.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
olmo2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
olmo.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
olmoe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
openai-moe.cpp
spec: add eagle3-v3 support for gpt-oss model (#25794)
2026-07-28 09:58:16 +03:00
openelm.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
orion.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
paddleocr.cpp
model : NvFP4 quantized LM head support (#23046)
2026-05-16 11:09:27 +02:00
pangu-embed.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
phi2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
phi3.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
phimoe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
plamo2.cpp
ggml : recurrent state rollback for ggml_ssm_scan (#26623)
2026-08-14 17:20:40 +03:00
plamo3.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
plamo.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
plm.cpp
model: use ggml_rope_set_offset() (#27382)
2026-08-21 18:54:29 +02:00
pockettts.cpp
mtmd: support pocket-tts (#26871)
2026-08-11 14:18:30 +02:00
qwen2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
qwen2moe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
qwen2vl.cpp
model : NvFP4 quantized LM head support (#23046)
2026-05-16 11:09:27 +02:00
qwen3.cpp
spec: add EAGLE3 speculative decoding support (#18039)
2026-06-12 10:21:06 +03:00
qwen3moe.cpp
spec: add EAGLE3 speculative decoding support (#18039)
2026-06-12 10:21:06 +03:00
qwen3next.cpp
model: MTP support for Qwen3-Next (#25589)
2026-08-03 17:15:01 +09:00
qwen3tts.cpp
mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary) (#26254)
2026-08-04 17:26:15 +02:00
qwen3vl.cpp
mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary) (#26254)
2026-08-04 17:26:15 +02:00
qwen3vlmoe.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
qwen4exp.cpp
qwen4exp: sum the indexer heads by slices (#28023)
2026-09-01 06:23:59 +02:00
qwen35.cpp
llama : load MTP tensors only if they are really used (#26296)
2026-07-31 14:57:02 +02:00
qwen35moe.cpp
llama : load MTP tensors only if they are really used (#26296)
2026-07-31 14:57:02 +02:00
qwen.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
refact.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
rnd1.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
rwkv6-base.cpp
models : deduplicate delta-net graphs for Qwen family (#19597)
2026-02-16 14:35:04 +02:00
rwkv6.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
rwkv6qwen2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
rwkv7-base.cpp
models : deduplicate delta-net graphs for Qwen family (#19597)
2026-02-16 14:35:04 +02:00
rwkv7.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
seed-oss.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
smallthinker.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
smollm3.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
stablelm.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
starcoder2.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
starcoder.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
step35.cpp
llama : load MTP tensors only if they are really used (#26296)
2026-07-31 14:57:02 +02:00
t5.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
t5encoder.cpp
model: move load_hparams and load_tensors to per-model definition (#22004)
2026-05-04 12:36:59 +02:00
talkie.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00
wavtokenizer-dec.cpp
model : NvFP4 quantized LM head support (#23046)
2026-05-16 11:09:27 +02:00
xverse.cpp
hparams : refactor hparams.n_layer (#24060)
2026-06-05 11:09:36 +03:00