* vocab : implement PLaMo-3 tokenizer pre-segmentation
The PLaMo-3 tokenizer inserts hard boundaries before running the Unigram
DP, around <|plamo:...|>-looking text, and around runs of at least 4
identical characters or 2 spaces. Without them llama.cpp tokenizes code
indentation and repeated punctuation differently from the reference.
Reproduce the two re.sub() passes in llm_tokenizer_plamo2 by encoding
each segment independently.
* add vocab type "plamo3"
* Update src/llama-vocab.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* misc change
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: K2 Horizon gguf conversion code
* model: loading hparams and tensors in k2-horizon.cpp
* model: K2 Horizon compute graph
* model: K2 Horizon compute graph adjustment and registering tokenizers
* model: K2 Horizon chat template and accomodate safetensors naming
* unicode : add the K2-Horizon pre-tokenizer splitter
The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.
On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.
Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal and alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.
The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.
tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.
* tests: expand K2 Horizon unicode splitter coverage
* unicode: handle K2 Horizon case folding and empty input
Assisted-by: Codex
* jinja : support sequence indices in selectattr and rejectattr
Assisted-by: Codex
* model : add K2 Horizon dense and MoVA support
Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.
Assisted-by: Codex
* chat : support K2 Horizon reasoning and tool calls
Assisted-by: Codex
* conversion: remove obsolete K2 Aurora alias
Assisted-by: Codex
* k2-horizon: enforce response schemas and load YaRN betas
Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.
Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.
Assisted-by: Codex
* renaming template fixture
* adressing cisc follows ups
* desloppify the parser / adress aldehir comments
* clean test-chat
* remove fallback : model trained mostly on high anyway
* fix k2 attn_v_exp tn splitting and metal fusion baseline
* k2-horizon : forward expand views before sums
* k2-horizon: copy embds before group norm to fix TP
* disable tesnor parallelism
---------
Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
The gather path attended over the selected latents with a plain
matmul and softmax. It only ran with n_ubatch <= 16, and the flash
attention backends now skip the masked rows through n_kv_max, so
the scatter path covers every case.
Drop the gather flag, the gathered attention branch and
gather_mla_rows. set_input_kpool always maps padding to the n_kv
sentinel, and the slot mask becomes sel_mask since only the
scatter reads it.
* ggml: fix CLAMP on non-contiguous views (CPU, CUDA)
CUDA clamped ggml_nelements values flat and ignored the view strides.
CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the
strides of dims 1..3; CUDA supports_op requires contiguous rows, like
Metal. test_clamp gains a non-contiguous view case.
* cuda: clamp kernel uses fastdiv for the view strides
A 1.0 loader (e.g. Android 8.1) has no vkEnumerateInstanceVersion, so
backend init called a null pointer. Treat it like any loader under 1.2.
Fixes#29871.
Assisted-by: Claude Opus 5.5
* mimo2 : always emit h_nextn
the other nextn-capable models set it unconditionally
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* models : consolidate nextn row cropping into shared helpers
- replace the duplicated crop conditions and the per-model flags (narrow_early,
crop_before_ffn, crop_last_layer, emit_h_nextn) with two helpers on llm_graph_context:
crop_before_nextn() / crop_after_nextn()
- models that only tested embeddings_nextn_masked now share the same condition, so they
crop the last layer before the nextn capture whenever extraction is off
- t_h_nextn is now set unconditionally in mimo2, qwen4exp and deepseek4 (as in the other
nextn-capable models); host-side reads stay gated by cparams.embeddings_nextn
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
The speculative MTP init enables NextN extraction on the target and draft
contexts after both were created and their schedulers reserved. With
unmasked extraction the trunk graph keeps every token through the last
layer instead of cropping to the output rows, so the first decode
reallocates to that batch's shape and the next, wider batch trips
GGML_SCHED_DEBUG_REALLOC. Invalidate the reserve when the flags change so
the next compute re-reserves with the new graph shape.
Assisted-by: Claude
* ggml: refactor selective expert copying to user code
* tests: enroll two models into selective expert copy test
* tests: use deepseek2 as test model
* improve comment in ggml-backend.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont: fix whitespace
* cont : better comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml-openvino: skip unselected graph branches and support DUP
Upstream #29622 adds a mixed token/embd branch to every input
embedding graph through ggml_build_forward_select(). Its nodes are
not flagged for compute, but the backend translated them anyway,
and the DUP in that branch was unsupported, so the scheduler split
the graph and passed the embeddings across the split with a fixed
token count. The first single-token decode then failed
(test-thread-safety on CPU and GPU).
Build the OV model from the compute nodes only, and translate a
same-type contiguous DUP like CONT so the graph stays on one backend.
* ggml-openvino: make inp_scale_rows token dim dynamic
#29622 also moves the per-token embedding scale (gemma3, gemma3n,
gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim
and pad it per chunk on the static (NPU) path.
* ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights
Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU
plugin fails to compile that form for some row counts with "clFinish,
error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on
the MUL_MAT cases added in #29869 (e.g. m=1000, n=2, k=1024). Model
weights use a u4 zero point and are not affected.
Report these cases as unsupported on GPU until the plugin is fixed.
Op tests check support before allocating, so the check matches unbound
weights only; model loading probes with a dummy buffer and keeps its
weights on the GPU.
* ggml-openvino: create FILL in the output type
translate_fill always built an f32 constant, so an f16 FILL produced
f32 data and the copy back overran the f16 output buffer. Use the
output type for the constant.
* ggml-openvino: reject CONCAT with a quantized type
Quantized inputs are dequantized when translated, so the backend cannot
write a quantized CONCAT output. Report it as unsupported, as for CPY
to a quantized type.
* ggml-openvino: handle the single recurrent state gather of build_rs
#29856 changed build_rs to gather all recurrent states with one GET_ROWS
on the s_copy leaf and take the ubatch and extra states as views of it.
The stateful path matched only the previous form, a GET_ROWS per view of
s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU
("is_axis_valid(axis, r)" in a Concat).
For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the
active-state gather, keep the rank-4 layout of reshapes that read a view
of it, and map the copy of the empty extra-state view to the single-slot
remainder writeback. Do not warn about the dynamic dim of empty views.
* openvino: align eltwise operand ranks to work around a GPU-plugin defect
* openvino: match the MoE fusion on the rank-3 stateful graph
* ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks
The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract
whose operand ranks differ. In gemma-3 the lower-rank operand of the
post-attention residual add is the norm output, and unsqueezing it makes
the GPU plugin compute the layer wrongly: gemma-3 returns empty answers
on GPU with stateful execution.
Skip the rewrite when the lower-rank operand is an RMS norm output.
* docs : update OpenVINO validated models
---------
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
* llama: re-pool each shared k-pool rep once
With shared cells every pool is re-pooled, and since the pooled keys
are always scattered, the pools a seq_cp shares between sequences
wrote the same rep row from several scatter entries, a data race on
the CPU backend. Mark each rep once: the sharing sequences read the
same row through pool_cells.
* llama: assert whole-sequence seq_cp in the hybrid idx memory
The recurrent state is always copied whole whatever the range, and a
k-pool cell shared by a partial copy could carry two pool groupings
with a single pooled row. Every caller copies whole sequences, so
reject partial ranges instead of supporting them.
* llama: drop the k-pool cache_safe mode
With whole-sequence seq_cp, sequences sharing cells share their pools
too, so the pooled row of a shared rep is valid for all of them. Mark
each rep once in every ubatch instead of re-pooling everything while
cells are shared, which removes the sharing scan and the stale-all
workarounds in seq_rm, state_read and state_drop. seq_cp now only
stales the destination.
* hexagon: head-parallel flash_attn partitioning for row-split multicore
In row-split mode each core computes its output row shard of every
MUL_MAT, but flash_attn was previously partitioning by Q tokens
(flat qrow split) instead of by heads. This forced every core to
read the full KV cache (all n_kv_heads), negating the memory
bandwidth benefit of multicore on flash_attn.
Change both HMX and HVX flash_attn kernels to partition by KV heads
when n_kv_heads is divisible by n_cores: core i processes heads
[i*n_kv_heads/N, (i+1)*n_kv_heads/N) exclusively, reading only its
head shard of the KV cache. Falls back to the original token-block
split when n_kv_heads % n_cores != 0 (e.g. Gemma-4 with 2 KV heads
on 4 cores).
Controlled by GGML_HEXAGON_FA_HEAD_SPLIT (default 1 = on).
The flag is packed into bit 1 of the existing is_dst_fp32 kparams
byte to stay within the 128-byte kernel_params blob limit.
Measured gains at 4c row-split (PP t/s, ubatch=1024):
Qwen3-0.6B: 6977 -> 11026 (+58%)
llama-3.2-3B: 3717 -> 5522 (+49%)
Qwen3.5-4B: 2739 -> 2855 (+4%)
Gemma-4 MoE: no change (MoE FFN dominates, fallback path)
TG is unchanged (flash_attn is a small fraction of decode time
relative to the matmul+barrier cost per layer).
* hex-fa: cleanup kern_params and head-split selection
* hex-fa: add -fa-head-split option to run.py
* hex-mdev: update matmul solver to account for reduced work in row-split scenarios
* hex-mmid: better work splitting by expers in multi-dev scenarios
* hex-fa: update HMX gating based on the model/n-hvx/ctx-len sweep
* hex-fa: precompute softcap/scale on the host
* hexagon: flatten matmul into 2d to use HMX in multi-sequence
* hex-mm: cleanup kparams and use collapse to 3/4D -> 2D mapping
* hex-mm: fix typo in collapse fallback
* hex-mm: another pass at consistent naming for act tensors
* hex-mm: add support for colapsing dims in fused matmuls
* hex-build: fix WoS build errors
* hex-mm: make sure to enforce dst stride in can_collapse
* hex-fa: add a onliner commit for head-split check
* hex-fa: remove unused local head_split var
* hex-fa: tighten up can_split checks
* hex-mm: update unfused paths to use act instead src1
* hex-mm: make sure to check all dsts for splitting
* hexagon: fix the second weight chunk address in the batched HMX matmul prologue
* hexagon: F16 activation and ragged N in the HMX matmul
* hex-mm: tighten the ragged/split checks in mdev cases
* hex-mm: enable MM fusion for F16 activations
* hex-mm: pass tiled sizes to the solver in fused paths
* hex-mmid: remove scalar divs from expert mapping loops
* hex-mmid: proper cacheline safety enforcement for mdev splits
* hex-mm: improve solver for mdev split scanarios and tail handling
* hex-mm: remove redundant checks
* hex-mm: fix fused HMX MUL_MAT_NX drops the final partial tile for quantized weights
* hex-mm: better handling of ragged shapes (removes scalar memset of vtcm)
---------
Co-authored-by: ebateni <ebateni@qti.qualcomm.com>
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* cuda: stage the lightning indexer queries in head passes for MUSA
MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile
kernel staged the queries of all four heads next to the key tile for
33 KB. The queries are now staged in passes of
LIGHTNING_INDEXER_TILE_HEADS_PER_PASS heads: two on MUSA for 25 KB,
four elsewhere where the single pass folds to the previous kernel.
* cuda: use the vector lightning indexer kernel on MUSA
Address review from am17an: the tile kernel stays off MUSA, whose archs
21 and 22 cap static shared memory at 28 KB, below the 33 KB the tile
needs, so MUSA keeps the vector kernel it ran before. This replaces the
head passes, CUDA and ROCm run the merged kernel unchanged.
* server: support vision input for Clef
* move input_attn_causal to private
* extend old server_batch::embd
* server_batch::token::pos to multi dim
* nits
* fix abort
* fix img tokens cap
* fix yield_to_queue mutate data
* server: reject partial media truncation
* server: keep only the keep_first fix
Drop the mtmd test helper change, which no longer builds since
clip_image_f32_batch stores its entries by value, and drop the
vision test: no test fixture reaches a cut between two adjacent
media chunks with a reused cache (tinygemma3 uses SWA and wraps
images in text tokens, tinyopenjev and small-test are recurrent),
so the test passed or failed independently of the fix.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan: sparse flash attention for quantized K/V
Assisted-by: Claude
* vulkan: single-scan sparse FA index compaction
The compaction ran one workgroup per mask row and walked the row in
BLOCK_SIZE chunks, with a workgroup scan per chunk. For decode that is
one workgroup doing KV/1024 barrier-bound iterations, so at 128k cells
it cost more than the sparse attention it feeds.
Split the row into contiguous segments instead: one per subgroup with
ballot counting over coalesced loads, or one per thread without
subgroups. A single scan over the segment counts then gives each
segment its output offset. The index list stays ascending.
* llama : fix unexpected graph reallocation in the k-pool models
Both k-pool models built a graph shape that depends on state the
full-context reserve cannot know:
- qwen4exp branched on inp->cache_safe, which turns false as soon as
llama_memory_seq_cp shares cells (e.g. batched-bench -pps): the QSA
layers swapped scatter+gather for fill+concat and dropped the
new_pool_rep leaf, so the decode graph had 12 fewer nodes than the
reserved one
- glm5-next branched on gather = n_tokens <= 16 && n_kv > n_sel, so the
TG decode built the gather shape (7564 nodes) while the last reserve,
the PP one, had the dense shape (7762 nodes)
Either mismatch forces a decode-time re-reserve that drops the
worst-case sizing and bakes in the current state, so the next state
growth (n_pool, n_kv, n_new) needs more room at an unchanged graph size
and aborts under GGML_SCHED_DEBUG_REALLOC=1. Reproduce with, e.g.:
GGML_SCHED_DEBUG_REALLOC=1 ./bin/llama-batched-bench \
-hf ggml-org/GLM-5.3-Flash-GGUF:Q2_K -npp 2500 -ntg 32 -npl 1,2 \
-c 32768 -pps -kvu
Always scatter+gather the pooled keys, and pick gather from context
constants only: n_ubatch bounds every ubatch, top_k + kpool - 1 bounds
n_sel. Every graph of a context then shares one shape, which the
reserve covers, and the dense path measured faster than the gather path
at 2.5k and 16k context.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* llama : drop the unused k-pool cache_safe graph API
The k-pool graphs no longer branch on cache_safe, so nothing reads
get_kpool_cache_safe() or the conditional new_pool_rep any more: both
models always pass the scatter target, which set_input_kpool now
requires instead of merely preferring.
Also drop the cache_safe copy in kpool_build_sizes(), a sizes-only
helper. The layout and state flag itself stays, it still decides which
pools a layout with shared cells must re-pool.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* tests : add a shared-seq graph reserve regression test
Decode a prompt into seq 0, share its cells with seq 1 via
llama_memory_seq_cp (what llama-batched-bench does for -pps), then keep
decoding both sequences. For the k-pool models sharing clears
cache_safe, which changes the graph topology while the pools keep
growing, so a scheduler that re-reserves with the current state
instead of the worst-case one aborts under GGML_SCHED_DEBUG_REALLOC=1.
The test registration sets that flag, and the test aborts on both
k-pool models before 2220411ec1.
kimi-linear and minimax-01 are skipped: they reserve the final pp graph
with n_seqs = 1 (see [TAG_RESERVE_DIAG_DECAY] in llama-context.cpp), so
every multi-seq graph has a different layout and re-reserves by design.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : add TODOs
* cont : fix comment
* cuda: match the moe weighted reduction on empty ubatches
ggml_cuda_match_moe_weighted_reduction rejected tensors with zero
rows. A ubatch without outputs shrinks the last layer to zero rows
through inp_out_ids, so graph_optimize dropped its alloc dep there and
the scheduler graph lost one node compared to the reserved one. The
scheduler then re-reserved at the size of that ubatch, and the next
ubatch with the same node count but larger tensors aborted under
GGML_SCHED_DEBUG_REALLOC=1.
The compute loop already skips empty nodes before trying any fusion,
so the guard only made the alloc deps depend on the row count.
* tests: build the rollback test only where internal symbols link
The shared-seq case calls llm_arch_from_string, which libllama does
not export through LLAMA_API, so linking test-recurrent-state-rollback
fails on Windows with shared libraries. Its build now sits in the
NOT WIN32 OR NOT BUILD_SHARED_LIBS block, next to test-llama-archs and
the test registration it already lives under.
* tests: skip archs by name in the shared-seq reserve test
The skip of kimi-linear and minimax-01 went through llm_arch_from_string,
which libllama does not export through LLAMA_API, so the test could not
link on Windows with shared libraries. It now compares the
general.architecture string directly, and the test builds on every
platform again.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* kv-cache: save exact KV rotation metadata, reject restoring mismatched rotation
* tests : move the state rotation test to test-save-load-state
the test is now part of the save/load test matrix and runs against
every model under test, like the rest of the suite
it probes the KV cache type combinations supported by the model and
treats models that do not use attention rotation as passing vacuously
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : skip unsupported KV caches
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60
On the SpacemiT X60, IME matrix acceleration only covered Q4_0/Q4_1/Q4_K.
Q8_0 had no IME1 kernel, and since the SpacemiT build sets
GGML_CPU_REPACK=OFF there was no repack path compiled in either, so Q8_0
had no accelerated path at all and ran roughly ten times slower than
Q4_0 for prefill on the same board.
- add make_block_q8_0x16 and the Q8_0 repack entry: interleave the
weights into the 16-column layout the IME1 vmadot sequence expects
- add ime1::gemm_kernel_i8i8, an int8 x int8 IME1 kernel with a
single-row and a 4-row A path; the 4-row path loads each B panel once
and reuses it across 4 rows of A
- add quantize_a_4row_i8 for the 4-row activation quantization
- wire both into forward_mul_mat and the repack factory for Q8_0
- docs: mark Q8_0 as supported on X60
Correctness was checked against a quant-exact integer reference for
K = 32 up to 4096, with a max relative error of about 1e-6, and by
checking that generation stays coherent across several prompts.
Tested on Milk-V Jupiter (SpacemiT X60), Bianbu 2.1.1, gcc 14.2, with
Qwen2.5-0.5B-Instruct Q8_0. llama-bench -t 4 under taskset -c 0-3, 5
repetitions on an idle board: pp128 goes from 10.70 to 93.87 t/s. Q4_0
is unchanged at 106.40 -> 107.51 t/s, as expected since this does not
touch that path.
* ggml-cpu : move q8_0_16x32 decl to IME1 section
* ggml-cpu : align q8_0 IME1 kernel assignments
Toki NasinandSigbjørn Skjæret
Pascal
Xuan-Son Nguyen
lhez
Aman GuptaandGeorgi Gerganov
thelittlefiremanandJohannes Gäßler
Foad Abo Dahood
Ian McKellar
Daniel Bevenius
Kartik GuliaandSigbjørn Skjæret
Piotr Wilkin (ilintar)
Ihar Hrachyshka
Alessandro de Oliveira Faria (A.K.A.CABELO)
Ravi PanchumarthyandMustafa Cavus
Todor BoinovskiandMax Krasnyansky
Aparna M PandMax Krasnyansky
Adrien Gallouët
Ruben Ortlam
virajwad
Giovanni Rivera
Konrad Moren
Yufeng HeandPascal
François-Xavier Gsell
Evan HuusandGeorgi Gerganov
Alan Tseng
Ed Addario
Masashi Yoshimura