* model : use exact GELU for ModernBERT encoders
Assisted-by: Codex
* model : keep tanh GELU aliases on ggml_geglu
Assisted-by: Claude Opus 5.5
* model : map gelu_python to ggml_geglu_erf
Assisted-by: Claude Opus 5.5
* model : support classifier_activation for rerankers
Assisted-by: Claude Opus 5.5
* model : map classifier gelu to gelu_erf and accept tanh
Assisted-by: Claude Opus 5.5
* model : default act_cls to tanh, ModernBERT falls back to gelu_erf
Assisted-by: Claude Opus 5.5
* model : add LiquidAI/d1-omni-600M decision model
Assisted-by: Claude Opus 5.5
* mtmd : keep conformer GLU sigmoid on CUDA
Assisted-by: Claude Opus 5.5
* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal
Assisted-by: Claude Opus 5.5
* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files
Assisted-by: Claude Opus 5.5
* llama: share the nextn tensor flags between models
Follow-up of the TODO in glm5-next: move the trunk-only and MTP-only
detection that each model copied into a nextn_flags helper of
llama_model_base. It probes the first trunk layer and the first NextN
layer, and adds TENSOR_SKIP when MTP is not loaded. qwen4exp probes
hc_attn_norm since it has no attn_norm.
deepseek4, nemotron-h, qwen35, qwen35moe, qwen3next and qwen4exp now
also accept a trunk-only file, like the other models.
* llama: avoid capturing structured bindings in the nextn flags
Lambdas that capture structured bindings need C++20, and GCC 15
rejects them under -Werror, so the models read the trunk and MTP
flags into plain variables.
* llama : add GLM5-Next NextN (MTP) graph
Build the GLM5-Next multi-token-prediction head as graph_mtp: the NextN block
embeds enorm(tok)+hnorm(h) through eh_proj, runs one plain DSA layer and the
shared lm_head, reusing the trunk's builders through the no_build tag ctor.
llama_memory_recurrent also tolerates a partial seq_rm when the context holds
no recurrent layers, which is what the MTP draft context needs.
Assisted-by: Claude
* llama : glm5-next: skip dead compute in headless NextN forwards
A NextN forward with no output rows (the MTP catch-up and the draft-context
prefill) persists only through its cache writes, so the headless graph keeps
the MLA latent, indexer key|gate and pooled-key writes and drops the query
path, the indexer selection, the attention body, the FFN and the LM head. The
4-token catch-up falls from 6.9 ms to 0.33 ms of kernels; the greedy output
hashes and the draft acceptance are unchanged.
Assisted-by: Claude
* llama : glm5-next: fix NextN extraction contracts and shared-tail rollback
Three fixes from the architectural review. The headless graph prune now also
requires that no unmasked nextn extraction is live, because that mode reads
n_tokens hidden rows regardless of the logits flags. Masked extraction
publishes the hidden rows gathered by the output ids, so a batch whose output
flags are not a prefix exports the right rows. A partial recurrent rollback
whose tail cell is shared with another sequence is now rejected instead of
silently moving that sequence's tail.
Assisted-by: Claude
* llama : glm5-next: tidy comments in the MTP changes
Assisted-by: Claude
* llama : glm5-next: crop the MTP graph to the output rows instead of pruning it
Replace the headless NextN prune with the crop pattern the other MTP
graphs use: gather the attention output and the block input at the
output ids before the position-wise FFN and the shared head. A NextN
forward with no output rows (the MTP catch-up and the draft-context
prefill) then runs the FFN and the head over zero rows. The 4-token
catch-up falls from 6.9 ms to 2.9 ms of kernels; greedy output hashes
are unchanged.
Assisted-by: Claude
* glm5-next: use the nextn crop helpers in the MTP graph
Replace the local crop condition and the masked select of t_h_nextn
with crop_before_nextn and crop_after_nextn, so the MTP graph narrows
its rows the same way as the main graph and the other models.
Describe the shared cell and empty filter branches of the recurrent
partial rollback.
* glm5-next: load MTP-only and trunk-only GGUF files
Make the trunk tensors optional when the file only holds the NextN
layer, and the NextN tensors optional when the file only holds the
trunk, so the split MTP GGUF loads as a draft model.
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* model: K2 Horizon gguf conversion code
* model: loading hparams and tensors in k2-horizon.cpp
* model: K2 Horizon compute graph
* model: K2 Horizon compute graph adjustment and registering tokenizers
* model: K2 Horizon chat template and accomodate safetensors naming
* unicode : add the K2-Horizon pre-tokenizer splitter
The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.
On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.
Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal and alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.
The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.
tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.
* tests: expand K2 Horizon unicode splitter coverage
* unicode: handle K2 Horizon case folding and empty input
Assisted-by: Codex
* jinja : support sequence indices in selectattr and rejectattr
Assisted-by: Codex
* model : add K2 Horizon dense and MoVA support
Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.
Assisted-by: Codex
* chat : support K2 Horizon reasoning and tool calls
Assisted-by: Codex
* conversion: remove obsolete K2 Aurora alias
Assisted-by: Codex
* k2-horizon: enforce response schemas and load YaRN betas
Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.
Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.
Assisted-by: Codex
* renaming template fixture
* adressing cisc follows ups
* desloppify the parser / adress aldehir comments
* clean test-chat
* remove fallback : model trained mostly on high anyway
* fix k2 attn_v_exp tn splitting and metal fusion baseline
* k2-horizon : forward expand views before sums
* k2-horizon: copy embds before group norm to fix TP
* disable tesnor parallelism
---------
Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
The gather path attended over the selected latents with a plain
matmul and softmax. It only ran with n_ubatch <= 16, and the flash
attention backends now skip the masked rows through n_kv_max, so
the scatter path covers every case.
Drop the gather flag, the gathered attention branch and
gather_mla_rows. set_input_kpool always maps padding to the n_kv
sentinel, and the slot mask becomes sel_mask since only the
scatter reads it.
* mimo2 : always emit h_nextn
the other nextn-capable models set it unconditionally
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* models : consolidate nextn row cropping into shared helpers
- replace the duplicated crop conditions and the per-model flags (narrow_early,
crop_before_ffn, crop_last_layer, emit_h_nextn) with two helpers on llm_graph_context:
crop_before_nextn() / crop_after_nextn()
- models that only tested embeddings_nextn_masked now share the same condition, so they
crop the last layer before the nextn capture whenever extraction is off
- t_h_nextn is now set unconditionally in mimo2, qwen4exp and deepseek4 (as in the other
nextn-capable models); host-side reads stay gated by cparams.embeddings_nextn
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* llama: re-pool each shared k-pool rep once
With shared cells every pool is re-pooled, and since the pooled keys
are always scattered, the pools a seq_cp shares between sequences
wrote the same rep row from several scatter entries, a data race on
the CPU backend. Mark each rep once: the sharing sequences read the
same row through pool_cells.
* llama: assert whole-sequence seq_cp in the hybrid idx memory
The recurrent state is always copied whole whatever the range, and a
k-pool cell shared by a partial copy could carry two pool groupings
with a single pooled row. Every caller copies whole sequences, so
reject partial ranges instead of supporting them.
* llama: drop the k-pool cache_safe mode
With whole-sequence seq_cp, sequences sharing cells share their pools
too, so the pooled row of a shared rep is valid for all of them. Mark
each rep once in every ubatch instead of re-pooling everything while
cells are shared, which removes the sharing scan and the stale-all
workarounds in seq_rm, state_read and state_drop. seq_cp now only
stales the destination.
* server: support vision input for Clef
* move input_attn_causal to private
* extend old server_batch::embd
* server_batch::token::pos to multi dim
* nits
* fix abort
* fix img tokens cap
* fix yield_to_queue mutate data
* llama : fix unexpected graph reallocation in the k-pool models
Both k-pool models built a graph shape that depends on state the
full-context reserve cannot know:
- qwen4exp branched on inp->cache_safe, which turns false as soon as
llama_memory_seq_cp shares cells (e.g. batched-bench -pps): the QSA
layers swapped scatter+gather for fill+concat and dropped the
new_pool_rep leaf, so the decode graph had 12 fewer nodes than the
reserved one
- glm5-next branched on gather = n_tokens <= 16 && n_kv > n_sel, so the
TG decode built the gather shape (7564 nodes) while the last reserve,
the PP one, had the dense shape (7762 nodes)
Either mismatch forces a decode-time re-reserve that drops the
worst-case sizing and bakes in the current state, so the next state
growth (n_pool, n_kv, n_new) needs more room at an unchanged graph size
and aborts under GGML_SCHED_DEBUG_REALLOC=1. Reproduce with, e.g.:
GGML_SCHED_DEBUG_REALLOC=1 ./bin/llama-batched-bench \
-hf ggml-org/GLM-5.3-Flash-GGUF:Q2_K -npp 2500 -ntg 32 -npl 1,2 \
-c 32768 -pps -kvu
Always scatter+gather the pooled keys, and pick gather from context
constants only: n_ubatch bounds every ubatch, top_k + kpool - 1 bounds
n_sel. Every graph of a context then shares one shape, which the
reserve covers, and the dense path measured faster than the gather path
at 2.5k and 16k context.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* llama : drop the unused k-pool cache_safe graph API
The k-pool graphs no longer branch on cache_safe, so nothing reads
get_kpool_cache_safe() or the conditional new_pool_rep any more: both
models always pass the scatter target, which set_input_kpool now
requires instead of merely preferring.
Also drop the cache_safe copy in kpool_build_sizes(), a sizes-only
helper. The layout and state flag itself stays, it still decides which
pools a layout with shared cells must re-pool.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* tests : add a shared-seq graph reserve regression test
Decode a prompt into seq 0, share its cells with seq 1 via
llama_memory_seq_cp (what llama-batched-bench does for -pps), then keep
decoding both sequences. For the k-pool models sharing clears
cache_safe, which changes the graph topology while the pools keep
growing, so a scheduler that re-reserves with the current state
instead of the worst-case one aborts under GGML_SCHED_DEBUG_REALLOC=1.
The test registration sets that flag, and the test aborts on both
k-pool models before 2220411ec1.
kimi-linear and minimax-01 are skipped: they reserve the final pp graph
with n_seqs = 1 (see [TAG_RESERVE_DIAG_DECAY] in llama-context.cpp), so
every multi-seq graph has a different layout and re-reserves by design.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : add TODOs
* cont : fix comment
* cuda: match the moe weighted reduction on empty ubatches
ggml_cuda_match_moe_weighted_reduction rejected tensors with zero
rows. A ubatch without outputs shrinks the last layer to zero rows
through inp_out_ids, so graph_optimize dropped its alloc dep there and
the scheduler graph lost one node compared to the reserved one. The
scheduler then re-reserved at the size of that ubatch, and the next
ubatch with the same node count but larger tensors aborted under
GGML_SCHED_DEBUG_REALLOC=1.
The compute loop already skips empty nodes before trying any fusion,
so the guard only made the alloc deps depend on the row count.
* tests: build the rollback test only where internal symbols link
The shared-seq case calls llm_arch_from_string, which libllama does
not export through LLAMA_API, so linking test-recurrent-state-rollback
fails on Windows with shared libraries. Its build now sits in the
NOT WIN32 OR NOT BUILD_SHARED_LIBS block, next to test-llama-archs and
the test registration it already lives under.
* tests: skip archs by name in the shared-seq reserve test
The skip of kimi-linear and minimax-01 went through llm_arch_from_string,
which libllama does not export through LLAMA_API, so the test could not
link on Windows with shared libraries. It now compares the
general.architecture string directly, and the test builds on every
platform again.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama: support both embd + raw tokens in batch
* add to test-llama-archs
* also check case llm_arch_supports_mixed_batch = false
* constant graph topology
* have dedicated input for mixed case
* rm set_tensor_backend
* is_embd --> type
* consolidate m-rope pos handling into one place
* nits
* qwen4exp : halve the indexer score memory
The indexer scored all heads in one product and rectified a copy of it,
so two [n_pool, n_idx_h, n_tokens] f32 tensors were live at once, the
largest buffers of the graph at long context. Each head now gets its
own product, rectified and summed in place into one [n_pool, n_tokens]
score.
* qwen4exp: let the allocator reuse the indexer score buffers
Address review from CISC: use plain ggml_add and ggml_relu in the
indexer head loop. The graph allocator already runs them in place when
their source has no other consumer, so the _inplace variants are not
needed. The compute buffer and the speed are unchanged.
* cuda: support 4 heads in the lightning indexer
Dispatch 4 heads to the vector kernel, too few for a wmma tile, and
accept them in supports_op. test-backend-ops covers 4 heads.
* metal: take the lightning indexer head count as a function constant
The kernel reads the head count from a function constant and zero fills
the last head tile, so any head count runs and 64 heads is unchanged.
* qwen4exp: compute the indexer score with the lightning indexer
Address review from am17an: the unweighted sum of the rectified head
scores scaled by 1/sqrt(head_dim) is the lightning indexer with every
head weight set to that scale, so the indexer calls
ggml_lightning_indexer on the pooled keys with an f16 pool mask. The
keys are read once for all heads and no per head score is
materialized.
* vulkan: tile the lightning indexer over keys and tokens
A workgroup scores 64 keys against 8 tokens: the keys are staged once
in shared memory, the queries one head at a time, and each invocation
owns one key for two tokens, so no dot product needs a cross invocation
reduction. The subgroup variant and the flat dispatch are gone, the grid
is keys x tokens x streams.
* vectorize vulkan loads and use fp16 dot product
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* init conversion
* convert: ok
* model loaded
* add server code
* improve conversion script
* support shared prompt prefix
* add docs, imorove UX a bit
* add vision support
* add openjev tiny model for testing
* add dev docs
* support lev & kev
* clean up
* fix lev noul
* fix py lint
* nits docs
* clarify about not supporting date_facts
#27941 disabled -sm tensor for qwen4exp because test-llama-archs asserted on the
Meta device once the fixture carried a PLE layer:
GGML_ASSERT(ggml_backend_buffer_is_meta(tensor->buffer)) at ggml-backend-meta.cpp:476.
With host-resident embeddings the PLE gather is a CPU node and hc_init (the REPEAT
that fans the embedding out to the hc streams) was first reached through layer 0's
PLE path, after that gather. ggml_backend_sched_split_graph pass 2 expands a device
assignment upwards only until it meets a CPU node, so the REPEAT stayed on the CPU
and the later reshape of hc_init inside the meta split viewed a host-resident node.
Expanding hc_init right after it is built puts the REPEAT directly before the first
device node, where pass 2 assigns it; the embedding reshape stays in the CPU split
and is copied in as a split input, as in deepseek4.
The sparse indexer mask is built with a set_rows scatter. Padded pools,
absent sequences and missing tail cells all pointed to the same n_kv
sentinel row, and invisible pools picked by top_k to fill the selection
overlap the tail cells of the token, so several CPU threads wrote the
same element (ThreadSanitizer data race in the sanitize CI).
Allocate the slot mask for both selection paths and route every dead
slot to its own dump row n_kv + slot. Live slots address disjoint cells,
so the scatter indices of a token are unique.
* llama: llama_prefetch_rows
* llama: support row prefetch on Windows
Apply the Windows port contributed by @praneshgo unchanged.
Source: https://github.com/ggml-org/llama.cpp/pull/29599#issuecomment-5887721014
* avoid exposing llama-mmap in model code, route via llama-impl
* add windows check, only prefetch in lazy mode
* cont : clean-up
* cont : fix build
* cont : clarify padding token for gemma4
---------
Co-authored-by: Pranesh Gonegandla <pranesh.iitp@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx
* Add initial MTP support
* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.
* Review driven changes, remove env vars, protect tensors
* Strip MTP for initial PR
* Clean up after mtp strip
* Clean up after mtp strip
* Update speculative.cpp
* Update llama-context.h
* Clean up after mtp strip
* Fix tokenizer ignore merges
* Improve quantization protection selection
* Refactor mhc helpers, graph base
* Lint Fixes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Skip glm5-next in model saver, fix CRLF
* Skip glm5-next in sweep
* Remove T4 fallback
* Review cleanup
* Review suggestions
* Defer separate MTP gguf handling to MTP PR, drop filter
* Repad n_head_kv
* kpool init apply
* Order by descending score
* Drop guard
* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)
* Add glm5-next support to model saver and add arch test fixture
* Review cleanup
* Kpool pooled caching clarify
* Add multi stream support
* Finish Rebase
* Sparse FA fir DSA prefill
* Const
* Update llama-model.cpp to fix rebase error
* gguf-py : merge tensor map entries for HC tensors
* model : use build_gdn_l2_norm in GLM5_NEXT implementation
* chore : remove trailing whitespace
* model : use new OP precision setting API in GLM5_NEXT implementation
* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit
The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.
Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)
* llama : keep the GLM5-Next k-pool layout across ubatches
The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.
A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.
Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)
* model : write the GLM5-Next recurrent rollback checkpoints
The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.
Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.
Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)
* llama: fix PR #27773 test-save-load-state restore failure
Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.
Assisted-by: Codex
* llama: fix PR #27773 gpu-rocm graph reallocation
Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.
Assisted-by: Codex
* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown
Two defects in the cross-ubatch k-pool layout added by the k-pool commit:
1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
count changed, so if the first ubatch after an edit added back exactly as many
cells as were removed, the stale position-to-cell list survived. With a unified
cache and more than one sequence, where another sequence takes the freed cells,
the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
Rebuild whenever the sequence is stale, not only on a size mismatch.
2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
rebuild, but sharing also ends when the other sequence is removed. The survivor
kept shared = true, pinning cache_safe off and re-pooling every pool on every
ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
so one rebuild re-derives sharing and cache_safe returns to 1.
Assisted-by: Claude Opus 5
* llama : fix build_attn_mha stream stride for non-contiguous q
build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.
Compute the stride from the token dimension, which is identical for a
contiguous q.
Assisted-by: Claude Opus 5
* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop
The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.
Assisted-by: Claude Opus 5
* quant : drop duplicate GLM5-Next hc_ filter
The hc_ name filter was listed twice in the GLM5_NEXT protection block.
Assisted-by: Claude Opus 5
* glm5-next: scope K-pool cache access to indexed operations
* glm5-next: keep K-pool access in hybrid index memory
* glm5-next: keep mHC graph builders model-local
* glm5-next: mark only touched pools per ubatch
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
* model : support classifier_pooling for ModernBERT rerankers
Assisted-by: Claude Opus 5.5
* model : read classifier pooling type in load_hparams
Write classifier.pooling_type from _try_set_pooling_type whenever the
config has classifier_pooling, and read it in
llama_model_base::load_hparams. ModernBERT falls back to mean when it
is unspecified.
Assisted-by: Claude Opus 5.5
* conversion : only accept cls and mean for classifier_pooling
Assisted-by: Claude Opus 5.5
* model : rename classifier_pooling_type to pooling_type_cls
Assisted-by: Claude Opus 5.5
* models: pad on the left with ggml_pad_ext
The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders
build a left padding as a right pad followed by a roll, and DFlash2
concatenates a zero filled block in front of the previous tokens.
ggml_pad_ext does both in one node now that every backend supports a
left padding. The Gemma 4 audio embeddings are bit identical.
* models: skip the DFlash2 taps that only read padding
A tap at or past block_size shifts every row out of the block, so its
term is zero. The loop runs min(kernel_size, block_size) taps.
* context : do not re-reserve the scheduler when toggling causal_attn
`llama_context::set_causal_attn()` marks the scheduler to do a full re-reserve on every change of the flag. For vision inputs, this flag is flipped twice around each non-causal image chunk for Gemma models, resulting in two expensive `sched_reserve()` passes per image. This is especially slow for multi-image or video inputs.
The cost of a re-reserve scales with context and ubatch configurations, so larger settings pay more per image (see table below).
The re-reserve is unnecessary in this case because `causal_attn` only changes the values written to KQ mask, not tensor shapes or any other buffer sizes.
Note: `causal_attn` is a graph reuse key (`llm_graph_params` via `cparams`), so a new graph is built regardless of `sched_need_reserve`, so this doesn't change the graph rebuilding behaviour.
llama-server with gemma-4-26B-A4B Q4_0 + BF16 mmproj, 130-token images,
cache_prompt=false, prompt_ms median of 3 (before -> after):
| images | config | H200 before -> after | RTX 4090 before -> after |
|-|-|-|-|
| 1 | `-c 8192 -ub 512` | 134 -> 105 ms (1.27×) | 201 -> 119 ms (1.69×) |
| 24 | `-c 8192 -ub 512` | 2278 -> 1562 ms (1.46×) | 3559 -> 1748 ms (2.04×) |
| 24 | `-c 32768 -ub 2048` | 5379 -> 1584 ms (3.40×) | 13377 -> 1759 ms (7.61×) |
Generated output remains identical before and after.
* qwen4exp : make the indexer bias shape independent of causal_attn
The block/cell bias path was selected on cparams.causal_attn, so the
causal and non-causal graphs differed in tensor shapes and ops. With the
re-reserve removed (previous commit), a runtime flip resulted in
reallocating the compute buffers, which would fail under
GGML_SCHED_NO_REALLOC.
This commit selects the block path from the mask shape only, independent
of causal_attn. causal_attn is instead passed to set_input_qsa.
causal_attn is fixed per graph as it's part of the reuse key. Causal
values are unchanged. Non-causal values now follow the reference rule,
where every visible block competes on score and only unpooled cells are
always selected.
* context : state the causal_attn shape rule in the comment
* cont : add TODOs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* model : fold Ling 3.0 VL into the BailingMoeV3 architecture
Assisted-by: Scout
* model : keep shared NORM rope list intact when gating bailingmoe3 on mrope sections
---------
Co-authored-by: aetherbird <aetherbird@users.noreply.github.com>
* dspark: add Gemma 4 draft support
Add GGUF conversion and runtime support for full-attention and SWA Gemma 4
DSpark drafts, including tied output weights and boolean backbone metadata.
Assisted-by: Codex
* dflash: infer Gemma draft features from metadata
* model : add DFlash layer-input taps for HunyuanVL
DFlash speculative decoding needs the target graph to expose the residual
stream entering each layer (res->t_layer_inp[il]) - the draft model reads
those tensors to build its cross-context. Qwen3 and the other DFlash-capable
targets register them, but the Hunyuan graphs do not, so serving a DFlash
draft against a HunyuanOCR target aborts during the first graph build:
GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")
Register the tensor at the top of the layer loop, mirroring qwen3. The
layer input is the residual stream entering layer il, i.e. the output of
layer il-1, which is what the draft's target_layers metadata refers to
(the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this
graph, so it is covered as well; hunyuan-moe has a separate graph and is
untouched.
The vector is only read when a speculative implementation enables those
layer ids, so there is no behaviour change without a draft model.
Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now
run, draft acceptance is ~0.5 and the OCR output is byte-identical to the
non-speculative run.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft conversion against HunYuan targets
Converting a DFlash draft with a HunYuan target failed in two ways.
1. DFlashModel.set_vocab() reuses the target class' vocab handling by
calling it unbound with the draft instance, but HunYuanModel.set_vocab()
called self._fix_special_tokens(), a method that only exists on
HunYuanModel, so the conversion always aborted with
AttributeError: 'DFlashModel' object has no attribute '_fix_special_tokens'
Make the vocab helpers module-level functions taking the model
explicitly, so they do not depend on the instance being a HunYuanModel.
They have no other callers, so the two id lookups are folded into
_fix_special_tokens().
2. The delegated call runs with self.dir_model pointed at the target but
keeps the draft's self.hparams, so config lookups inside the target's
vocab code (the pad_token_id < 0 guard, eod_token_id) read the draft's
config instead of the target's. That aborts on targets with
pad_token_id = -1 (e.g. the HunyuanOCR v1.0 checkpoint) and otherwise
writes special token ids that disagree with the target.
Add _vocab_hparams(): it returns the target's config (with text_config
merged to the root, as TextModel does) when the model is a draft
converted with --target-model-dir, and the model's own hparams
otherwise, so a normal conversion is unaffected.
Tested: converting tencent/HunyuanOCR/dflash succeeds with both the 1.5 and
the v1.0 target; converting the base model without --target-model-dir
produces a byte-identical GGUF to before.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft vocab against HunYuan targets
Switch hparams to the target config for the duration of the borrowed
set_vocab(), matching the existing dir_model swap, instead of teaching
HunYuanModel::set_vocab about draft models.
* convert : fix HunYuan special token ids for DFlash drafts
* convert : use load_hparams for HunYuan special token ids
* llama: read the SWA pattern as a period or a per-layer array
Add llama_model_base::load_swa_pattern(), which reads
sliding_window_pattern either as one flag per layer or as a period
expanded by set_swa_pattern(), and use it in every loader that reads
the key as a period.
These loaders silently ignored an array and applied their default
period, although the converters of olmo2, gemma3n and exaone4 write
arrays. The published GGUFs match the defaults, so their outputs do
not change. The loaders that already accepted both forms lose their
duplicated scalar-then-array block, and use their declared default
period when the key is absent.
* model-saver: write the SWA pattern and the MLA SWA geometry
Write sliding_window_pattern as one flag per layer, nextn layers
included, for every model using SWA. The array is never collapsed to
a scalar, since the loaders read a scalar as a period.
Also write the MLA key/value lengths and KV LoRA rank of the SWA
layers, required by dots3note.
This enables the saver for plamo3, gemma3, cohere2, cohere2moe,
olmo2, exaone-moe, afmoe, mimo2, spark2_5, muse-glimmer, mellum,
laguna, granite_swa, dots3note and maple, all passing the bit-exact
roundtrip of test-llama-archs.
* model : add support for HrmTextForCausalLM (DFM Mimir 1B)
HRM-Text runs two transformer stacks (low, high) in an alternating cycle over the same token stream. The low-cycle state z_l starts from a learned [n_embd] tensor and is broadcast over positions.
- conversion: new writer for the fused gqkv projection (order gate,q,k,v) remapped to llama.cpp q/k/v plus a separate sigmoid gate tensor
- loader: block_count = lps * h_cycles * (l_cycles + 1) cache slots aliasing 2*lps physical blocks via struct copies
- graph: looped build with sigmoid-gated attention, SwiGLU FFN and parameterless RMS norms; learned embedding_scale applied in build_inp_embd
- saver: pointer-deduplicated layer loop (looped archs alias tensors)
- tests: hrm_text fixture (lps 1, h 2, l 3) in test-llama-archs
Limitations:
causal attention only - the upstream prefix-LM mode is not implemented (the prefix_lm GGUF key round-trips unused).
The KV cache holds one entry per pass: 128 layers for Mimir 1B, i.e. 4x a same-width 32-layer model - about 3072 MiB at ctx 4096 in F16 (halves with q8_0 KV + FA).
Every token runs all 128 block passes, so decode cost is roughly 4x a dense model of equal width (2.65 t/s BF16, 8-thread desktop CPU).
Verified against the HF reference: identical argmax at 334/334 positions across 20 prompts (BF16 GGUF vs FP32 golden).
q8_0 requant: 95.8% top-1, all remaining misses inside the HF top-5 (accumulated error over 128 sequential blocks).
AI usage disclosure: YES
Used GLM-5.3 for the majority of code AI-generated under my direction, all gates verified locally.
All in all I could say that I have written less than 20% of the code and most of the heavy lifting has been done by the model. As such, this should be considered experimental.
* Update conversion/hrm_text.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : add gguf_writer methods for hrm_text metadata
replace raw add_uint32/add_bool calls with dedicated GGUFWriter methods, following the add_embedding_scale pattern
Assisted-by: GLM-5.3
* convert : map regular hrm_text tensors via tensor_mapping
delegate unfused checkpoints to the base tensor mapping; training-style attn. names are renamed to self_attn. so the patterns match
Assisted-by: GLM-5.3
* model : format hrm-text build_* calls as in other models
one argument group per line, matching sibling model files
Assisted-by: GLM-5.3
* llama : move hrm z_l_init table entries out of the nemotron group
place the name and tensor-info entries with the other global input tensors
Assisted-by: GLM-5.3
* convert : slim down hrm_text comments
Assisted-by: GLM-5.3
* convert : build hrm_text block tensor names from the {bid} template
The tensor map holds concrete per-block names, so format the template
with the computed layer index before handing it to super().
* llama : name hrm metadata keys in their own hrm. namespace
The four keys are arch-independent, unlike the arch-substituted
Keys.LLM entries, so group them under Keys.HRM (like Keys.Split) and
rename the llm_kv entries to LLM_KV_HRM_*. Only our own GGUFs carry
the old hrm_text.* keys; they are regenerated.
* Update src/llama-model-saver.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : keep hrm metadata keys arch-substituted
Per review: the GGUF keys stay "{arch}.h_cycles" style, so the Python
members drop the LLM_KV_HRM_ prefix and keep arch templates; C++ keeps
the LLM_KV_HRM_* enums. GGUF output is unchanged - existing files and
HF uploads stay valid.
* Update gguf-py/gguf/constants.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : rename hrm writer methods to add_hrm_*
Generic names like add_h_cycles/add_prefix_lm are too broad on the
shared GGUFWriter; prefix them with hrm_ like the metadata keys.
* model : fix meta-split lookup for archs with aliased cache slots
Cache tensors of archs that alias physical blocks across looped slots
(hrm_text, nanbeige with num_loops > 1) can reference block indices
without weight tensor names. Take the output projection from the layer
array instead of asserting; all other lookups are unchanged.
* model : replicate hrm_text tensors on meta devices instead of splitting
The aliased cache slots rotate split states differently from their
physical weights, so the meta-split execution invariants (set_rows
requires the cache state to match the token indices) cannot hold for
any device count. Replicate all hrm_text tensors on every meta device
instead; single-device and non-meta paths are unchanged.
Assisted-by: Claude Sonnet
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>