Commit Graph
167 Commits
Author SHA1 Message Date
fzowlandSigbjørn Skjæret 412a363c65 qwen3 : bidirectional attention and dense_2 output projection (#30607)
* qwen3 : bidirectional attention and dense_2 output projection

Assisted-by: Grok

* convert : read use_bidirectional_attention in base

Assisted-by: Grok

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-19 09:29:02 +03:00
Xue ZhenliangandSigbjørn Skjæret 62318cd57d model : add Kolibri-1 (Aleph-Alpha) support (#30673)
* model : add Kolibri-1 (Aleph-Alpha) support

Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.

Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.

The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.

Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Add fusion records for kolibri1

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-17 13:00:17 +02:00
Gabe Goodhart 1268edac57 model : Granite switch decision 256 options (#30693)
* feat(convert): Attempt to come up with 256 single-token, semantic option markers

Each switch model needs to identify a collection of single-token option
markers that can be mapped to their respective logits for the output
probability distribution. The simple upper/lower/numbers set only yields 66
unique markers, but the model can support up to 255. The challenge is that
which combinations of characters result in single-token markers is model-
specific, so we do a discovery exercise at conversion time by looking
through the specific model's vocab for single-token sequences of 1+ plain
letter strings (eg `A`, `AA`, `BB`, etc).

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Note granite-switch with 256 option markers

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Remove extra newline

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(convert): Use the computed labels in the template so c++ side is source of truth

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Use LEV case for GRANITE_SWITCH w/ computed labels

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Consolidate g-s into 255 bucket

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-17 09:14:32 +02:00
Daniel Han 01fc3aae84 model: add Beam (Reflection) (#30670) 2026-10-17 00:07:55 +02:00
Gabe Goodhart 23f7aabc16 model : Granite Switch Decision (#30676)
* feat(convert): Add decision.chat to support decision models that can also chat

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(convert): Add conversion for Granite Switch as decision model

The template follows from the usage sample:
https://huggingface.co/ibm-granite/granite-switch-4.1-8b#guardian-and-decision-side-by-side

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(common): Add support for chat+decision models in common

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(common): Add granite switch decision model enum

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(server): Plumb through decision_chat bool to support decision+chat models

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(server): Add Granite Switch decision support

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Update server README for granite switch decisions / chat+decisions

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* Revert "feat(convert): Add decision.chat to support decision models that can also chat"

This reverts commit 952ef567647e53ac9beda7a1f79d6927240d068d.

* Revert "feat(common): Add support for chat+decision models in common"

This reverts commit 93ed782e3a8b2fd5968732c6525c7ad29ad4bea2.

* Revert "feat(server): Plumb through decision_chat bool to support decision+chat models"

This reverts commit 97e9af60ddaccbd82f1eb0a488adff43b16ee095.

* fix: If Granite Switch supports decisions, it always supports text too

Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Remove reference to decision.chat in server README

Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-16 20:00:41 +02:00
aiqubits 2a667032dc convert : respect Mistral format for DeepseekV2Model (#30669)
Use the instance's is_mistral_format flag when DeepseekV2Model reloads
configuration, allowing Mistral MoE checkpoints to read params.json.

Fixes #29899

Signed-off-by: aiqubits <aiqubits@hotmail.com>
2026-10-16 15:17:30 +02:00
Mikhail Podvitskii 9a16f1dbe4 model : MTP support for Mellum2.1 (#30612)
* model : MTP support for Mellum

* model : inline rope construction in Mellum
2026-10-16 10:20:03 +02:00
FINAL-Bench 0f4376268f convert : support text-only checkpoints in pplx-decider (#30575)
* pplx-decider-text-only

convert : support text-only checkpoints in pplx-decider

* Apply suggestions from code review

* convert : remove unused import
2026-10-16 08:56:00 +02:00
Xuan-Son Nguyen 2851b3bcbf mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)
* mtmd: add mtmd_audio_stream + support voxtral-realtime

* address review
2026-10-16 01:46:42 +02:00
Nigel JonesandGabe Goodhart 59d5fad7c1 convert : fix Granite Switch block_count for MultiSwitch checkpoints (#30621)
* convert : fix Granite Switch block_count for MultiSwitch checkpoints

Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

* Update granite.py

Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>

---------

Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-16 00:41:01 +02:00
ba3c05f71d model: add Muse Spark Open (text, vision and audio) (#30616)
* model: add Muse Spark Open (text, vision and audio)

Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.

Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.

The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Muse Spark: drop the redundant forward expand

* Muse Spark: drop the redundant forward expand and tighten the comments

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 23:27:12 +02:00
Kartik Gulia cd0a5b6a18 models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
* add support

* remove unneeded bos fix

* add video flag

* fix flake error

* remove video-specific budgeting

* convert : fix lilicorr-dflash bid clash

* spec : enforce lilicorr n_tokens condition

* spec : fix lilicorr walk

[no release]
2026-10-15 16:34:56 +03:00
Mikhail PodvitskiiandSigbjørn Skjæret a6b91d8c5e model : Mellum - read MoE gating params from GGUF, unify RoPE (#30578)
* model : mellum - read MoE gating params from GGUF, unify RoPE

- read/write expert_weights_norm and expert_gating_func instead of hardcoded values (defaults unchanged)
- use per-layer rope params via get_rope_freq_base/scale
- check n_expert once, outside the layer loop

* Update conversion/mellum.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

[no release]
2026-10-15 16:24:47 +03:00
Kartik GuliaandGeorgi Gerganov 843e7dcc15 spec : add lilicorr support (#30506)
* spec : add lilicorr support

* convert : use convention for hparam assignment

* add todo for hparam use in llama-batch.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* typo fix

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-15 05:49:25 +02:00
Takamichi Ohara eca5458ce0 convert : export YaRN truncation config (#30288)
* convert : export YaRN truncation config

* rename
2026-10-14 08:31:40 +03:00
Xuan-Son Nguyen efe3626b11 model: support vllm d3 models (#30383)
* models: support pplx-decider

* vllm d2

* update

* clarify naming rule

* stricter validation

* d3
2026-10-12 15:01:52 +02:00
Xuan-Son Nguyen 097cf33d94 model: support vllm decision 2 model (#30378)
* models: support pplx-decider

* vllm d2

* update

* clarify naming rule

* stricter validation

* fix ci

* rm redundant filter_tensors
2026-10-12 13:39:55 +02:00
bri-prismandGeorgi Gerganov a9d27ac693 model : support for Prism Bonsai 2 27B (#29600)
* Runtime support for Prism Bonsai 2 27B

Assisted-by: Claude Code

* address prism hadamard runtime feedback

* move hadamard tensors into method, define folded weight

* load_* hadamard fixes

* cont : clean-up

* conversion cleanup

Assisted-by: Claude Code

* prism hadamard key and methods for converter

Assisted-by: Claude Code

* address review bot feedback

Assisted-by: Claude Code

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-11 14:39:16 +02:00
69f201a205 model : support MiniCPM-V 4.7 (#29416)
* mtmd : add MiniCPM-V 4.7 support

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* model : allow mrope time from an extra position slot

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* Update conversion/minicpm.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Slim down comments

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix for "do not hand-wrap comments"

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix ci

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* rm 3d repo for pr one

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* gguf: add rope.section_order metadata

* fix comments

* handle grid layout

* allow compat

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-10 20:41:59 +02:00
ynankani 609290be6b convert : support compressed-tensor mixed-precision NVFP4 checkpoint (#28636)
Signed-off-by: ynankani <ynankani@nvidia.com>
2026-10-09 13:54:58 +02:00
Hrishith Thadicherla a11f57ba93 model : fix DFlash output head sharing (#30111)
* llama : fix DFlash output head sharing

Assisted-by: Codex

* dflash : read tied output weights from GGUF metadata

Assisted-by: Codex

* llama : share tied word embedding metadata

Assisted-by: Codex

* llama : remove DFlash embedding head fallback

Assisted-by: Codex
2026-10-08 17:42:28 +02:00
bosh 37ac634566 model : support classifier_activation for rerankers (#29692)
* model : support classifier_activation for rerankers

Assisted-by: Claude Opus 5.5

* model : map classifier gelu to gelu_erf and accept tanh

Assisted-by: Claude Opus 5.5

* model : default act_cls to tanh, ModernBERT falls back to gelu_erf

Assisted-by: Claude Opus 5.5
2026-10-08 09:29:20 +03:00
Sam Malayek 75118a3a59 convert : support Qwen3.5 embedding models (#27920)
* Update convert_hf_to_gguf: support Qwen3.5 embedding models

* Behavior-preserving refactor for conventions.

* conversion: simplify pooling comment
2026-10-08 08:40:01 +03:00
Tarek Dakhran a657f7e981 model : add LiquidAI/d1-omni-600M decision model (#30114)
* model : add LiquidAI/d1-omni-600M decision model

Assisted-by: Claude Opus 5.5

* mtmd : keep conformer GLU sigmoid on CUDA

Assisted-by: Claude Opus 5.5

* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal

Assisted-by: Claude Opus 5.5

* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files

Assisted-by: Claude Opus 5.5
2026-10-08 02:07:49 +02:00
Tarek Dakhran 88dcc460d6 model : add LiquidAI/d1-3B decision model (#30110)
* model : add LiquidAI/d1-3b decision model

mtmd : read LFM2 image resize algo from GGUF

Assisted-by: Claude Opus 5.5

* common : rename decision type d1 to lfm2-d1

Assisted-by: Claude Opus 5.5
2026-10-07 22:04:39 +02:00
Terrence Zhao 50a6c5cf7c mtmd: add cohere2 vision support (#30062)
* cohere2 vision model

* address comments

* remove redundant mapping

* follow existing patterns

* fused linear_1
2026-10-07 20:07:50 +02:00
SIDDARTHA REDDY 48499d2e1c qwen3tts : guard speaker_encoder_config patch for CustomVoice variant (#29179)
* Fix KeyError converting Qwen3-TTS CustomVoice variant without speaker_encoder_config (fixes #29088)

Signed-off-by: SIDDARTHA REDDY <75976672+SIDDARTHAREDDY8@users.noreply.github.com>

* Address review feedback: drop tests/test-convert-qwen3tts.py

Per reviewer feedback on PR #29179, remove the regression test file.
The fix itself is unchanged.

Signed-off-by: SIDDARTHA REDDY <75976672+SIDDARTHAREDDY8@users.noreply.github.com>

---------

Signed-off-by: SIDDARTHA REDDY <75976672+SIDDARTHAREDDY8@users.noreply.github.com>
2026-10-07 12:57:48 +02:00
Toki Nasin 7481354a17 convert : fix token configuration for PLaMo-3 (#29843)
* convert : Fix token configuration for PLaMo-3

The reasoning and tool calling tags in PLaMo-3 consist of three tokens
each. For example, for reasoning:

* reasoning start: `<|plamo:begin_`, `think`, `:plamo|>`
* reasoning end: `<|plamo:end_`, `think`, `:plamo|>`

Registering `<|plamo:begin_`, `<|plamo:end_`, and `:plamo|>` as
`USER_DEFINED` so that they are parsed correctly.

Also PLaMo-3 models use <|plamo:tag|> as EOT, while PLaMo-2 models
use <|plamo:op|>.

Take EOT token as a parameter and look it up so that PLaMo-2 and
PLaMo-3 can use their appropriate ones.

* use NORMAL instead of USER_DEFINED
2026-10-07 11:34:16 +02:00
Toki NasinandSigbjørn Skjæret abeada335e vocab : implement PLaMo-3 tokenizer pre-segmentation (#30045)
* vocab : implement PLaMo-3 tokenizer pre-segmentation

The PLaMo-3 tokenizer inserts hard boundaries before running the Unigram
DP, around <|plamo:...|>-looking text, and around runs of at least 4
identical characters or 2 spaces. Without them llama.cpp tokenizes code
indentation and repeated punctuation differently from the reference.

Reproduce the two re.sub() passes in llm_tokenizer_plamo2 by encoding
each segment independently.

* add vocab type "plamo3"

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* misc change

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-06 19:07:54 +02:00
4625240437 model : add K2 Horizon dense and MoVA support (#29535)
* model: K2 Horizon gguf conversion code

* model: loading hparams and tensors in k2-horizon.cpp

* model: K2 Horizon compute graph

* model: K2 Horizon compute graph adjustment and registering tokenizers

* model: K2 Horizon chat template and accomodate safetensors naming

* unicode : add the K2-Horizon pre-tokenizer splitter

The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.

On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.

Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal ‌ and ‍ alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.

The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.

tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.

* tests: expand K2 Horizon unicode splitter coverage

* unicode: handle K2 Horizon case folding and empty input

Assisted-by: Codex

* jinja : support sequence indices in selectattr and rejectattr

Assisted-by: Codex

* model : add K2 Horizon dense and MoVA support

Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.

Assisted-by: Codex

* chat : support K2 Horizon reasoning and tool calls

Assisted-by: Codex

* conversion: remove obsolete K2 Aurora alias

Assisted-by: Codex

* k2-horizon: enforce response schemas and load YaRN betas

Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.

Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.

Assisted-by: Codex

* renaming template fixture

* adressing cisc follows ups

* desloppify the parser / adress aldehir comments

* clean test-chat

* remove fallback : model trained mostly on high anyway

* fix k2 attn_v_exp tn splitting and metal fusion baseline

* k2-horizon : forward expand views before sums

* k2-horizon: copy embds before group norm to fix TP

* disable tesnor parallelism

---------

Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
2026-10-06 18:42:24 +02:00
Xuan-Son Nguyen 4fbc76dec5 model: support embeddinggemma2 (text+vision+audio) (#30054) 2026-10-06 18:20:31 +02:00
Xuan-Son Nguyen da263e7275 models: support pplx-decider (#30044) 2026-10-06 16:07:46 +02:00
Kartik GuliaandSigbjørn Skjæret 6c73b3e12d convert : add text_config as fallback [transformers 5.18] (#30040)
* add text_config as fallback

* remove redundant llm_config check

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-06 11:04:26 +02:00
Xuan-Son Nguyen 9871df5911 server: support vision input for Clef (#29969)
* server: support vision input for Clef

* move input_attn_causal to private

* extend old server_batch::embd

* server_batch::token::pos to multi dim

* nits

* fix abort

* fix img tokens cap

* fix yield_to_queue mutate data
2026-10-05 14:02:40 +02:00
Tarek Dakhran cb7934c52c model : Add LFM2.5-Encoder-350M and LFM2.5-Encoder-230M (#29862)
Register `Lfm2BidirectionalForMaskedLM` architecture for LFM2.5-Encoder
models.
2026-10-03 08:44:45 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 99b95488ca model: add support for clef decision model (text-only) (#29831)
* init support for clef (text only)

* more static graph

* clean up

* nits

* nits 2

* Update gguf-py/gguf/constants.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-03 02:50:48 +02:00
Xuan-Son Nguyen 46ca246de9 model: support nimble decision model (#29844) 2026-10-02 14:56:53 +02:00
Xuan-Son Nguyen a4cb4c61fd llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) (#29818)
* init conversion

* convert: ok

* model loaded

* add server code

* improve conversion script

* support shared prompt prefix

* add docs, imorove UX a bit

* add vision support

* add openjev tiny model for testing

* add dev docs

* support lev & kev

* clean up

* fix lev noul

* fix py lint

* nits docs

* clarify about not supporting date_facts
2026-10-02 11:56:04 +02:00
Yu Chengye 2b36825cbc convert : write Gemma embedding scale for DFlash drafts (#29802)
* convert : write Gemma embedding scale for DFlash drafts

A DFlash draft shares the target's token embeddings. Gemma scales them by sqrt(hidden_size) in the forward pass, and the draft config does not state that scale, so the converted draft read unscaled embeddings.

Take the scale from the target config when the draft config has none.

Assisted-by: Claude

* convert : check with get_model_architecture for gemma models
2026-10-01 16:34:32 +02:00
Aman GuptaandGeorgi Gerganov c061df1983 Qwen4Exp: add MTP (#29761)
* Qwen4Exp: add MTP

* remove has_state member, check via ctx_bufs being non-empty

* consistent naming + less verbose comments

* cont : clean-up recurrent memory

* cont : clean-up comments

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-01 14:13:27 +03:00
Toki NasinandSigbjørn Skjæret db33d3cb89 vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 (#29734)
* vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3

The original tokenizer configs for PLaMo-2 and PLaMo-3 have
`add_bos_token: true` and `add_eos_token: false` , but
_set_vocab_plamo() did not write the BOS/EOS metadata. The
PLAMO2 tokenizer path also ignored add_bos/add_eos during
tokenization.

Write the settings from tokenizer_config.json and honor them in
the PLAMO2 tokenization path. GGUFs without these keys keep the
previous behavior.

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-01 08:31:42 +03:00
b016f461be convert : fix LoRA conversion crash for Qwen3.5 V-head reorder (#28324)
* convert: fix LoRA conversion crash for Qwen3.5 V-head reorder

_reorder_v_heads does reshape+permute+reshape to reorder V heads from
grouped to tiled order.  LoraTorchTensor.reshape() cannot split its
row dimension (A matrix), so converting Qwen3.5 LoRA adapters that
target out_proj crashes with NotImplementedError.

Fix: detect LoRA tensors and apply the equivalent index permutation
directly — column reorder (dim=last) permutes A's columns, row
reorder (dim=0) permutes B's rows.  This is mathematically identical:
  (B @ A)[:, perm] == B @ A[:, perm]
  (B @ A)[perm, :] == B[perm, :] @ A

Verified: both paths produce exactly zero diff against the full-tensor
reorder on random (rank=32, 4096×4096) matrices.

Fixes #21125

Signed-off-by: Radu Swigler <radu@swigler.com>

* convert: add ty: ignore for hasattr-guarded LoRA call

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix comment

* nowrap

---------

Signed-off-by: Radu Swigler <radu@swigler.com>
Co-authored-by: Radu Swigler <radu@swigler.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-09-30 19:30:21 +02:00
Georgi GerganovandSigbjørn Skjæret 22bdcc4cdd mimo : support dflash (convert + feature extraction) (#29650)
* convert : update to support dflash

* cont : fix

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-09-30 17:04:35 +03:00
649dcb1036 add GLM-5.3-Flash (GLM5-Next) support (#27773)
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx

* Add initial MTP support

* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.

* Review driven changes, remove env vars, protect tensors

* Strip MTP for initial PR

* Clean up after mtp strip

* Clean up after mtp strip

* Update speculative.cpp

* Update llama-context.h

* Clean up after mtp strip

* Fix tokenizer ignore merges

* Improve quantization protection selection

* Refactor mhc helpers, graph base

* Lint Fixes

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Skip glm5-next in model saver, fix CRLF

* Skip glm5-next in sweep

* Remove T4 fallback

* Review cleanup

* Review suggestions

* Defer separate MTP gguf handling to MTP PR, drop filter

* Repad n_head_kv

* kpool init apply

* Order by descending score

* Drop guard

* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)

* Add glm5-next support to model saver and add arch test fixture

* Review cleanup

* Kpool pooled caching clarify

* Add multi stream support

* Finish Rebase

* Sparse FA fir DSA prefill

* Const

* Update llama-model.cpp to fix rebase error

* gguf-py : merge tensor map entries for HC tensors

* model : use build_gdn_l2_norm in GLM5_NEXT implementation

* chore : remove trailing whitespace

* model : use new OP precision setting API in GLM5_NEXT implementation

* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit

The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.

Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)

* llama : keep the GLM5-Next k-pool layout across ubatches

The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.

A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.

Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)

* model : write the GLM5-Next recurrent rollback checkpoints

The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.

Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.

Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)

* llama: fix PR #27773 test-save-load-state restore failure

Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.

Assisted-by: Codex

* llama: fix PR #27773 gpu-rocm graph reallocation

Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.

Assisted-by: Codex

* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown

Two defects in the cross-ubatch k-pool layout added by the k-pool commit:

1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
   count changed, so if the first ubatch after an edit added back exactly as many
   cells as were removed, the stale position-to-cell list survived. With a unified
   cache and more than one sequence, where another sequence takes the freed cells,
   the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
   Rebuild whenever the sequence is stale, not only on a size mismatch.

2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
   rebuild, but sharing also ends when the other sequence is removed. The survivor
   kept shared = true, pinning cache_safe off and re-pooling every pool on every
   ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
   copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
   so one rebuild re-derives sharing and cache_safe returns to 1.

Assisted-by: Claude Opus 5

* llama : fix build_attn_mha stream stride for non-contiguous q

build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.

Compute the stride from the token dimension, which is identical for a
contiguous q.

Assisted-by: Claude Opus 5

* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop

The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.

Assisted-by: Claude Opus 5

* quant : drop duplicate GLM5-Next hc_ filter

The hc_ name filter was listed twice in the GLM5_NEXT protection block.

Assisted-by: Claude Opus 5

* glm5-next: scope K-pool cache access to indexed operations

* glm5-next: keep K-pool access in hybrid index memory

* glm5-next: keep mHC graph builders model-local

* glm5-next: mark only touched pools per ubatch

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
2026-09-30 14:20:32 +08:00
bosh 48de2a1bcb model : support classifier_pooling for rerankers (#29627)
* model : support classifier_pooling for ModernBERT rerankers

Assisted-by: Claude Opus 5.5

* model : read classifier pooling type in load_hparams

Write classifier.pooling_type from _try_set_pooling_type whenever the
config has classifier_pooling, and read it in
llama_model_base::load_hparams. ModernBERT falls back to mean when it
is unspecified.

Assisted-by: Claude Opus 5.5

* conversion : only accept cls and mean for classifier_pooling

Assisted-by: Claude Opus 5.5

* model : rename classifier_pooling_type to pooling_type_cls

Assisted-by: Claude Opus 5.5
2026-09-29 22:33:05 +02:00
Toki Nasin 9adc7f420c convert : export YaRN scaling parameters for PLaMo-3 (#29528)
Recent PLaMo-3 models use YaRN, while some earlier PLaMo-3 models do not.
The recent PLaMo-3 store their YaRN settings as flat config keys
(rope_scaling_factor, initial_context_length) and build the dict at runtime
in Plamo3Config.rope_parameters. The current converter misses these settings
and writes plain RoPE metadata to GGUF. Mirror the runtime settings into
rope_parameters so the corresponding rope.scaling.* is written to GGUF.
2026-09-27 13:45:47 +02:00
Sigbjørn Skjæret 6fd50a4094 ci : bump ty to 0.0.84 (#29529)
* bump ty to 0.0.84

* fix assertion bug caught by ty
2026-09-27 13:43:08 +02:00
Sigbjørn Skjæret d81aef1994 gguf-py : TemplateProcessing has final word on add_special_token (#29417)
* templateprocessing must win over tokenizer config

* remove obsolete override
2026-09-25 11:55:38 +02:00
e9f824d8c0 llama : add llama_prec_policy + model-driven W4A4 path (#24364)
* Rebase and update based on #26675

Signed-off-by: ynankani <ynankani@nvidia.com>

* CI failure fix(launh_bounds overload on HIP) and cleanup

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Use ggml tensor instead of name in act policy map

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review comments and cleanup

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Rename changes

Signed-off-by: ynankani <ynankani@nvidia.com>

* Update ggml/src/ggml-cuda/mmq.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* MXFP4 dispatch changes for higher src prec

Signed-off-by: ynankani <ynankani@nvidia.com>

* Refactor and address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Updates based on review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Apply batched suggestions from code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Apply patch from review

Signed-off-by: ynankani <ynankani@nvidia.com>

---------

Signed-off-by: ynankani <ynankani@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-09-25 11:36:35 +03:00
Aman Gupta f805c57a2d llama : fix tensor split for fused qkv with uneven K/V head sizes (#29294)
* llama : fix tensor split for fused qkv with uneven K/V head sizes

Assisted-by: Qwen3.8-27B

* fix v granularity

* convert: fix mtp conversion

* convert: add support for mtp flags

* fix loader
2026-09-25 11:15:33 +08:00