Xue ZhenliangandSigbjørn Skjæret
62318cd57d
* model : add Kolibri-1 (Aleph-Alpha) support Kolibri-1 is a bilingual German/English MoE reasoning model with 78B parameters and 3.46B active per token: 50 layers, 384 routed experts plus one shared expert. Attention interleaves four sliding layers at a 513 window with one full layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE. GQA runs 48 query heads against 4 KV heads at head_dim 128 with a QK-RMSNorm, and every block carries four RMSNorms, so the attention and the MLP output are normalized before they rejoin the residual stream. Routing picks the experts from logits + expert_bias and weights them with the sigmoid of the unbiased logits, without renorm. The converter reuses AfmoeModel for the per-expert checkpoint layout and the expert selection bias, and writes the shared expert and the per layer sliding window pattern. The released FP8 checkpoint needs no special code, it goes through the generic weight_scale_inv path. Three tensor names are added to the mapping (post_attn_norm, post_ffn_norm, moe.router.expert_bias) and the 128k vocabulary registers the qwen2 pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA and the sliding window pattern. Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash) * Update src/models/kolibri1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Add fusion records for kolibri1 * Update src/models/kolibri1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Fusion baselines
Per-device baselines for test-fusion, one CSV per backend (e.g. MTL.csv). Rows are
arch,moe,mode,label,count. Regenerate a CSV whenever fusion patterns change.
Update a baseline
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON # enable the target backend
cmake --build build --config Release --target test-llama-archs --target test-fusion -j
rm -rf build-ci-models && mkdir -p build-ci-models
./build/bin/test-llama-archs -o build-ci-models
./build/bin/test-fusion --models build-ci-models --device MTL0 --record MTL.csv
Validate
./build/bin/test-fusion --models build-ci-models --device MTL0 --check MTL.csv
Non-zero exit means a row differs from the baseline. Use --model FILE to run a single
architecture. --check also reports baseline rows that the run no longer produces, so a fusion
that stops matching fails the run instead of silently dropping out of the CSV.