The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a
128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough
LDS free for a second workgroup per CU, while the larger one does not.
Extend the 64-row tile to the (type, J) pairs where it measured faster on
gfx1151, and leave every other entry untouched. J=96 is deliberately excluded
because it did not benefit.
test-backend-ops perf on gfx1151, 100 MUL_MAT shapes over the affected types
(N in 48..512, M both padded and unpadded), 3 rounds: median +2.1%, best
+26.0%, no shape outside run-to-run noise. 5 MUL_MAT_ID shapes: median +8.0%.
Assisted-by: Claude Opus 5
* llama : build the k-pool selection in the causal mask layout
- the glm5-next indexer folded the causal mask in through a [n_kv, n_tokens]
view: one reshape of the mask ahead of the add, and one of the result ahead
of every layer reading the selection, including the shared-indexer layers
- build the selection directly in the mask layout, as qwen4exp already does,
and drop the now-degenerate reshape in its attention as well
- assert the selection width matches the input it is taken from
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
* unicode : add opt-in NFC normalization for Unigram tokenizers
Note: used help from Codex, Sol 6.1 Medium
* Fixing initial bot comments + linter error (using existing out instead of print)
* removed test_nfc_tokenizer
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.
server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
* hexagon: add configurable HTP DCVS/HMX power mode
* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)
* hexagon: add --hex-power-mode flag to run.py
Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.
* hexagon: keep HTP awake for DCVS power profiles
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache attention pads K, V and the mask to 256, but T5 also adds
a relative position bias of [n_tokens, n_tokens], so the add with KQ
fails unless n_tokens divides the padded width. Pad the bias with
zeros up to the KV dimension of the mask, like K and V.
* llama : do not map foreign ranges into a backend buffer
A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.
For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.
Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : enable prefetch on macOS
The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.
With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : use a plain buffer list per context
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* graph: skip the LM head for pooled embeddings
A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.
The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.
* graph: drop the redundant t_embd expand
Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.
* graph: add build_logits() for the LM head output
Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.
* graph: call build_logits() before build_sampling()
Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.
* nit
* tests : consolidate the context state tests into test-llama-context
test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.
They are one binary now, and the case is the unit of the test:
tests/llama-context/test-case.h the interface a case implements
tests/llama-context/cases-kv.cpp the cases of the KV cache
tests/llama-context/cases-rs.cpp the cases of the recurrent state
tests/llama-context/test-util.cpp tokens, generation, comparison
tests/llama-context/test-runner.cpp the models of a case, on threads
tests/llama-context/test-log.cpp the log file of a model
A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.
Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.
ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : run single-threaded by default
* cont : try to build the tests on all platforms
* cont : trim down some body fat
* cont : fix
* cont : fix the fix
* ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG
* ggml-cuda: cache env var lookups in static const instead of re-reading per call
* ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_cuda_env_int
* ggml-cuda: consolidate env flag reads into single static accessors
* ggml-cuda : drop env helper comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : remove NDEBUG gates for CUDA graph logs
* vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount
* vulkan : fix narrowing in chunked matmul dispatch
* vulkan : zero fill the A side of mul_mmq for k steps past the end
The A side skipped the shared memory write for the k steps past end_k, while the B side was already told the bound and zero filled. The A registers are read for all BK_STEP steps, so the stale scale went into the dot product and produced NaN at k = 32.
Pass the bound to block_a_to_shmem and write zeros when it is false, for every DATA_A type, mirroring the B side.
Also assert that the B and D subbuffer sizes are not VK_WHOLE_SIZE before they are adjusted per chunk, and add two batched cases of the chunked shape.
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* vulkan : remove redundant matmul tile adjustment
---------
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* tests: cover weighted scalar broadcasts
Assisted-by: OpenCode
* ggml : support extended scalar DMA binary ops
Assisted-by: OpenCode
* hex-binary: remove all non-dma paths
* hex-binary: add support for non-contig strides and cleanup dma preable a bit
* hex-binary: cleanup the gating and simplify dma retry
* hex-geglu-erf: add a gate for F32 only, we dont support F16 for this op
* tests: cover weighted repeat broadcast
Assisted-by: OpenCode
* hex-binary: update REPEAT and COMPLEX kernels to use dma as well (missed them last time)
* hex-add-id: add dma support for src2 (idx)
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V
unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels
for head size 512 require a multiple of FATTN_KQ_STRIDE, so the op falls
back to the CPU for most lengths, and the -fa auto probe (1 token)
disables flash attention for such models.
Pad the mask like llama_kv_cache::get_n_kv and pad K and V with zeros up
to the KV dimension of the mask. clef builds its own causal mask and
pads it the same way.
* hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound
* hex-cpy: cleanup cpy cursor handling
* hex-cpy-concat: introduce threaded dma/hvx gather-row helper
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* hex-cpy: remove leftover rejections for extended buffers
* hex-cpy: relax the gate for fast-path
* hex-cpy: remove scalar divs and vtcm access and use explicit gather_sync
* hex-cpy: add proper support for mdev work splitting to fastpath
* hex-cpy: optimize dense gather-row tasks
* hex-cpy: unroll gather-rows loops
* hex-cpy: add fastpath for f16
* hex-cpy-concat: support for colapsing 3/4D dims
* hex-cpy: double-buffer support
* hex-cpy: factor out gather-rows into more of an op by itself
* hex-cpy-concat: dedup kernel-params
* hex-cpy-concat: step-down nr to fit instead of hard fail
* hex-gather: replace div with a mask
* hex-gather-rows: cleanup dma pipeline and remove the stalls
* hex-gather: specialize compute funcs
* hex-gather: reduce scalar reg.spills and func calls
* hex-gather: do not recompute rows-per-thread on the npu
* hex-build: add support for building/installing just hexagon
* hex-cpy-concat: remove redundant barrier in concat and move cpy barrier under fence
* hex-gather: annotations for dma-out
* hex-mdev: make all key vtcm_acquire sync state atomic (fixes random hangs in mdev setups)
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* vulkan: pack FMA for f16 accumulation, no dot product op
* vulkan: dot_product packs two rows per f16 FMA
* vulkan: rename dot_product to dot_product_v2
Give llama_build() and llama_build_and_test() a LINK keyword (default:
llama-common) so call sites can declare what a test actually needs:
- ggml-only tests now link just ggml: test-opt, test-metal-graph-optimize,
test-backend-ops, test-tiled-mulmat, test-barrier, test-quantize-fns,
test-quantize-perf, test-rope, test-col2im-1d, test-gguf, test-alloc
- llama-only tests drop llama-common: test-unicode, test-sampling,
test-grammar-parser, test-llama-grammar, test-batch-alloc,
test-gbnf-validator, test-quant-type-selection
- both mtmd tests link mtmd only, dropping the LLAMA_TEST_NAME /
target_link_libraries workarounds
Both helpers now set cxx_std_17 explicitly, since the tests no longer
inherit it from llama-common's PUBLIC interface.
tests/testing.h no longer includes common.h; the only symbol it used from
there (string_join) is inlined.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* metal : fix mul_mat_id limits and kernel name
* metal : fix mul_mat_id with duplicate ids
kernel_mul_mm_id_map0 assumed at most one match per token and a fixed
capacity of ne21 entries per expert, which produced wrong or dropped
output rows and could overflow into the slice of the next expert when
the ids contain duplicates.
build a compact per-expert list of the (token, slot) entries instead:
- pass 1 counts the entries of every expert, thread 0 computes the
offsets of the expert slices (tpe now holds [base, count])
- pass 2 writes the entries with one unconditional store per token
(conditional stores of divergent lanes are very slow), the dummy
value of the tokens without a match lands on a padding slot of the
slice that is never read - duplicates are handled by a rare branch
- kernel_mul_mm_id reads the base/count of its expert and loops over
the entry tiles that do not fit into the grid
The ids buffer is sized as ne20*ne21 + ne02 (entries + padding).
883/883 test-backend-ops MUL_MAT_ID tests pass, llama-bench is within
run-to-run variance of the previous behavior.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* metal : make mul_mat_id map0 ne02 a function constant
use FOR_UNROLL for the id loops of kernel_mul_mm_id_map0 and turn ne02
into the function constant FC_mul_mm_id_map0_ne02 (FC_MUL_MM + 7), so
that the threadgroup offsets of cnt/bas/sids and the prefix-sum bound
fold at compile time.
the pipeline name now encodes ne02 (same convention as the ssm_conv
nc/silu constants) and the map0 pipeline is compiled with the constant
value set. ne02 is no longer part of ggml_metal_kargs_mul_mm_id_map0 -
the dispatch grid still comes from the tensor ne[2] on the host.
no measurable perf change: 883/883 test-backend-ops MUL_MAT_ID tests
pass, op-level and llama-bench results are within run variance.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : clean-up + TODO
* metal : dispatch mul_mat_id entry tiles from the prepass
follow the approach of the vulkan implementation (#29998): instead of
launching a grid over experts x ne21 entry tiles and letting most of
the threadgroups early exit, kernel_mul_mm_id_map0 now also emits a
row map - the total tile count plus one (expert, first entry, valid
rows) descriptor per 32-entry tile that has work.
kernel_mul_mm_id is launched with a static upper bound on the number
of tiles (full tiles over all entries plus at most one partial tile
per expert), reads its descriptor and processes a single tile - the
tail early exits. This removes the round loop that was needed to cover
duplicate ids past the grid (register pressure: th_max 768 -> back to
the plain mul_mm level) and duplicates are now handled implicitly,
since they only produce more descriptors within the bound.
The grid shrinks by ~8-16x on typical shapes (e.g. 128 experts,
8 used, 512 tokens: 128 tiles instead of 2048 entry tiles, of which
94% only launched to exit) and ne21 is no longer needed by the mm
kernel kargs.
883/883 test-backend-ops MUL_MAT_ID tests pass. llama-bench
(Qwen3.6-35B-A3B Q8_0) is now faster than master on all pp sizes
(pp2048/4096/8192 +2%, pp512 +0.5%), tg32 unchanged.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : add smem assert
liangliangchang
Ruben Ortlam
Georgi Gerganov
Masato Nakasaka
Matt Corallo
Sandro Hanea
Takamichi Ohara
Raffaele Di DonnaandRaffaele Di Donna
Masashi Yoshimura
Yuri Khrustalev
Xuan-Son Nguyen
Adrien Gallouët
Aaron Teo
Charles Xu
Jeff Bolz
Igor Matuszewski
Pranav DhinakarandMax Krasnyansky
Pascal
Adam CaldwellandAdam Caldwell
homeofeandSigbjørn Skjæret
losewayy
Oliver Simons
Chanho Lee
Animesh
Muhammad SaadandSigbjørn Skjæret
kurquharandMax Krasnyansky
Pratyush Kumar
Alex Morgan