Commit Graph
3534 Commits
Author SHA1 Message Date
Tarek Dakhran 39da130fcd ggml-cuda : support ssm_conv kernel size 7 (#30873) 2026-10-20 17:47:44 +02:00
Georgi Gerganov df11702136 metal : optimize argmax and pool_1d (#30862)
* metal : optimize argmax and pool_1d

Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* tests : add pool_1d perf cases

Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : minor fix
2026-10-20 16:46:00 +03:00
lhezandHongqiang Wang a01ce9ffc6 opencl: add several gemm variants for small batches and some fixes (#30850)
* opencl: fix OOB read in `mul_mv_mxfp4_f32_flat`

* opencl: remove large-N guard that forces CPU weight placement

* this condition claims that large batch produces incorrect results on
  adreno and cannot be reproduced
* model loader uses N512 to probe op support, resulting in cpu fallback
* thus this condition does not seem necessary

* opencl: cap num of subgroups by workgroup size for q4_0 mc3

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add small batch variant for q4_k dp4a GEMM

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add a narrow variant of mul_mm_f16_f32_l4_lm

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-20 14:37:28 +03:00
Neo Zhang 8115b4c31f sycl : replace memset due to not always ordered (#30812) 2026-10-20 14:35:48 +03:00
Bernard Ladenthin 119f68d7d4 ggml-cpu : select the NEON fp32->fp16 conversion (#30505)
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects
only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable
software conversion while neon_compute_fp32_to_fp16, a single fcvt,
stays defined and referenced by nothing. __POWER9_VECTOR__ and
__riscv_zfhmin select both.

Add an #error pair that refuses a branch defining a COMPUTE macro
without selecting it. That is how this stayed unnoticed: the fallback
silently supplies a correct software implementation, so the only
symptom is a hardware instruction nothing ever reaches. The
FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR
30343 keeps that direction on the lookup table on purpose.

Replace the comment above the fallback: it singled out NEON and POWER9
and pointed at a selection that now happens in the branch itself.
2026-10-20 14:35:06 +03:00
uvos b684498579 HIP/CUDA: Fix reduction in mmf getting tile widths not devisiable by 64 (#30836) 2026-10-20 14:32:38 +03:00
Aaron Teo 522f2b6eed ggml-zdnn: introduce flash attention op (#30512)
* ggml-zdnn: add scratch buffer pool

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ggml-zdnn: implement fattn, ditch scratch slots in favour of scratch buffer pool

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: add note about fattn

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-20 14:31:35 +03:00
Bálint Bokros 13786fbb3a CUDA: Clear CUDA graphs after legacy memory pool flush (#26771)
Assisted-by: Claude Opus 5.5
2026-10-20 14:30:50 +03:00
Caladrel 704366d76f metal : fix missing MoE kernel variant for 9 experts per token (#30610) (#30699) 2026-10-20 13:02:05 +02:00
Max Krasnyansky 8466395353 hexagon: update to snapdragon-toolchain v0.8 and enable hexagon v85 (#30854)
* hex-build: bump toolchain, enable v85

* hex-build: bump toolchain to v0.8 in the CI and build scripts

* hex-build: add a little python script for swiv needed on some iot devices

* hex-build: fix flake8 warnings

* hex-arch: update version cap to v85

* hex-inspect: flexible TOOLS_ROOT

* hex-build: update to snake_case naming

* hex-build: update leftover old toolchain refs

* hex-target: update build and run scripts to support ubuntu with adb transport
2026-10-19 23:49:59 -07:00
Michael Taylor dd002b91f5 sycl: fix compiler warnings - missing prototypes, dead code, extra semicolons (#30271) 2026-10-20 00:47:25 -04:00
Sam MurdochandSam 3bb63cbcbf sycl : fix get_rows on reordered quantized weights (#30430)
Co-authored-by: Sam <sam@garagellm.tail7dd858.ts.net>
2026-10-20 07:47:21 +08:00
lhez 1b29621bac opencl: add cooperative-k dp4a gemm for q4_0, q4_k and q6_k (#30741)
* opencl: add cooperative-k dp4a GEMM for q4_0

* opencl: add cooperative-k dp4a GEMM for q4_k

* opencl: add cooperative-k dp4a GEMM for q6_k

* opencl: fix alignment
2026-10-19 12:30:39 -07:00
lhez 6fa4f6cd3e opencl: add get_rows for i32, cpy for i32 and f32 (#30756)
* opencl: support i32 GET_ROWS

* opencl: support i32 and f32 copy conversions
2026-10-19 12:22:34 -07:00
Georgi Gerganov 098ce526b5 metal : fix moe_reduce fusion race when dst aliases its inputs (#30795)
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a
single dispatch, reading experts and weights while writing dst. The graph
allocator computes tensor liveness on the unfused graph, where the inputs are
fully consumed by earlier nodes of the fused group, so it can legally recycle
the input memory as dst. In the fused kernel this becomes a
threadgroup-vs-threadgroup race that produces rare non-deterministic wrong
results (observed with test-llama-archs -a kolibri on the meta backend).

Reject the fusion when dst overlaps either input, same guard as the
mul_mat_add fusion. The unfused MUL + ADD path is always correct.

Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
2026-10-19 11:55:02 +03:00
Raul Torres f3fb2fcdbc cann : refactor #ifdef blocks to avoid unreachable code after return (#25273) 2026-10-19 11:17:54 +03:00
Johannes Gäßler 4866db350d CUDA: fix warning for nonexistent extra semicolon (#30774) 2026-10-19 10:08:21 +02:00
Aaron Teo 42b70032b1 ggml-cpu: vectorize ggml_v_silu and ggml_v_expf (#30791)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-19 09:59:49 +02:00
JoelandJohannes Gäßler 4c1004c492 cuda: cap MMQ Stream-K blocks at available work (#30553)
* cuda: cap Stream-K blocks at the number of k-blocks

Stream-K normally launches one CUDA block per SM. For small matrices,
this can exceed the total number of output-tile/k-block pairs, leaving
blocks with no work and adding launch and fixup overhead. Cap the grid
at that count to remove these excess blocks without changing the useful
rounded K ranges or numerical results.

Assisted-by: Codex

* cuda: cap Stream-K blocks at the number of k iterations

Cap the grid at the number of K_vram iterations across all output tiles, rounding up partial iterations. This removes extra CUDA blocks left by the k-block cap.

Partial iterations can change FP32 summation grouping. MUL_MAT and MUL_MAT_ID backend tests pass on RTX 5090.

* Update ggml/src/ggml-cuda/mmq.cuh

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-19 09:41:14 +02:00
Paul MairhoferandJohannes Gäßler 99db719bdf CUDA: fix shared-memory WAR hazard in flash_attn_ext_vec (#30573)
* CUDA: fix KQ write-after-read race in flash_attn_ext_vec

Assisted-by: ChatGPT

* CUDA: apply review suggestions for KQ sync comments

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-19 13:25:04 +08:00
Max Krasnyansky 31288c9c66 hexagon: FLASH_ATTN_EXT improvements for longer context and SWA models (#30731)
* hex-fa: support for masked block skipping to reduce DMA fetching

* hex-fa: finetuning the hmx->hvx crossover thresholds (boosting gemma-4)

* hex-fa: update hmx gate to account for mdev.count
2026-10-18 18:08:11 -07:00
Jhen-Jie Hong fe8841a78a hexagon: allow multistream FA (#30218) 2026-10-18 18:07:53 -07:00
Aman Gupta ff5a233767 cuda : fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain (#30602)
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the
hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a
gated template flag and computes scale_out*sigmoid(scale_in*p) from the
raw weights, so the chain runs as one kernel instead of four.
2026-10-19 01:48:47 +08:00
Andrea Rossetti 0878f7c304 ggml: support non-contiguous SUM (CPU, CUDA, Vulkan, WebGPU) (#28851)
* ggml: support non-contiguous SUM (CPU, CUDA)

* tests: add non-contiguous SUM cases

* tests: SUM over non-contiguous views

* vulkan: support non-contiguous SUM

* webgpu: support non-contiguous SUM
2026-10-18 18:57:36 +02:00
thelittlefireman 5aeac37a35 ggml-cuda: fattn do not check oob if k_VKQ_max = (k_VKQ_max - nbatch_fa) (#30572) 2026-10-18 10:59:06 +02:00
Todd Malsbary 547208dca2 Restore ESIMD kernels on FFN path (#30239)
* Reenable ESIMD on the FFN path

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Extend ESIMD fusion on FFN path to include Q8_0

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Remove comment explicitly referencing ESIMD in test-backend-ops

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-10-18 00:02:38 -04:00
Todor BoinovskiandMax Krasnyansky 848bbda904 hexagon: add support for hadamard MUL_MAT and GET_ROW for flat and tiled Q1_0 and Q2_0 (#30627)
* hex-hadamard: add support for hadamard ops

* hex-mm: reduce pkt counts in the q8_1 quantizer and vec_dot_q1

* hex-q1/q2: add support for flat and tiled GET_ROWS for Q1_0 and Q2_0

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-17 19:17:33 -07:00
Todor BoinovskiandMax Krasnyansky 797de63b2f hexagon: add support for Q1_0 and Q2_0 (#30122)
* hexagon: add support for q1_0 quant

* hexagon: enable Q1_0 weights in the 2D HMX MUL_MAT path

* hex-q1: cleanup gates, generalize tiling and remove in-loop divs

* hex-tiling: document tiled formats in developer.md and update htp-tensor.h

* hex-q2: add support for Q2_0

* hex-q2: remove stale asserts and speacial cases

* hex-mm: fix MUL_MAT_NX assumption about square tiling, Q1/Q2 have non-square tiles

* hex-q1/2: minor optimizations (unrolls/un-unrolls/...)

* hex-q1: optimize Q1_0 layout to reduce unpack/vrmpy overhead

* hex-q1/q2_0: further optimize vec_dots

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-17 13:36:38 -07:00
thelittlefireman 9d1878a435 ggml-cuda: HIP: use __builtin_amdgcn_perm for q1_0 unpacking (#29927) 2026-10-17 19:04:39 +02:00
Aman Guptaandthelittlefireman 2744e48d1e CUDA: faster MMVQ for low-bit types (#30517)
* CUDA: faster MMVQ for low-bit types

* ggml-cuda: hip: small nwarps GCN fix on Q3_K(ncols=2,3), Q6_K(ncols=4)

---------

Co-authored-by: thelittlefireman <thelittlefireman3@gmail.com>
2026-10-17 15:35:30 +02:00
Adrien Gallouët 9d1e03616e cuda : fix -Wtautological-compare in fattn vec dispatch (#30519)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-17 14:36:17 +02:00
Sigbjørn Skjæret f2f667b02d cuda : add grouped experts top-k fusion (#29181)
* fuse grouped experts

* add tests

* hip fix

* add to graph_optimize

* apply suggestion from am17an

* hip fix

* fix hip fix

* fix fix fix

* gate on hip version instead
2026-10-17 13:55:33 +02:00
Marcus f74e2fa6a3 sycl: fix prompt-processing regression after Q4_0/Q8_0 weight reorder (#30374) 2026-10-16 22:03:41 -04:00
wes 27c7a769da hexagon: vectorize softplus with HVX (#30435)
softplus_f32 and tile_softplus_f32 ran a scalar logf(1.0f + expf(x)) loop.
Replace both with a shared hvx_softplus_f32_aa() helper built from the
backend's existing hvx_vec_exp_f32 / hvx_vec_log_f32 primitives, so no new
numerical approximation is introduced.

hvx_vec_exp_f32 clamps its input to (-88, 88) before exponentiating, so a
naive log(1 + exp(x)) composition would saturate to ~88.72 for large x
instead of returning x. The Q6_V_vmux_QVV select keeps the scalar
reference's x > 20 guard (ggml_compute_softplus_f32 in ggml-impl.h),
which is what makes the vector path match the CPU result.

Assisted-by: Claude
2026-10-16 15:20:43 -07:00
Shawn GuandLi He 2f2763db6e opencl: Add MxFP4 MoE bin kernel for Adreno X1 devices (#30691)
* opencl: add bin kernel `gemm_moe_mxfp4_q8_1_dp4a_ila_ldck` for X1

* opencl: fix whitespace

* opencl: undo unnecessary change

* opencl: enable dp4a clc for X1E

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-16 15:15:04 -07:00
Jhen-Jie HongandMax Krasnyansky ceefd16daa hexagon: correct HMX FA sinks (#30581)
* hexagon: correct HMX FA sinks

* hex-fa: vectorize sinks in hmx kernels

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-16 16:51:14 +08:00
Michael Taylor a30fb4924d [SYCL]: fix memory errors in mul_mat, split buffer, host pool (#29889) 2026-10-16 03:28:50 -04:00
uvos 623c16393c CUDA: HIP: rms_norm: add variants that load float4 at a time and store x in registers (#29720)
* CUDA: HIP: rms_norm: load float4 at a time and store x in registers

* Add template paramter comments

* use ggml_cuda_is_aligned for src

* nits

* also use ggml_cuda_is_aligned for src in fused variants

* fix nits
2026-10-16 10:07:34 +03:00
Trivalik 3dbcb7fca5 cpu : reduce AVX2 tiled accumulators together (#30296)
Assisted-by: Codex
2026-10-16 09:48:23 +03:00
Aaron Teo c804754529 ggml-cpu: refactor s309x repack helper functions (#30588)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-16 09:45:35 +03:00
Titaniumtown 13de11e78f sycl: wide stores for the q4_K/q5_K weight dequant (#29696) 2026-10-16 09:43:07 +03:00
Ruben Ortlam a8d9a7e1e5 vulkan: fix gpu-visible host memory issues (#30468)
* vulkan: fix log spam if host pointer import fails

* exclude MoltenVK from shared memory import

* remove unreachable code, assert instead. Shorten warnings

* disable host memory buffer sharing on AMD due to amdgpu perf issue
2026-10-16 08:29:23 +02:00
Max Krasnyansky b797b9ca81 hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round

Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors

* hex-get-rows: vectorize raw k-quant ukernels

* hex-binary: refactor dma-loop thread functions (no need for macros, etc)

* hex-quant: move/cleanup flat and tiled dequant ukernels

* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights

* hex-get-rows: remove redundant cast

* hex-repack: proper filtering for BF16 that were leaking through during the tests

* hex-repack: streamling tiling rules (broken MM for now)

* hex-repack: fix mmid after tiling updates

* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars

* hex-mm: fix valid_rows calc in various kernels

* hex-mm: remove unused hvx_mm_nx_2d_flat

* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16

* hex-mm: another pass at hvx-tiled f16 matmul

* hex-get-rows: optimize small src1

* hex-get-rows: one more pass at recovering get-rows perf regression

* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py

* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul

* hex-mm: fold tiled_f16 checks into the general flow

* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op

* hex-quant: proper tail handling in new dequant ukernels

* hex-get-rows: remove unused code and cleanup support checks

* hex-mm: cleanup ne1 padding/tiling checks

* hex-mm: cleanup fusion checks for tiled tensors

* hex-repack: add a little can_repack helper to reduce duplication

* hex-repack: never repack host buffers

* hex-repack: factor out repack functions into separate .h and .cpp

* hex-tensor: cleanup tensor ifaces for tiled/padded sizes

* hex-repack: update k-quant tiling comments and cleanup #defines

* hex-dcache: factor out l2/dcache dirty tracking and flushing

* hex-tensor: update docs and remove #defs duplication

* hex-tensor: remove unused/incorrect get_data interface

* hex-get-rows: cleanup vtcm layout build

* tests: disable huge MM tests

* hex-dcache: add missing header
2026-10-15 20:30:11 -07:00
Gaurav Garg b6334e37e6 Limit max CUDA pinned host buffer size to 4 GB (#30449) 2026-10-16 08:20:26 +05:30
Frosty40 e707790c73 sycl: support reordered Q8_0 MoE weights (#29337)
Preserve the existing MoE accumulation order.

Assisted-by: Codex
Assisted-by: Angel
2026-10-15 21:59:38 -04:00
Georgi Gerganov 4ed1c8507e ggml : bump version to 0.27.0 2026-10-15 16:20:14 +03:00
losewayy 47f5a64ae1 vulkan : fix concurrent first device init (#29766)
* vulkan : fix concurrent first device init

* tests : load models in parallel in test-thread-safety

Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test.

* vulkan : create fence before retaining device in buffer context
2026-10-15 10:37:35 +02:00
lhezandHongqiang Wang 9166ceaf16 opencl: fix OOB tail access for adreno noshuffle gemv kernels (#30421)
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-15 10:21:18 +03:00
Adrien Gallouët 23f83e056d ggml-cpu : optimize pad_f32 (#30526)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-15 09:13:18 +02:00
lhez 77c1f39929 opencl: apply flash attn softcap before mask (#30546) 2026-10-15 10:11:51 +03:00