* sycl : add DPAS flash attention kernel for quantized KV cache
Decode and small batches (1 to 31 query tokens) now run flash attention
straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or
converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8
DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix
has no s4.
Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts
stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off.
B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K
context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master.
Assisted-by: Claude
* sycl : 192 head dropped, Wildcat Lake added to docs
Assisted-by: Claude
* sycl : stricter checks from review
Assisted-by: Claude
* [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS
Assisted-by: Claude
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps
Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8
* hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap
Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d
* hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes
Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c
* Multi-column on Intel XMX matrix engines refactored
Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).
* Fixed regression bug
real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4
* Removed GGML_TYPE_Q8_0 activation
* rewritten xmx_mul_mat
64-byte alignment and 16-byte pitch check on the activations
* Implemented fused gate/up kernel
One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.
* Removed the temporary XMX_NOGLU environment switch
* Q2_K min-sum optimization
Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group
* Added dispatch check and GRF sweep
tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.
* Renamed variables
* Added 2 environment variables
GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.
* Update esimd.hpp
1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.
2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.
The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
* sycl: fix CONV_2D wrong results for non-contiguous input
The kernel indexes the `input` as a contiguous [W, H, C, N] tensor, but
supports_op claimed the op for any layout. A non-contiguous (e.g. `CWHN`
permuted) input was read with the wrong element offsets, silently
producing wrong results.
Gate the CONV_2D op on `input` and `kernel` contiguity, matching the
CUDA backend contract, and assert input contiguity in the op.
`test-backend-ops -b SYCL0 -o CONV_2D`: goes from 16 failures to 0,
none.
```text
2006/2006 tests passed
Backend SYCL0: OK
Backend 2/2: CPU
Skipping
2/2 backends passed
OK
```
Assisted-by: pi (unsloth/Qwen3.8-27B-GGUF:Q4_K_M via llama.cpp)
* sycl : CONV_2D cwhn=1 support
Make the SYCL backend compute `GGML_OP_CONV_2D` correctly for `cwhn=1` (CWHN
channel-order) inputs and kernels, so the previously failing test-
backend-ops CONV_2D cases pass on SYCL instead of being skipped or
miscomputed.
- Implement CWHN layout support in `ggml/src/ggml-sycl/conv2d.cpp`
(`ggml_sycl_op_conv2d`).
- Update SYCL `supports_op` for `GGML_OP_CONV_2D` in
`ggml/src/ggml-sycl/ggml-sycl.cpp` to accept both layouts.
Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF via llama.cpp)
* sycl docs : update ops for SYCL
* hex-build: bump toolchain, enable v85
* hex-build: bump toolchain to v0.8 in the CI and build scripts
* hex-build: add a little python script for swiv needed on some iot devices
* hex-build: fix flake8 warnings
* hex-arch: update version cap to v85
* hex-inspect: flexible TOOLS_ROOT
* hex-build: update to snake_case naming
* hex-build: update leftover old toolchain refs
* hex-target: update build and run scripts to support ubuntu with adb transport
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round
Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors
* hex-get-rows: vectorize raw k-quant ukernels
* hex-binary: refactor dma-loop thread functions (no need for macros, etc)
* hex-quant: move/cleanup flat and tiled dequant ukernels
* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights
* hex-get-rows: remove redundant cast
* hex-repack: proper filtering for BF16 that were leaking through during the tests
* hex-repack: streamling tiling rules (broken MM for now)
* hex-repack: fix mmid after tiling updates
* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars
* hex-mm: fix valid_rows calc in various kernels
* hex-mm: remove unused hvx_mm_nx_2d_flat
* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16
* hex-mm: another pass at hvx-tiled f16 matmul
* hex-get-rows: optimize small src1
* hex-get-rows: one more pass at recovering get-rows perf regression
* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py
* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul
* hex-mm: fold tiled_f16 checks into the general flow
* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op
* hex-quant: proper tail handling in new dequant ukernels
* hex-get-rows: remove unused code and cleanup support checks
* hex-mm: cleanup ne1 padding/tiling checks
* hex-mm: cleanup fusion checks for tiled tensors
* hex-repack: add a little can_repack helper to reduce duplication
* hex-repack: never repack host buffers
* hex-repack: factor out repack functions into separate .h and .cpp
* hex-tensor: cleanup tensor ifaces for tiled/padded sizes
* hex-repack: update k-quant tiling comments and cleanup #defines
* hex-dcache: factor out l2/dcache dirty tracking and flushing
* hex-tensor: update docs and remove #defs duplication
* hex-tensor: remove unused/incorrect get_data interface
* hex-get-rows: cleanup vtcm layout build
* tests: disable huge MM tests
* hex-dcache: add missing header
* common : add --spec-draft-n-ubatch
Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).
The default 0 keeps the current behavior of following --n-ubatch.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
[no ci]
Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately
* hexagon: add configurable HTP DCVS/HMX power mode
* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)
* hexagon: add --hex-power-mode flag to run.py
Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.
* hexagon: keep HTP awake for DCVS power profiles
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* bugfix: infinite recursion caused by a tool named 'call'(#29967)
* chat : use index for schema and argument rules
* tests : remove tests
* tests : add expect_rules to peg test parser
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
* hex-cpy: replace more paths with dma and simplify l2flush
* hex-cpy: use DMA in all sametype paths
* hex-cpy: rewrite the rest of the copy paths (diff type) to use dma
* hex-concat: use dma for multi-dev path which also removes the need for l2-line alignment
* hex-concat: proper support for mdev splitting
* hex-concat: cleanup ctx and kern params usage
* hex-cpy: cleanup contex and remove left-over non-dma checks
* hex-cpy: clean dma_cpy naming
* hex-cpy: proper kernel params and kernel selection
* hex-build: resolve left-over rebase conflicts
* hex-cpy: update dev guide to clarify 128 byte alignment requirement
* hex-concat: make sure we go through mdev barrier
* hex-concat: make sure to flush dma-queue
* hex-cpy/concat: cleanup kparams and vtcm layout handling
* hex-cpy: remove dead check for contig (routed to diff kernel) and update comments
* hex-dev: update developer guide based on latest changes
* hex-cpy: safe skip of noop copies
* hex-dup: route DUP to CPY
The XIELU CUDA kernel template is already generic over the element
type; only the F32/F16 type assertion and the else-if dispatch were
missing. Add the nv_bfloat16 branch to the launcher, and drop the
temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU.
test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]).
docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row
flips from no to yes as well, i.e. the previous record was stale.
Tested:
- Mac CPU: xIELU F32/F16/BF16, 6/6
- Mac Metal: existing F32/F16, 4/4; BF16 still unsupported
- RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6
- RTX 4090 CUDA BF16-only: 2/2
- git diff --check passes
* ggml-openvino: skip unselected graph branches and support DUP
Upstream #29622 adds a mixed token/embd branch to every input
embedding graph through ggml_build_forward_select(). Its nodes are
not flagged for compute, but the backend translated them anyway,
and the DUP in that branch was unsupported, so the scheduler split
the graph and passed the embeddings across the split with a fixed
token count. The first single-token decode then failed
(test-thread-safety on CPU and GPU).
Build the OV model from the compute nodes only, and translate a
same-type contiguous DUP like CONT so the graph stays on one backend.
* ggml-openvino: make inp_scale_rows token dim dynamic
#29622 also moves the per-token embedding scale (gemma3, gemma3n,
gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim
and pad it per chunk on the static (NPU) path.
* ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights
Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU
plugin fails to compile that form for some row counts with "clFinish,
error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on
the MUL_MAT cases added in #29869 (e.g. m=1000, n=2, k=1024). Model
weights use a u4 zero point and are not affected.
Report these cases as unsupported on GPU until the plugin is fixed.
Op tests check support before allocating, so the check matches unbound
weights only; model loading probes with a dummy buffer and keeps its
weights on the GPU.
* ggml-openvino: create FILL in the output type
translate_fill always built an f32 constant, so an f16 FILL produced
f32 data and the copy back overran the f16 output buffer. Use the
output type for the constant.
* ggml-openvino: reject CONCAT with a quantized type
Quantized inputs are dequantized when translated, so the backend cannot
write a quantized CONCAT output. Report it as unsupported, as for CPY
to a quantized type.
* ggml-openvino: handle the single recurrent state gather of build_rs
#29856 changed build_rs to gather all recurrent states with one GET_ROWS
on the s_copy leaf and take the ubatch and extra states as views of it.
The stateful path matched only the previous form, a GET_ROWS per view of
s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU
("is_axis_valid(axis, r)" in a Concat).
For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the
active-state gather, keep the rank-4 layout of reshapes that read a view
of it, and map the copy of the empty extra-state view to the single-slot
remainder writeback. Do not warn about the dynamic dim of empty views.
* openvino: align eltwise operand ranks to work around a GPU-plugin defect
* openvino: match the MoE fusion on the rank-3 stateful graph
* ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks
The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract
whose operand ranks differ. In gemma-3 the lower-rank operand of the
post-attention residual add is the norm output, and unsqueezing it makes
the GPU plugin compute the layer wrongly: gemma-3 returns empty answers
on GPU with stateful execution.
Skip the rewrite when the lower-rank operand is an RMS norm output.
* docs : update OpenVINO validated models
---------
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
* ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60
On the SpacemiT X60, IME matrix acceleration only covered Q4_0/Q4_1/Q4_K.
Q8_0 had no IME1 kernel, and since the SpacemiT build sets
GGML_CPU_REPACK=OFF there was no repack path compiled in either, so Q8_0
had no accelerated path at all and ran roughly ten times slower than
Q4_0 for prefill on the same board.
- add make_block_q8_0x16 and the Q8_0 repack entry: interleave the
weights into the 16-column layout the IME1 vmadot sequence expects
- add ime1::gemm_kernel_i8i8, an int8 x int8 IME1 kernel with a
single-row and a 4-row A path; the 4-row path loads each B panel once
and reuses it across 4 rows of A
- add quantize_a_4row_i8 for the 4-row activation quantization
- wire both into forward_mul_mat and the repack factory for Q8_0
- docs: mark Q8_0 as supported on X60
Correctness was checked against a quant-exact integer reference for
K = 32 up to 4096, with a max relative error of about 1e-6, and by
checking that generation stays coherent across several prompts.
Tested on Milk-V Jupiter (SpacemiT X60), Bianbu 2.1.1, gcc 14.2, with
Qwen2.5-0.5B-Instruct Q8_0. llama-bench -t 4 under taskset -c 0-3, 5
repetitions on an idle board: pp128 goes from 10.70 to 93.87 t/s. Q4_0
is unchanged at 106.40 -> 107.51 t/s, as expected since this does not
touch that path.
* ggml-cpu : move q8_0_16x32 decl to IME1 section
* ggml-cpu : align q8_0 IME1 kernel assignments
* ggml-openvino : Qwen3.5 MoE perf (#312)
Squash of ravi9/llama.cpp#312:
- ggml-openvino: add detailed inference profiling (Yu, Zijun)
- ggml-openvino: use remote output tensors by default (Yu, Zijun)
- ggml-openvino: optimize single-sequence recurrent state (Yu, Zijun)
- opt1: remove recurrent reset for single sequence, opt2: direct gdn outputs (break parallel sequence) (Yu, Zijun)
- fix parallel sequences (Yu, Zijun)
- ggml-openvino: simplify graph cache key (ynimmaga)
- enable stateful for qwen35 single sequence (Yu, Zijun)
- Fix after rebasing (Yu, Zijun)
- Add k-requant option q4_asym64 (Yu, Zijun)
- Fix qwen35 llama-bench -p 0 (Yu, Zijun)
- Simplify RESHAPE translation (Yu, Zijun)
- openvino: fuse MoE routing (Yu, Zijun)
- openvino: fuse GDN qk normalization (Yu, Zijun)
- openvino: enable GPU MoE fusion by default (Yu, Zijun)
- ggml-openvino: add cache_only mode to import cached compiled model on disk directly (Yu, Zijun)
- openvino : report the device allocation limit to ggml (Łukasz Ślusarczyk)
- Fix windows build (Yu, Zijun)
Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com>
Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
* ggml-openvino: Update doc of compiled model cache
* openvino: implement PRD-compliant device enumeration and memory reporting
* openvino: fix multi-device listing issues from review
- Only the device selected by GGML_OPENVINO_DEVICE reports as GPU; the
other OpenVINO devices report as IGPU so llama.cpp does not offload to
them. Initializing a non-selected device logs a warning.
- Name devices OPENVINO<i> again and show the OpenVINO id in the
description. Raw "CPU" names shadowed the ggml CPU backend.
- Support GPU.N: create the OpenCL queue on OpenVINO's own context for
the selected device, and replace "GPU"/"NPU" string comparisons with
ggml_openvino_is_gpu()/ggml_openvino_is_npu().
- An unavailable GGML_OPENVINO_DEVICE is now an error that lists the
available devices, instead of silently falling back to CPU.
- Memory: cap iGPU/NPU free memory at system available memory, fall back
to system memory instead of 0/0 when the plugin lacks memory
properties, and ignore host USM allocations in GPU usage.
- Initialize the device config once under a lock, even if OpenCL setup
fails.
- Fix supports_op return type for non-selected devices (build error).
* openvino : take USM entry points from the selected device platform
clGetExtensionFunctionAddressForPlatform was called on the first platform
returned by clGetPlatformIDs. The address it returns is only valid for the
platform it was queried on, and the first platform is not always the one that
holds the device OpenVINO selected.
On a host whose first platform comes from another vendor the lookup returns
null, and then every read, write and memset on a GPU buffer fails with
"clEnqueueMemcpyINTEL not available".
Look both entry points up in init(), on the platform of the device OpenVINO
picked, and keep them in the device config next to the command queue.
Assisted-by: Claude Opus 5
* openvino: fuse MoE experts for models with a fused gate_up weight
FuseMoeCompressed only matches models whose gate and up projections are
separate GatherMatmul ops. gemma-4 packs both into one expert weight and
splits the result after the GEMM, so its MoE block stayed unfused and ran
the expert GEMMs as per-token GEMVs.
Add FuseMoeCompressedFusedGateUp, which matches that shape
(one GatherMatmul -> Slice/Slice -> Gelu(ERF) -> Multiply) and folds it into
the same MOECompressed op, using GEMM3_SWIGLU with GEGLU_ERF. The fused
weight, scale and zero point are split into gate/up halves by copying raw
bytes, since a graph Slice would be rewritten to StridedSlice and constant
folded, whose reference evaluator crashes on sub-byte types.
gemma-4 also applies a per-expert output scale to the down projection before
the router weights. MOECompressed takes only one per-expert weight, so that
scale is folded into the routing weights, which is exact.
The op reads the zero point straight off a weight port and needs an integer
Constant there, so the matcher requires one and leaves natively quantized
experts (exact f16 zp) to the unfused path.
gemma-4-26B-A4B on Arc B390, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all,
llama-bench -p 512 -n 128 -r 2, against a GGML_OPENVINO_MOE_OP=0 baseline:
pp512 66.16 -> 1608.73 t/s, tg128 25.94 -> 26.46 t/s. Perplexity over 12
chunks is unchanged (1451.3 +/- 177.9 unfused vs 1427.6 +/- 175.1 fused).
No effect without that requant option, on models with separate gate/up
weights, or on CPU. test-backend-ops -b OPENVINO0 is unchanged by this
commit: two MUL_MAT_ID m_v cases fail, the same two on the unmodified base.
* openvino: fix rank-3 axis handling so MoE works under stateful execution
Stateful execution drops the leading size-1 batch dim, so OV tensors are rank
3 while GgmlOvDecoder::get_shape/get_stride still report GGML_MAX_DIMS=4
reversed entries. Several MoE ops derive OV axis indices straight from that
metadata, so they picked the wrong axis. A MoE model with
GGML_OPENVINO_STATEFUL_EXECUTION=1 aborts while building the graph:
Check 'is_axis_valid(axis, r)' failed at src/core/src/validation_util.cpp:336
While validating node 'opset11::TopK ... _ffn_moe_probs ...'
Axis 3 out of the tensor rank range [-3, 2].
Fix idiom throughout: take the axis from the real OV rank, or shift a
metadata-derived axis down by metadata_rank - actual_rank.
argsort.cpp the router top-k axis is 2 on rank 3, not 3. This is the
abort quoted above.
add.cpp the MoE expert-sum bypass collapses the 8-ADD chain into one
ReduceSum on hardcoded axis 2, which on rank 3 reduces n_embd
instead of the expert axis. Now rank-2, with the following
Unsqueeze at rank-3.
get_rows.cpp squeezing a hardcoded {0,1} also strips the batch dim
whenever it is 1, which is every decode step. Squeeze down to
the trailing two dims instead.
mul_mat_id.cpp pick the reshape dims by actual rank, and skip the trailing
Unsqueeze that re-adds the batch dim.
view.cpp the expert-plane slice had the Slice axis, dst_ov_axis, the
ShapeOf+Gather index and the Reshape target all rank-4.
utils.cpp process_view_input_new's "translate_view already resolved
this VIEW, skip re-slicing" shortcut required equal ranks. 4
vs 3 never matched, so every resolved expert plane got
re-sliced. Now compares the common trailing dims. Same axis
shift for the Slice in the view-chain walker.
Stateless is unchanged by construction: every edit is gated on the actual
rank, so axis_shift == 0 reproduces the previous code exactly. Checked on
OV-CPU by diffing greedy output against the unmodified base for dense
gemma-4-E2B, granite-1b-a400m and gemma-4-26B-A4B; all identical.
granite-1b-a400m on OV-CPU aborts with the error above before this change;
after it, it generates and is byte-identical to stateless. Dense gemma-4-E2B
is identical stateless vs stateful both before and after. test-backend-ops
-b OPENVINO0 is unchanged: two pre-existing MUL_MAT_ID m_v cases fail, the
same two on the unmodified base.
gemma-4-26B-A4B is a poor correctness vehicle here. On OV it already drifts
into degenerate repetition a few tokens in, in stateless as much as stateful,
and the two modes diverge somewhere inside that degenerate region instead of
matching token for token. Each mode is self-reproducible across runs.
Known limitation: FuseMoeCompressedFusedGateUp does not match the rank-3
graph, so a MoE model run with GGML_OPENVINO_STATEFUL_EXECUTION=1 loses the
prefill fusion while gaining decode. gemma-4-26B-A4B on Arc B390,
GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all, llama-bench -p 512 -n 128 -r 2:
unfused (GGML_OPENVINO_MOE_OP=0) pp512 66.16 tg128 25.94
fused, stateless (default) pp512 1608.73 tg128 26.46
fused, stateful pp512 66.18 tg128 29.91
Stateful is opt-in and off by default, and MoE did not run there at all
before this, so nothing that previously worked regresses. Making the pass
match rank 3 is the follow-up.
* OpenVINO Backend: Upgrade graph cache to use node_idx, src_idx, node type
* ggml-openvino : enable more comprehensive conv fusion
* enable conv ops
* Reject kernel size 0 and support IM2COL_3D
* openvino : abort when the GPU remote context cannot be created
init() logged the error and returned, which left the device name a GPU but
remote_context empty. The remote buffer and tensor paths assert only on the
device being a GPU and then dereference that empty optional.
Those paths have no host fallback, and a device that OpenVINO listed should
have a working OpenCL context, so stop instead of continuing. An OpenCL stack
that is broken as a whole is still caught earlier by the device availability
check, which falls back to CPU.
Assisted-by: Claude Opus 5
* openvino : fix build warnings
The single-argument form of the OpenVINO RTTI macros is the intended one, but
their selector macro leaves __VA_ARGS__ empty, which -Wpedantic reports on
every pass and op header. Turn that warning off for this backend only, the
way ggml-cuda and ggml-sycl already do for their own third-party warnings.
Also drop a break and a dead assignment around a GGML_ABORT, which is noreturn.
Assisted-by: Claude Opus 5
* OpenVINO Backend: Support common MTMD ops
* ggml-openvino: give a reshaping view its own ov::Tensor
* ggml-openvino : compute HARDSIGMOID and EXPM1 in f32
HARDSIGMOID used a 1/6 constant in the input type, which is not exact
in bf16, and EXPM1 lost precision for small inputs in f16. Both now
compute in f32 and convert back, except on NPU where the f32 path
gives wrong results.
Fixes the HARDSIGMOID/EXPM1 test-backend-ops failures on GPU.
* ggml-openvino : update device selection and --list-devices
Show the selecting GGML_OPENVINO_DEVICE value and active device in
--list-devices, startup logs, and backend tests.
Clarify OpenVINO selection uses GGML_OPENVINO_DEVICE, not -dev.
* openvino : remove unreachable OpenCL queue checks
A remote buffer exists only on a GPU device, and init() aborts there if the
queue cannot be created, so the queue is never null at these call sites.
Assisted-by: Claude Opus 5
* openvino : update OpenVINO to 2026.4.1 and GPU drivers to 26.35.39758.10
* docs : update OpenVINO validated models and GPU driver version
* ggml-openvino : skip empty views when giving a reshaping view its own tensor
A zero-size view can sit at the end of a GPU USM buffer (Qwen3.5 recurrent cache). Wrapping it as a remote tensor throws "shared USM buffer has smaller size (0)".
Assisted-by: Claude
* ggml-openvino : rebind the cached decoder when llama passes a different graph
llama keeps separate graphs for batches with and without outputs. llama-server splits the prompt into chunks for context checkpoints, so a cached decoder could be reused with a graph built in other memory and bind the previous chunk's input tensors. SWA and recurrent models then lost most of the prompt in llama-cli and llama-server.
Assisted-by: Claude
* docs : update OpenVINO validated models
Smoke test on Lunar Lake (32 GB) with the two fixes above. Re-add the Qwen3.5 and gemma models.
Assisted-by: Claude
---------
Co-authored-by: Yu, Zijun <zijun.yu@intel.com>
Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com>
Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Co-authored-by: haarika-madaka <haarika.madaka@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
* init conversion
* convert: ok
* model loaded
* add server code
* improve conversion script
* support shared prompt prefix
* add docs, imorove UX a bit
* add vision support
* add openjev tiny model for testing
* add dev docs
* support lev & kev
* clean up
* fix lev noul
* fix py lint
* nits docs
* clarify about not supporting date_facts
* Adding wide-load mmvq for Q8_0 and esimd dmmv for q8_0
Assisted-by: Codex
* remove guard for q8_0
* remove docs
* Simplify by committing to clean code without fallback
* Add feature flag as requested
Assisted-by: Claude Opus 5
---------
Co-authored-by: cwriter <cwriter@localhost>
Pinning to >= 3.4.3 is required to enable DeviceTopK, which was affected by
a race condition https://github.com/NVIDIA/cccl/pull/10627.
We will relax this for future CTK versions which will bundle CCCL >
3.4.X (CTK 13.5 will bundle CCCL 3.5.0 for example)
* BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS
* AOCL-Blas : Add an AOCL-BLAS Quick Start and drop the fixed version path
* AOCL-BLAS doc : Note on ZenDNN
LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space
after every special token and parallel tool calls are separated by
<|end|>. The GPT-OSS handler rejects this output, so add a dedicated
handler, selected by the chat_format=llm-jp-harmony-v1 declaration in
the chat template.
Assisted-by: Claude Fable 5.1
The committed docs/ops/CPU.csv is out of sync with the current
test-backend-ops suite: 11 ops with CPU support (COL2IM_1D,
MUL_MAT_HADAMARD, SWIGLU_CLAMP, MUL_MAT_W4A4/W4A8, MUL_MAT_ID_W4A4/W4A8,
DSV4_HC_COMB/PRE/POST, LIGHTNING_INDEXER) are missing entirely, and
many other ops have fewer test cases than the suite generates now.
docs/ops.md (which CI requires to match the CSVs) therefore
understates CPU support.
Regenerated with:
test-backend-ops support -b CPU --output csv > docs/ops/CPU.csv
scripts/create_ops_docs.py
Note: ADD1 now reads unsupported on CPU because ggml_add1 is
GGML_DEPRECATED and the suite no longer generates test cases for it;
the CPU implementation itself is still present.
Assisted-by: Xing
* ci : update the oneAPI toolkit to 2026.1
oneDNN is removed from Intel Deep Learning Essentials in 2026.0, so
staying on the deep-learning-essentials path would silently lose oneDNN
support when the toolkit version is updated. Switch both the Ubuntu and
Windows CI jobs to the new unified Intel oneAPI Toolkit installer,
which still includes oneDNN (until 2027.0) and keeps the component IDs
unchanged for the Windows install script.
Measured with the same code (b10899) built with oneAPI 2026.1 vs the
2025.3-based release build on Arc B570: prompt processing 1331 vs 434
t/s (3.1x), token generation 50.1 vs 45.3-48.0 t/s.
Assisted-by: GLM (z-ai/glm-5.3-flash)
* docs : update the SYCL backend build requirements for oneAPI 2026.1
With the 2026.0 release the Base toolkit and the HPC toolkit are
combined into the oneAPI Toolkit, and oneDNN is removed from the Deep
Learning Essentials package. Update the install instructions, the
verified release table and the news section accordingly.
Assisted-by: GLM (z-ai/glm-5.3-flash)
* ci : update the release workflow for oneAPI 2026.1 and Level Zero SDK 1.33.1
Align the release package build with the CI build update:
- oneAPI toolkit 2025.3.3 -> 2026.1 (the unified oneAPI Toolkit)
- Level Zero SDK 1.28.2 -> 1.33.1, and the Debian package names
(level-zero/level-zero-devel -> libze1/libze-dev)
- The Windows DLL copy list for the 2026.1 runtime: sycl9.dll and the
.6/.3 MKL library versions
Assisted-by: GLM (z-ai/glm-5.3-flash)
* ci : remove the removed .spv fallback files from the Windows DLL copy list
oneAPI 2026.1 no longer ships libsycl-fallback-bfloat16.spv and
libsycl-native-bfloat16.spv (the OpenCL fallback mechanism changed), so
the copy step failed with exit 1.
Assisted-by: GLM (z-ai/glm-5.3-flash)
* devops : update the oneAPI toolkit image in the Intel Dockerfile
Assisted-by: GLM (z-ai/glm-5.3-flash)
---------
Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
* hex-topk: trying to improve/cleanup the pipeline
* hex-sampling: add STEP op
* hex-sampler: add SUM op
* hex-sampler: update CPY to support sampling cases
* hex-binary: add support for chunking to handle large logits
* hex-argmax: super basic version of ARGMAX
* hex-binary: support for scalars in extended buffers
* hex-binary: fix wrong indexing for dim 1 broadcasts across dim 2 slices
* hex-argsort: fix missing header
* hex-sampler: cleanup dma usage in the sampler related ops, and binary
* hex-build: disable autovectorizer, it is better to use explicit hints for critical loops
* hex-binary: fix perf regression due to is_1d fallback
* hex-ops: update supported ops
xrip
shaofeiqiandLi He
Jesse Symons
Abhijith Jajam
Ehsan Bateni
Max Krasnyansky
Todd Malsbary
budiony
Michael Taylor
Aaron Teo
Mendy Berger
Todor BoinovskiandMax Krasnyansky
Georgi Gerganov
Francisco Herrera
Raffaele Di DonnaandRaffaele Di Donna
Pranav DhinakarandMax Krasnyansky
R0CKSTAR
Ilya Maslennikov
cwriterandcwriter
Frost-54andAlde Rojas
qiao_px
Ravi PanchumarthyandMustafa Cavus
Alan Tseng
Oliver Simons
Pradeep Rao
e-mon
CaramelizedCUDA
Asahi-PrvandAsahi-Prv
Sarah Wu
Trivikram Reddy
