Frosty40
39febd2adb
* sycl: work-group argmax for short TOP_K rows Used angelX to optimize SYCL/drama 3 coordination, results are as follows Rows with ncols <= max work-group size: one work-item per column, k rounds of work-group argmax. Ties take the lower column, NaN is never picked. Longer rows keep the existing paths. Arc Pro B70, oneAPI 2026.1, test-backend-ops perf (existing short-row cases), base9b56ccf00vs patch, median ratio of 6 rotating A/B/A rounds: | ncols | nrows | k | base us | patch us | speedup | |---|---|---|---|---|---| | 16 | 1024 | 1 | 19.97 | 2.88 | 6.95x | | 16 | 1024 | 2 | 23.83 | 2.90 | 8.25x | | 16 | 1024 | 8 | 66.12 | 4.66 | 14.21x | | 16 | 8192 | 1 | 110.42 | 5.96 | 18.51x | | 16 | 8192 | 2 | 130.21 | 8.82 | 14.77x | | 16 | 8192 | 8 | 420.62 | 27.27 | 15.42x | | 128 | 1024 | 1 | 20.49 | 2.94 | 6.97x | | 128 | 1024 | 2 | 25.89 | 3.45 | 7.52x | | 128 | 1024 | 8 | 101.67 | 8.95 | 11.36x | | 128 | 8192 | 1 | 112.19 | 13.29 | 8.44x | | 128 | 8192 | 2 | 141.18 | 20.60 | 6.85x | | 128 | 8192 | 8 | 643.21 | 67.60 | 9.51x | | 1024 | 1024 | 1 | 17.32 | 14.98 | 1.16x | | 1024 | 1024 | 2 | 28.77 | 24.02 | 1.20x | | 1024 | 1024 | 8 | 151.87 | 77.88 | 1.95x | | 1024 | 8192 | 1 | 156.49 | 138.78 | 1.13x | | 1024 | 8192 | 2 | 201.47 | 209.18 | 0.96x | | 1024 | 8192 | 8 | 1006.25 | 632.60 | 1.59x | The 1024 x 8192, k=2 case is 3.7% slower. Correctness: test-backend-ops -o TOP_K 533/533 and -o ARGSORT 98/98 on the patch, again on the rebase onto3f03b1772. Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch. One CPY f16->q2_0 [256,2,3,4] failure was seen once on the patch, and 3/3 re-runs pass on base and patch. All other patch-run failures also fail on base. Assisted-by: Claude Opus 5.5 * made changes as suggested Assisted-by: Claude Opus 5.5
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Quick start
A few options to get llama.cpp installed on your machine:
# curl
curl -LsSf https://llama.app/install.sh | sh
# powershell
irm https://llama.app/install.ps1 | iex
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
Supported backends
| Backend | Target devices |
|---|---|
| BLAS | All |
| BLIS | All |
| CANN | Ascend NPU |
| CUDA | Nvidia GPU |
| HIP | AMD GPU |
| Hexagon | Snapdragon |
| IBM zDNN | IBM Z & LinuxONE |
| MUSA | Moore Threads GPU |
| Metal | Apple Silicon |
| OpenCL | Adreno GPU |
| OpenVINO [In Progress] | Intel CPUs, GPUs, and NPUs |
| RPC | All |
| SYCL | Intel GPU |
| VirtGPU | VirtGPU APIR |
| Vulkan | GPU |
| WebGPU | All |
| ZenDNN | AMD CPU |
Documentation
Tools
Development
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
Acknowledgements
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain