llama.cpp-xrip/BENCH_JOURNAL.md

Benchmark journal — SM75 / Ivy Bridge box

Running record of every measurement, the exact command behind it, and what it showed. Purpose: be able to attribute each gain to a specific cause.

Hardware / software baseline

GPU	NVIDIA CMP 50HX, cc 7.5 (Turing), 10240 MiB
CPU	2× Xeon E5-2670 v2 (Ivy Bridge), 2 NUMA nodes × 10 cores, 40 threads
CPU ISA	AVX yes, AVX2 no — so the fork's AVX1 paths are the live path
RAM	62 GiB (32 GiB per node)
CUDA	13.3, CCCL 3.3.4 (CUB_VERSION 300304)
Driver	610.43.03

Clock note: nvidia-smi -lgc is accepted but ignored on this card — it reports success and still boosts to ~2100 MHz. No measurement here is clock-pinned. Long-running cases are reliable only because they reach steady boost.

Builds under test

tag	tree	commit	cuBLAS
UP	/home/xrip/upstream-pristine	f114f91f9 (pristine ggml-org)	off
UPc	same	same	on
US	/home/xrip/llama.cpp	d5919ca1f (our master)	off
USc	same	same	on

Both at the same upstream base, so differences are the fork's work only.

cmake -B build-cublas -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=75 -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DLLAMA_CURL=OFF -DGGML_CCACHE=ON -DGGML_CUDA_FORCE_CUBLAS=ON

---

1. Dense 9B, fully on GPU — Qwen3.5-9B-UD-Q4_K_XL

llama-server -m Qwen3.5-9B-UD-Q4_K_XL.gguf -ngl 99 \
  -b 2048 -ub 2048 -c 17408 --host 127.0.0.1 --port 8099 -t 8 --no-warmup
# 16384-token prompt sent as an exact token-id array, cache_prompt=false,
# greedy (temp 0, top_k 1, seed 1234), 3 reps, median.

build	prefill t/s	tg t/s
UPc	2234.12	66.24
USc	2236.98	67.00
UP	1789.13	66.13
US	1955.07	66.92

Attribution

• -DGGML_CUDA_FORCE_CUBLAS=ON: +24.9% prefill (UP→UPc, 1789→2234). This is a stock upstream CMake option, not fork code. Upstream gets it too.
• Fork vs upstream, cuBLAS config: prefill +0.13%, tg +1.14%.
• Fork vs upstream, default config: prefill +9.27% (the sm75_ffn dispatch block), tg +1.19%. Subsumed by FORCE_CUBLAS, not additive with it.
• llama-bench pp8 (default cfg only): +47.7% — the Turing mmvq threshold work. Guarded off under FORCE_CUBLAS, so it does not reach production builds.

Caveat: this model is fully GPU-resident, so no CPU kernel ran. This test says nothing about the AVX1/NUMA work.

---

2. MoE 35B — Qwen3.6-35B-A3B-UD-Q4_K_XL (22.85 GB, experts on CPU)

GGML_CUDA_GRAPH_OPT=1 CUDA_SCALE_LAUNCH_QUEUES=4x \
numactl --interleave=all llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --temp 0.0 --jinja --no-mmproj-offload \
  -c 64000 -np 1 -fa on -ctk q4_0 -ctv q8_0 \
  --threads 20 --threads-batch 20 -b 2048 -ub 2048 \
  --fit on --fit-target 256 --load-mode none \
  --host 127.0.0.1 --port 8099
# 2048-token prompt, n_predict=128, ignore_eos=true, cache_prompt=false

2a. Methodology correction (important)

moe3 and moe4 gave opposite signs for the same binaries:

run	upstream tg	ours tg	apparent verdict
moe3	22.81	21.35	ours −6.4%
moe4	20.90	23.21	ours +11.0%

Same upstream binary swung 8.4% between runs while reps within a run agreed to <1%. Reps inside one load share NUMA page placement, page cache and thermal state, so they are correlated and prove nothing. Alternate arms across separate model loads. Two earlier conclusions drawn from within-run reps were withdrawn.

Also: ignore_eos: true is mandatory. Without it a synthetic random-token prompt at temp 1.0 hits EOS after 2–63 tokens and "tg" is startup overhead.

2b. Fork vs upstream — 4 alternating loads per arm

arm	pp median	pp spread	tg median	tg spread
upstream	166.52	0.4%	21.37	1.8%
ours	167.02	0.7%	22.90	0.4%
ours + --threads 40 -Cr 0-39 --cpu-strict 1	173.26	1.1%	20.39	1.3%

• Fork tg +7.1% vs upstream — reproducible, spreads far below the effect. This is the AVX1 Q4_K path in repack.cpp paying off. (48 q4_K mentions in the CPU diff; an earlier grep for the enum GGML_TYPE_Q4_K missed them.)
• Prefill: ~0%.
• 40 threads: prefill +3.7%, tg −11.0% — 40 threads means hyperthreads on a 2×10-core box. Prefill is compute-bound and gains; generation is memory-latency-bound and loses.

2c. CPU profile during generation (perf record -F 499 -g)

%	symbol	class
18.3	rep_movs_alternative (kernel)	fread of model file
13.4	ggml_vec_dot_q4_K_q8_K	matmul
~36.5	libgomp (barrier spin)	thread sync
7.7	ggml_vec_dot_q5_K_q8_K	matmul
4.2	clear_page_erms (kernel)	page zeroing
1.6	tinyBLAS Q8_0 gemm	matmul
1.0	ggml_vec_dot_q6_K_q8_K	matmul

Only ~24% of generation CPU time is actual matmul. Call graph attributes the 18.3%:

llama_file::impl::read_raw_unsafe() -> _IO_fread -> read()
  -> ext4_file_read_iter -> filemap_read -> copy_page_to_iter
    -> rep_movs_alternative

i.e. page-cache→userspace copies of model data, consistent with --load-mode none (mmap disabled). Note no tensor in this model exceeds 1 GiB, so --lazy-mode auto (threshold 4 GiB) never triggers — lazy mode is not involved. --lazy-mode on-direct (PR #28136) targets the much larger Qwen3.8-Flash and is not applicable here; it is also not in our master (it lives on perf/numa-dense-20260902).

Implication: tuning ggml_vec_dot_q4_K_q8_K by 20% would move the total by ~2.7%. The sync and I/O costs are far larger targets.

2d. Thread / OpenMP sweep (3 alternating loads per arm)

Base = --threads 20 --threads-batch 20.

arm	pp median	tg median	Δpp	Δtg
t20-b20 (base)	166.95	22.93	—	—
t20-b40	173.44	22.38	+3.9%	−2.4%
t10-b40	174.31	19.09	+4.4%	−16.7%
t20 + OMP_WAIT_POLICY=passive	172.94	12.13	+3.6%	−47.1%
t20-b40 + OMP_PROC_BIND=close OMP_PLACES=cores	174.64	22.95	+4.6%	+0.1%

• OMP_WAIT_POLICY=passive is catastrophic (−47% tg). The libgomp spinning is load-bearing: MoE graphs have many small nodes, so barrier wake-up latency costs far more than spinning. The 36.5% is largely inherent, not recoverable by configuration. (This refuted the hypothesis that motivated the sweep.)
• OMP_PROC_BIND=close + OMP_PLACES=cores is a free +4.6% prefill at no tg cost — stops thread migration across the two NUMA nodes.

2e. Best known MoE launch so far

GGML_CUDA_GRAPH_OPT=1 CUDA_SCALE_LAUNCH_QUEUES=4x \
OMP_PROC_BIND=close OMP_PLACES=cores \
numactl --interleave=all llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --temp 0.0 --jinja --no-mmproj-offload \
  -c 64000 -np 1 -fa on -ctk q4_0 -ctv q8_0 \
  --threads 20 --threads-batch 40 -b 2048 -ub 2048 \
  --fit on --fit-target 256 \
  --host 0.0.0.0 --port 8080

vs the original --threads 20 --threads-batch 20 --load-mode none: prefill +4.6%, tg unchanged.

---

3. Open / in flight

• --load-mode sweep (none vs auto vs mmap vs mmap+mlock) — testing whether mmap removes the 18.3% fread cost. Running.
• SM75 IQ2_XXS decode reuse — implemented (f39c81415), not yet built or measured. Load-widening is impossible for IQ2_XXS (66-byte block, qs at offset 2, only 2-byte alignment), so only decode reuse applies.
• IQ2_XXS Qwen3.8-27B-UD-IQ2_XXS downloaded (6.77 GB). Prediction on record: ~0% from the fork, since there is no IQ2 code in it and at 7.27 GB the model is fully GPU-resident.

4. Method rules learned here

1. Alternate arms across model loads, not reps within a load (§2a).
2. ignore_eos: true for any tg measurement with synthetic prompts.
3. Check the GPU is idle with pgrep -x llama-server (name match — pgrep -f matches other command lines that merely mention the binary).
4. A stray server holding VRAM silently invalidates results — verify 0 MiB.
5. Prefer test-backend-ops perf (thousands of iterations) over llama-bench at tiny sizes, where clock ramp gives ±7–17% error bars.

---

5. --load-mode sweep — mmap is worse, none is correct

3 alternating loads per arm, best OpenMP settings applied to all arms (OMP_PROC_BIND=close OMP_PLACES=cores, --threads 20 --threads-batch 40).

load-mode	load s	pp median	tg median	Δpp	Δtg
none (what you run)	29	174.11	23.10	—	—
auto	19	165.11	21.44	−5.2%	−7.2%
mmap	19	165.38	20.78	−5.0%	−10.1%
mmap+mlock	19	163.62	21.30	−6.0%	−7.8%

--load-mode none wins; mmap costs 5–10%. Cause is NUMA: numactl --interleave=all only stripes anonymous memory. With none the weights are malloc'd buffers and get interleaved across both nodes; with mmap the pages are page-cache-backed and land wherever the kernel put them, so a large share of expert reads become remote-node accesses on this 2-socket box. mmap loads ~10 s faster and then runs slower.

This also refutes the §2c reading of the 18.3% fread. If that cost were in the generation steady state, mmap (which has no read syscalls) would have helped. It hurt. The v1 profile window (20 s after issuing the request) most likely caught first-touch page population rather than steady-state generation. Re-profiling with two warmups and a 45 s offset before recording; §2c should be treated as unreliable until that lands.

6. Hypotheses tested and rejected (kept so they are not retried)

hypothesis	expected	measured	verdict
OMP_WAIT_POLICY=passive recovers the 36% libgomp spin	large tg gain	−47.1% tg	rejected — spinning is load-bearing
mmap removes the 18% fread cost	tg gain	−10.1% tg	rejected — NUMA interleave beats mmap
fewer threads help the memory-bound tg	tg gain	−16.7% (t10)	rejected
--lazy-mode is causing on-demand reads	—	no tensor >1 GiB, threshold is 4 GiB	not applicable
PR 28366 (radix top-k) affects our prefill	prefill gain	compiled out (CCCL 3.3.4); our models never call TOP_K	no-op here
fork's Q4_K CPU work absent	—	48 q4_K hits in CPU diff	wrong — grep was for the enum name

7. Confirmed gains, ranked by size

#	change	workload	gain	whose
1	-DGGML_CUDA_FORCE_CUBLAS=ON	dense 9B prefill	+24.9%	upstream option
2	fork AVX1 Q4_K CPU path	MoE tg	+7.1%	fork
3	OMP_PROC_BIND=close OMP_PLACES=cores + --threads-batch 40	MoE prefill	+4.6%	launch flags
4	Turing mmvq threshold tuning	dense pp8, non-cuBLAS only	+47.7%	this session
5	upstream branchless Q4_K/Q5_K unpack	mmvq kernel n≥5	+4–10%	upstream
6	fork sm75_ffn cuBLAS dispatch	dense 9B prefill, non-cuBLAS only	+9.3%	fork (subsumed by #1)

Note #4 and #6 apply only to builds without FORCE_CUBLAS, so they do not reach the production configuration.

---

8. SM75 IQ2_XXS decode reuse — implemented, measured, rejected

Applied the technique that works for IQ4_XS (sm75_iq4_decode): hoist the weight-side decode out of the ncols_dst loop so the four iq2xxs_grid lookups and sign expansion happen once per block instead of once per output column. Gated to Turing, ncols_dst 2..4.

Load widening — the other IQ4_XS trick — is impossible here:

type	block size	qs offset	widening
IQ4_XS	136 B	8	int2 loads OK (existing fork path)
IQ2_XXS	66 B	2	only 2-byte aligned; get_int_b2 is required

66 is 2 mod 4, so block bases are not even 4-byte aligned.

Result, test-backend-ops perf -p type_a=iq2_xxs, medians of 5 reps after a discarded warmup, interleaved, FORCE_CUBLAS builds:

n	base us	exp us	delta
1	94.19	94.19	+0.00%	control, identical code
2	97.09	96.90	−0.20%	reuse active
3	118.96	119.73	+0.65%	reuse active
4	145.49	146.02	+0.36%	reuse active
5	157.10	157.33	+0.15%
8	221.22	221.06	−0.07%
512	1589.06	1583.31	−0.36%

n=2..4 mean +0.27%; noise floor −0.07% (identical-code rows). No gain. Correctness passed (2/2 backends).

Why it fails where IQ4_XS succeeds: the decoded state is 8 ints of grid plus scale and d — about double sm75_iq4_decoded — so occupancy drops on Turing, while the work saved is nearly free to begin with: iq2xxs_grid is 256 x 8 B = 2 KB and L1-resident. IQ4_XS reuse eliminates genuinely expensive work; IQ2_XXS reuse eliminates L1 hits.

Reverted in a261f5e95. Implementation was correct; the premise was wrong.

Generalisation for future SM75 work: decode reuse only pays when the decoded state is small and the eliminated work is expensive (cache-missing loads or long dependency chains). Check the grid/table size first — if it fits in L1, reuse will not pay for the register pressure.

---

9. SM75 shared-memory codebooks — +6% tg on a real model

Found by profiling, after two kernel ideas written from plausible reasoning both failed. Nsight works on this card via the CMP unlock (Ubuntu nsight-compute 2022.4.1 + the hash-gated patch); ERR_NVCMPGPU is not a hard limit.

Diagnosis

mul_mat_vec_q<IQ2_XXS, ncols=1>, grid 4096 x block (32,4):

metric	value	reading
lg_throttle	19.81/issue	43x the next stall — the bottleneck
wait	1.43
long_scoreboard	0.46	not memory latency
mio_throttle	0.06	shared/MIO pipe idle
L1 hit	98.04%	not a cache problem
DRAM / SM	22.99% / 28.54%	neither saturated
occupancy	92.82%	not warp-starved

The load/store instruction queue is full. A vec_dot issues ~16 loads (4x16-bit from get_int_b2, 4x64-bit codebook, 8x32-bit activations), and a load costs an LG issue slot even when it hits L1.

Fix

Stage the codebook in shared memory once per block: LDG -> LDS, moving that traffic to the idle MIO pipe. Grids are 1-8 KiB and fit easily.

Counters after: lg_throttle 19.81 -> 11.59, mio_throttle 0.06 -> 0.14, DRAM 23% -> 27.9%, SM 28.5% -> 36.1%.

Kernel results (n=1, m=4096, k=14336)

type	delta
iq3_s	-15.7%
iq2_xxs	-13.6%
iq3_xxs	-11.0%
iq2_xs	-10.2%
iq2_s	-9.2%
iq1_s	-5.2%

IQ1_S/IQ1_M are gated to ncols_dst <= 2: they win at n=1/n=2 (-5.2%/-11.1%) and lose at n>=4 (+7.7%/+2.2%) because the 8 KiB iq1s_grid_gpu costs a block of occupancy per SM. IQ2_S is also 8 KiB and wins at every n, so size alone is not the deciding factor.

End to end — llama-server, Qwen3.8-27B-UD-IQ2_XXS, 16384-token prompt

arm	prefill t/s	tg t/s
base	699.17	19.213
shared codebooks	697.15	20.361
	-0.29% (control)	+5.98%

Raw tg, 3 alternating model loads: base 19.448 / 19.213 / 19.144, sgrid 20.455 / 20.361 / 20.290 — distributions do not overlap.

The prefill column is the control: at -ub 2048 prefill goes through cuBLAS, which the change does not touch, so it must stay flat. It does. Only the mmvq path moved. Correctness: all 7 grid types and MUL_MAT_ID pass 2/2 backends.

Two measurement traps hit while doing this

1. Ordering bias. Running all-base-then-all-exp let the first base invocation of each pass catch the 645 MHz idle clock, producing a bogus -78% for iq2_xxs. Alternate base/exp per type and discard a warm-up invocation.
2. Gating on the wrong batch size. IQ1 was first excluded outright on the strength of its n=4/n=8 regression — but token generation is ncols_dst == 1, where IQ1_S is 5.2% faster. Excluding it would have removed a win from the only path that mattered. Check which n the workload actually uses.