From 27af58168afde4fbc4ce35677c03f3557e76d266 Mon Sep 17 00:00:00 2001 From: Georgi Gerganov Date: Mon, 19 Oct 2026 09:11:38 +0300 Subject: [PATCH] tests : update Metal fusion baseline for beam (#30764) * tests: add beam fusion rows to MTL baseline Regenerate tests/fusion/MTL.csv with test-fusion --record. The newly added beam architecture has no baseline rows, so test-fusion fails with 2 failing rows (MUL+ADD x6, RMS_NORM+MUL x25). All other arch counts are unchanged. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * skills: add update-fusion-table skill Add a skill for regenerating a per-device test-fusion baseline. It keeps the workflow from tests/fusion/README.md and the parts that are easy to get wrong: diff the recorded file before applying it, the distinct meanings of a FAIL row, and never recording a partial run. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) --- skills/update-fusion-table/SKILL.md | 39 +++++++++++++++++++++++++++++ tests/fusion/MTL.csv | 2 ++ 2 files changed, 41 insertions(+) create mode 100644 skills/update-fusion-table/SKILL.md diff --git a/skills/update-fusion-table/SKILL.md b/skills/update-fusion-table/SKILL.md new file mode 100644 index 000000000..4d545418d --- /dev/null +++ b/skills/update-fusion-table/SKILL.md @@ -0,0 +1,39 @@ +--- +name: update-fusion-table +description: Regenerate a per-device test-fusion baseline (tests/fusion/MTL.csv, ...) after fusion patterns or the dummy-model set change. Use when test-fusion --check fails. +--- + +# Update a fusion baseline + +`tests/fusion/.csv` is the `test-fusion` golden baseline: `arch,moe,mode,label,count`, +where `count` is how many times a fusion label matched per graph run. See `tests/fusion/README.md`. + +Re-record only when the counts moved for the reason the contributor intended. A row failing on +`nmse_fus` (> 1e-4) is a fused-vs-unfused correctness bug, not baseline drift - fix it, never +record it away. + +## Steps + +```sh +cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON # enable the target backend +cmake --build build --config Release --target test-llama-archs --target test-fusion -j + +rm -rf build-ci-models && mkdir -p build-ci-models # always regenerate +./build/bin/test-llama-archs -o build-ci-models + +D=MTL0; B=tests/fusion/MTL.csv; T=/tmp/$(basename $B) # keep the target backend/device in sync +./build/bin/test-fusion --models build-ci-models --device $D --check $B # read the FAIL rows +./build/bin/test-fusion --models build-ci-models --device $D --record $T +diff $B $T +cp $T $B +./build/bin/test-fusion --models build-ci-models --device $D --check $B # 0 failed, exit 0 +``` + +- Read the diff before applying it. Every added, removed or changed row must be explainable by the + change under review; an unexplained row means the run picked up something unintended. +- `expected 0` in a FAIL row = no baseline row for that label (new arch, or a fusion newly + firing); non-zero `expected` = count changed; `fusion ... is no longer matched` = row stopped + firing. +- `--record` rewrites the whole file, so never record a partial run (`--model FILE`, a filtered + models dir) - it drops the rows of every architecture that was not run. +- `--record` marks every row `ok`; only the `--check` run validates. diff --git a/tests/fusion/MTL.csv b/tests/fusion/MTL.csv index 9ab1f1f3f..e6cd43a82 100644 --- a/tests/fusion/MTL.csv +++ b/tests/fusion/MTL.csv @@ -20,6 +20,8 @@ bailingmoe3 ,1 ,any ,GATED_DELTA_NET+CPY , 1 bailingmoe3 ,1 ,any ,MUL+ADD , 2 bailingmoe3 ,1 ,any ,RMS_NORM+MUL , 8 bailingmoe3 ,1 ,any ,RMS_NORM+SCALE , 2 +beam ,0 ,any ,MUL+ADD , 6 +beam ,0 ,any ,RMS_NORM+MUL , 25 bloom ,0 ,any ,NORM+MUL+ADD , 6 chatglm ,0 ,any ,RMS_NORM+MUL , 5 codeshell ,0 ,any ,NORM+MUL+ADD , 5