From e14d7024f7c8ddbf46d37acd0e4849caffdf881f Mon Sep 17 00:00:00 2001 From: xrip Date: Wed, 9 Sep 2026 11:37:58 +0300 Subject: [PATCH] sm75: record tests and memory results --- sm75/SHARED-CHAT-PERFORMANCE.md | 214 ++++++++++++++++++++++++++++++ sm75/SHARED-CHAT-TESTS.md | 46 +++++++ sm75/profile-shared-chat.sh | 20 +++ sm75/run-shared-chat-model.py | 92 +++++++++++++ sm75/run-shared-chat-mtp.py | 60 +++++++++ sm75/run-shared-chat-trace.py | 40 ++++++ sm75/shared-chat-mtp-request.json | 1 + 7 files changed, 473 insertions(+) create mode 100644 sm75/SHARED-CHAT-PERFORMANCE.md create mode 100644 sm75/SHARED-CHAT-TESTS.md create mode 100644 sm75/profile-shared-chat.sh create mode 100644 sm75/run-shared-chat-model.py create mode 100644 sm75/run-shared-chat-mtp.py create mode 100644 sm75/run-shared-chat-trace.py create mode 100644 sm75/shared-chat-mtp-request.json diff --git a/sm75/SHARED-CHAT-PERFORMANCE.md b/sm75/SHARED-CHAT-PERFORMANCE.md new file mode 100644 index 000000000..007f514fa --- /dev/null +++ b/sm75/SHARED-CHAT-PERFORMANCE.md @@ -0,0 +1,214 @@ +# SM75 shared-chat performance log + +Started: 2026-09-09. + +## Scope and base + +- Approved order: prefill SwiGLU/Q8_1 fusion, recurrent-state memory savings, shared target/draft compute memory. +- Source under review: BrunoPPassini/llama.cpp at `daa7e974f5b7`. +- Initial local control source was `03acdbdc5e6470470a1eeca11a2c87834dd36cf6`. A live fetch advanced origin/master to `e653dac15`; the isolated experiment was moved to this fresh base before source changes. +- Isolated worktree: `G:/llama/.codex-worktrees/sm75-shared-chat-20260909`. +- Ancestry check passed: this base includes kernel integration `888371054`. +- Main checkout remains at `912eaf4c0`; its existing staged and untracked work is outside this experiment. + +## Hardware and access + +The older two-card reports are not a current hardware check. `PREFILL-EXECUTION.md` records a later user-approved target of one CMP50HX with 10 GiB. Confirm the live test host before choosing model placement. Kernel results on one card and CPU-offloaded model results must be separate from two-card full-GPU results. + +Initial access to `xrip@192.168.1.224` failed with `Permission denied (publickey,password)` in batch mode. The SSH executable is `C:/Program Files/Git/usr/bin/ssh.exe`; `ssh` is not on this session's PATH. No remote state was changed. + +Access resolved with the existing `C:/Users/xr1p/.ssh/servers` key and `IdentitiesOnly=yes`. Live check: one CMP50HX, 10240 MiB total, 0 MiB used. No test/server process found. Production source is clean at `d5919ca1f`; it was not switched or rebuilt. Remote isolated worktree: `/home/xrip/tmp/sm75-shared-chat-20260909`. CUDA 13.3, GCC14, native SM75 Release build started. The idle governor was active at setup. + +## Test rules + +- Keep source SHA, build flags, model identity, actual tensor types, command, device placement, and raw log paths with each result. +- First pass: existing backend numerical tests for the changed operation and boundary shapes. +- Check CUDA Graph replay and the unfused control. A sanitizer failure to start is not a sanitizer pass. +- Use a warmup and repeated paired control/candidate runs; record median and min/max, not just the best run. +- Model tests: actual PP16384, fixed generation budget, no prompt cache reuse, same sampling, and peak VRAM. +- MTP tests: acceptance, partial acceptance, rejection/rollback, MTP2/MTP3, and long-context generation. +- Keep hardware settings fixed. Do not combine speed percentages from separate operations. +- No production promotion without measured benefit and correctness evidence. + +## Experiment journal + +| ID | Change | Build / correctness | Performance | Decision | +|---|---|---|---|---| +| 00 | Control setup | Clean isolated worktree; integration ancestry passed; live single-card check passed | No measurements | Use fresh fork base e653dac15 | +| 01 | SwiGLU to Q8_1 prefill fusion | Corrected build passed; both modes passed 50/50 cases; enabled dispatch trace confirmed; sanitizer unavailable | Fused graph throughput +4.40% to +6.38%; all active timing ranges separate | Whole-model check running | +| 02 | Recurrent-state transaction log | Not implemented | Not measured | After experiment 01 | +| 03 | Shared target/draft compute memory | Not implemented | Not measured | Device count must be confirmed | + +## Design constraints found + +- The source fusion uses fixed FFN width 17408, layer numbers, and Q6_K down projections. Use actual graph consumer type and shape instead. +- The local CUDA backend already has graph fusion checks. Use these checks for a short producer/consumer lifetime; avoid a persistent activation cache. +- MMQ consumes type-dependent Q8_1 layouts. Fusion must match the actual down-projection type and retain the ordinary path when it is not eligible. +- Existing gate/up/GLU decode fusion must retain its current dispatch. +- Fresh fork code already sends large Q4_K/IQ4_XS FFN batches to cuBLAS. The first experiment excludes Q4_K/IQ4_XS N1024-2048 so fusion cannot undo that measured path. +- Experiment 01 is opt-in with `GGML_CUDA_SM75_SWIGLU_Q8_1=1`. It fuses adjacent split-F32 SwiGLU and MMQ using existing graph dependency and memory-range checks. The quantizer uses the actual weight type. No activation cache is added. +- Source shared compute memory requires exactly one CUDA device. Two-device support is a separate change, not an environment-only setting. +- Its context link also requires target and MTP to use the same model object and one sequence each. A separately loaded draft GGUF may not enter this path; confirm the actual context/model link in the chosen MTP run. +- Source transaction logging is restricted to QWEN35, GPU-offloaded F32 recurrent state, and one sequence. Check the actual model architecture before use. + +## Experiment 01 evidence + +- `first-numerical.log` under the remote worktree: 30/30 CUDA numerical cases passed with the feature enabled. Cases cover Q4_K, IQ4_XS, Q6_K, N1/9/32/256/1024, K256/768, and 128/129 output rows. +- Test fixtures reuse `test_mul_mat` in the existing `tests/test-backend-ops.cpp`. They are selected only by `GGML_TEST_SWIGLU_MMQ`. +- The perf fixture times the complete SwiGLU/down graph. It keeps one graph instance per timing iteration, because repeating only the final matmul would add extra consumers and prevent fusion. +- Runner: `sm75/run-shared-chat-kernels.py`. Both modes use the same build. It keeps raw output and telemetry, does one warmup and five paired measured passes, and restores the idle governor's starting state. +- First paired runner attempt stopped after its warmup because CUDA Graph messages split timing lines. This was a parser error, not a kernel error. Raw run: `kernel-results-20260909-070356`. The reader now accepts those line breaks; a new run keeps the failed attempt intact. +- The second attempt (`kernel-results-20260909-070526`) showed near-zero changes. Source inspection found that `ggml_can_fuse` requires equal node shapes, so the SwiGLU/down pair was not fused. That run was stopped; its numerical passes and timings are not evidence for the new kernel. The corrected code uses `ggml_can_fuse_subgraph`, and the numerical runner now requires a dispatch trace before timing. +- The traced run `kernel-results-20260909-071237` passed the backend checks, but its runner counted only unsplit result lines. The reader now uses the backend's exact `30/30 tests passed` summary and separately requires the fusion marker. This second reader fix did not change the CUDA code. + +### Minimal memory-port design + +1. Transaction log: keep the existing persistent recurrent state, save the GDN update operands for the speculative suffix, and replay only its accepted prefix. Keep full snapshots as the default and as the fallback for unsupported model/state/sequence layouts. Do not add phase-arena reuse or lazy replay in this first change. +2. Shared compute: share only compatible target/draft scheduler buffer layouts. Keep a shared owner for the allocation, synchronize the previous context before allocation or input writes, and detach to private storage on growth. Keep the single-device restriction until a two-device test host is available. +3. Test both changes separately with the chosen MTP target and the same quant, placement, KV types, context, and draft length as its control. Record actual physical VRAM savings, not a sum of two logical scheduler size reports. +4. Only combine candidates after separate checks pass. CUDA Graph replay must survive alternating target/draft work and a graph-size change. + +## Experiment 01 kernel results + +Five paired measured passes after warmup. Same build, feature off/on. Units are microseconds per complete SwiGLU/down graph, including activation preparation. Throughput change is `control_time / candidate_time - 1`. + +| Weight | Width | Output rows | Tokens | Control median us | Fused median us | Throughput change | +|---|---:|---:|---:|---:|---:|---:| +| Q4_K | 12288 | 4096 | 256 | 825.53 | 780.60 | +5.76% | +| Q4_K | 17408 | 5120 | 256 | 1358.93 | 1299.24 | +4.59% | +| IQ4_XS | 12288 | 4096 | 256 | 732.94 | 688.98 | +6.38% | +| IQ4_XS | 17408 | 5120 | 256 | 1196.69 | 1135.66 | +5.37% | +| Q6_K | 12288 | 4096 | 256 | 935.83 | 885.70 | +5.66% | +| Q6_K | 17408 | 5120 | 256 | 1541.45 | 1476.55 | +4.40% | +| Q6_K | 12288 | 4096 | 1024 | 3264.06 | 3089.57 | +5.65% | +| Q6_K | 17408 | 5120 | 1024 | 5511.86 | 5270.61 | +4.58% | +| Q6_K | 12288 | 4096 | 2048 | 6532.27 | 6196.89 | +5.41% | +| Q6_K | 17408 | 5120 | 2048 | 11011.22 | 10437.29 | +5.50% | + +All ten active-case min/max ranges are separate. The eight unchanged Q4_K/IQ4_XS cuBLAS controls at N1024/2048 have overlapping ranges; median changes are -0.40% to +0.47%. Do not add these graph gains to older kernel gains or call them whole-model gains. + +Raw results and full min/max ranges: `sm75/shared-chat-results/kernel-results-20260909-071303/`. The same directory remains under the remote worktree. GPU settings were not changed; the idle governor was restored after the run. + +Both installed 9B model files have 15 Q6_K, 5 Q5_K, and 12 Q4_K `ffn_down.weight` tensors, all with width 12288 and 4096 output rows. Thus 20 of 32 down projections can use this path at ubatch 2048; the Q4_K layers retain cuBLAS. Q5_K uses the same Q8_1 scale/sum layout as Q4_K. Extra numerical cases cover Q5_K and Q2_K, which exercises the third layout. + +The expanded suite passed 50/50 with the feature off and 50/50 with it on. CUDA implementation files are unchanged from the timed build; only the test fixture grew. Evidence: remote `numerical-50-off.log` and `numerical-50-on.log`. + +Compute Sanitizer was attempted on one Q6_K N32/K256 case. The numerical case passed, but the tool reported `Error: GPU debugging features are disabled` and exited nonzero. This is unavailable memory-check coverage, not a reported kernel memory violation and not a sanitizer pass. Evidence: `sanitizer.log`. No driver or GPU setting was changed to bypass this limit. + +## Whole-model results and output gate + +Completed control/candidate/control runs are saved in `shared-chat-results/model-results-20260909-072523/`. Each phase has one warmup and three measured requests with the same 16384 input tokens and 128 output tokens. + +| Model | Control before PP tok/s | Fusion PP tok/s | Control after PP tok/s | Result | +| --- | ---: | ---: | ---: | --- | +| Ornith 9B UD-Q4_K_XL | 1960.23 | 1956.10 | 1939.32 | Within control drift | +| Qwen3.5 9B UD-Q4_K_XL | 1934.09 | 1946.02 | 1932.94 | +0.62% to +0.68%, subject to output caveat | + +Peak GPU memory was unchanged: 7045 MiB for Ornith and 7031 MiB for Qwen. No whole-model memory saving is claimed. Decode medians stayed in the 65.87 to 66.37 tok/s range across both models and all phases; no decode gain is claimed. + +The equal-output gate did not pass. All four responses within each phase differ, including both control phases, despite temperature 0 and seed 1234 in the returned settings. This establishes control-run variation, not its cause, and does not prove the candidate preserves model output. The timing data remain useful observations but are not an accepted equal-output model speedup. Numerical operator checks passed separately. Keep the feature opt-in pending a stable model correctness check. + +The server hides GGML info logs at its default log level. A separate `-lv 5` trace confirms 20 of 32 FFN layers use fusion, with contiguous inputs and a passing fusion memory-range check. The other 12 Q4_K layers retain cuBLAS. Earlier absence of trace lines at the default level was not evidence of absent fusion. Evidence: `shared-chat-results/model-dispatch-trace.log`. + +## Nsight check + +Used the saved private Nsight Compute 2022.4.1.0 build 32308335 recipe on the current host. The executable SHA256 matches `ce0edb2afde930bc517a6a2da2cbcd5d53f4c8ec0947623e98ea478a8b444bd0`. The old matching section folder, `sudo -n`, and `--clock-control none` are required. Runner: `profile-shared-chat.sh`. + +Explicit LaunchStats, Occupancy, SchedulerStats, WarpStateStats, SpeedOfLight, and MemoryWorkloadAnalysis sections collected real counters. `--set basic` found no metrics; a first explicit-section capture with the idle governor active ran at low clocks and is not timing evidence. The final capture paused the governor and restored it after the run. + +The final report records the fused `quantize_mmq_q8_1` template with the SwiGLU boolean true: grid (2048,24,1), block (128,1,1), 28 registers/thread, SM frequency 2.07 GHz, DRAM throughput 523.75 GB/s (88.14%), and profiled duration 436.90 us. This is a profiled kernel observation, not an off/on speed comparison. Report and log: `shared-chat-results/fused-quant-metrics-p0.ncu-rep` and `.log`. The separate Compute Sanitizer limitation remains unresolved. + +Final host check: idle governor active; no experiment server/test process left running. Production source and installed profiler were not changed. Experimental code remains in the isolated worktree, with opt-in trace diagnostics; it has not been promoted or committed. + +## Memory-stage preparation + +### Selected 27B IQ2 + MTP baseline + +The user selected the installed 27B IQ2 + MTP setup on 2026-09-09. Baseline source is the isolated e653dac15 worktree with prefill fusion off. Files: `Qwen3.8-27B-UD-IQ2_XXS.gguf` and `mtp-Qwen3.8-27B-Q4_0.gguf`. Runner: `run-shared-chat-mtp.py`; raw evidence: `shared-chat-results/mtp-baseline-20260909-074348/`. + +Settings: CUDA0, all model layers offloaded, context 4096, one slot, F16 KV, batch/ubatch 512, `--fit off`, `--spec-type draft-mtp --spec-draft-n-max 3`, PP1024, 128 output tokens, greedy sampling, seed 1234, no prompt cache. These are a small initial baseline, not a long-context capacity test. Log level 5 is held fixed; its overhead means absolute speed is not a deployment best-case result. + +- One warmup and three measured requests completed. All four output texts are equal. +- Measured median PP: 457.27 tok/s (455.97 to 458.34). +- Measured median TG: 38.399 tok/s (38.349 to 38.407). +- Each request accepted 85 of 124 draft tokens, 68.55%. +- Sampled peak GPU memory: 8813 MiB out of 10240 MiB. +- Target recurrent memory: 598.50 MiB, comprising R 22.50 MiB and S 576.00 MiB; three rollback slots plus active state. +- Target/draft CUDA compute buffers: about 126.02 MiB each. These are distinct model objects, which must be considered before any shared-compute port. +- Idle governor was stopped for the run and restored in the runner's finally block. The experiment server exited normally. + +Two initial launches failed before model load due to CLI names: `mtp` must be `draft-mtp`, and removed `--draft-max` must be `--spec-draft-n-max`. Corrected in the runner; no performance data from those launches are used. + +The user approved the cross-layer design and further in-scope work. First scope: one SM75 CUDA device, one sequence, Qwen35 architecture with state dimension 128, two F32 S state planes plus per-token update records; replay the accepted prefix on partial rollback. The current snapshot path remains the default and fallback. No F16 snapshots, phase-arena reuse, or lazy replay are included. Shared compute is a separate opt-in experiment. + +### Transaction implementation and checks + +Opt-in: `LLAMA_SM75_GDN_TXN=1`. The kernel records the last `n_rs_seq + 1` updates and their prior state, including when prefill is larger than that window. S row zero is always current; R retains existing snapshot indexing. Replay waits for outstanding GPU writes before it runs. Sequence serialization reads current S separately from the pending R snapshot. State restore and clear invalidate old transaction metadata. Unsupported configurations use snapshots. + +The new standalone `sm75/test-gdn-transaction.cpp` checks 20 forward cases and 80 replay prefixes (state dimensions 16/32/64/128, token counts 1/2/4/9/32, four log slots). All pass bit for bit on CUDA 13.3/SM75. Initial replay checks failed because the generated forward multiply-add order differs for small state dimensions: 16/32 round decay times state first; 64/128 round key times delta first. Explicit replay intrinsics matching that order passed. This is tested-build evidence, not a claim of parity on other CUDA toolchains or architectures. The production opt-in is gated to SM75 and dimension 128. + +`sm75/test-mtp-state.cpp` checks the target model's accepted-prefix lengths 1 through 4 after a four-token verification batch, rejects a second pending rollback as the snapshot path does, checks save/load logits bitwise, and clears/restarts the sequence. Both modes pass. All nine resulting sequence-state files compare byte for byte between off and on. The files remain remote in `txn-state-off/` and `txn-state-on/`; logs are `txn-state-off.log` and `txn-state-on.log`. Prefix 1 corresponds to rejecting all three draft tokens while retaining the target token. Operator replay tests also cover retaining zero logged updates. + +First model candidate: `mtp-baseline-20260909-075527`. All four texts equal the original baseline; acceptance stays 85/124. Peak GPU memory falls from 8813 to 8533 MiB (280 MiB). Target S is 288 instead of 576 MiB; the update log costs about 9.04 MiB. Total recurrent allocation is 319.54 instead of 598.50 MiB. This establishes a memory saving, not a speed gain. + +Fresh off/on/off runs: `mtp-baseline-20260909-080032`, `080104`, `080135`. Median TG is 38.422 / 38.356 / 38.015 tok/s and PP is 453.420 / 449.417 / 448.536 tok/s. Control drift spans the candidate result, so no speed improvement is accepted. One SM75 guard build failed due to a misspelled constant (`CC_TURING`); corrected to `GGML_CUDA_CC_TURING`, and the next build completed before these runs. + +The benchmark runner now reuses the original saved request token IDs. It must not tokenize changing source files for later comparisons. Fusion stays off during memory tests. + +### Shared compute experiment + +Separate opt-in: `LLAMA_SM75_MTP_SHARED_COMPUTE=1`. Scope is a Qwen35 MTP context and its target, one SM75 device, one sequence, matching scheduler devices/buffer types, no pipeline copies. Model weights and memory are not shared. Host compute buffers stay separate. Device buffers have shared ownership; growth detaches an allocator rather than invalidating the other allocator's tensors. + +The context handoff covers encode/decode, including allocation, input writes, compute, and queued output copies. A new owner synchronizes the old owner before use. Shared buffer ownership is independent of context destruction order. Shared scheduler use must remain serialized; this is not support for concurrent graph execution on one arena. No VMM or sparse-memory feature is included. + +`sm75/test-shared-compute.cpp` passes alternating graph execution, buffer growth from either allocator, use after the other allocator is freed, and both destruction orders. Evidence: `shared-chat-results/shared-allocator.log`. The test was then extended to select the larger of two reserved buffers after discarding old graphs. + +The first shared run (`mtp-baseline-20260909-081204`) kept identical text and acceptance but still used 8533 MiB, so its enabled log is not proof of a saving. Setup initially shared before final capacity was known. The draft buffer is slightly larger than the target buffer, so the allocator now selects the larger fully reserved buffer and invalidates both old graphs before replacement. A second run (`081613`) still had no extra saving: enabling MTP hidden-state output triggers a later scheduler rebuild. The context pair now repeats sharing after scheduler reservation, while removing dead context pointers on destruction. + +The completed final-setup run (`081835`) shows sharing after both reservations. All four texts and all 85/124 acceptance counts equal the baseline. Steady GPU memory is 8405 MiB, versus 8533 MiB with transactions alone and 8813 MiB with both options off: 408 MiB saved in total. The 200 ms telemetry also sampled a peak of 8405 MiB in this run, but it can miss short allocation transients while two buffers are being reserved; this is not a guarantee of the instantaneous startup peak. + +Final standalone logs: `txn-operator-final.log` (80 bitwise replay and 20 forward checks), `shared-allocator-final.log` (including larger-buffer selection), and `txn-gdn-regression-final.log` (36/36 existing backend tests). These are copied under `shared-chat-results/`. Builds and `git diff --check` pass; Git emits normal LF/CRLF warnings on this Windows checkout. A check with `core.autocrlf` temporarily overridden falsely treated unchanged CRLF files as whole-file edits; the normal project configuration was used for the valid check. No line-ending normalization was applied. + +### Final control / both options / control + +Same final build, fixed original request IDs, one warmup and three measured requests per phase: + +| Run suffix | Transactions | Shared compute | PP median tok/s | TG median tok/s | GPU memory MiB | +| --- | --- | --- | ---: | ---: | ---: | +| 081951 | off | off | 449.380 | 38.220 | 8813 | +| 082023 | on | on | 446.190 | 38.005 | 8405 | +| 082054 | off | off | 445.147 | 37.841 | 8813 | + +All twelve texts equal the original baseline. Every response accepts 85 of 124 draft tokens. The candidate speed lies between the two controls; no speed gain is claimed. The accepted result is 408 MiB less working GPU memory on this 27B IQ2 + MTP / single CMP50HX setup. Raw files are under `shared-chat-results/mtp-baseline-20260909-/`. + +Use both environment variables with the isolated experiment build: + +```sh +LLAMA_SM75_GDN_TXN=1 LLAMA_SM75_MTP_SHARED_COMPUTE=1 ./build/bin/llama-server ... +``` + +Both options are off by default. Prefill fusion remains a separate opt-in and was off throughout these memory tests. No production checkout, installed service, GPU clocks, or driver was changed. The idle governor is active again and GPU memory is back to 0 MiB after the test processes exit. No commit or push was made. Scope limits: only this SM75/CUDA toolchain and model setup were measured; no multi-GPU, multi-sequence, long-context capacity, or sanitizer coverage is claimed. + +- The donor published these features together in `1a63cbf12`; transaction logging is not an isolated commit. Its recurrent source/header and CUDA GDN changes total 1239 added and 127 removed lines relative to the common base. This count includes related features and is not the size of a minimal transaction-log port. +- Keep only the transaction log first. Phase-arena reuse and lazy replay need separate results. Required tests include accept-all, reject-all, partial prefix, repeated rollback, and state save/load. +- Shared compute requires allocator buffer ownership, scheduler compatibility checks, buffer-growth detachment, and target/draft synchronization. Sparse VMM and host-KV are separate features and are outside this port. +- The donor hands off shared compute at `graph_compute`. Our `process_ubatch` sets inputs before that call. A port must acquire buffer ownership before allocation/input writes, not only before kernel launch. This is a source-level ordering concern, not a reproduced failure. +- The older intended MTP model was 27B Q4 on two 20 GiB cards. The user has now selected the current single 10 GiB card and installed 27B IQ2 target plus draft. Do not compare its absolute speed or memory use directly with the older two-card Q4 setup. +- Independent prefill model checks can use the two installed 9B UD-Q4_K_XL models from the existing single-card test setup. Runner: `run-shared-chat-model.py`; control/candidate/control, PP16384, 128 output tokens, greedy sampling, F16 KV, full GPU placement, one warmup and three measured requests per phase. + +## Commit review, 2026-09-09 + +The user requested logical commits and a push. Work is grouped on `perf/sm75-shared-chat-20260909`, based on fork commit `e653dac15`. The main worktree's staged work is not included. + +- `bd4df3e34`: opt-in prefill fusion, existing backend test cases, paired kernel runner. +- `59cbd8f6f`: opt-in recurrent replay, operator checks, full-model state checks. +- `73be57915`: opt-in shared compute buffers and allocator checks. +- A separate documentation commit holds this journal, model runners, profiler recipe, fixed MTP request, and test instructions. + +KISS/DRY review: fusion reuses the existing Q8_1 kernel and MMQ path; replay uses the existing recurrent state flow; sharing uses reference counts on the existing buffers. Two temporary dispatch-check trace blocks were removed. The success trace remains useful for proving that fusion ran. Empty-buffer selection now prefers an existing buffer over an absent zero-size peer. The allocator contract states that allocated graphs must be discarded before sharing. No new pool, VMM, lazy replay, or general benchmark framework is added. Device and shape gates remain narrow and all options stay off by default. + +Fresh checks after review: `llama-server` and `test-backend-ops` build passed; fusion-on 50/50 passed; replay 80 bitwise checks and 20 forward checks passed; shared-buffer alternation, growth, free-order, and larger-buffer checks passed. All committed Python runners pass syntax checks. + +The final candidate smoke run `mtp-baseline-20260909-083610` uses the committed fixed request and both memory options. All four texts match the original baseline and all accept 85 of 124 draft tokens. Sampled GPU memory reaches 8405 MiB. Measured medians are PP 451.462 tok/s and TG 38.604 tok/s. This is a smoke check, not a new paired speed claim. The accepted paired result above remains 408 MiB less working GPU memory with no proven speed gain. + +See `SHARED-CHAT-TESTS.md` for reproduction commands. Large raw logs, model files, saved states, and profiler reports stay outside Git. Earlier statements that no commit or push was made describe the state at the end of the experiment, before this review. diff --git a/sm75/SHARED-CHAT-TESTS.md b/sm75/SHARED-CHAT-TESTS.md new file mode 100644 index 000000000..c7061f7ab --- /dev/null +++ b/sm75/SHARED-CHAT-TESTS.md @@ -0,0 +1,46 @@ +# SM75 test build + +These are local experiment tools, not production launch scripts. Run from a Linux CUDA build with one SM75 device. The measured setup uses CUDA 13.3 and GCC 14. All three code options are off by default. + +## Scope + +- `GGML_CUDA_SM75_SWIGLU_Q8_1=1`: fuse split SwiGLU and Q8_1 input packing for eligible prefill MMQ graphs. Keep the existing cuBLAS path. +- `LLAMA_SM75_GDN_TXN=1`: use a prior state and replay log for Qwen35 recurrent rollback, one sequence, one SM75 device, F32 state, state width 128. +- `LLAMA_SM75_MTP_SHARED_COMPUTE=1`: share target/draft device compute buffers. Both contexts must use the same single SM75 device and run in turn. + +The shared-compute option does not require the transaction option to be on. Their commits are ordered because they use the same device capability check. No VMM, host KV cache, lazy replay, multi-GPU support, or new allocation pool is added. + +## Checks + +Build `llama-server` and `test-backend-ops` in `build/`. From the repository root: + +```sh +GGML_TEST_SWIGLU_MMQ=1 GGML_CUDA_SM75_SWIGLU_Q8_1=0 ./build/bin/test-backend-ops test -b CUDA0 -o SWIGLU_MMQ +GGML_TEST_SWIGLU_MMQ=1 GGML_CUDA_SM75_SWIGLU_Q8_1=1 ./build/bin/test-backend-ops test -b CUDA0 -o SWIGLU_MMQ +./build/bin/test-backend-ops test -b CUDA0 -o GATED_DELTA_NET + +g++-14 -std=c++17 -O2 sm75/test-gdn-transaction.cpp -Iggml/include -Lbuild/bin -Wl,-rpath,'$ORIGIN/build/bin' -lggml -lggml-base -o test-gdn-transaction +g++-14 -std=c++17 -O2 sm75/test-shared-compute.cpp -Iggml/include -Lbuild/bin -Wl,-rpath,'$ORIGIN/build/bin' -lggml -lggml-base -o test-shared-compute +g++-14 -std=c++17 -O2 sm75/test-mtp-state.cpp -Iinclude -Iggml/include -Lbuild/bin -Wl,-rpath,'$ORIGIN/build/bin' -lllama -lggml -lggml-base -o test-mtp-state +./test-gdn-transaction +./test-shared-compute +LLAMA_SM75_GDN_TXN=0 ./test-mtp-state "$SM75_MODELS/Qwen3.8-27B-UD-IQ2_XXS.gguf" txn-state-off +LLAMA_SM75_GDN_TXN=1 ./test-mtp-state "$SM75_MODELS/Qwen3.8-27B-UD-IQ2_XXS.gguf" txn-state-on +for f in txn-state-off/*.bin; do cmp "$f" "txn-state-on/${f##*/}" || exit 1; done +``` + +## Model and speed runs + +Set `SM75_MODELS` to the installed GGUF folder. The model scripts default to the measured host's folder. They require the exact filenames in each script. The MTP runner uses the fixed token request in `shared-chat-mtp-request.json`; it does not make a prompt from the current source tree. + +```sh +python3 sm75/run-shared-chat-kernels.py +python3 sm75/run-shared-chat-model.py +LLAMA_SM75_GDN_TXN=0 LLAMA_SM75_MTP_SHARED_COMPUTE=0 python3 sm75/run-shared-chat-mtp.py +LLAMA_SM75_GDN_TXN=1 LLAMA_SM75_MTP_SHARED_COMPUTE=1 python3 sm75/run-shared-chat-mtp.py +LLAMA_SM75_GDN_TXN=0 LLAMA_SM75_MTP_SHARED_COMPUTE=0 python3 sm75/run-shared-chat-mtp.py +``` + +Run only one model script at a time; they use port 18089. Timing scripts stop `cmp-idle-governor.service` if active and restore it on normal exit or error. They need passwordless sudo for that service. Check the service after an interrupted run. `profile-shared-chat.sh` uses the host's private Nsight Compute installation; inspect its two paths before use. + +Raw output goes to dated folders at the repository root. Keep large logs, state files, GGUF files, and profiler reports out of Git. See `SHARED-CHAT-PERFORMANCE.md` for the measured results and limits. diff --git a/sm75/profile-shared-chat.sh b/sm75/profile-shared-chat.sh new file mode 100644 index 000000000..387fe771f --- /dev/null +++ b/sm75/profile-shared-chat.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -euo pipefail +root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) +profiler=/home/xrip/tmp/ncu-old-compare-20260903/private/linux-desktop-glibc_2_11_3-x64/ncu +active=$(systemctl is-active cmp-idle-governor.service || true) +restore() { + if [ "$active" = active ]; then + sudo -n systemctl start cmp-idle-governor.service + fi +} +trap restore EXIT +sudo -n systemctl stop cmp-idle-governor.service +sudo -n env GGML_TEST_SWIGLU_MMQ=1 GGML_CUDA_SM75_SWIGLU_Q8_1=1 "$profiler" \ + --section-folder /home/xrip/tmp/ncu-old-compare-20260903/root/usr/lib/x86_64-linux-gnu/nsight-compute/sections \ + --clock-control none --kernel-name regex:quantize_mmq_q8_1 --launch-count 1 \ + --section LaunchStats --section Occupancy --section SchedulerStats --section WarpStateStats \ + --section SpeedOfLight --section MemoryWorkloadAnalysis \ + -o "$root/fused-quant-metrics-p0" \ + "$root/build/bin/test-backend-ops" perf -b CUDA0 -o SWIGLU_MMQ \ + -p 'type_a=q6_K.*m=4096,n=2048,k=12288' diff --git a/sm75/run-shared-chat-model.py b/sm75/run-shared-chat-model.py new file mode 100644 index 000000000..60835b873 --- /dev/null +++ b/sm75/run-shared-chat-model.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import statistics +import subprocess +import time +import urllib.request + +root = Path(__file__).resolve().parent.parent +models = Path(os.environ.get('SM75_MODELS', '/home/xrip/tmp/prefill-single-20260907/models')) +out = root / ('model-results-' + time.strftime('%Y%m%d-%H%M%S')) +out.mkdir() +url = 'http://127.0.0.1:18089' +server = None +telemetry = None +rows = [] +requests = {} +active = subprocess.run(['systemctl', 'is-active', '--quiet', 'cmp-idle-governor.service']).returncode == 0 + +def api(path, data=None): + body = None if data is None else json.dumps(data).encode() + req = urllib.request.Request(url + path, data=body, headers={'Content-Type': 'application/json'}) + with urllib.request.urlopen(req, timeout=1800) as response: + return json.load(response) + +try: + if active: + subprocess.run(['sudo', '-n', 'systemctl', 'stop', 'cmp-idle-governor.service'], check=True) + for model in ['Ornith-1.0-9B', 'Qwen3.5-9B']: + for phase, enabled in enumerate([0, 1, 0]): + label = f'{model}-{phase}-{enabled}' + command = [str(root / 'build/bin/llama-server'), '-m', str(models / (model + '-UD-Q4_K_XL.gguf')), + '--device', 'CUDA0', '-ngl', 'all', '-fa', 'on', '-c', '32768', '-np', '1', + '-ctk', 'f16', '-ctv', 'f16', '-b', '2048', '-ub', '2048', '--fit', 'off', + '--no-mmproj', '--jinja', '--host', '127.0.0.1', '--port', '18089'] + env = dict(os.environ, GGML_CUDA_SM75_SWIGLU_Q8_1=str(enabled), CUDA_SCALE_LAUNCH_QUEUES='4x', GGML_CUDA_GRAPH_OPT='1') + (out / (label + '-command.json')).write_text(json.dumps(dict(command=command, env={k: env[k] for k in ['GGML_CUDA_SM75_SWIGLU_Q8_1', 'CUDA_SCALE_LAUNCH_QUEUES', 'GGML_CUDA_GRAPH_OPT']}), indent=2)) + with (out / (label + '-server.log')).open('w') as log, (out / (label + '-gpu.csv')).open('w') as gpu_log: + telemetry = subprocess.Popen(['nvidia-smi', '--query-gpu=memory.used,pstate,clocks.sm,clocks.mem,temperature.gpu', '--format=csv,noheader,nounits', '-lms', '200'], stdout=gpu_log) + server = subprocess.Popen(command, env=env, stdout=log, stderr=log) + for attempt in range(300): + if server.poll() is not None: + raise RuntimeError(f'{label}: server exit {server.returncode}') + try: + api('/health') + break + except Exception: + time.sleep(1) + else: + raise RuntimeError('server readiness timeout') + if model not in requests: + text = (root / 'src/llama-context.cpp').read_text() + tokens = api('/tokenize', {'content': text * 4, 'add_special': True})['tokens'] + tail = api('/tokenize', {'content': '\nExplain the main responsibilities of this code and list five correctness risks.\n', 'add_special': False})['tokens'] + prompt = tokens[:16384-len(tail)] + tail + assert len(prompt) == 16384 + requests[model] = dict(prompt=prompt, n_predict=128, cache_prompt=False, temperature=0.0, + seed=1234, ignore_eos=True) + (out / (model + '-request.json')).write_text(json.dumps(requests[model])) + for repeat in range(4): + response = api('/completion', requests[model]) + (out / f'{label}-{repeat}.json').write_text(json.dumps(response, indent=2)) + timings = response['timings'] + if timings['prompt_n'] != 16384 or timings['predicted_n'] != 128: + raise RuntimeError(f'{label}: token count mismatch {timings}') + rows.append(dict(model=model, phase=phase, enabled=enabled, repeat=repeat, timings=timings, content=response['content'])) + (out / 'records.json').write_text(json.dumps(rows, indent=2)) + print(label, repeat, timings['prompt_per_second'], timings['predicted_per_second'], flush=True) + server.terminate() + server.wait(timeout=60) + server = None + telemetry.terminate() + telemetry.wait(timeout=10) + telemetry = None + summary = [] + for model in requests: + measured = [r for r in rows if r['model'] == model and r['repeat'] > 0] + result = dict(model=model, same_output=len({r['content'] for r in measured}) == 1) + for phase in [0, 1, 2]: + group = [r for r in measured if r['phase'] == phase] + result[str(phase)] = {metric: dict(median=statistics.median(r['timings'][metric] for r in group), minimum=min(r['timings'][metric] for r in group), maximum=max(r['timings'][metric] for r in group)) for metric in ['prompt_per_second', 'predicted_per_second']} + summary.append(result) + (out / 'summary.json').write_text(json.dumps(summary, indent=2)) + print('COMPLETE', out, flush=True) +finally: + for process in [server, telemetry]: + if process is not None and process.poll() is None: + process.terminate() + process.wait(timeout=60) + if active: + subprocess.run(['sudo', '-n', 'systemctl', 'start', 'cmp-idle-governor.service'], check=True) diff --git a/sm75/run-shared-chat-mtp.py b/sm75/run-shared-chat-mtp.py new file mode 100644 index 000000000..73fe00d62 --- /dev/null +++ b/sm75/run-shared-chat-mtp.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import subprocess +import time +import urllib.request + +root = Path(__file__).resolve().parent.parent +models = Path(os.environ.get('SM75_MODELS', '/home/xrip/tmp/prefill-single-20260907/models')) +out = root / ('mtp-baseline-' + time.strftime('%Y%m%d-%H%M%S')) +out.mkdir() +url = 'http://127.0.0.1:18089' +command = [str(root / 'build/bin/llama-server'), '-m', str(models / 'Qwen3.8-27B-UD-IQ2_XXS.gguf'), + '-md', str(models / 'mtp-Qwen3.8-27B-Q4_0.gguf'), '--spec-type', 'draft-mtp', '--spec-draft-n-max', '3', + '--device', 'CUDA0', '-ngl', 'all', '-fa', 'on', '-c', '4096', '-np', '1', + '-ctk', 'f16', '-ctv', 'f16', '-b', '512', '-ub', '512', '--fit', 'off', + '--no-mmproj', '-lv', '5', '--host', '127.0.0.1', '--port', '18089'] +env = dict(os.environ, GGML_CUDA_SM75_SWIGLU_Q8_1='0', CUDA_SCALE_LAUNCH_QUEUES='4x', GGML_CUDA_GRAPH_OPT='1', LLAMA_SM75_GDN_TXN=os.environ.get('LLAMA_SM75_GDN_TXN', '0'), LLAMA_SM75_MTP_SHARED_COMPUTE=os.environ.get('LLAMA_SM75_MTP_SHARED_COMPUTE', '0')) +(out / 'command.json').write_text(json.dumps(dict(command=command, env={k: env[k] for k in ['GGML_CUDA_SM75_SWIGLU_Q8_1', 'CUDA_SCALE_LAUNCH_QUEUES', 'GGML_CUDA_GRAPH_OPT', 'LLAMA_SM75_GDN_TXN', 'LLAMA_SM75_MTP_SHARED_COMPUTE']}), indent=2)) + +def api(path, data=None): + req = urllib.request.Request(url + path, data=None if data is None else json.dumps(data).encode(), headers={'Content-Type': 'application/json'}) + with urllib.request.urlopen(req, timeout=600) as response: + return json.load(response) + +active = subprocess.run(['systemctl', 'is-active', '--quiet', 'cmp-idle-governor.service']).returncode == 0 +server = None +telemetry = None +print(out, flush=True) +try: + if active: + subprocess.run(['sudo', '-n', 'systemctl', 'stop', 'cmp-idle-governor.service'], check=True) + with (out / 'server.log').open('w') as log, (out / 'gpu.csv').open('w') as gpu: + telemetry = subprocess.Popen(['nvidia-smi', '--query-gpu=memory.used,pstate,clocks.sm,clocks.mem,temperature.gpu', '--format=csv,noheader,nounits', '-lms', '200'], stdout=gpu) + server = subprocess.Popen(command, env=env, stdout=log, stderr=log) + for attempt in range(300): + if server.poll() is not None: + raise RuntimeError(f'server exit {server.returncode}') + try: + api('/health') + break + except Exception: + time.sleep(1) + else: + raise RuntimeError('readiness timeout') + request = json.loads((Path(__file__).with_name('shared-chat-mtp-request.json')).read_text()) + (out / 'request.json').write_text(json.dumps(request)) + for repeat in range(4): + result = api('/completion', request) + assert result['timings']['prompt_n'] == 1024 and result['timings']['predicted_n'] == 128 + (out / f'response-{repeat}.json').write_text(json.dumps(result, indent=2)) + print(repeat, result['timings'], flush=True) +finally: + for process in [server, telemetry]: + if process is not None and process.poll() is None: + process.terminate() + process.wait(timeout=60) + if active: + subprocess.run(['sudo', '-n', 'systemctl', 'start', 'cmp-idle-governor.service'], check=True) diff --git a/sm75/run-shared-chat-trace.py b/sm75/run-shared-chat-trace.py new file mode 100644 index 000000000..b4b70f81a --- /dev/null +++ b/sm75/run-shared-chat-trace.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import subprocess +import time +import urllib.request + +root = Path(__file__).resolve().parent.parent +models = Path(os.environ.get('SM75_MODELS', '/home/xrip/tmp/prefill-single-20260907/models')) +url = 'http://127.0.0.1:18089' +command = [str(root / 'build/bin/llama-server'), '-m', str(models / 'Qwen3.5-9B-UD-Q4_K_XL.gguf'), + '--device', 'CUDA0', '-ngl', 'all', '-fa', 'on', '-c', '8192', '-np', '1', + '-ctk', 'f16', '-ctv', 'f16', '-b', '2048', '-ub', '2048', '--fit', 'off', + '--no-mmproj', '-lv', '5', '--host', '127.0.0.1', '--port', '18089'] + +def api(path, data=None): + req = urllib.request.Request(url + path, data=None if data is None else json.dumps(data).encode(), headers={'Content-Type': 'application/json'}) + with urllib.request.urlopen(req, timeout=600) as response: + return json.load(response) + +with (root / 'model-dispatch-trace.log').open('w') as log: + server = subprocess.Popen(command, env=dict(os.environ, GGML_CUDA_SM75_SWIGLU_Q8_1='1', GGML_CUDA_SM75_SWIGLU_Q8_1_TRACE='1'), stdout=log, stderr=log) + try: + for attempt in range(300): + if server.poll() is not None: + raise RuntimeError(f'server exit {server.returncode}') + try: + api('/health') + break + except Exception: + time.sleep(1) + else: + raise RuntimeError('readiness timeout') + tokens = api('/tokenize', {'content': (root / 'src/llama-context.cpp').read_text(), 'add_special': True})['tokens'][:2048] + result = api('/completion', dict(prompt=tokens, n_predict=1, temperature=0, cache_prompt=False)) + print(result['timings']) + finally: + server.terminate() + server.wait(timeout=60) diff --git a/sm75/shared-chat-mtp-request.json b/sm75/shared-chat-mtp-request.json new file mode 100644 index 000000000..94d31384b --- /dev/null +++ b/sm75/shared-chat-mtp-request.json @@ -0,0 +1 @@ +{"prompt": [1032, 328, 636, 2933, 61485, 834, 1, 271, 1032, 328, 13985, 980, 834, 1, 198, 1032, 328, 636, 2933, 12, 1077, 834, 1, 198, 1032, 328, 636, 2933, 70534, 834, 1, 198, 1032, 328, 636, 2933, 12, 6182, 834, 1, 198, 1032, 328, 636, 2933, 1402, 730, 834, 1, 198, 1032, 328, 636, 2933, 83290, 834, 1, 198, 1032, 328, 636, 2933, 61932, 834, 1, 198, 1032, 328, 636, 2933, 1395, 2117, 834, 1, 198, 1032, 328, 636, 2933, 27327, 834, 1, 198, 1032, 328, 636, 2933, 63871, 834, 1, 198, 1032, 328, 636, 2933, 1284, 33356, 834, 1, 198, 1032, 328, 636, 2933, 834, 1, 271, 1032, 361, 66, 390, 8969, 29, 198, 1032, 361, 32608, 29, 198, 1032, 361, 37618, 29, 198, 1032, 361, 39721, 29, 198, 1032, 361, 91247, 29, 198, 1032, 361, 889, 29, 271, 320, 198, 320, 90448, 8207, 198, 320, 271, 1911, 9046, 76, 14312, 1756, 5458, 1756, 2270, 14312, 1756, 34347, 2933, 8207, 1756, 5458, 1756, 8, 313, 198, 262, 3289, 318, 3645, 1756, 8, 313, 198, 285, 1105, 18912, 36840, 28610, 4051, 13415, 25, 460, 436, 10673, 56397, 4051, 13415, 26, 198, 285, 1105, 18912, 36840, 28610, 4051, 1203, 4100, 262, 535, 460, 436, 10673, 56397, 4051, 23029, 2001, 622, 1203, 4100, 26, 198, 262, 333, 198, 262, 2430, 1407, 476, 21564, 3960, 437, 40503, 5458, 913, 4876, 198, 92, 271, 1193, 9046, 76, 736, 2506, 9981, 47460, 313, 198, 262, 9046, 76, 736, 2506, 9981, 1139, 26, 198, 262, 713, 1122, 348, 803, 26, 198, 262, 2535, 18, 17, 516, 307, 27896, 5500, 14071, 26, 198, 10738, 271, 1911, 713, 9046, 76, 736, 2506, 9981, 47460, 9046, 76, 736, 2506, 9981, 29239, 3625, 1463, 47460, 283, 313, 198, 262, 1343, 13, 445, 2215, 283, 1749, 436, 10673, 1352, 15332, 13509, 46899, 8403, 28217, 11, 198, 262, 1343, 13, 591, 1734, 283, 1749, 328, 23129, 59930, 487, 198, 262, 1343, 13, 77, 27896, 5500, 14071, 283, 1749, 220, 16, 11, 198, 10738, 271, 1911, 713, 9046, 76, 736, 2506, 9981, 47460, 9046, 76, 736, 2506, 9981, 1824, 16884, 25018, 47460, 283, 313, 198, 262, 1343, 13, 445, 2215, 283, 1749, 436, 10673, 1352, 15332, 13509, 2559, 30237, 22671, 11, 198, 262, 1343, 13, 591, 1734, 283, 1749, 328, 69, 2506, 469, 639, 24167, 9095, 318, 191280, 45157, 11250, 198, 262, 1343, 13, 77, 27896, 5500, 14071, 283, 1749, 220, 16, 11, 198, 10738, 271, 1911, 713, 9046, 76, 736, 2506, 9981, 47460, 9046, 76, 736, 2506, 9981, 1824, 16884, 4002, 47460, 283, 313, 198, 262, 1343, 13, 445, 2215, 283, 1749, 436, 10673, 1352, 15332, 13509, 2559, 30237, 6264, 11, 198, 262, 1343, 13, 591, 1734, 283, 1749, 328, 69, 2506, 469, 639, 24167, 9095, 318, 25148, 290, 11250, 198, 262, 1343, 13, 77, 27896, 5500, 14071, 283, 1749, 220, 16, 21, 11, 198, 10738, 271, 1911, 713, 9046, 76, 736, 2506, 9981, 47460, 9046, 76, 736, 2506, 9981, 879, 306, 47460, 283, 313, 198, 262, 1343, 13, 445, 2215, 283, 1749, 436, 10673, 1352, 15332, 13509, 34437, 28894, 14097, 622, 11, 198, 262, 1343, 13, 591, 1734, 283, 1749, 328, 13512, 1187, 7757, 261, 487, 198, 262, 1343, 13, 77, 27896, 5500, 14071, 283, 1749, 220, 16, 11, 198, 10738, 271, 1911, 713, 9046, 76, 736, 2506, 9981, 47460, 9046, 76, 736, 2506, 9981, 788, 3378, 19, 1469, 66, 10132, 47460, 283, 313, 198, 262, 1343, 13, 445, 2215, 283, 1749, 436, 10673, 1352, 15332, 13509, 1502, 17272, 19, 95566, 16912, 11, 198, 262, 1343, 13, 591, 1734, 283, 1749, 328, 69, 2506, 17642, 38049, 629, 19, 44301, 829, 487, 198, 262, 1343, 13, 77, 27896, 5500, 14071, 283, 1749, 220, 16, 11, 198, 10738, 271, 1911, 713, 9046, 76, 736, 2506, 9981, 47460, 9046, 76, 736, 2506, 9981, 788, 3378, 19, 1469, 66, 33310, 47460, 283, 313, 198, 262, 1343, 13, 445, 2215, 283, 1749, 436, 10673, 1352, 15332, 13509, 1502, 17272, 19, 95566, 6662, 33, 11, 198, 262, 1343, 13, 591, 1734, 283, 1749, 328, 69, 2506, 17642, 38049, 629, 19, 44301, 3493, 487, 198, 262, 1343, 13, 77, 27896, 5500, 14071, 283, 1749, 220, 16, 11, 198, 10738, 271, 1911, 713, 9046, 76, 736, 2506, 9981, 47460, 9046, 76, 736, 2506, 9981, 788, 3378, 19, 1469, 66, 6133, 47460, 283, 313, 198, 262, 1343, 13, 445, 2215, 283, 1749, 436, 10673, 1352, 15332, 13509, 1502, 17272, 19, 95566, 19881, 11, 198, 262, 1343, 13, 591, 1734, 283, 1749, 328, 69, 2506, 17642, 38049, 629, 19, 44301, 1675, 487, 198, 262, 1343, 13, 77, 27896, 5500, 14071, 283, 1749, 220, 16, 11, 198, 10738, 271, 636, 2933, 8207, 476, 636, 2933, 8207, 7, 198, 285, 713, 90448, 4885, 594, 1558, 11, 198, 1025, 90448, 8207, 6533, 3506, 8, 535, 198, 262, 1558, 7392, 681, 198, 262, 272, 3947, 5029, 476, 6711, 20567, 48592, 2933, 42236, 648, 3947, 29, 11519, 198, 262, 324, 62462, 5029, 476, 6711, 20567, 48592, 2933, 42236, 879, 62462, 29, 11519, 198, 262, 292, 4589, 5029, 476, 6711, 20567, 48592, 2933, 14116, 14373, 81, 2163, 2448, 834, 3403, 1210, 6276, 5500, 54962, 67, 9831, 313, 198, 262, 434, 5175, 9663, 948, 6611, 90448, 8207, 440, 27817, 5458, 1331, 421, 369, 524, 264, 2279, 314, 220, 17, 11, 198, 262, 434, 257, 1189, 1144, 310, 381, 18583, 41403, 198, 262, 18912, 36840, 8159, 8795, 4285, 82, 25, 47692, 90448, 8207, 1639, 487, 1259, 2739, 84550, 271, 262, 259, 4747, 10979, 283, 1558, 714, 4747, 10979, 26, 198, 262, 259, 12055, 10979, 220, 283, 1558, 714, 12055, 10979, 26, 271, 262, 713, 3129, 594, 304, 3403, 283, 1558, 834, 3403, 26, 271, 262, 272, 3403, 1210, 14071, 6144, 283, 1407, 476, 2721, 7, 16, 84, 11, 3506, 1210, 14071, 6144, 1174, 198, 262, 413, 318, 66, 3403, 1210, 14071, 6144, 835, 18912, 36840, 6592, 48360, 8, 313, 198, 285, 2430, 1407, 476, 21564, 3960, 437, 77, 14071, 6144, 1902, 381, 2564, 328, 478, 1407, 476, 951, 3773, 86254, 36840, 6592, 48360, 5722, 198, 262, 333, 271, 262, 272, 3403, 1210, 45542, 14071, 283, 3506, 1210, 45542, 14071, 26, 198, 262, 413, 318, 66, 3403, 1210, 45542, 14071, 835, 220, 15, 976, 729], "n_predict": 128, "temperature": 0, "seed": 1234, "cache_prompt": false, "ignore_eos": true} \ No newline at end of file