feat: update to Q5_K_P @ 128K ctx, non-blocking proxy sessions, and full multi-GPU benchmark suite

benchmarks/BENCHMARK_RESULTS.md

@@ -0,0 +1,58 @@
+# Multi-GPU Quantization Benchmark: Qwen3.8-27B vs Gemma4-26B-A4B (Turing SM75)
+
+Comprehensive performance benchmark, memory footprint comparison, context window scaling, and architecture analysis across 3x NVIDIA Turing GPUs (RTX 2060 12GB + 2x CMP 50HX 10GB).
+
+---
+
+## 1. Hardware & System Architecture
+
+- **Host Processor**: Intel Xeon E5-2697 v2 (12C/24T, Ivy Bridge @ 2.70 GHz base, AVX1 support)
+- **Active GPUs**: 3x NVIDIA Turing GPUs (**32,768 MiB physical VRAM total**)
+  - `CUDA0` (Bus `21:00.0`): **NVIDIA GeForce RTX 2060 12GB** (11,841 MiB usable) @ PCIe 3.0 x16
+  - `CUDA1` (Bus `24:00.0`): **NVIDIA CMP 50HX 10GB** (9,798 MiB usable) @ PCIe 3.0 x16
+  - `CUDA2` (Bus `27:00.0`): **NVIDIA CMP 50HX 10GB** (9,798 MiB usable) @ PCIe 3.0 x16
+- **Interconnect**: NVIDIA NCCL / Layer Split pipeline across 3 PCIe buses
+
+---
+
+## 2. Head-to-Head Benchmark Matrix: Dense vs MoE
+
+| Metric / Benchmark Stage | **Gemma4-26B-A4B-QAT @ 256K Ctx** *(Active Now)* | **Qwen3.8-27B (Q5_K_P @ 192K)** | **Qwen3.8-27B (Q4_K_P @ 256K)** |
+| :--- | :--- | :--- | :--- |
+| **Architecture Type** | **Sparse MoE (128 experts / 8 active)** | Dense (64 layers) | Dense (64 layers) |
+| **Active Params / Total Params** | **4.0B active / 26B total** | 27.5B active / 27.5B total | 27.5B active / 27.5B total |
+| **Model Disk Size** | **15.64 GB** (`Q4_K_M`) | 20.22 GB (`Q5_K_P`) | 17.92 GB (`Q4_K_P`) |
+| **KV Cache Precision** | **`q5_0` (5.5-bit)** | `q5_0` (5.5-bit) | `q4_0` (4.5-bit) |
+| **Max Context Window** | **256K** (`262,144 tokens`) | 192K (`196,608 tokens`) | 256K (`262,144 tokens`) |
+| **Multimodal Support** | **Text + Vision + Video (`mmproj-BF16`)** | Text + Vision (`mmproj-f16`) | Text + Vision (`mmproj-f16`) |
+| **Split Mode** | **Layer Split (`--split-mode layer`)** | NCCL Tensor Split | NCCL Tensor Split |
+| **Active Memory Footprint** | **9.09 GB / 9.25 GB / 7.67 GB** | 11.60 GB / 9.07 GB / 9.05 GB | 11.75 GB / 8.89 GB / 8.87 GB |
+| **Total VRAM Allocated** | **26.00 GB (83%)** | 29.72 GB (95%) | 29.51 GB (94%) |
+| **Free Headroom / GPU** | **+2.75 GB / +0.55 GB / +2.13 GB** | +240 MB / +720 MB / +740 MB | +90 MB / +905 MB / +923 MB |
+| **1. Short Prompt (100 tok)** | | | |
+| • Prefill Speed | **95.87 tok/s** | 53.66 tok/s | 52.46 tok/s |
+| • Decode Speed | **46.00 tok/s** | 37.14 tok/s | 38.12 tok/s |
+| • Total Wall Time | **2.44 s** | 3.76 s | 3.47 s |
+| **2. Medium Context (~5k tok)** | | | |
+| • Prefill Speed | **2,279.22 tok/s** *(5.8x faster)* | 388.85 tok/s | 391.23 tok/s |
+| • Decode Speed | **45.42 tok/s** | 33.81 tok/s | 41.42 tok/s |
+| • Total Wall Time | **4.40 s** *(3.8x faster)* | 16.79 s | 16.33 s |
+| **3. Long Context (~6k tok)** | | | |
+| • Prefill Speed | **2,374.57 tok/s** *(6.0x faster)* | 396.52 tok/s | 398.61 tok/s |
+| • Decode Speed | **43.67 tok/s** | 35.99 tok/s | 41.00 tok/s |
+| • Total Wall Time | **4.97 s** *(3.9x faster)* | 19.60 s | 19.47 s |
+| **4. Sustained Decode (256 tok)** | | | |
+| • Decode Speed | **47.50 tok/s** *(+58% faster)* | 30.10 tok/s | 31.97 tok/s |
+| • Total Wall Time | **5.82 s** | 10.71 s | 10.17 s |
+
+---
+
+## 3. Context Scaling Matrix on Gemma4-26B-A4B
+
+| Context Window | KV Cache Quant Type | Total VRAM Across Rig | RTX 2060 12GB | CMP 50HX #1 10GB | CMP 50HX #2 10GB | Status |
+| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
+| **64K** (`65,536`) | `q5_0` (5.5-bit) | 19.5 GB | 6.8 GB | 7.1 GB | 5.6 GB | Rock Solid |
+| **128K** (`131,072`) | `q5_0` (5.5-bit) | 22.0 GB | 7.7 GB | 7.9 GB | 6.5 GB | Rock Solid |
+| **192K** (`196,608`) | `q5_0` (5.5-bit) | 24.0 GB | 8.4 GB | 8.6 GB | 7.0 GB | Stable |
+| **256K** (`262,144`) | `q5_0` (5.5-bit) | **26.0 GB** | **9.09 GB** | **9.25 GB** | **7.67 GB** | **Full Native Ceiling** (+550MB margin) |
+| **192K** (`196,608`) | `q8_0` (8.5-bit) | 27.8 GB | 9.8 GB | 9.6 GB | 8.4 GB | Max Precision Ceiling |