wmantly/turing-multi-gpu-llm-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: Update README with 38.7-44.6 tok/s decode, 502 tok/s prefill, and 33W cluster idle power benchmarks

Browse Source
This commit is contained in:
wmantly
2026-09-02 01:04:49 +00:00
parent 4a03c75710
commit 38b308c6fd
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3 -3
View File
@@ -9,9 +9,9 @@ A production-grade, highly optimized software and hardware configuration for run
* **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights) * **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights)
* **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes * **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes
* **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition) * **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition)
* **Decode Speed**: **`17.66 – 18.03 tok/s`** (sustained across 3 GPUs in NCCL tensor parallel) * **Decode Speed**: **`38.7 – 44.6 tok/s`** 🚀 *(2.2x speedup using native Multi-Token Prediction (MTP) speculation & kernel unlocks)*
* **Prompt Processing Speed**: **`140.3 – 179.4 tok/s`** (2x speedup using low-latency NCCL ring buffers & Flash Attention) * **Prompt Processing Speed**: **`325.0 – 502.6 tok/s`** ⚡ *(0.5k tok/s prefill on 4k context with ubatch 512 & PCIe Gen2)*
* **Idle Power**: Automatically downclocks CMP cards to 300–600 MHz, cutting idle draw from **200W+ down to ~100W** (~100W wall power savings!) * **Idle Power**: Driver unlock & NVAPI P-state governor drop idle power from **~240W down to `~33W total`** across all 3 GPUs!
* **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev. * **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev.
--- ---