wmantly/turing-multi-gpu-llm-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: Update HARDWARE_LEARNINGS and README with low-latency NCCL ring buffer benchmarks and speculative decoding findings

Browse Source
This commit is contained in:
wmantly
2026-09-01 16:06:32 +00:00
parent 261631457c
commit 5f261c191f
2 changed files with 30 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -2
View File
@@ -9,8 +9,8 @@ A production-grade, highly optimized software and hardware configuration for run
* **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights)
* **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes
* **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition)
* **Decode Speed**: **`18.03 tok/s`** (sustained across 3 GPUs in NCCL tensor parallel)
* **Prompt Processing Speed**: **`~127 tok/s`** (using Flash Attention & Turing cuBLAS GEMM)
* **Decode Speed**: **`17.66 – 18.03 tok/s`** (sustained across 3 GPUs in NCCL tensor parallel)
* **Prompt Processing Speed**: **`140.3 – 179.4 tok/s`** (2x speedup using low-latency NCCL ring buffers & Flash Attention)
* **Idle Power**: Automatically downclocks CMP cards to 300–600 MHz, cutting idle draw from **200W+ down to ~100W** (~100W wall power savings!)
* **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev.