🚀 Turing Multi-GPU LLM Inference Rig (32GB VRAM Cluster)

A production-grade, highly optimized software and hardware configuration for running large 27B+ parameter reasoning and multimodal vision LLMs (Qwen3.8-27B-Uncensored, Qwen 2.5 32B, DeepSeek) across heterogeneous NVIDIA Turing GPUs (RTX 2060 12GB + 2x CMP 50HX 10GB = 32 GB Total VRAM) on an HPE ProLiant DL380p Gen8 server with full 256K Context Window (262,144 tokens), Multimodal Vision, and Automated Idle Power Management.

📊 Rig Highlights & Performance

Model: Qwen3.8-27B-Uncensored (Q4_K_P Quantization, 16.68 GB weights)
Context Window: 262,144 tokens (Full 256K Context) with zero OOM crashes
Multimodal Vision: Active (mmproj ViT GGUF projector for image recognition)
Decode Speed: 17.66 – 18.03 tok/s (sustained across 3 GPUs in NCCL tensor parallel)
Prompt Processing Speed: 140.3 – 179.4 tok/s (2x speedup using low-latency NCCL ring buffers & Flash Attention)
Idle Power: Automatically downclocks CMP cards to 300–600 MHz, cutting idle draw from 200W+ down to ~100W (~100W wall power savings!)
API Compatibility: Fully compatible with Ollama (http://<ip>:11434), OpenAI (/v1/chat/completions), and Anthropic Messages (/v1/messages) for OpenWebUI, Claude Code, and Continue.dev.

🏗️ Hardware & Cluster Architecture

Host System: HPE ProLiant DL380p Gen8 (Dual Intel Xeon E5-2620 v2 Ivy Bridge-EP)
Total VRAM:  32 GB GDDR6 across 3 GPUs (Shared PCIe Root Complex)

[GPU 0: NVIDIA GeForce RTX 2060 12GB] (TU106, 1920 CUDA cores, 336 GB/s) -> Primary Vision & KV Slot
[GPU 1: NVIDIA CMP 50HX 10GB Mining]  (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
[GPU 2: NVIDIA CMP 50HX 10GB Mining]  (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split

📂 Repository Contents

├── README.md                           # This overview and quickstart
├── scripts/
│   ├── build-nccl-llama.sh             # Compiles llama.cpp with CUDA 12 + NCCL + sm_75 optimizations
│   ├── ollama-proxy.py                 # High-performance Ollama/OpenAI/Anthropic proxy with thinking control
│   ├── power-governor.py               # Dynamic GPU power & clock governor for CMP 50HX cards
│   ├── start-server.sh                 # Production start script for llama-server
│   └── tune-gpus.sh                    # Helper script for power limits & persistence mode
├── systemd/
│   ├── llama-server.service            # Systemd unit with NUMA socket pinning
│   ├── ollama-proxy.service            # Systemd unit for API proxy
│   └── gpu-power-governor.service      # Systemd unit for automated power governor
├── vbios/
│   └── msi_cmp50hx_90.02.60.00.17.rom  # Tuned MSI VBIOS ROM (31W idle & 25% min fan curve)
└── docs/
    ├── SETUP_GUIDE.md                  # Complete step-by-step setup guide on any Linux server
    ├── PROXMOX_LXC_GUIDE.md            # GPU passthrough & NVML permissions in Proxmox LXC
    ├── HARDWARE_LEARNINGS.md           # Deep dive: CMP VBIOS, PCIe Gen1 vs Gen3, riser cables, 20GB mods
    ├── 20GB_MOD_GUIDE.md               # Hardware strapping & driver patching for 20GB TU102 mods
    ├── CMP_ECOSYSTEM_AND_COMMUNITY.md  # Master ecosystem guide: projects, techniques & community
    └── API_AND_THINKING_GUIDE.md       # Controlling reasoning/thinking effort in OpenWebUI & API

⚡ Quick Start

1. Build llama.cpp with NCCL Support

bash scripts/build-nccl-llama.sh /opt/llama.cpp

2. Configure Systemd Services

sudo cp systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service

3. Connect from OpenWebUI

Set your Ollama URL in OpenWebUI to:

http://<your-server-ip>:11434

Pick qwen:fast for instant answers (0 thinking delay) or qwen:think for deep multi-step reasoning!

4.5 KiB Raw Blame History Unescape Escape