Initial commit: Complete deployment scripts, power governor, systemd units, and architecture documentation for Turing multi-GPU LLM rig

2026-09-01 01:55:19 +00:00
commit aeb72cecfa
13 changed files with 1925 additions and 0 deletions
@@ -0,0 +1,82 @@
+# 🚀 Turing Multi-GPU LLM Inference Rig (32GB VRAM Cluster)
+
+A production-grade, highly optimized software and hardware configuration for running large 27B+ parameter reasoning and multimodal vision LLMs (`Qwen3.8-27B-Uncensored`, `Qwen 2.5 32B`, `DeepSeek`) across heterogeneous NVIDIA Turing GPUs (**RTX 2060 12GB + 2x CMP 50HX 10GB** = **32 GB Total VRAM**) on an **HPE ProLiant DL380p Gen8** server with full **256K Context Window (262,144 tokens)**, **Multimodal Vision**, and **Automated Idle Power Management**.
+
+---
+
+## 📊 Rig Highlights & Performance
+
+* **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights)
+* **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes
+* **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition)
+* **Decode Speed**: **`18.03 tok/s`** (sustained across 3 GPUs in NCCL tensor parallel)
+* **Prompt Processing Speed**: **`~127 tok/s`** (using Flash Attention & Turing cuBLAS GEMM)
+* **Idle Power**: Automatically downclocks CMP cards to 300–600 MHz, cutting idle draw from **200W+ down to ~100W** (~100W wall power savings!)
+* **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev.
+
+---
+
+## 🏗️ Hardware & Cluster Architecture
+
+```text
+Host System: HPE ProLiant DL380p Gen8 (Dual Intel Xeon E5-2620 v2 Ivy Bridge-EP)
+Total VRAM:  32 GB GDDR6 across 3 GPUs (Shared PCIe Root Complex)
+
+[GPU 0: NVIDIA GeForce RTX 2060 12GB] (TU106, 1920 CUDA cores, 336 GB/s) -> Primary Vision & KV Slot
+[GPU 1: NVIDIA CMP 50HX 10GB Mining]  (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
+[GPU 2: NVIDIA CMP 50HX 10GB Mining]  (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
+```
+
+---
+
+## 📂 Repository Contents
+
+```
+├── README.md                           # This overview and quickstart
+├── scripts/
+│   ├── build-nccl-llama.sh             # Compiles llama.cpp with CUDA 12 + NCCL + sm_75 optimizations
+│   ├── ollama-proxy.py                 # High-performance Ollama/OpenAI/Anthropic proxy with thinking control
+│   ├── power-governor.py               # Dynamic GPU power & clock governor for CMP 50HX cards
+│   ├── start-server.sh                 # Production start script for llama-server
+│   └── tune-gpus.sh                    # Helper script for power limits & persistence mode
+├── systemd/
+│   ├── llama-server.service            # Systemd unit with NUMA socket pinning
+│   ├── ollama-proxy.service            # Systemd unit for API proxy
+│   └── gpu-power-governor.service      # Systemd unit for automated power governor
+└── docs/
+    ├── SETUP_GUIDE.md                  # Complete step-by-step setup guide on any Linux server
+    ├── PROXMOX_LXC_GUIDE.md            # GPU passthrough & NVML permissions in Proxmox LXC
+    ├── HARDWARE_LEARNINGS.md           # Deep dive: CMP VBIOS, PCIe Gen1 vs Gen3, riser cables, 20GB mods
+    └── API_AND_THINKING_GUIDE.md       # Controlling reasoning/thinking effort in OpenWebUI & API
+```
+
+---
+
+## ⚡ Quick Start
+
+### 1. Build llama.cpp with NCCL Support
+```bash
+bash scripts/build-nccl-llama.sh /opt/llama.cpp
+```
+
+### 2. Configure Systemd Services
+```bash
+sudo cp systemd/*.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
+```
+
+### 3. Connect from OpenWebUI
+Set your Ollama URL in OpenWebUI to:
+```text
+http://<your-server-ip>:11434
+```
+Pick **`qwen:fast`** for instant answers (0 thinking delay) or **`qwen:think`** for deep multi-step reasoning!
+
+---
+
+## 📜 Documentation Links
+* [Detailed Step-by-Step Setup Guide](docs/SETUP_GUIDE.md)
+* [Proxmox LXC Passthrough & Permissions Guide](docs/PROXMOX_LXC_GUIDE.md)
+* [Hardware Architecture & CMP Learnings](docs/HARDWARE_LEARNINGS.md)
+* [API, OpenWebUI & Thinking Control Guide](docs/API_AND_THINKING_GUIDE.md)