85 lines
4.2 KiB
Markdown
85 lines
4.2 KiB
Markdown
# 🚀 Turing Multi-GPU LLM Inference Rig (32GB VRAM Cluster)
|
||
|
||
A production-grade, highly optimized software and hardware configuration for running large 27B+ parameter reasoning and multimodal vision LLMs (`Qwen3.8-27B-Uncensored`, `Qwen 2.5 32B`, `DeepSeek`) across heterogeneous NVIDIA Turing GPUs (**RTX 2060 12GB + 2x CMP 50HX 10GB** = **32 GB Total VRAM**) on an **HPE ProLiant DL380p Gen8** server with full **256K Context Window (262,144 tokens)**, **Multimodal Vision**, and **Automated Idle Power Management**.
|
||
|
||
---
|
||
|
||
## 📊 Rig Highlights & Performance
|
||
|
||
* **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights)
|
||
* **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes
|
||
* **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition)
|
||
* **Decode Speed**: **`17.66 – 18.03 tok/s`** (sustained across 3 GPUs in NCCL tensor parallel)
|
||
* **Prompt Processing Speed**: **`140.3 – 179.4 tok/s`** (2x speedup using low-latency NCCL ring buffers & Flash Attention)
|
||
* **Idle Power**: Automatically downclocks CMP cards to 300–600 MHz, cutting idle draw from **200W+ down to ~100W** (~100W wall power savings!)
|
||
* **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev.
|
||
|
||
---
|
||
|
||
## 🏗️ Hardware & Cluster Architecture
|
||
|
||
```text
|
||
Host System: HPE ProLiant DL380p Gen8 (Dual Intel Xeon E5-2620 v2 Ivy Bridge-EP)
|
||
Total VRAM: 32 GB GDDR6 across 3 GPUs (Shared PCIe Root Complex)
|
||
|
||
[GPU 0: NVIDIA GeForce RTX 2060 12GB] (TU106, 1920 CUDA cores, 336 GB/s) -> Primary Vision & KV Slot
|
||
[GPU 1: NVIDIA CMP 50HX 10GB Mining] (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
|
||
[GPU 2: NVIDIA CMP 50HX 10GB Mining] (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
|
||
```
|
||
|
||
---
|
||
|
||
## 📂 Repository Contents
|
||
|
||
```
|
||
├── README.md # This overview and quickstart
|
||
├── scripts/
|
||
│ ├── build-nccl-llama.sh # Compiles llama.cpp with CUDA 12 + NCCL + sm_75 optimizations
|
||
│ ├── ollama-proxy.py # High-performance Ollama/OpenAI/Anthropic proxy with thinking control
|
||
│ ├── power-governor.py # Dynamic GPU power & clock governor for CMP 50HX cards
|
||
│ ├── start-server.sh # Production start script for llama-server
|
||
│ └── tune-gpus.sh # Helper script for power limits & persistence mode
|
||
├── systemd/
|
||
│ ├── llama-server.service # Systemd unit with NUMA socket pinning
|
||
│ ├── ollama-proxy.service # Systemd unit for API proxy
|
||
│ └── gpu-power-governor.service # Systemd unit for automated power governor
|
||
├── vbios/
|
||
│ └── msi_cmp50hx_90.02.60.00.17.rom # Tuned MSI VBIOS ROM (31W idle & 25% min fan curve)
|
||
└── docs/
|
||
├── SETUP_GUIDE.md # Complete step-by-step setup guide on any Linux server
|
||
├── PROXMOX_LXC_GUIDE.md # GPU passthrough & NVML permissions in Proxmox LXC
|
||
├── HARDWARE_LEARNINGS.md # Deep dive: CMP VBIOS, PCIe Gen1 vs Gen3, riser cables, 20GB mods
|
||
└── API_AND_THINKING_GUIDE.md # Controlling reasoning/thinking effort in OpenWebUI & API
|
||
```
|
||
|
||
---
|
||
|
||
## ⚡ Quick Start
|
||
|
||
### 1. Build llama.cpp with NCCL Support
|
||
```bash
|
||
bash scripts/build-nccl-llama.sh /opt/llama.cpp
|
||
```
|
||
|
||
### 2. Configure Systemd Services
|
||
```bash
|
||
sudo cp systemd/*.service /etc/systemd/system/
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
|
||
```
|
||
|
||
### 3. Connect from OpenWebUI
|
||
Set your Ollama URL in OpenWebUI to:
|
||
```text
|
||
http://<your-server-ip>:11434
|
||
```
|
||
Pick **`qwen:fast`** for instant answers (0 thinking delay) or **`qwen:think`** for deep multi-step reasoning!
|
||
|
||
---
|
||
|
||
## 📜 Documentation Links
|
||
* [Detailed Step-by-Step Setup Guide](docs/SETUP_GUIDE.md)
|
||
* [Proxmox LXC Passthrough & Permissions Guide](docs/PROXMOX_LXC_GUIDE.md)
|
||
* [Hardware Architecture & CMP Learnings](docs/HARDWARE_LEARNINGS.md)
|
||
* [API, OpenWebUI & Thinking Control Guide](docs/API_AND_THINKING_GUIDE.md)
|