90 lines
4.6 KiB
Markdown
90 lines
4.6 KiB
Markdown
# 🚀 Turing Multi-GPU LLM Inference Rig (32GB VRAM Cluster)
|
||
|
||
A production-grade, highly optimized software and hardware configuration for running large 27B+ parameter reasoning and multimodal vision LLMs (`Qwen3.8-27B-Uncensored`, `Qwen 2.5 32B`, `DeepSeek`) across heterogeneous NVIDIA Turing GPUs (**RTX 2060 12GB + 2x CMP 50HX 10GB** = **32 GB Total VRAM**) on an **HPE ProLiant DL380p Gen8** server with full **256K Context Window (262,144 tokens)**, **Multimodal Vision**, and **Automated Idle Power Management**.
|
||
|
||
---
|
||
|
||
## 📊 Rig Highlights & Performance
|
||
|
||
* **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights)
|
||
* **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes
|
||
* **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition)
|
||
* **Decode Speed**: **`38.7 – 44.6 tok/s`** 🚀 *(2.2x speedup using native Multi-Token Prediction (MTP) speculation & kernel unlocks)*
|
||
* **Prompt Processing Speed**: **`325.0 – 502.6 tok/s`** ⚡ *(0.5k tok/s prefill on 4k context with ubatch 512 & PCIe Gen2)*
|
||
* **Idle Power**: Driver unlock & NVAPI P-state governor drop idle power from **~240W down to `~33W total`** across all 3 GPUs!
|
||
* **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev.
|
||
|
||
---
|
||
|
||
## 🏗️ Hardware & Cluster Architecture
|
||
|
||
```text
|
||
Host System: HPE ProLiant DL380p Gen8 (Dual Intel Xeon E5-2620 v2 Ivy Bridge-EP)
|
||
Total VRAM: 32 GB GDDR6 across 3 GPUs (Shared PCIe Root Complex)
|
||
|
||
[GPU 0: NVIDIA GeForce RTX 2060 12GB] (TU106, 1920 CUDA cores, 336 GB/s) -> Primary Vision & KV Slot
|
||
[GPU 1: NVIDIA CMP 50HX 10GB Mining] (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
|
||
[GPU 2: NVIDIA CMP 50HX 10GB Mining] (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
|
||
```
|
||
|
||
---
|
||
|
||
## 📂 Repository Contents
|
||
|
||
```
|
||
├── README.md # This overview and quickstart
|
||
├── scripts/
|
||
│ ├── build-nccl-llama.sh # Compiles llama.cpp with CUDA 12 + NCCL + sm_75 optimizations
|
||
│ ├── ollama-proxy.py # High-performance Ollama/OpenAI/Anthropic proxy with thinking control
|
||
│ └── start-server.sh # Production start script for llama-server
|
||
├── systemd/
|
||
│ ├── llama-server.service # Systemd unit with NUMA socket pinning
|
||
│ └── ollama-proxy.service # Systemd unit for API proxy
|
||
├── tuning/
|
||
│ └── cmp-tune.conf # Host-level CMP 50HX tuning profiles (stable-fast, efficient, insane)
|
||
├── vbios/
|
||
│ └── msi_cmp50hx_90.02.60.00.17.rom # Tuned MSI VBIOS ROM (31W idle & 25% min fan curve)
|
||
└── docs/
|
||
├── SETUP_GUIDE.md # Complete step-by-step setup guide on any Linux server
|
||
├── PROXMOX_LXC_GUIDE.md # GPU passthrough, CPU socket pinning & host power management
|
||
├── HARDWARE_LEARNINGS.md # Deep dive: CMP VBIOS, PCIe Gen1 vs Gen3, riser cables, 20GB mods
|
||
├── 20GB_MOD_GUIDE.md # Hardware strapping & driver patching for 20GB TU102 mods
|
||
├── CMP_ECOSYSTEM_AND_COMMUNITY.md # Master ecosystem guide: projects, techniques & community
|
||
└── API_AND_THINKING_GUIDE.md # Controlling reasoning/thinking effort in OpenWebUI & API
|
||
```
|
||
|
||
---
|
||
|
||
## ⚡ Quick Start
|
||
|
||
### 1. Build llama.cpp with NCCL Support
|
||
```bash
|
||
bash scripts/build-nccl-llama.sh /opt/llama.cpp
|
||
```
|
||
|
||
### 2. Configure Systemd Services
|
||
```bash
|
||
sudo cp systemd/*.service /etc/systemd/system/
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl enable --now llama-server.service ollama-proxy.service
|
||
```
|
||
|
||
### 3. Connect from OpenWebUI
|
||
Set your Ollama URL in OpenWebUI to:
|
||
```text
|
||
http://<your-server-ip>:11434
|
||
```
|
||
Pick **`qwen:fast`** for instant answers (0 thinking delay) or **`qwen:think`** for deep multi-step reasoning!
|
||
|
||
---
|
||
|
||
## 📜 Documentation & Benchmark Links
|
||
* [Comprehensive Multi-GPU Benchmark Results (Q4/Q5/Q6/Gemma4 MoE)](benchmarks/BENCHMARK_RESULTS.md)
|
||
* [Detailed Client Usage Guide (OpenWebUI, Ollama, OpenAI, Anthropic, Continue.dev)](docs/USAGE-CLIENTS.md)
|
||
* [Detailed Step-by-Step Setup Guide](docs/SETUP_GUIDE.md)
|
||
* [Proxmox LXC Passthrough & Permissions Guide](docs/PROXMOX_LXC_GUIDE.md)
|
||
* [Hardware Architecture & CMP Learnings](docs/HARDWARE_LEARNINGS.md)
|
||
* [CMP 20GB VRAM Modding & Driver Patch Guide](docs/20GB_MOD_GUIDE.md)
|
||
* [CMP Reverse Engineering Ecosystem & Community Map](docs/CMP_ECOSYSTEM_AND_COMMUNITY.md)
|
||
* [API, OpenWebUI & Thinking Control Guide](docs/API_AND_THINKING_GUIDE.md)
|