# 🚀 Turing Multi-GPU LLM Inference Rig (32GB VRAM Cluster)

A production-grade, highly optimized software and hardware configuration for running large 27B+ parameter reasoning and multimodal vision LLMs (`Qwen3.8-27B-Uncensored`, `Qwen 2.5 32B`, `DeepSeek`) across heterogeneous NVIDIA Turing GPUs (**RTX 2060 12GB + 2x CMP 50HX 10GB** = **32 GB Total VRAM**) on an **HPE ProLiant DL380p Gen8** server with full **256K Context Window (262,144 tokens)**, **Multimodal Vision**, and **Automated Idle Power Management**.

---

## 📊 Rig Highlights & Performance

* **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights)
* **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes
* **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition)
* **Decode Speed**: **`18.03 tok/s`** (sustained across 3 GPUs in NCCL tensor parallel)
* **Prompt Processing Speed**: **`~127 tok/s`** (using Flash Attention & Turing cuBLAS GEMM)
* **Idle Power**: Automatically downclocks CMP cards to 300–600 MHz, cutting idle draw from **200W+ down to ~100W** (~100W wall power savings!)
* **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev.

---

## 🏗️ Hardware & Cluster Architecture

```text
Host System: HPE ProLiant DL380p Gen8 (Dual Intel Xeon E5-2620 v2 Ivy Bridge-EP)
Total VRAM:  32 GB GDDR6 across 3 GPUs (Shared PCIe Root Complex)

[GPU 0: NVIDIA GeForce RTX 2060 12GB] (TU106, 1920 CUDA cores, 336 GB/s) -> Primary Vision & KV Slot
[GPU 1: NVIDIA CMP 50HX 10GB Mining]  (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
[GPU 2: NVIDIA CMP 50HX 10GB Mining]  (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
```

---

## 📂 Repository Contents

```
├── README.md                           # This overview and quickstart
├── scripts/
│   ├── build-nccl-llama.sh             # Compiles llama.cpp with CUDA 12 + NCCL + sm_75 optimizations
│   ├── ollama-proxy.py                 # High-performance Ollama/OpenAI/Anthropic proxy with thinking control
│   ├── power-governor.py               # Dynamic GPU power & clock governor for CMP 50HX cards
│   ├── start-server.sh                 # Production start script for llama-server
│   └── tune-gpus.sh                    # Helper script for power limits & persistence mode
├── systemd/
│   ├── llama-server.service            # Systemd unit with NUMA socket pinning
│   ├── ollama-proxy.service            # Systemd unit for API proxy
│   └── gpu-power-governor.service      # Systemd unit for automated power governor
└── docs/
    ├── SETUP_GUIDE.md                  # Complete step-by-step setup guide on any Linux server
    ├── PROXMOX_LXC_GUIDE.md            # GPU passthrough & NVML permissions in Proxmox LXC
    ├── HARDWARE_LEARNINGS.md           # Deep dive: CMP VBIOS, PCIe Gen1 vs Gen3, riser cables, 20GB mods
    └── API_AND_THINKING_GUIDE.md       # Controlling reasoning/thinking effort in OpenWebUI & API
```

---

## ⚡ Quick Start

### 1. Build llama.cpp with NCCL Support
```bash
bash scripts/build-nccl-llama.sh /opt/llama.cpp
```

### 2. Configure Systemd Services
```bash
sudo cp systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
```

### 3. Connect from OpenWebUI
Set your Ollama URL in OpenWebUI to:
```text
http://<your-server-ip>:11434
```
Pick **`qwen:fast`** for instant answers (0 thinking delay) or **`qwen:think`** for deep multi-step reasoning!

---

## 📜 Documentation Links
* [Detailed Step-by-Step Setup Guide](docs/SETUP_GUIDE.md)
* [Proxmox LXC Passthrough & Permissions Guide](docs/PROXMOX_LXC_GUIDE.md)
* [Hardware Architecture & CMP Learnings](docs/HARDWARE_LEARNINGS.md)
* [API, OpenWebUI & Thinking Control Guide](docs/API_AND_THINKING_GUIDE.md)