Initial commit: Complete deployment scripts, power governor, systemd units, and architecture documentation for Turing multi-GPU LLM rig
This commit is contained in:
wmantly
@@ -0,0 +1,82 @@
|
||||
# 🚀 Turing Multi-GPU LLM Inference Rig (32GB VRAM Cluster)
|
||||
|
||||
A production-grade, highly optimized software and hardware configuration for running large 27B+ parameter reasoning and multimodal vision LLMs (`Qwen3.8-27B-Uncensored`, `Qwen 2.5 32B`, `DeepSeek`) across heterogeneous NVIDIA Turing GPUs (**RTX 2060 12GB + 2x CMP 50HX 10GB** = **32 GB Total VRAM**) on an **HPE ProLiant DL380p Gen8** server with full **256K Context Window (262,144 tokens)**, **Multimodal Vision**, and **Automated Idle Power Management**.
|
||||
|
||||
---
|
||||
|
||||
## 📊 Rig Highlights & Performance
|
||||
|
||||
* **Model**: `Qwen3.8-27B-Uncensored` (`Q4_K_P` Quantization, 16.68 GB weights)
|
||||
* **Context Window**: **`262,144 tokens` (Full 256K Context)** with zero OOM crashes
|
||||
* **Multimodal Vision**: Active (`mmproj` ViT GGUF projector for image recognition)
|
||||
* **Decode Speed**: **`18.03 tok/s`** (sustained across 3 GPUs in NCCL tensor parallel)
|
||||
* **Prompt Processing Speed**: **`~127 tok/s`** (using Flash Attention & Turing cuBLAS GEMM)
|
||||
* **Idle Power**: Automatically downclocks CMP cards to 300–600 MHz, cutting idle draw from **200W+ down to ~100W** (~100W wall power savings!)
|
||||
* **API Compatibility**: Fully compatible with **Ollama** (`http://<ip>:11434`), **OpenAI** (`/v1/chat/completions`), and **Anthropic Messages** (`/v1/messages`) for OpenWebUI, Claude Code, and Continue.dev.
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Hardware & Cluster Architecture
|
||||
|
||||
```text
|
||||
Host System: HPE ProLiant DL380p Gen8 (Dual Intel Xeon E5-2620 v2 Ivy Bridge-EP)
|
||||
Total VRAM: 32 GB GDDR6 across 3 GPUs (Shared PCIe Root Complex)
|
||||
|
||||
[GPU 0: NVIDIA GeForce RTX 2060 12GB] (TU106, 1920 CUDA cores, 336 GB/s) -> Primary Vision & KV Slot
|
||||
[GPU 1: NVIDIA CMP 50HX 10GB Mining] (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
|
||||
[GPU 2: NVIDIA CMP 50HX 10GB Mining] (TU102, 3584 CUDA cores, 448 GB/s) -> Tensor Parallel Split
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📂 Repository Contents
|
||||
|
||||
```
|
||||
├── README.md # This overview and quickstart
|
||||
├── scripts/
|
||||
│ ├── build-nccl-llama.sh # Compiles llama.cpp with CUDA 12 + NCCL + sm_75 optimizations
|
||||
│ ├── ollama-proxy.py # High-performance Ollama/OpenAI/Anthropic proxy with thinking control
|
||||
│ ├── power-governor.py # Dynamic GPU power & clock governor for CMP 50HX cards
|
||||
│ ├── start-server.sh # Production start script for llama-server
|
||||
│ └── tune-gpus.sh # Helper script for power limits & persistence mode
|
||||
├── systemd/
|
||||
│ ├── llama-server.service # Systemd unit with NUMA socket pinning
|
||||
│ ├── ollama-proxy.service # Systemd unit for API proxy
|
||||
│ └── gpu-power-governor.service # Systemd unit for automated power governor
|
||||
└── docs/
|
||||
├── SETUP_GUIDE.md # Complete step-by-step setup guide on any Linux server
|
||||
├── PROXMOX_LXC_GUIDE.md # GPU passthrough & NVML permissions in Proxmox LXC
|
||||
├── HARDWARE_LEARNINGS.md # Deep dive: CMP VBIOS, PCIe Gen1 vs Gen3, riser cables, 20GB mods
|
||||
└── API_AND_THINKING_GUIDE.md # Controlling reasoning/thinking effort in OpenWebUI & API
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Quick Start
|
||||
|
||||
### 1. Build llama.cpp with NCCL Support
|
||||
```bash
|
||||
bash scripts/build-nccl-llama.sh /opt/llama.cpp
|
||||
```
|
||||
|
||||
### 2. Configure Systemd Services
|
||||
```bash
|
||||
sudo cp systemd/*.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
|
||||
```
|
||||
|
||||
### 3. Connect from OpenWebUI
|
||||
Set your Ollama URL in OpenWebUI to:
|
||||
```text
|
||||
http://<your-server-ip>:11434
|
||||
```
|
||||
Pick **`qwen:fast`** for instant answers (0 thinking delay) or **`qwen:think`** for deep multi-step reasoning!
|
||||
|
||||
---
|
||||
|
||||
## 📜 Documentation Links
|
||||
* [Detailed Step-by-Step Setup Guide](docs/SETUP_GUIDE.md)
|
||||
* [Proxmox LXC Passthrough & Permissions Guide](docs/PROXMOX_LXC_GUIDE.md)
|
||||
* [Hardware Architecture & CMP Learnings](docs/HARDWARE_LEARNINGS.md)
|
||||
* [API, OpenWebUI & Thinking Control Guide](docs/API_AND_THINKING_GUIDE.md)
|
||||
@@ -0,0 +1,82 @@
|
||||
# 🤖 API, OpenWebUI & Thinking / Reasoning Control Guide
|
||||
|
||||
The API proxy (`ollama-proxy.py`) translates incoming **Ollama**, **OpenAI**, and **Anthropic Messages** API calls into optimized requests for the `llama-server` backend.
|
||||
|
||||
---
|
||||
|
||||
## 1. Controlling Reasoning & Thinking
|
||||
|
||||
By default, Qwen3.8 and DeepSeek reasoning models have their Jinja template default set to `xhigh` effort, which can cause excessive thinking on simple queries. The proxy provides full granular control over thinking modes.
|
||||
|
||||
### Option A: Model Tag Aliases (Recommended for OpenWebUI Dropdown)
|
||||
|
||||
Select any of the registered alias tags directly in your client:
|
||||
|
||||
| Model Tag | Thinking Mode | Behavior |
|
||||
| :--- | :--- | :--- |
|
||||
| **`qwen:fast`** / **`qwen:nothink`** | **Disabled** (0 reasoning tokens) | Answers immediately with zero thinking delay! |
|
||||
| **`qwen`** / **`qwen:latest`** | **Low Effort** (Default) | Concise, focused 1–3 sentence reasoning trace before answering. |
|
||||
| **`qwen:think`** / **`qwen:deep`** | **High Effort (`xhigh`)** | Full deep multi-step reasoning for complex math/coding. |
|
||||
|
||||
---
|
||||
|
||||
### Option B: OpenWebUI UI Controls & Parameters
|
||||
|
||||
* **Thinking Toggle**: Toggle "Thinking" ON/OFF in the chat interface.
|
||||
* **Reasoning Effort Setting**:
|
||||
* `none` / `off` $\rightarrow$ Thinking disabled.
|
||||
* `low` $\rightarrow$ Brief, focused reasoning.
|
||||
* `medium` $\rightarrow$ Balanced reasoning.
|
||||
* `high` / `xhigh` $\rightarrow$ Deep reasoning.
|
||||
|
||||
---
|
||||
|
||||
### Option C: API Payload Parameters
|
||||
|
||||
#### 1. Disabling Thinking (Ollama Format)
|
||||
```json
|
||||
{
|
||||
"model": "qwen",
|
||||
"messages": [{"role": "user", "content": "What is 2+2?"}],
|
||||
"options": {
|
||||
"enable_thinking": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Specifying Thinking Token Budget (Anthropic / OpenAI Format)
|
||||
```json
|
||||
{
|
||||
"model": "qwen",
|
||||
"messages": [{"role": "user", "content": "Solve this equation: 3x + 12 = 45"}],
|
||||
"thinking": {
|
||||
"type": "enabled",
|
||||
"budget_tokens": 512
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Multimodal Vision Support
|
||||
|
||||
Send images directly via standard base64 strings in the `images` array (Ollama format) or `image_url` data URLs (OpenAI/Anthropic format).
|
||||
|
||||
The proxy features automatic **magic-byte MIME detection** supporting `image/png`, `image/jpeg`, `image/webp`, and `image/gif`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Supported API Endpoints
|
||||
|
||||
* **Ollama Endpoints**:
|
||||
* `POST /api/chat` (Streaming & non-streaming)
|
||||
* `POST /api/generate` (Streaming & non-streaming)
|
||||
* `GET /api/tags`
|
||||
* `POST /api/show`
|
||||
* `GET /api/ps`
|
||||
* `POST /api/embed` & `POST /api/embeddings`
|
||||
* **Anthropic Messages Endpoint**:
|
||||
* `POST /v1/messages` (Claude Code, Continue.dev, Anthropic SDK)
|
||||
* `POST /v1/messages/count_tokens`
|
||||
* **OpenAI Backend**:
|
||||
* `POST /v1/chat/completions` (Forwarded directly to `llama-server`)
|
||||
@@ -0,0 +1,70 @@
|
||||
# 🔬 Hardware Architecture & Key Technical Learnings
|
||||
|
||||
This document details the practical hardware behaviors, quirks, and engineering solutions discovered while building and optimizing this 3-GPU Turing inference rig on an **HPE ProLiant DL380p Gen8** server.
|
||||
|
||||
---
|
||||
|
||||
## 1. NVIDIA CMP 50HX Mining GPUs: VBIOS & Power States
|
||||
|
||||
### Why CMP 50HX Mining Cards Idle at ~75W–85W Stock
|
||||
* **No Display / Headless Architecture**: CMP 50HX mining cards lack display outputs and display engines.
|
||||
* **Missing Deep P8 VBIOS Tables**: Standard GeForce VBIOSes drop to P8 state (405 MHz GDDR6 / ~10W) when no display is active. Mining VBIOSes lock the cards in P0 state (1,890 MHz core / 7,000 MHz GDDR6) by default.
|
||||
|
||||
### OEM Reference VBIOS vs. MSI VBIOS
|
||||
Comparing the hardware profiles of two CMP 50HX cards on the exact same rig:
|
||||
* **GPU 2 (MSI Board `0x1462`, VBIOS `90.02.60.00.17`)**:
|
||||
* Idles at **`~31.8W`** when core is locked to 300–600 MHz.
|
||||
* Minimum fan speed: **`25%`**.
|
||||
* Aggressive P3 voltage gating tables.
|
||||
* **GPU 1 (NVIDIA Reference OEM `0x10DE`, VBIOS `90.02.60.00.01`)**:
|
||||
* Idles at **`~61.2W`** when core is locked to 300–600 MHz.
|
||||
* Minimum fan speed: **`40%`** (locked in VBIOS).
|
||||
* Higher static VRM voltage rail leakage.
|
||||
* **Actionable Solution**: Cross-flash GPU 2's MSI VBIOS (`90.02.60.00.17`) onto GPU 1 using `nvflash -6` to cut GPU 1's idle draw by ~30W and match fan curves.
|
||||
|
||||
---
|
||||
|
||||
## 2. Power Cap vs Inference Throughput
|
||||
|
||||
* **100W Power Cap (`-pl 100`)**: Capping CMP 50HX cards to 100W throttles GPU 1's core clock down to **1,305 MHz** during tensor-parallel splits, dropping generation speed from **18.0 tok/s $\rightarrow$ ~14.0 tok/s**.
|
||||
* **150W Power Cap (`-pl 150`)**: Provides sufficient headroom for full **1,800–1,900 MHz boost clocks**, sustaining **18.03 tok/s** while protecting against unnecessary 225W power spikes and heat.
|
||||
|
||||
---
|
||||
|
||||
## 3. Automated Dynamic Power Governor ([`power-governor.py`](../scripts/power-governor.py))
|
||||
|
||||
* **The Problem with GPU Utilization Polling**: On Turing GPUs during batch=1 token generation, `utilization.gpu` fluctuates between 0% and 5% between token steps. Relying on GPU utilization causes false idle triggers.
|
||||
* **The Solution**: Direct slot polling against `llama-server` (`http://127.0.0.1:8080/slots` `is_processing: True`).
|
||||
* **Governor Behavior**:
|
||||
* When `is_processing: True`: Instantly sets unconstrained clocks (`nvidia-smi -rgc`) $\rightarrow$ **18.03 tok/s**.
|
||||
* When idle for > 15 seconds: Sets low-power 300–600 MHz core clocks $\rightarrow$ **drops power to ~31W/card**.
|
||||
|
||||
---
|
||||
|
||||
## 4. HPE ProLiant DL380p Gen8 Platform Learnings
|
||||
|
||||
### Sandy Bridge-EP (v1) vs. Ivy Bridge-EP (v2) PCIe 3.0 Jitter
|
||||
* **The Sandy Bridge-EP v1 Bug**: Intel's first-generation PCIe 3.0 controller on `Xeon E5-2600 v1` (2012) suffered from transmitter signal margin attenuation (*Intel Errata BD78/BD105*). Over riser cables, the 8.0 GHz eye diagram degrades, forcing links down to Gen 1 (2.5 GT/s).
|
||||
* **The Ivy Bridge-EP v2 Fix**: `Xeon E5-2600 v2` (22nm Tri-Gate) completely redesigned the PCIe 3.0 PHY with CTLE equalization, reliably locking Gen 3 speeds across risers.
|
||||
|
||||
### NUMA Socket Pinning
|
||||
* Dual-socket Xeon servers have two distinct NUMA nodes.
|
||||
* If all GPUs are plugged into **Primary Riser 1**, they are physically connected to **CPU Socket 1 (NUMA Node 1)**.
|
||||
* Binding `llama-server` to NUMA Node 1 (`CPUAffinity` & `NUMAPolicy=bind`) eliminates all host-to-device memory traffic over the cross-socket Intel QPI bus.
|
||||
|
||||
### External PSU & Common Grounding
|
||||
* When GPUs receive PCIe data signals from the server motherboard but 12V power from an external PSU:
|
||||
* Ensure the **server chassis** and **external PSU casing** share a solid common ground to eliminate high-frequency ground loop noise on PCIe differential clock lines (`REFCLK`).
|
||||
|
||||
---
|
||||
|
||||
## 5. 20GB VRAM Modding Feasibility (CMP 50HX TU102)
|
||||
|
||||
* **Architecture**: CMP 50HX uses the **TU102 PCB layout (320-bit bus, 10 memory pads)**.
|
||||
* **Memory Swap**:
|
||||
* Desolder 10x 1GB (8Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4Z80325BC-HC14`).
|
||||
* Solder 10x 2GB (16Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4ZAF325BM-HC14` or Micron `D9ZCL`).
|
||||
* Modify memory strapping resistor dividers to signal 16Gbit density to the TU102 memory controller.
|
||||
* **Payoff**:
|
||||
* 2x Modded Cards = **40 GB Total VRAM** (capable of running full 70B/72B models like `Llama-3.3-70B` or `Qwen2.5-72B` on just 2 dedicated x16 slots).
|
||||
* 4x Modded Cards = **80 GB Total VRAM** (enterprise A100-tier capacity for under $1,000).
|
||||
@@ -0,0 +1,64 @@
|
||||
# 📦 Proxmox VE LXC Container GPU Passthrough & Hardware Permissions
|
||||
|
||||
Running multi-GPU AI inference and hardware power management inside a Proxmox LXC container requires specific device mappings, cgroup permissions, and capability flags.
|
||||
|
||||
---
|
||||
|
||||
## 1. Proxmox Host Configuration (`/etc/pve/lxc/<CTID>.conf`)
|
||||
|
||||
Add the following lines to your container configuration file on the Proxmox host:
|
||||
|
||||
```ini
|
||||
# /etc/pve/lxc/<CTID>.conf
|
||||
|
||||
# 1. Unconfined AppArmor profile (Required for NVML clock/power limit modification)
|
||||
lxc.apparmor.profile: unconfined
|
||||
|
||||
# 2. Grant SYS_ADMIN capability for hardware clock management
|
||||
lxc.cap.keep: sys_admin sys_rawio
|
||||
|
||||
# 3. Allow all NVIDIA device cgroups
|
||||
lxc.cgroup2.devices.allow: c 195:* rwm
|
||||
lxc.cgroup2.devices.allow: c 235:* rwm
|
||||
lxc.cgroup2.devices.allow: c 510:* rwm
|
||||
lxc.cgroup2.devices.allow: c 511:* rwm
|
||||
|
||||
# 4. Pass-through NVIDIA character device nodes
|
||||
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
|
||||
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Verifying Permissions Inside the Container
|
||||
|
||||
Restart the container, then run:
|
||||
|
||||
```bash
|
||||
# Verify all GPUs are visible
|
||||
nvidia-smi
|
||||
|
||||
# Test NVML Persistence Mode (Requires CAP_SYS_ADMIN)
|
||||
nvidia-smi -pm 1
|
||||
|
||||
# Test Power Capping
|
||||
nvidia-smi -i 1,2 -pl 150
|
||||
|
||||
# Test Dynamic Clock Locking
|
||||
nvidia-smi -i 1,2 -lgc 600,600
|
||||
nvidia-smi -i 1,2 -rgc
|
||||
```
|
||||
|
||||
If all four commands return with code `0` and "All done", your container has full hardware rights.
|
||||
|
||||
---
|
||||
|
||||
## 3. Important Systemd Service Security Flags
|
||||
|
||||
When running services that execute `nvidia-smi` hardware commands inside systemd:
|
||||
* Do **NOT** set `NoNewPrivileges=true` in `gpu-power-governor.service`. `NoNewPrivileges=true` blocks processes from acquiring permissions to execute privileged NVML clock-locking calls.
|
||||
@@ -0,0 +1,101 @@
|
||||
# 🛠️ Complete Multi-GPU LLM Server Setup Guide
|
||||
|
||||
This guide walks through deploying the complete multi-GPU inference stack on any fresh Ubuntu/Debian server or Proxmox container with NVIDIA Turing GPUs.
|
||||
|
||||
---
|
||||
|
||||
## 1. System Prerequisites
|
||||
|
||||
### Install Base Dependencies & NVIDIA Drivers
|
||||
```bash
|
||||
sudo apt-get update && sudo apt-get install -y \
|
||||
build-essential cmake ninja-build git curl wget \
|
||||
python3 python3-pip python3-venv \
|
||||
libcurl4-openssl-dev libssl-dev pkg-config numactl
|
||||
|
||||
# Ensure NVIDIA driver and CUDA Toolkit (12.x+) are installed
|
||||
nvidia-smi
|
||||
nvcc --version
|
||||
```
|
||||
|
||||
### Install NVIDIA NCCL (for Multi-GPU Tensor Parallelism)
|
||||
```bash
|
||||
# In Python venv or system:
|
||||
pip3 install nvidia-nccl-cu12
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Compile `llama.cpp` with NCCL & Turing cuBLAS Optimization
|
||||
|
||||
Turing architecture (`sm_75`) requires specific CMake flags to avoid throttled DP4A integer paths and enable fast cuBLAS GEMM tensor parallel synchronization:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
|
||||
cd /opt/llama.cpp
|
||||
|
||||
mkdir -p build-nccl
|
||||
cd build-nccl
|
||||
|
||||
cmake .. \
|
||||
-GNinja \
|
||||
-DGGML_CUDA=ON \
|
||||
-DGGML_CUDA_GRAPHS=ON \
|
||||
-DGGML_CUDA_FORCE_CUBLAS=ON \
|
||||
-DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \
|
||||
-DGGML_CUDA_ARCHITECTURES="75" \
|
||||
-DCMAKE_BUILD_TYPE=Release
|
||||
|
||||
ninja llama-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Preparing Model Weights & Multimodal Vision Projector
|
||||
|
||||
### Download GGUF Model Weights
|
||||
```bash
|
||||
mkdir -p /opt/models/gguf
|
||||
cd /opt/models/gguf
|
||||
|
||||
# Download Qwen3.8-27B-Uncensored (or any Qwen2.5 / 27B / 32B model)
|
||||
wget -c "https://huggingface.co/.../Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf"
|
||||
```
|
||||
|
||||
### Extracting Vision Projector (`mmproj`)
|
||||
If converting from Hugging Face safetensors, extract the multimodal ViT projector to GGUF using `convert_image_encoder_to_gguf.py`:
|
||||
```bash
|
||||
python3 /opt/llama.cpp/examples/llava/convert_image_encoder_to_gguf.py \
|
||||
-m /opt/models/orcarouter_Qwen3.8-27B-Uncensored \
|
||||
--output-dir /opt/models/gguf \
|
||||
--llava-projector
|
||||
```
|
||||
Result: `/opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf` (931 MB).
|
||||
|
||||
---
|
||||
|
||||
## 4. Key Server Parameters Explained
|
||||
|
||||
In `scripts/start-server.sh`:
|
||||
* `--split-mode tensor`: Splits every attention head and FFN layer across all 3 GPUs simultaneously using NCCL AllReduce.
|
||||
* `-c 262144`: Enables the full 256K token context window.
|
||||
* `--parallel 1`: Allocates a single dedicated KV cache slot to prevent multi-slot VRAM duplication.
|
||||
* `--cache-type-k q4_0 --cache-type-v q4_0`: Quantizes the KV cache to 4-bit, shrinking 256K context memory footprint by 75% (down to ~8.8 GB).
|
||||
* `--image-max-tokens 2048`: Prevents out-of-memory spikes when decoding high-resolution 4K images.
|
||||
* `--jinja`: Uses native Jinja chat templates with reasoning control.
|
||||
|
||||
---
|
||||
|
||||
## 5. Systemd Production Deployment
|
||||
|
||||
Copy service files and enable on boot:
|
||||
```bash
|
||||
sudo cp systemd/*.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
|
||||
```
|
||||
|
||||
Verify services:
|
||||
```bash
|
||||
systemctl status llama-server ollama-proxy gpu-power-governor
|
||||
```
|
||||
Executable
+32
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
# ==============================================================================
|
||||
# Build script for compiling llama.cpp with NVIDIA NCCL tensor parallelism
|
||||
# Optimized for Turing GPUs (RTX 2060, CMP 50HX, sm_75)
|
||||
# ==============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
LLAMA_DIR="${1:-/opt/llama.cpp}"
|
||||
BUILD_DIR="${LLAMA_DIR}/build-nccl"
|
||||
|
||||
echo "==> Ensuring build dependencies are installed..."
|
||||
apt-get update -qq && apt-get install -y -qq \
|
||||
build-essential cmake git ninja-build \
|
||||
libcurl4-openssl-dev libssl-dev pkg-config
|
||||
|
||||
echo "==> Configuring CMake for llama.cpp with NCCL and CUDA sm_75..."
|
||||
mkdir -p "$BUILD_DIR"
|
||||
cd "$BUILD_DIR"
|
||||
|
||||
cmake .. \
|
||||
-GNinja \
|
||||
-DGGML_CUDA=ON \
|
||||
-DGGML_CUDA_GRAPHS=ON \
|
||||
-DGGML_CUDA_FORCE_CUBLAS=ON \
|
||||
-DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \
|
||||
-DGGML_CUDA_ARCHITECTURES="75" \
|
||||
-DCMAKE_BUILD_TYPE=Release
|
||||
|
||||
echo "==> Building llama-server..."
|
||||
ninja llama-server
|
||||
|
||||
echo "==> Build complete! Binary located at: ${BUILD_DIR}/bin/llama-server"
|
||||
File diff suppressed because it is too large
Load Diff
Executable
+67
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
power-governor.py — Dynamic clock and power governor for CMP 50HX GPUs.
|
||||
|
||||
Directly queries llama-server slot state for 100% accurate active/idle tracking:
|
||||
- When generating (is_processing: True): Instantly sets full 1900 MHz boost clocks.
|
||||
- When idle (is_processing: False for > 15s): Downclocks CMP cards to 600 MHz (~32W/card).
|
||||
"""
|
||||
|
||||
import time
|
||||
import subprocess
|
||||
import urllib.request
|
||||
import json
|
||||
import sys
|
||||
|
||||
CMP_GPUS = "1,2"
|
||||
IDLE_TIMEOUT_SEC = 15
|
||||
LOW_POWER_CLOCK = 300
|
||||
|
||||
def run_cmd(*args):
|
||||
try:
|
||||
subprocess.run(args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def is_server_processing():
|
||||
try:
|
||||
req = urllib.request.Request("http://127.0.0.1:8080/slots", headers={"User-Agent": "power-governor"})
|
||||
with urllib.request.urlopen(req, timeout=1.0) as r:
|
||||
slots = json.loads(r.read().decode())
|
||||
return any(bool(s.get("is_processing")) for s in slots)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def main():
|
||||
print("[power-governor] Initializing GPU persistence and 150W power cap...", flush=True)
|
||||
run_cmd("nvidia-smi", "-pm", "1")
|
||||
run_cmd("nvidia-smi", f"-i={CMP_GPUS}", "-pl", "150")
|
||||
run_cmd("nvidia-smi", f"-i={CMP_GPUS}", "-rgc")
|
||||
|
||||
is_low_power = False
|
||||
last_active = time.time()
|
||||
|
||||
print("[power-governor] Monitoring llama-server processing state...", flush=True)
|
||||
while True:
|
||||
try:
|
||||
active = is_server_processing()
|
||||
now = time.time()
|
||||
|
||||
if active:
|
||||
last_active = now
|
||||
if is_low_power:
|
||||
run_cmd("nvidia-smi", f"-i={CMP_GPUS}", "-rgc")
|
||||
is_low_power = False
|
||||
print(f"[{time.strftime('%X')}] Inference Active -> Boost clocks engaged (1900 MHz)", flush=True)
|
||||
else:
|
||||
if not is_low_power and (now - last_active) > IDLE_TIMEOUT_SEC:
|
||||
run_cmd("nvidia-smi", f"-i={CMP_GPUS}", f"-lgc={LOW_POWER_CLOCK},{LOW_POWER_CLOCK}")
|
||||
is_low_power = True
|
||||
print(f"[{time.strftime('%X')}] Inference Idle -> Low-power state engaged (600 MHz, ~32W/card)", flush=True)
|
||||
|
||||
time.sleep(0.5 if is_low_power else 1.0)
|
||||
except Exception as e:
|
||||
time.sleep(3.0)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+31
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# ==============================================================================
|
||||
# Production startup script for llama-server with NCCL tensor parallelism
|
||||
# Supports: 256K Context Window, Multimodal Vision Projector, Flash Attention,
|
||||
# q4_0 KV Cache quantization, and Jinja reasoning control.
|
||||
# ==============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
MODEL_PATH="${MODEL_PATH:-/opt/models/gguf/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf}"
|
||||
MMPROJ_PATH="${MMPROJ_PATH:-/opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf}"
|
||||
BINARY="${LLAMA_BINARY:-/opt/llama.cpp/build-nccl/bin/llama-server}"
|
||||
|
||||
export LD_LIBRARY_PATH="/opt/llama.cpp/build-nccl/bin:/opt/minicpm-venv/lib/python3.13/site-packages/nvidia/nccl/lib:/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}"
|
||||
|
||||
exec "$BINARY" \
|
||||
-m "$MODEL_PATH" \
|
||||
--mmproj "$MMPROJ_PATH" \
|
||||
-ngl 99 \
|
||||
-c 262144 \
|
||||
--parallel 1 \
|
||||
--image-max-tokens 2048 \
|
||||
--split-mode tensor \
|
||||
--flash-attn on \
|
||||
--batch-size 1024 \
|
||||
--ubatch-size 256 \
|
||||
--jinja \
|
||||
--threads 12 \
|
||||
--cache-type-k q4_0 \
|
||||
--cache-type-v q4_0 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8080
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/usr/bin/env bash
|
||||
# ==============================================================================
|
||||
# GPU Persistence Mode and Power Cap Tuner
|
||||
# Sets safe 150W power cap on CMP 50HX cards and activates persistence mode.
|
||||
# ==============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
echo "==> Enabling NVIDIA Persistence Mode..."
|
||||
nvidia-smi -pm 1
|
||||
|
||||
echo "==> Setting 150W Power Cap on CMP 50HX GPUs (index 1 and 2)..."
|
||||
nvidia-smi -i 1,2 -pl 150
|
||||
|
||||
echo "==> Resetting GPU clocks to unconstrained boost..."
|
||||
nvidia-smi -rgc
|
||||
|
||||
echo "==> Current GPU States:"
|
||||
nvidia-smi --query-gpu=index,name,power.draw,power.limit,clocks.gr,clocks.mem --format=csv
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Dynamic GPU Power & Clock Governor for CMP 50HX
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/python3 /opt/llama-server/power-governor.py
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,33 @@
|
||||
[Unit]
|
||||
Description=llama.cpp OpenAI-compatible server (Qwen3.8-27B Q4_K_P, 3x Turing GPUs, 256K ctx)
|
||||
Documentation=file:///opt/llama-server/README.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Required: without this the loader binds a stale libcuda.so.1 (550) from
|
||||
# /usr/lib/x86_64-linux-gnu/nvidia/current and CUDA init fails with
|
||||
# "system has unsupported display driver / cuda driver combination".
|
||||
Environment=LD_LIBRARY_PATH=/opt/llama.cpp/build-nccl/bin:/opt/minicpm-venv/lib/python3.13/site-packages/nvidia/nccl/lib:/lib/x86_64-linux-gnu
|
||||
ExecStart=/opt/llama-server/start.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
# Model load takes ~20s; give it time to come up cleanly
|
||||
TimeoutStartSec=300
|
||||
TimeoutStopSec=60
|
||||
# Basic hardening (server only listens on :8080, reads models read-only)
|
||||
ProtectSystem=strict
|
||||
ProtectHome=read-only
|
||||
ReadWritePaths=
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
# Do not cap memory/CPU; inference needs all of it
|
||||
LimitNOFILE=1048576
|
||||
# NUMA Affinity: Pin to CPU Socket 1 (NUMA Node 1) where Riser 1 GPUs physically connect (24-thread v2 CPU)
|
||||
CPUAffinity=6-11,18-23
|
||||
NUMAPolicy=bind
|
||||
NUMAMask=1
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Ollama+Anthropic API proxy backed by llama.cpp (Qwen3.8-27B)
|
||||
Documentation=file:///opt/llama-server/USAGE-CLIENTS.md
|
||||
After=network-online.target llama-server.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/python3 /opt/llama-server/ollama-proxy.py
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
TimeoutStopSec=30
|
||||
# proxy does not need GPU access itself; it forwards to llama-server on 127.0.0.1:8080
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
LimitNOFILE=1048576
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user