Initial commit: Complete deployment scripts, power governor, systemd units, and architecture documentation for Turing multi-GPU LLM rig

2026-09-01 01:55:19 +00:00
commit aeb72cecfa
13 changed files with 1925 additions and 0 deletions
@@ -0,0 +1,101 @@
+# 🛠️ Complete Multi-GPU LLM Server Setup Guide
+
+This guide walks through deploying the complete multi-GPU inference stack on any fresh Ubuntu/Debian server or Proxmox container with NVIDIA Turing GPUs.
+
+---
+
+## 1. System Prerequisites
+
+### Install Base Dependencies & NVIDIA Drivers
+```bash
+sudo apt-get update && sudo apt-get install -y \
+    build-essential cmake ninja-build git curl wget \
+    python3 python3-pip python3-venv \
+    libcurl4-openssl-dev libssl-dev pkg-config numactl
+
+# Ensure NVIDIA driver and CUDA Toolkit (12.x+) are installed
+nvidia-smi
+nvcc --version
+```
+
+### Install NVIDIA NCCL (for Multi-GPU Tensor Parallelism)
+```bash
+# In Python venv or system:
+pip3 install nvidia-nccl-cu12
+```
+
+---
+
+## 2. Compile `llama.cpp` with NCCL & Turing cuBLAS Optimization
+
+Turing architecture (`sm_75`) requires specific CMake flags to avoid throttled DP4A integer paths and enable fast cuBLAS GEMM tensor parallel synchronization:
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
+cd /opt/llama.cpp
+
+mkdir -p build-nccl
+cd build-nccl
+
+cmake .. \
+    -GNinja \
+    -DGGML_CUDA=ON \
+    -DGGML_CUDA_GRAPHS=ON \
+    -DGGML_CUDA_FORCE_CUBLAS=ON \
+    -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \
+    -DGGML_CUDA_ARCHITECTURES="75" \
+    -DCMAKE_BUILD_TYPE=Release
+
+ninja llama-server
+```
+
+---
+
+## 3. Preparing Model Weights & Multimodal Vision Projector
+
+### Download GGUF Model Weights
+```bash
+mkdir -p /opt/models/gguf
+cd /opt/models/gguf
+
+# Download Qwen3.8-27B-Uncensored (or any Qwen2.5 / 27B / 32B model)
+wget -c "https://huggingface.co/.../Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf"
+```
+
+### Extracting Vision Projector (`mmproj`)
+If converting from Hugging Face safetensors, extract the multimodal ViT projector to GGUF using `convert_image_encoder_to_gguf.py`:
+```bash
+python3 /opt/llama.cpp/examples/llava/convert_image_encoder_to_gguf.py \
+    -m /opt/models/orcarouter_Qwen3.8-27B-Uncensored \
+    --output-dir /opt/models/gguf \
+    --llava-projector
+```
+Result: `/opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf` (931 MB).
+
+---
+
+## 4. Key Server Parameters Explained
+
+In `scripts/start-server.sh`:
+* `--split-mode tensor`: Splits every attention head and FFN layer across all 3 GPUs simultaneously using NCCL AllReduce.
+* `-c 262144`: Enables the full 256K token context window.
+* `--parallel 1`: Allocates a single dedicated KV cache slot to prevent multi-slot VRAM duplication.
+* `--cache-type-k q4_0 --cache-type-v q4_0`: Quantizes the KV cache to 4-bit, shrinking 256K context memory footprint by 75% (down to ~8.8 GB).
+* `--image-max-tokens 2048`: Prevents out-of-memory spikes when decoding high-resolution 4K images.
+* `--jinja`: Uses native Jinja chat templates with reasoning control.
+
+---
+
+## 5. Systemd Production Deployment
+
+Copy service files and enable on boot:
+```bash
+sudo cp systemd/*.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
+```
+
+Verify services:
+```bash
+systemctl status llama-server ollama-proxy gpu-power-governor
+```