# 🛠️ Complete Multi-GPU LLM Server Setup Guide

This guide walks through deploying the complete multi-GPU inference stack on any fresh Ubuntu/Debian server or Proxmox container with NVIDIA Turing GPUs.

---

## 1. System Prerequisites

### Install Base Dependencies & NVIDIA Drivers
```bash
sudo apt-get update && sudo apt-get install -y \
    build-essential cmake ninja-build git curl wget \
    python3 python3-pip python3-venv \
    libcurl4-openssl-dev libssl-dev pkg-config numactl

# Ensure NVIDIA driver and CUDA Toolkit (12.x+) are installed
nvidia-smi
nvcc --version
```

### Install NVIDIA NCCL (for Multi-GPU Tensor Parallelism)
```bash
# In Python venv or system:
pip3 install nvidia-nccl-cu12
```

---

## 2. Compile `llama.cpp` with NCCL & Turing cuBLAS Optimization

Turing architecture (`sm_75`) requires specific CMake flags to avoid throttled DP4A integer paths and enable fast cuBLAS GEMM tensor parallel synchronization:

```bash
git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
cd /opt/llama.cpp

mkdir -p build-nccl
cd build-nccl

cmake .. \
    -GNinja \
    -DGGML_CUDA=ON \
    -DGGML_CUDA_GRAPHS=ON \
    -DGGML_CUDA_FORCE_CUBLAS=ON \
    -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \
    -DGGML_CUDA_ARCHITECTURES="75" \
    -DCMAKE_BUILD_TYPE=Release

ninja llama-server
```

---

## 3. Preparing Model Weights & Multimodal Vision Projector

### Download GGUF Model Weights
```bash
mkdir -p /opt/models/gguf
cd /opt/models/gguf

# Download Qwen3.8-27B-Uncensored (or any Qwen2.5 / 27B / 32B model)
wget -c "https://huggingface.co/.../Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf"
```

### Extracting Vision Projector (`mmproj`)
If converting from Hugging Face safetensors, extract the multimodal ViT projector to GGUF using `convert_image_encoder_to_gguf.py`:
```bash
python3 /opt/llama.cpp/examples/llava/convert_image_encoder_to_gguf.py \
    -m /opt/models/orcarouter_Qwen3.8-27B-Uncensored \
    --output-dir /opt/models/gguf \
    --llava-projector
```
Result: `/opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf` (931 MB).

---

## 4. Key Server Parameters Explained

In `scripts/start-server.sh`:
* `--split-mode tensor`: Splits every attention head and FFN layer across all 3 GPUs simultaneously using NCCL AllReduce.
* `-c 262144`: Enables the full 256K token context window.
* `--parallel 1`: Allocates a single dedicated KV cache slot to prevent multi-slot VRAM duplication.
* `--cache-type-k q4_0 --cache-type-v q4_0`: Quantizes the KV cache to 4-bit, shrinking 256K context memory footprint by 75% (down to ~8.8 GB).
* `--image-max-tokens 2048`: Prevents out-of-memory spikes when decoding high-resolution 4K images.
* `--jinja`: Uses native Jinja chat templates with reasoning control.

---

## 5. Systemd Production Deployment

Copy service files and enable on boot:
```bash
sudo cp systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
```

Verify services:
```bash
systemctl status llama-server ollama-proxy gpu-power-governor
```