🛠️ Complete Multi-GPU LLM Server Setup Guide

This guide walks through deploying the complete multi-GPU inference stack on any fresh Ubuntu/Debian server or Proxmox container with NVIDIA Turing GPUs.

1. System Prerequisites

Install Base Dependencies & NVIDIA Drivers

sudo apt-get update && sudo apt-get install -y \
    build-essential cmake ninja-build git curl wget \
    python3 python3-pip python3-venv \
    libcurl4-openssl-dev libssl-dev pkg-config numactl

# Ensure NVIDIA driver and CUDA Toolkit (12.x+) are installed
nvidia-smi
nvcc --version

Install NVIDIA NCCL (for Multi-GPU Tensor Parallelism)

# In Python venv or system:
pip3 install nvidia-nccl-cu12

2. Compile `llama.cpp` with NCCL & Turing cuBLAS Optimization

Turing architecture (sm_75) requires specific CMake flags to avoid throttled DP4A integer paths and enable fast cuBLAS GEMM tensor parallel synchronization:

git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
cd /opt/llama.cpp

mkdir -p build-nccl
cd build-nccl

cmake .. \
    -GNinja \
    -DGGML_CUDA=ON \
    -DGGML_CUDA_GRAPHS=ON \
    -DGGML_CUDA_FORCE_CUBLAS=ON \
    -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \
    -DGGML_CUDA_ARCHITECTURES="75" \
    -DCMAKE_BUILD_TYPE=Release

ninja llama-server

3. Preparing Model Weights & Multimodal Vision Projector

Download GGUF Model Weights

mkdir -p /opt/models/gguf
cd /opt/models/gguf

# Download Qwen3.8-27B-Uncensored (or any Qwen2.5 / 27B / 32B model)
wget -c "https://huggingface.co/.../Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf"

Extracting Vision Projector (`mmproj`)

If converting from Hugging Face safetensors, extract the multimodal ViT projector to GGUF using convert_image_encoder_to_gguf.py:

python3 /opt/llama.cpp/examples/llava/convert_image_encoder_to_gguf.py \
    -m /opt/models/orcarouter_Qwen3.8-27B-Uncensored \
    --output-dir /opt/models/gguf \
    --llava-projector

Result: /opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf (931 MB).

4. Key Server Parameters Explained

In scripts/start-server.sh:

--split-mode tensor: Splits every attention head and FFN layer across all 3 GPUs simultaneously using NCCL AllReduce.
-c 262144: Enables the full 256K token context window.
--parallel 1: Allocates a single dedicated KV cache slot to prevent multi-slot VRAM duplication.
--cache-type-k q4_0 --cache-type-v q4_0: Quantizes the KV cache to 4-bit, shrinking 256K context memory footprint by 75% (down to ~8.8 GB).
--image-max-tokens 2048: Prevents out-of-memory spikes when decoding high-resolution 4K images.
--jinja: Uses native Jinja chat templates with reasoning control.

5. Systemd Production Deployment

Copy service files and enable on boot:

sudo cp systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service

Verify services:

systemctl status llama-server ollama-proxy gpu-power-governor

3.0 KiB Raw Blame History