102 lines
3.0 KiB
Markdown
102 lines
3.0 KiB
Markdown
# 🛠️ Complete Multi-GPU LLM Server Setup Guide
|
|
|
|
This guide walks through deploying the complete multi-GPU inference stack on any fresh Ubuntu/Debian server or Proxmox container with NVIDIA Turing GPUs.
|
|
|
|
---
|
|
|
|
## 1. System Prerequisites
|
|
|
|
### Install Base Dependencies & NVIDIA Drivers
|
|
```bash
|
|
sudo apt-get update && sudo apt-get install -y \
|
|
build-essential cmake ninja-build git curl wget \
|
|
python3 python3-pip python3-venv \
|
|
libcurl4-openssl-dev libssl-dev pkg-config numactl
|
|
|
|
# Ensure NVIDIA driver and CUDA Toolkit (12.x+) are installed
|
|
nvidia-smi
|
|
nvcc --version
|
|
```
|
|
|
|
### Install NVIDIA NCCL (for Multi-GPU Tensor Parallelism)
|
|
```bash
|
|
# In Python venv or system:
|
|
pip3 install nvidia-nccl-cu12
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Compile `llama.cpp` with NCCL & Turing cuBLAS Optimization
|
|
|
|
Turing architecture (`sm_75`) requires specific CMake flags to avoid throttled DP4A integer paths and enable fast cuBLAS GEMM tensor parallel synchronization:
|
|
|
|
```bash
|
|
git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
|
|
cd /opt/llama.cpp
|
|
|
|
mkdir -p build-nccl
|
|
cd build-nccl
|
|
|
|
cmake .. \
|
|
-GNinja \
|
|
-DGGML_CUDA=ON \
|
|
-DGGML_CUDA_GRAPHS=ON \
|
|
-DGGML_CUDA_FORCE_CUBLAS=ON \
|
|
-DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \
|
|
-DGGML_CUDA_ARCHITECTURES="75" \
|
|
-DCMAKE_BUILD_TYPE=Release
|
|
|
|
ninja llama-server
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Preparing Model Weights & Multimodal Vision Projector
|
|
|
|
### Download GGUF Model Weights
|
|
```bash
|
|
mkdir -p /opt/models/gguf
|
|
cd /opt/models/gguf
|
|
|
|
# Download Qwen3.8-27B-Uncensored (or any Qwen2.5 / 27B / 32B model)
|
|
wget -c "https://huggingface.co/.../Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf"
|
|
```
|
|
|
|
### Extracting Vision Projector (`mmproj`)
|
|
If converting from Hugging Face safetensors, extract the multimodal ViT projector to GGUF using `convert_image_encoder_to_gguf.py`:
|
|
```bash
|
|
python3 /opt/llama.cpp/examples/llava/convert_image_encoder_to_gguf.py \
|
|
-m /opt/models/orcarouter_Qwen3.8-27B-Uncensored \
|
|
--output-dir /opt/models/gguf \
|
|
--llava-projector
|
|
```
|
|
Result: `/opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf` (931 MB).
|
|
|
|
---
|
|
|
|
## 4. Key Server Parameters Explained
|
|
|
|
In `scripts/start-server.sh`:
|
|
* `--split-mode tensor`: Splits every attention head and FFN layer across all 3 GPUs simultaneously using NCCL AllReduce.
|
|
* `-c 262144`: Enables the full 256K token context window.
|
|
* `--parallel 1`: Allocates a single dedicated KV cache slot to prevent multi-slot VRAM duplication.
|
|
* `--cache-type-k q4_0 --cache-type-v q4_0`: Quantizes the KV cache to 4-bit, shrinking 256K context memory footprint by 75% (down to ~8.8 GB).
|
|
* `--image-max-tokens 2048`: Prevents out-of-memory spikes when decoding high-resolution 4K images.
|
|
* `--jinja`: Uses native Jinja chat templates with reasoning control.
|
|
|
|
---
|
|
|
|
## 5. Systemd Production Deployment
|
|
|
|
Copy service files and enable on boot:
|
|
```bash
|
|
sudo cp systemd/*.service /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now llama-server.service ollama-proxy.service gpu-power-governor.service
|
|
```
|
|
|
|
Verify services:
|
|
```bash
|
|
systemctl status llama-server ollama-proxy gpu-power-governor
|
|
```
|