wmantly/turing-multi-gpu-llm-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: update to Q5_K_P @ 128K ctx, non-blocking proxy sessions, and full multi-GPU benchmark suite

Browse Source
This commit is contained in:
wmantly
2026-09-13 02:14:25 +00:00
parent cfeac35ad6
commit cd989b3e70
5 changed files with 406 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3 -1
View File
@@ -78,7 +78,9 @@ Pick **`qwen:fast`** for instant answers (0 thinking delay) or **`qwen:think`**
---
## 📜 Documentation Links
## 📜 Documentation & Benchmark Links
* [Comprehensive Multi-GPU Benchmark Results (Q4/Q5/Q6/Gemma4 MoE)](benchmarks/BENCHMARK_RESULTS.md)
* [Detailed Client Usage Guide (OpenWebUI, Ollama, OpenAI, Anthropic, Continue.dev)](docs/USAGE-CLIENTS.md)
* [Detailed Step-by-Step Setup Guide](docs/SETUP_GUIDE.md)
* [Proxmox LXC Passthrough & Permissions Guide](docs/PROXMOX_LXC_GUIDE.md)
* [Hardware Architecture & CMP Learnings](docs/HARDWARE_LEARNINGS.md)
benchmarks/BENCHMARK_RESULTS.md
+58
View File
@@ -0,0 +1,58 @@
# Multi-GPU Quantization Benchmark: Qwen3.8-27B vs Gemma4-26B-A4B (Turing SM75)
Comprehensive performance benchmark, memory footprint comparison, context window scaling, and architecture analysis across 3x NVIDIA Turing GPUs (RTX 2060 12GB + 2x CMP 50HX 10GB).
---
## 1. Hardware & System Architecture
- **Host Processor**: Intel Xeon E5-2697 v2 (12C/24T, Ivy Bridge @ 2.70 GHz base, AVX1 support)
- **Active GPUs**: 3x NVIDIA Turing GPUs (**32,768 MiB physical VRAM total**)
- `CUDA0` (Bus `21:00.0`): **NVIDIA GeForce RTX 2060 12GB** (11,841 MiB usable) @ PCIe 3.0 x16
- `CUDA1` (Bus `24:00.0`): **NVIDIA CMP 50HX 10GB** (9,798 MiB usable) @ PCIe 3.0 x16
- `CUDA2` (Bus `27:00.0`): **NVIDIA CMP 50HX 10GB** (9,798 MiB usable) @ PCIe 3.0 x16
- **Interconnect**: NVIDIA NCCL / Layer Split pipeline across 3 PCIe buses
---
## 2. Head-to-Head Benchmark Matrix: Dense vs MoE
| Metric / Benchmark Stage | **Gemma4-26B-A4B-QAT @ 256K Ctx** *(Active Now)* | **Qwen3.8-27B (Q5_K_P @ 192K)** | **Qwen3.8-27B (Q4_K_P @ 256K)** |
| :--- | :--- | :--- | :--- |
| **Architecture Type** | **Sparse MoE (128 experts / 8 active)** | Dense (64 layers) | Dense (64 layers) |
| **Active Params / Total Params** | **4.0B active / 26B total** | 27.5B active / 27.5B total | 27.5B active / 27.5B total |
| **Model Disk Size** | **15.64 GB** (`Q4_K_M`) | 20.22 GB (`Q5_K_P`) | 17.92 GB (`Q4_K_P`) |
| **KV Cache Precision** | **`q5_0` (5.5-bit)** | `q5_0` (5.5-bit) | `q4_0` (4.5-bit) |
| **Max Context Window** | **256K** (`262,144 tokens`) | 192K (`196,608 tokens`) | 256K (`262,144 tokens`) |
| **Multimodal Support** | **Text + Vision + Video (`mmproj-BF16`)** | Text + Vision (`mmproj-f16`) | Text + Vision (`mmproj-f16`) |
| **Split Mode** | **Layer Split (`--split-mode layer`)** | NCCL Tensor Split | NCCL Tensor Split |
| **Active Memory Footprint** | **9.09 GB / 9.25 GB / 7.67 GB** | 11.60 GB / 9.07 GB / 9.05 GB | 11.75 GB / 8.89 GB / 8.87 GB |
| **Total VRAM Allocated** | **26.00 GB (83%)** | 29.72 GB (95%) | 29.51 GB (94%) |
| **Free Headroom / GPU** | **+2.75 GB / +0.55 GB / +2.13 GB** | +240 MB / +720 MB / +740 MB | +90 MB / +905 MB / +923 MB |
| **1. Short Prompt (100 tok)** | | | |
| • Prefill Speed | **95.87 tok/s** | 53.66 tok/s | 52.46 tok/s |
| • Decode Speed | **46.00 tok/s** | 37.14 tok/s | 38.12 tok/s |
| • Total Wall Time | **2.44 s** | 3.76 s | 3.47 s |
| **2. Medium Context (~5k tok)** | | | |
| • Prefill Speed | **2,279.22 tok/s** *(5.8x faster)* | 388.85 tok/s | 391.23 tok/s |
| • Decode Speed | **45.42 tok/s** | 33.81 tok/s | 41.42 tok/s |
| • Total Wall Time | **4.40 s** *(3.8x faster)* | 16.79 s | 16.33 s |
| **3. Long Context (~6k tok)** | | | |
| • Prefill Speed | **2,374.57 tok/s** *(6.0x faster)* | 396.52 tok/s | 398.61 tok/s |
| • Decode Speed | **43.67 tok/s** | 35.99 tok/s | 41.00 tok/s |
| • Total Wall Time | **4.97 s** *(3.9x faster)* | 19.60 s | 19.47 s |
| **4. Sustained Decode (256 tok)** | | | |
| • Decode Speed | **47.50 tok/s** *(+58% faster)* | 30.10 tok/s | 31.97 tok/s |
| • Total Wall Time | **5.82 s** | 10.71 s | 10.17 s |
---
## 3. Context Scaling Matrix on Gemma4-26B-A4B
| Context Window | KV Cache Quant Type | Total VRAM Across Rig | RTX 2060 12GB | CMP 50HX #1 10GB | CMP 50HX #2 10GB | Status |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **64K** (`65,536`) | `q5_0` (5.5-bit) | 19.5 GB | 6.8 GB | 7.1 GB | 5.6 GB | Rock Solid |
| **128K** (`131,072`) | `q5_0` (5.5-bit) | 22.0 GB | 7.7 GB | 7.9 GB | 6.5 GB | Rock Solid |
| **192K** (`196,608`) | `q5_0` (5.5-bit) | 24.0 GB | 8.4 GB | 8.6 GB | 7.0 GB | Stable |
| **256K** (`262,144`) | `q5_0` (5.5-bit) | **26.0 GB** | **9.09 GB** | **9.25 GB** | **7.67 GB** | **Full Native Ceiling** (+550MB margin) |
| **192K** (`196,608`) | `q8_0` (8.5-bit) | 27.8 GB | 9.8 GB | 9.6 GB | 8.4 GB | Max Precision Ceiling |
docs/USAGE-CLIENTS.md
+303
View File
@@ -0,0 +1,303 @@
# Connecting clients to the llama.cpp server
This guide covers wiring the Qwen3.8-27B llama.cpp server (OpenAI-compatible
API on `:8080`) into Open WebUI, opencode, Claude Code, and other tools.
Server: `http://<HOST>:8080`
Base URL: `http://<HOST>:8080/v1` ← use this for OpenAI-style clients
> Note: llama.cpp exposes an **OpenAI-compatible** API. It is *not* an
> Ollama-protocol server (no `GET /api/tags`, `/api/chat`, etc.). Most tools
> accept OpenAI-style endpoints, so that's fine. If you need a true Ollama
> clone API, see §6.
---
## 0. Quick reference — endpoints
| Endpoint | Purpose |
|---|---|
| `GET /v1/models` | list models |
| `POST /v1/chat/completions` | chat (reasoning model: returns `reasoning_content`) |
| `POST /v1/completions` | raw completions |
| `POST /v1/embeddings` | embeddings |
| `GET /health` | liveness |
| `GET /props` | server params (context size, etc.) |
Auth is optional (no `--api-key` set). If you set one later, pass it as
`Authorization: Bearer <key>`.
---
## 1. Open WebUI
Open WebUI (openwebui.com) connects fine via its **OpenAI API** connection.
### 1.1 Docker (recommended)
```bash
docker run -d -p 3000:8080 \
-v open-webui:/app/backend/data \
--name open-webui \
--restart always \
ghcr.io/open-webui/open-webui:main
```
### 1.2 Point it at the llama.cpp server
1. Open `http://<HOST>:3000` and create an admin account.
2. **Admin panel → Settings → Connections → OpenAI API.**
- **API Base URL:** `http://<HOST>:8080/v1`
- **API Key:** anything non-empty, e.g. `local`
- Enable if you want: "Enable Reasoning Content" (shows `thinking` blocks).
3. Click refresh/save. The model `Qwen3.8-27B-Uncensored-...` should appear in
the model picker.
Notes:
- Open WebUI's **Ollama** connection type will *not* see this server (different
protocol). Use the **OpenAI API** connection type.
- Optional: set `OPENAI_API_BASE_URL` / `OPENAI_API_KEY` env vars instead of the
UI form.
---
## 2. opencode
opencode supports arbitrary OpenAI-compatible providers via the
`@ai-sdk/openai-compatible` driver.
### 2.1 `opencode.json` in your project
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"qwenlocal": {
"npm": "@ai-sdk/openai-compatible",
"name": "Qwen3.8-27B (local)",
"options": {
"baseURL": "http://<HOST>:8080/v1",
"apiKey": "local"
},
"models": {
"Qwen3.8-27B-Uncensored": {
"name": "Qwen3.8-27B Uncensored"
}
}
}
}
}
```
Then start with:
```bash
opencode
# model picker -> qwenlocal/Qwen3.8-27B-Uncensored
```
or force it per run:
```bash
opencode --model qwenlocal/Qwen3.8-27B-Uncensored
```
### 2.2 Global config (optional)
Put the same `provider` block in `~/.config/opencode/opencode.json` to make it
available in every project.
Note: the model is a **reasoning** model — opencode will show the
`reasoning_content` stream as thinking output.
---
## 3. Claude Code
Claude Code speaks the Anthropic Messages protocol, so it needs a small
translation layer to talk to llama.cpp's OpenAI API.
### 3.1 Use claude-code-router (CCR)
```bash
npm install -g @musistudio/claude-code-router
ccr --set-base-url http://<HOST>:8080/v1
ccr --set-provider openai
ccr
```
Point `ANTHROPIC_BASE_URL` at the router and run Claude Code as usual.
### 3.2 Alternative: a generic OpenAI→Anthropic proxy
Any tool that translates `/v1/chat/completions` (OpenAI) to the Anthropic
Messages shape, e.g. LiteLLM, works:
```bash
pip install litellm[proxy]
litellm --model openai/qwen3.8-27b --api_base http://<HOST>:8080/v1 --port 4000
# then: export ANTHROPIC_BASE_URL=http://localhost:4000
```
Caveat: the uncensored Q4_K_P model has no tools/functions baked in beyond
basic chat — agentic tool-calling may be unreliable.
---
## 4. Other OpenAI-compatible clients
### 4.1 curl
```bash
curl http://<HOST>:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen",
"messages": [{"role":"user","content":"Hello"}],
"max_tokens": 200
}'
```
### 4.2 Python (openai SDK)
```bash
pip install openai
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://<HOST>:8080/v1", api_key="local")
resp = client.chat.completions.create(
model="qwen",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=200,
)
print(resp.choices[0].message.content)
# reasoning available as: resp.choices[0].message.reasoning_content
```
### 4.3 Node.js
```bash
npm i openai
```
```js
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://<HOST>:8080/v1", apiKey: "local" });
const r = await client.chat.completions.create({
model: "qwen",
messages: [{ role: "user", content: "Hello" }],
});
console.log(r.choices[0].message.content);
```
### 4.4 Local web UIs that accept an OpenAI endpoint
- **AnythingLLM** — Settings → LLM → "OpenAI" → custom base URL.
- **LM Studio / Jan** — treat the llama.cpp server as a remote OpenAI endpoint.
- **SillyTavern** — Chat Completion → Custom OpenAI → set base URL.
- **Continue.dev (VS Code)** — `models.yaml` with an openai provider + base URL.
---
## 5. Using the reasoning model properly
Qwen3.8-27B emits `thinking` internally before its `response`.
- OpenAI-compatible clients that surface `reasoning_content` (Open WebUI,
opencode) will show it automatically.
- To disable/trim reasoning (lower latency, shorter answers), the chat template
supports `reasoning_effort`: `"low"` | `"medium"` | `"xhigh"` (default).
Most clients pass extra `chat_template_kwargs`; alternatively send it via the
`chat_template_kwargs` field or set `enable_thinking: false`:
```json
"chat_template_kwargs": {"reasoning_effort": "low"}
```
- If a client shows raw `<thinking>` / `<response>` tags, strip them, e.g. in
Python: `re.sub(r"</?thinking>", "", text)`.
---
## 6. "I want an Ollama clone API"
**Done — a built-in proxy provides one.** `/opt/llama-server/ollama-proxy.py`
fakes both the **Ollama API** (`/api/chat`, `/api/generate`, `/api/tags`,
`/api/ps`, `/api/show`, `/api/embed`, …) **and the Anthropic Messages API**
(`/v1/messages`, `/v1/messages/count_tokens`) on port **11434**, translating
every request to the llama.cpp OpenAI backend on `:8080`. It runs as a systemd
service (`ollama-proxy.service`), pure Python stdlib, no third-party deps.
### 6.1 What the proxy exposes
| Endpoint | Purpose | Status |
|---|---|---|
| `GET /api/version` | version string | ✅ |
| `GET /api/tags` | list models (`Qwen3.8-Uncensored`) | ✅ |
| `GET /api/ps` | running models | ✅ |
| `GET /api/status` | cloud status (launcher) | ✅ |
| `GET /api/experimental/model-recommendations` | launcher hint | ✅ |
| `POST /api/show` | model details | ✅ |
| `POST /api/chat` | chat (stream + non-stream) | ✅ |
| `POST /api/generate` | single-prompt completion | ✅ |
| `POST /api/embed` / `/api/embeddings` | embeddings | ✅ |
| `POST /v1/messages` | **Anthropic Messages API** (Claude Code) | ✅ |
| `POST /v1/messages/count_tokens` | rough token estimate | ✅ |
| `GET /v1/models` | OpenAI-style model list | ✅ |
| `POST /v1/chat/completions` | OpenAI passthrough | ✅ |
Reasoning output from Qwen is exposed as `reasoning_content` on the Ollama
shape and `thinking`/`text` blocks on the Anthropic shape.
**Context length:** all four discovery endpoints (`/api/tags`, `/api/ps`,
`/api/show`, `/v1/models`) report `context_length: 262144` so tools don't
down-scale to a default (e.g. 128k).
### 6.2 Usage
```bash
# any Ollama-native client, point it at this box
OLLAMA_HOST=http://192.168.1.198:11434 ollama run Qwen3.8-Uncensored
curl http://192.168.1.198:11434/api/chat -d '{
"model": "Qwen3.8-Uncensored",
"messages": [{"role": "user", "content": "hi"}]
}'
```
### 6.3 Claude Code via `ollama launch claude`
`ollama launch claude` makes Claude Code talk to the **Anthropic `/v1/messages`**
endpoint at `OLLAMA_HOST`. The proxy implements it, so:
```bash
OLLAMA_HOST="http://192.168.1.198:11434" \
CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536 \
ollama launch claude --model Qwen3.8-Uncensored
```
Important:
- **Use `192.168.1.198`**, this box's LAN IP — not a different address. Earlier
guidance referenced `.165`, which is not this host.
- The proxy presents `ANTHROPIC_BASE_URL` = `OLLAMA_HOST`, so Claude Code talks
directly to `:11434/v1/messages`.
- If you don't use `ollama launch`, the equivalent manual setup is:
```bash
export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=
export ANTHROPIC_BASE_URL=http://192.168.1.198:11434
claude --model Qwen3.8-Uncensored
```
- **Tool calling: supported.** Anthropic `tools`, `tool_use`, and `tool_result`
blocks translate to/from llama.cpp OpenAI function calls (verified: the model
returns proper `tool_use` blocks, and multi-turn tool results are answered
correctly). Streaming emits `input_json_delta` events for tool_use blocks.
### 6.4 Service management
```bash
systemctl status llama-server # backend (llama.cpp, :8080)
systemctl status ollama-proxy # API faker (:11434)
journalctl -u ollama-proxy -f # proxy logs
```
Both are enabled at boot. The proxy needs the backend up; the unit has
`After=llama-server.service`.
---
## 7. Troubleshooting
| Symptom | Fix |
|---|---|
| Connection refused | Server stopped (`systemctl start llama-server`); wrong host/port |
| `401` | A `--api-key` was set; add `Authorization: Bearer <key>` |
| Model not in client list | Client cached models; hit refresh, or `curl /v1/models` to confirm |
| Slow first token | Reasoning model thinking; set `reasoning_effort: "low"` |
| OOM / VRAM errors | Reduce `-c`, or drop `--cache-type-k/v q4_0` trade-offs (see README §11) |
| Client needs Ollama API | Use the built-in proxy on `:11434` (§6) |
| `ollama launch <tool>` fails "something went wrong" | Check `journalctl -u ollama-proxy -f`; the launcher does `HEAD /` (heartbeat) + `GET /api/tags` first — both must return 200 |
| Tool calls return empty / no tool_use | Confirm the model supports tools via `POST /api/chat` with a `tools` array; the Q4_K_P GGUF does |
scripts/ollama-proxy.py
+15 -13
View File
@@ -92,39 +92,39 @@ def extract_session_id(headers, payload=None):
return "default"
import traceback
def ensure_session(session_id):
"""Ensure slot 0 contains the KV cache for session_id, saving/restoring as needed."""
"""Ensure slot 0 contains the KV cache for explicit named sessions, safely and non-blockingly."""
try:
global CURRENT_SESSION
if not session_id:
session_id = "default"
return
clean_id = re.sub(r'[^a-zA-Z0-9_\-\.]', '_', str(session_id))[:64]
# Only manage disk snapshots for explicit user session IDs (ignore transient sys- hashes)
if clean_id == "default" or clean_id.startswith("sys-"):
return
with SESSION_LOCK:
if clean_id == CURRENT_SESSION:
return
# 1. Save old session if not default
if CURRENT_SESSION and CURRENT_SESSION != "default":
if CURRENT_SESSION and CURRENT_SESSION != "default" and not CURRENT_SESSION.startswith("sys-"):
save_file = f"{CURRENT_SESSION}.bin"
print(f"[ollama-proxy] Saving slot 0 for session '{CURRENT_SESSION}' -> {save_file}", flush=True)
_slot_action("save", save_file)
# 2. Restore new session or erase
target_file = f"{clean_id}.bin"
target_path = os.path.join(SLOT_SAVE_PATH, target_file)
if os.path.exists(target_path):
print(f"[ollama-proxy] Restoring slot 0 for session '{clean_id}' <- {target_file}", flush=True)
res = _slot_action("restore", target_file)
if not res:
print(f"[ollama-proxy] Restore failed for '{clean_id}', falling back to erase", flush=True)
_slot_action("erase")
else:
print(f"[ollama-proxy] Starting fresh slot for session '{clean_id}'", flush=True)
_slot_action("erase")
_slot_action("restore", target_file)
CURRENT_SESSION = clean_id
except Exception as e:
print(f"[ollama-proxy] Warning: ensure_session({session_id}) error (non-fatal): {e}", flush=True)
MODEL_TAGS = [
"Qwen3.8-Uncensored:latest",
@@ -1444,6 +1444,8 @@ class Handler(http.server.BaseHTTPRequestHandler):
detail = ""
self._send(e.code, {"error": detail or str(e)})
except Exception as e:
print(f"[ollama-proxy] Exception in POST {path}: {e}", flush=True)
traceback.print_exc()
self._send(500, {"error": f"{type(e).__name__}: {e}"})
def do_DELETE(self):
scripts/start-server.sh
+13 -9
View File
@@ -1,5 +1,5 @@
#!/bin/bash
# llama-server launcher for Qwen3.8-27B on 3x Turing GPUs (RTX 2060 12GB + 2x CMP 50HX 10GB)
# llama-server launcher for Qwen3.8-27B on Turing GPUs (RTX 2060 12GB + CMP 50HX 10GB)
# Built with NVIDIA NCCL for hardware-accelerated multi-GPU tensor parallelism.
# Accelerated with HauhauCS FastMTP 32K draft sidecar and official thinking parameters.
export LD_LIBRARY_PATH=/opt/llama.cpp-xrip/build-nccl/bin:/opt/minicpm-venv/lib/python3.13/site-packages/nvidia/nccl/lib:/lib/x86_64-linux-gnu
@@ -12,16 +12,20 @@ export NCCL_P2P_DISABLE=0
export NCCL_ALGO=RING
export NCCL_PROTO=SIMPLE
# Dynamically target the 12GB RTX 2060 for mmproj and FastMTP draft model
RTX_DEV=$(/opt/llama.cpp-xrip/build-nccl/bin/llama-cli --list-devices | grep -i "RTX 2060" | awk '{print $1}' | tr -d ':')
RTX_DEV=${RTX_DEV:-CUDA0}
exec /opt/llama.cpp-xrip/build-nccl/bin/llama-server \
-m /opt/models/gguf/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf \
-m /opt/models/gguf/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q5_K_P.gguf \
--spec-draft-model /opt/models/gguf/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-FastMTP-32K.gguf \
--spec-draft-device CUDA2 \
--spec-draft-device "$RTX_DEV" \
--spec-draft-ngl all \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--spec-draft-p-min 0 \
--mmproj /opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf \
--mmproj-device CUDA2 \
--mmproj-device "$RTX_DEV" \
--image-min-tokens 1024 \
--temp 1.0 \
--top-k 20 \
@@ -35,17 +39,17 @@ exec /opt/llama.cpp-xrip/build-nccl/bin/llama-server \
--reasoning-format deepseek \
--numa split \
-ngl 99 \
-c 204800 \
-c 131072 \
--parallel 1 \
--slot-save-path /var/cache/llama-slots \
--cache-ram 16384 \
--split-mode tensor \
--flash-attn on \
--batch-size 1024 \
--ubatch-size 512 \
--batch-size 2048 \
--ubatch-size 1024 \
--jinja \
--threads 12 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--cache-type-k q5_0 \
--cache-type-v q5_0 \
--host 0.0.0.0 \
--port 8080