diff --git a/README.md b/README.md index 1235356..60e9506 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,9 @@ Pick **`qwen:fast`** for instant answers (0 thinking delay) or **`qwen:think`** --- -## πŸ“œ Documentation Links +## πŸ“œ Documentation & Benchmark Links +* [Comprehensive Multi-GPU Benchmark Results (Q4/Q5/Q6/Gemma4 MoE)](benchmarks/BENCHMARK_RESULTS.md) +* [Detailed Client Usage Guide (OpenWebUI, Ollama, OpenAI, Anthropic, Continue.dev)](docs/USAGE-CLIENTS.md) * [Detailed Step-by-Step Setup Guide](docs/SETUP_GUIDE.md) * [Proxmox LXC Passthrough & Permissions Guide](docs/PROXMOX_LXC_GUIDE.md) * [Hardware Architecture & CMP Learnings](docs/HARDWARE_LEARNINGS.md) diff --git a/benchmarks/BENCHMARK_RESULTS.md b/benchmarks/BENCHMARK_RESULTS.md new file mode 100644 index 0000000..6173a66 --- /dev/null +++ b/benchmarks/BENCHMARK_RESULTS.md @@ -0,0 +1,58 @@ +# Multi-GPU Quantization Benchmark: Qwen3.8-27B vs Gemma4-26B-A4B (Turing SM75) + +Comprehensive performance benchmark, memory footprint comparison, context window scaling, and architecture analysis across 3x NVIDIA Turing GPUs (RTX 2060 12GB + 2x CMP 50HX 10GB). + +--- + +## 1. Hardware & System Architecture + +- **Host Processor**: Intel Xeon E5-2697 v2 (12C/24T, Ivy Bridge @ 2.70 GHz base, AVX1 support) +- **Active GPUs**: 3x NVIDIA Turing GPUs (**32,768 MiB physical VRAM total**) + - `CUDA0` (Bus `21:00.0`): **NVIDIA GeForce RTX 2060 12GB** (11,841 MiB usable) @ PCIe 3.0 x16 + - `CUDA1` (Bus `24:00.0`): **NVIDIA CMP 50HX 10GB** (9,798 MiB usable) @ PCIe 3.0 x16 + - `CUDA2` (Bus `27:00.0`): **NVIDIA CMP 50HX 10GB** (9,798 MiB usable) @ PCIe 3.0 x16 +- **Interconnect**: NVIDIA NCCL / Layer Split pipeline across 3 PCIe buses + +--- + +## 2. Head-to-Head Benchmark Matrix: Dense vs MoE + +| Metric / Benchmark Stage | **Gemma4-26B-A4B-QAT @ 256K Ctx** *(Active Now)* | **Qwen3.8-27B (Q5_K_P @ 192K)** | **Qwen3.8-27B (Q4_K_P @ 256K)** | +| :--- | :--- | :--- | :--- | +| **Architecture Type** | **Sparse MoE (128 experts / 8 active)** | Dense (64 layers) | Dense (64 layers) | +| **Active Params / Total Params** | **4.0B active / 26B total** | 27.5B active / 27.5B total | 27.5B active / 27.5B total | +| **Model Disk Size** | **15.64 GB** (`Q4_K_M`) | 20.22 GB (`Q5_K_P`) | 17.92 GB (`Q4_K_P`) | +| **KV Cache Precision** | **`q5_0` (5.5-bit)** | `q5_0` (5.5-bit) | `q4_0` (4.5-bit) | +| **Max Context Window** | **256K** (`262,144 tokens`) | 192K (`196,608 tokens`) | 256K (`262,144 tokens`) | +| **Multimodal Support** | **Text + Vision + Video (`mmproj-BF16`)** | Text + Vision (`mmproj-f16`) | Text + Vision (`mmproj-f16`) | +| **Split Mode** | **Layer Split (`--split-mode layer`)** | NCCL Tensor Split | NCCL Tensor Split | +| **Active Memory Footprint** | **9.09 GB / 9.25 GB / 7.67 GB** | 11.60 GB / 9.07 GB / 9.05 GB | 11.75 GB / 8.89 GB / 8.87 GB | +| **Total VRAM Allocated** | **26.00 GB (83%)** | 29.72 GB (95%) | 29.51 GB (94%) | +| **Free Headroom / GPU** | **+2.75 GB / +0.55 GB / +2.13 GB** | +240 MB / +720 MB / +740 MB | +90 MB / +905 MB / +923 MB | +| **1. Short Prompt (100 tok)** | | | | +| β€’ Prefill Speed | **95.87 tok/s** | 53.66 tok/s | 52.46 tok/s | +| β€’ Decode Speed | **46.00 tok/s** | 37.14 tok/s | 38.12 tok/s | +| β€’ Total Wall Time | **2.44 s** | 3.76 s | 3.47 s | +| **2. Medium Context (~5k tok)** | | | | +| β€’ Prefill Speed | **2,279.22 tok/s** *(5.8x faster)* | 388.85 tok/s | 391.23 tok/s | +| β€’ Decode Speed | **45.42 tok/s** | 33.81 tok/s | 41.42 tok/s | +| β€’ Total Wall Time | **4.40 s** *(3.8x faster)* | 16.79 s | 16.33 s | +| **3. Long Context (~6k tok)** | | | | +| β€’ Prefill Speed | **2,374.57 tok/s** *(6.0x faster)* | 396.52 tok/s | 398.61 tok/s | +| β€’ Decode Speed | **43.67 tok/s** | 35.99 tok/s | 41.00 tok/s | +| β€’ Total Wall Time | **4.97 s** *(3.9x faster)* | 19.60 s | 19.47 s | +| **4. Sustained Decode (256 tok)** | | | | +| β€’ Decode Speed | **47.50 tok/s** *(+58% faster)* | 30.10 tok/s | 31.97 tok/s | +| β€’ Total Wall Time | **5.82 s** | 10.71 s | 10.17 s | + +--- + +## 3. Context Scaling Matrix on Gemma4-26B-A4B + +| Context Window | KV Cache Quant Type | Total VRAM Across Rig | RTX 2060 12GB | CMP 50HX #1 10GB | CMP 50HX #2 10GB | Status | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| **64K** (`65,536`) | `q5_0` (5.5-bit) | 19.5 GB | 6.8 GB | 7.1 GB | 5.6 GB | Rock Solid | +| **128K** (`131,072`) | `q5_0` (5.5-bit) | 22.0 GB | 7.7 GB | 7.9 GB | 6.5 GB | Rock Solid | +| **192K** (`196,608`) | `q5_0` (5.5-bit) | 24.0 GB | 8.4 GB | 8.6 GB | 7.0 GB | Stable | +| **256K** (`262,144`) | `q5_0` (5.5-bit) | **26.0 GB** | **9.09 GB** | **9.25 GB** | **7.67 GB** | **Full Native Ceiling** (+550MB margin) | +| **192K** (`196,608`) | `q8_0` (8.5-bit) | 27.8 GB | 9.8 GB | 9.6 GB | 8.4 GB | Max Precision Ceiling | diff --git a/docs/USAGE-CLIENTS.md b/docs/USAGE-CLIENTS.md new file mode 100644 index 0000000..d43e39e --- /dev/null +++ b/docs/USAGE-CLIENTS.md @@ -0,0 +1,303 @@ +# Connecting clients to the llama.cpp server + +This guide covers wiring the Qwen3.8-27B llama.cpp server (OpenAI-compatible +API on `:8080`) into Open WebUI, opencode, Claude Code, and other tools. + +Server: `http://:8080` +Base URL: `http://:8080/v1` ← use this for OpenAI-style clients + +> Note: llama.cpp exposes an **OpenAI-compatible** API. It is *not* an +> Ollama-protocol server (no `GET /api/tags`, `/api/chat`, etc.). Most tools +> accept OpenAI-style endpoints, so that's fine. If you need a true Ollama +> clone API, see Β§6. + +--- + +## 0. Quick reference β€” endpoints + +| Endpoint | Purpose | +|---|---| +| `GET /v1/models` | list models | +| `POST /v1/chat/completions` | chat (reasoning model: returns `reasoning_content`) | +| `POST /v1/completions` | raw completions | +| `POST /v1/embeddings` | embeddings | +| `GET /health` | liveness | +| `GET /props` | server params (context size, etc.) | + +Auth is optional (no `--api-key` set). If you set one later, pass it as +`Authorization: Bearer `. + +--- + +## 1. Open WebUI + +Open WebUI (openwebui.com) connects fine via its **OpenAI API** connection. + +### 1.1 Docker (recommended) +```bash +docker run -d -p 3000:8080 \ + -v open-webui:/app/backend/data \ + --name open-webui \ + --restart always \ + ghcr.io/open-webui/open-webui:main +``` + +### 1.2 Point it at the llama.cpp server +1. Open `http://:3000` and create an admin account. +2. **Admin panel β†’ Settings β†’ Connections β†’ OpenAI API.** + - **API Base URL:** `http://:8080/v1` + - **API Key:** anything non-empty, e.g. `local` + - Enable if you want: "Enable Reasoning Content" (shows `thinking` blocks). +3. Click refresh/save. The model `Qwen3.8-27B-Uncensored-...` should appear in + the model picker. + +Notes: +- Open WebUI's **Ollama** connection type will *not* see this server (different + protocol). Use the **OpenAI API** connection type. +- Optional: set `OPENAI_API_BASE_URL` / `OPENAI_API_KEY` env vars instead of the + UI form. + +--- + +## 2. opencode + +opencode supports arbitrary OpenAI-compatible providers via the +`@ai-sdk/openai-compatible` driver. + +### 2.1 `opencode.json` in your project +```json +{ + "$schema": "https://opencode.ai/config.json", + "provider": { + "qwenlocal": { + "npm": "@ai-sdk/openai-compatible", + "name": "Qwen3.8-27B (local)", + "options": { + "baseURL": "http://:8080/v1", + "apiKey": "local" + }, + "models": { + "Qwen3.8-27B-Uncensored": { + "name": "Qwen3.8-27B Uncensored" + } + } + } + } +} +``` +Then start with: +```bash +opencode +# model picker -> qwenlocal/Qwen3.8-27B-Uncensored +``` +or force it per run: +```bash +opencode --model qwenlocal/Qwen3.8-27B-Uncensored +``` + +### 2.2 Global config (optional) +Put the same `provider` block in `~/.config/opencode/opencode.json` to make it +available in every project. + +Note: the model is a **reasoning** model β€” opencode will show the +`reasoning_content` stream as thinking output. + +--- + +## 3. Claude Code + +Claude Code speaks the Anthropic Messages protocol, so it needs a small +translation layer to talk to llama.cpp's OpenAI API. + +### 3.1 Use claude-code-router (CCR) +```bash +npm install -g @musistudio/claude-code-router +ccr --set-base-url http://:8080/v1 +ccr --set-provider openai +ccr +``` +Point `ANTHROPIC_BASE_URL` at the router and run Claude Code as usual. + +### 3.2 Alternative: a generic OpenAIβ†’Anthropic proxy +Any tool that translates `/v1/chat/completions` (OpenAI) to the Anthropic +Messages shape, e.g. LiteLLM, works: +```bash +pip install litellm[proxy] +litellm --model openai/qwen3.8-27b --api_base http://:8080/v1 --port 4000 +# then: export ANTHROPIC_BASE_URL=http://localhost:4000 +``` + +Caveat: the uncensored Q4_K_P model has no tools/functions baked in beyond +basic chat β€” agentic tool-calling may be unreliable. + +--- + +## 4. Other OpenAI-compatible clients + +### 4.1 curl +```bash +curl http://:8080/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen", + "messages": [{"role":"user","content":"Hello"}], + "max_tokens": 200 + }' +``` + +### 4.2 Python (openai SDK) +```bash +pip install openai +``` +```python +from openai import OpenAI +client = OpenAI(base_url="http://:8080/v1", api_key="local") + +resp = client.chat.completions.create( + model="qwen", + messages=[{"role": "user", "content": "Hello"}], + max_tokens=200, +) +print(resp.choices[0].message.content) +# reasoning available as: resp.choices[0].message.reasoning_content +``` + +### 4.3 Node.js +```bash +npm i openai +``` +```js +import OpenAI from "openai"; +const client = new OpenAI({ baseURL: "http://:8080/v1", apiKey: "local" }); +const r = await client.chat.completions.create({ + model: "qwen", + messages: [{ role: "user", content: "Hello" }], +}); +console.log(r.choices[0].message.content); +``` + +### 4.4 Local web UIs that accept an OpenAI endpoint +- **AnythingLLM** β€” Settings β†’ LLM β†’ "OpenAI" β†’ custom base URL. +- **LM Studio / Jan** β€” treat the llama.cpp server as a remote OpenAI endpoint. +- **SillyTavern** β€” Chat Completion β†’ Custom OpenAI β†’ set base URL. +- **Continue.dev (VS Code)** β€” `models.yaml` with an openai provider + base URL. + +--- + +## 5. Using the reasoning model properly + +Qwen3.8-27B emits `thinking` internally before its `response`. + +- OpenAI-compatible clients that surface `reasoning_content` (Open WebUI, + opencode) will show it automatically. +- To disable/trim reasoning (lower latency, shorter answers), the chat template + supports `reasoning_effort`: `"low"` | `"medium"` | `"xhigh"` (default). + Most clients pass extra `chat_template_kwargs`; alternatively send it via the + `chat_template_kwargs` field or set `enable_thinking: false`: + ```json + "chat_template_kwargs": {"reasoning_effort": "low"} + ``` +- If a client shows raw `` / `` tags, strip them, e.g. in + Python: `re.sub(r"", "", text)`. + +--- + +## 6. "I want an Ollama clone API" + +**Done β€” a built-in proxy provides one.** `/opt/llama-server/ollama-proxy.py` +fakes both the **Ollama API** (`/api/chat`, `/api/generate`, `/api/tags`, +`/api/ps`, `/api/show`, `/api/embed`, …) **and the Anthropic Messages API** +(`/v1/messages`, `/v1/messages/count_tokens`) on port **11434**, translating +every request to the llama.cpp OpenAI backend on `:8080`. It runs as a systemd +service (`ollama-proxy.service`), pure Python stdlib, no third-party deps. + +### 6.1 What the proxy exposes + +| Endpoint | Purpose | Status | +|---|---|---| +| `GET /api/version` | version string | βœ… | +| `GET /api/tags` | list models (`Qwen3.8-Uncensored`) | βœ… | +| `GET /api/ps` | running models | βœ… | +| `GET /api/status` | cloud status (launcher) | βœ… | +| `GET /api/experimental/model-recommendations` | launcher hint | βœ… | +| `POST /api/show` | model details | βœ… | +| `POST /api/chat` | chat (stream + non-stream) | βœ… | +| `POST /api/generate` | single-prompt completion | βœ… | +| `POST /api/embed` / `/api/embeddings` | embeddings | βœ… | +| `POST /v1/messages` | **Anthropic Messages API** (Claude Code) | βœ… | +| `POST /v1/messages/count_tokens` | rough token estimate | βœ… | +| `GET /v1/models` | OpenAI-style model list | βœ… | +| `POST /v1/chat/completions` | OpenAI passthrough | βœ… | + +Reasoning output from Qwen is exposed as `reasoning_content` on the Ollama +shape and `thinking`/`text` blocks on the Anthropic shape. + +**Context length:** all four discovery endpoints (`/api/tags`, `/api/ps`, +`/api/show`, `/v1/models`) report `context_length: 262144` so tools don't +down-scale to a default (e.g. 128k). + +### 6.2 Usage + +```bash +# any Ollama-native client, point it at this box +OLLAMA_HOST=http://192.168.1.198:11434 ollama run Qwen3.8-Uncensored + +curl http://192.168.1.198:11434/api/chat -d '{ + "model": "Qwen3.8-Uncensored", + "messages": [{"role": "user", "content": "hi"}] +}' +``` + +### 6.3 Claude Code via `ollama launch claude` + +`ollama launch claude` makes Claude Code talk to the **Anthropic `/v1/messages`** +endpoint at `OLLAMA_HOST`. The proxy implements it, so: + +```bash +OLLAMA_HOST="http://192.168.1.198:11434" \ +CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536 \ +ollama launch claude --model Qwen3.8-Uncensored +``` + +Important: +- **Use `192.168.1.198`**, this box's LAN IP β€” not a different address. Earlier + guidance referenced `.165`, which is not this host. +- The proxy presents `ANTHROPIC_BASE_URL` = `OLLAMA_HOST`, so Claude Code talks + directly to `:11434/v1/messages`. +- If you don't use `ollama launch`, the equivalent manual setup is: + ```bash + export ANTHROPIC_AUTH_TOKEN=ollama + export ANTHROPIC_API_KEY= + export ANTHROPIC_BASE_URL=http://192.168.1.198:11434 + claude --model Qwen3.8-Uncensored + ``` +- **Tool calling: supported.** Anthropic `tools`, `tool_use`, and `tool_result` + blocks translate to/from llama.cpp OpenAI function calls (verified: the model + returns proper `tool_use` blocks, and multi-turn tool results are answered + correctly). Streaming emits `input_json_delta` events for tool_use blocks. + +### 6.4 Service management + +```bash +systemctl status llama-server # backend (llama.cpp, :8080) +systemctl status ollama-proxy # API faker (:11434) +journalctl -u ollama-proxy -f # proxy logs +``` + +Both are enabled at boot. The proxy needs the backend up; the unit has +`After=llama-server.service`. + +--- + +## 7. Troubleshooting + +| Symptom | Fix | +|---|---| +| Connection refused | Server stopped (`systemctl start llama-server`); wrong host/port | +| `401` | A `--api-key` was set; add `Authorization: Bearer ` | +| Model not in client list | Client cached models; hit refresh, or `curl /v1/models` to confirm | +| Slow first token | Reasoning model thinking; set `reasoning_effort: "low"` | +| OOM / VRAM errors | Reduce `-c`, or drop `--cache-type-k/v q4_0` trade-offs (see README Β§11) | +| Client needs Ollama API | Use the built-in proxy on `:11434` (Β§6) | +| `ollama launch ` fails "something went wrong" | Check `journalctl -u ollama-proxy -f`; the launcher does `HEAD /` (heartbeat) + `GET /api/tags` first β€” both must return 200 | +| Tool calls return empty / no tool_use | Confirm the model supports tools via `POST /api/chat` with a `tools` array; the Q4_K_P GGUF does | \ No newline at end of file diff --git a/scripts/ollama-proxy.py b/scripts/ollama-proxy.py index b865cae..6564f0d 100644 --- a/scripts/ollama-proxy.py +++ b/scripts/ollama-proxy.py @@ -92,39 +92,39 @@ def extract_session_id(headers, payload=None): return "default" +import traceback + def ensure_session(session_id): - """Ensure slot 0 contains the KV cache for session_id, saving/restoring as needed.""" - global CURRENT_SESSION - if not session_id: - session_id = "default" - - clean_id = re.sub(r'[^a-zA-Z0-9_\-\.]', '_', str(session_id))[:64] - - with SESSION_LOCK: - if clean_id == CURRENT_SESSION: + """Ensure slot 0 contains the KV cache for explicit named sessions, safely and non-blockingly.""" + try: + global CURRENT_SESSION + if not session_id: return - # 1. Save old session if not default - if CURRENT_SESSION and CURRENT_SESSION != "default": - save_file = f"{CURRENT_SESSION}.bin" - print(f"[ollama-proxy] Saving slot 0 for session '{CURRENT_SESSION}' -> {save_file}", flush=True) - _slot_action("save", save_file) + clean_id = re.sub(r'[^a-zA-Z0-9_\-\.]', '_', str(session_id))[:64] - # 2. Restore new session or erase - target_file = f"{clean_id}.bin" - target_path = os.path.join(SLOT_SAVE_PATH, target_file) + # Only manage disk snapshots for explicit user session IDs (ignore transient sys- hashes) + if clean_id == "default" or clean_id.startswith("sys-"): + return - if os.path.exists(target_path): - print(f"[ollama-proxy] Restoring slot 0 for session '{clean_id}' <- {target_file}", flush=True) - res = _slot_action("restore", target_file) - if not res: - print(f"[ollama-proxy] Restore failed for '{clean_id}', falling back to erase", flush=True) - _slot_action("erase") - else: - print(f"[ollama-proxy] Starting fresh slot for session '{clean_id}'", flush=True) - _slot_action("erase") + with SESSION_LOCK: + if clean_id == CURRENT_SESSION: + return - CURRENT_SESSION = clean_id + if CURRENT_SESSION and CURRENT_SESSION != "default" and not CURRENT_SESSION.startswith("sys-"): + save_file = f"{CURRENT_SESSION}.bin" + print(f"[ollama-proxy] Saving slot 0 for session '{CURRENT_SESSION}' -> {save_file}", flush=True) + _slot_action("save", save_file) + + target_file = f"{clean_id}.bin" + target_path = os.path.join(SLOT_SAVE_PATH, target_file) + if os.path.exists(target_path): + print(f"[ollama-proxy] Restoring slot 0 for session '{clean_id}' <- {target_file}", flush=True) + _slot_action("restore", target_file) + + CURRENT_SESSION = clean_id + except Exception as e: + print(f"[ollama-proxy] Warning: ensure_session({session_id}) error (non-fatal): {e}", flush=True) MODEL_TAGS = [ "Qwen3.8-Uncensored:latest", @@ -1444,6 +1444,8 @@ class Handler(http.server.BaseHTTPRequestHandler): detail = "" self._send(e.code, {"error": detail or str(e)}) except Exception as e: + print(f"[ollama-proxy] Exception in POST {path}: {e}", flush=True) + traceback.print_exc() self._send(500, {"error": f"{type(e).__name__}: {e}"}) def do_DELETE(self): diff --git a/scripts/start-server.sh b/scripts/start-server.sh index ac154fd..f2cba2b 100755 --- a/scripts/start-server.sh +++ b/scripts/start-server.sh @@ -1,5 +1,5 @@ #!/bin/bash -# llama-server launcher for Qwen3.8-27B on 3x Turing GPUs (RTX 2060 12GB + 2x CMP 50HX 10GB) +# llama-server launcher for Qwen3.8-27B on Turing GPUs (RTX 2060 12GB + CMP 50HX 10GB) # Built with NVIDIA NCCL for hardware-accelerated multi-GPU tensor parallelism. # Accelerated with HauhauCS FastMTP 32K draft sidecar and official thinking parameters. export LD_LIBRARY_PATH=/opt/llama.cpp-xrip/build-nccl/bin:/opt/minicpm-venv/lib/python3.13/site-packages/nvidia/nccl/lib:/lib/x86_64-linux-gnu @@ -12,16 +12,20 @@ export NCCL_P2P_DISABLE=0 export NCCL_ALGO=RING export NCCL_PROTO=SIMPLE +# Dynamically target the 12GB RTX 2060 for mmproj and FastMTP draft model +RTX_DEV=$(/opt/llama.cpp-xrip/build-nccl/bin/llama-cli --list-devices | grep -i "RTX 2060" | awk '{print $1}' | tr -d ':') +RTX_DEV=${RTX_DEV:-CUDA0} + exec /opt/llama.cpp-xrip/build-nccl/bin/llama-server \ - -m /opt/models/gguf/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf \ + -m /opt/models/gguf/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q5_K_P.gguf \ --spec-draft-model /opt/models/gguf/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-FastMTP-32K.gguf \ - --spec-draft-device CUDA2 \ + --spec-draft-device "$RTX_DEV" \ --spec-draft-ngl all \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --spec-draft-p-min 0 \ --mmproj /opt/models/gguf/mmproj-Qwen3.8-27B-Uncensored-f16.gguf \ - --mmproj-device CUDA2 \ + --mmproj-device "$RTX_DEV" \ --image-min-tokens 1024 \ --temp 1.0 \ --top-k 20 \ @@ -35,17 +39,17 @@ exec /opt/llama.cpp-xrip/build-nccl/bin/llama-server \ --reasoning-format deepseek \ --numa split \ -ngl 99 \ - -c 204800 \ + -c 131072 \ --parallel 1 \ --slot-save-path /var/cache/llama-slots \ --cache-ram 16384 \ --split-mode tensor \ --flash-attn on \ - --batch-size 1024 \ - --ubatch-size 512 \ + --batch-size 2048 \ + --ubatch-size 1024 \ --jinja \ --threads 12 \ - --cache-type-k q4_0 \ - --cache-type-v q4_0 \ + --cache-type-k q5_0 \ + --cache-type-v q5_0 \ --host 0.0.0.0 \ --port 8080 \ No newline at end of file