feat: implement session-managed persistent KV cache architecture with slot persistence and management API

2026-09-04 02:53:54 +00:00
parent 5daafe24e8
commit 9ab0583cab
4 changed files with 244 additions and 3 deletions
@@ -0,0 +1,88 @@
+# Session-Managed KV Cache for llama-server + ollama-proxy
+
+## Goal
+
+Make the multi-GPU llama.cpp backend (exposed via `ollama-proxy.py`) correctly manage KV cache **per session**, so that:
+
+1. Different users / agents / OpenWebUI chats do not pollute each other’s context.
+2. Large shared system prompts (2–30k tokens) used by Maki, OpenWebUI, and other tools are reused instead of being re-prefilled every time.
+3. Session state **survives reboots** (disk-backed).
+
+Frontends:
+- OpenWebUI
+- Maki (https://github.com/wmantly/maki)
+
+Backend stack:
+- `llama-server` (port 8080, `--parallel 1`, `--slot-save-path /var/cache/llama-slots`, `--cache-ram 16384`)
+- `ollama-proxy.py` (port 11434) – Ollama / OpenAI / Anthropic compatible surface with automatic session affinity
+
+---
+
+## Architecture
+
+```text
+OpenWebUI  ─┐
+            ├──► ollama-proxy.py (port 11434) ──► llama-server (port 8080)
+Maki       ─┘         │
+                      └── session_id → /var/cache/llama-slots/<id>.bin
+```
+
+- Single slot (`--parallel 1`) for maximum context and dedicated tensor parallelism across all 3 GPUs.
+- Proxy owns session affinity and decides when to save / restore / erase the slot.
+- Disk persistence via `--slot-save-path /var/cache/llama-slots`.
+- Hot prefix reuse via `--cache-ram 16384` + `cache_prompt: true`.
+
+---
+
+## 1. Session Identity Extraction
+
+The proxy extracts a stable `session_id` on every request according to this priority:
+
+1. **HTTP Headers**:
+   - `X-Session-Id`
+   - `X-Conversation-Id`
+   - `Session-Id`
+   - `Conversation-Id`
+2. **JSON Body Fields**:
+   - `session_id`
+   - `conversation_id`
+   - `chat_id`
+   - `id` (when structured as a chat identifier)
+3. **Fallback**:
+   - `sys-<sha256[:16]>` (hash of the system prompt so identical system prompts share a base KV slot)
+   - `"default"`
+
+---
+
+## 2. Proxy Session Lifecycle
+
+When a request arrives at `ollama-proxy.py`:
+
+```text
+with session_lock:
+    if session_id == current_session:
+        proceed
+
+    # 1. Persist previous session slot
+    if current_session and current_session != "default":
+        POST /slots/0?action=save {"filename": f"{current_session}.bin"}
+
+    # 2. Restore new session or start fresh
+    if os.path.exists(f"/var/cache/llama-slots/{session_id}.bin"):
+        POST /slots/0?action=restore {"filename": f"{session_id}.bin"}
+    else:
+        POST /slots/0?action=erase
+
+    current_session = session_id
+```
+
+---
+
+## 3. Session Management Endpoints
+
+Exposed on `ollama-proxy` (`port 11434`):
+
+* `GET /api/sessions/current`: Returns active `session_id` and slot file status.
+* `GET /api/sessions`: Lists all saved session `.bin` slot files with file sizes and timestamps.
+* `POST /api/sessions/clear`: Forces erase on slot 0 and resets active session to `"default"`.
+* `DELETE /api/sessions/<id>`: Deletes the persisted `.bin` cache file for a specific session.