89 lines
2.8 KiB
Markdown
89 lines
2.8 KiB
Markdown
# Session-Managed KV Cache for llama-server + ollama-proxy
|
||
|
||
## Goal
|
||
|
||
Make the multi-GPU llama.cpp backend (exposed via `ollama-proxy.py`) correctly manage KV cache **per session**, so that:
|
||
|
||
1. Different users / agents / OpenWebUI chats do not pollute each other’s context.
|
||
2. Large shared system prompts (2–30k tokens) used by Maki, OpenWebUI, and other tools are reused instead of being re-prefilled every time.
|
||
3. Session state **survives reboots** (disk-backed).
|
||
|
||
Frontends:
|
||
- OpenWebUI
|
||
- Maki (https://github.com/wmantly/maki)
|
||
|
||
Backend stack:
|
||
- `llama-server` (port 8080, `--parallel 1`, `--slot-save-path /var/cache/llama-slots`, `--cache-ram 16384`)
|
||
- `ollama-proxy.py` (port 11434) – Ollama / OpenAI / Anthropic compatible surface with automatic session affinity
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```text
|
||
OpenWebUI ─┐
|
||
├──► ollama-proxy.py (port 11434) ──► llama-server (port 8080)
|
||
Maki ─┘ │
|
||
└── session_id → /var/cache/llama-slots/<id>.bin
|
||
```
|
||
|
||
- Single slot (`--parallel 1`) for maximum context and dedicated tensor parallelism across all 3 GPUs.
|
||
- Proxy owns session affinity and decides when to save / restore / erase the slot.
|
||
- Disk persistence via `--slot-save-path /var/cache/llama-slots`.
|
||
- Hot prefix reuse via `--cache-ram 16384` + `cache_prompt: true`.
|
||
|
||
---
|
||
|
||
## 1. Session Identity Extraction
|
||
|
||
The proxy extracts a stable `session_id` on every request according to this priority:
|
||
|
||
1. **HTTP Headers**:
|
||
- `X-Session-Id`
|
||
- `X-Conversation-Id`
|
||
- `Session-Id`
|
||
- `Conversation-Id`
|
||
2. **JSON Body Fields**:
|
||
- `session_id`
|
||
- `conversation_id`
|
||
- `chat_id`
|
||
- `id` (when structured as a chat identifier)
|
||
3. **Fallback**:
|
||
- `sys-<sha256[:16]>` (hash of the system prompt so identical system prompts share a base KV slot)
|
||
- `"default"`
|
||
|
||
---
|
||
|
||
## 2. Proxy Session Lifecycle
|
||
|
||
When a request arrives at `ollama-proxy.py`:
|
||
|
||
```text
|
||
with session_lock:
|
||
if session_id == current_session:
|
||
proceed
|
||
|
||
# 1. Persist previous session slot
|
||
if current_session and current_session != "default":
|
||
POST /slots/0?action=save {"filename": f"{current_session}.bin"}
|
||
|
||
# 2. Restore new session or start fresh
|
||
if os.path.exists(f"/var/cache/llama-slots/{session_id}.bin"):
|
||
POST /slots/0?action=restore {"filename": f"{session_id}.bin"}
|
||
else:
|
||
POST /slots/0?action=erase
|
||
|
||
current_session = session_id
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Session Management Endpoints
|
||
|
||
Exposed on `ollama-proxy` (`port 11434`):
|
||
|
||
* `GET /api/sessions/current`: Returns active `session_id` and slot file status.
|
||
* `GET /api/sessions`: Lists all saved session `.bin` slot files with file sizes and timestamps.
|
||
* `POST /api/sessions/clear`: Forces erase on slot 0 and resets active session to `"default"`.
|
||
* `DELETE /api/sessions/<id>`: Deletes the persisted `.bin` cache file for a specific session.
|