turing-multi-gpu-llm-server/docs/SESSION_KV_CACHE.md

Session-Managed KV Cache for llama-server + ollama-proxy

Goal

Make the multi-GPU llama.cpp backend (exposed via ollama-proxy.py) correctly manage KV cache per session, so that:

1. Different users / agents / OpenWebUI chats do not pollute each other’s context.
2. Large shared system prompts (2–30k tokens) used by Maki, OpenWebUI, and other tools are reused instead of being re-prefilled every time.
3. Session state survives reboots (disk-backed).

Frontends:

• OpenWebUI
• Maki (https://github.com/wmantly/maki)

Backend stack:

• llama-server (port 8080, --parallel 1, --slot-save-path /var/cache/llama-slots, --cache-ram 16384)
• ollama-proxy.py (port 11434) – Ollama / OpenAI / Anthropic compatible surface with automatic session affinity

---

Architecture

OpenWebUI  ─┐
            ├──► ollama-proxy.py (port 11434) ──► llama-server (port 8080)
Maki       ─┘         │
                      └── session_id → /var/cache/llama-slots/<id>.bin

• Single slot (--parallel 1) for maximum context and dedicated tensor parallelism across all 3 GPUs.
• Proxy owns session affinity and decides when to save / restore / erase the slot.
• Disk persistence via --slot-save-path /var/cache/llama-slots.
• Hot prefix reuse via --cache-ram 16384 + cache_prompt: true.

---

1. Session Identity Extraction

The proxy extracts a stable session_id on every request according to this priority:

1. HTTP Headers:
   • X-Session-Id
   • X-Conversation-Id
   • Session-Id
   • Conversation-Id
2. JSON Body Fields:
   • session_id
   • conversation_id
   • chat_id
   • id (when structured as a chat identifier)
3. Fallback:
   • sys-<sha256[:16]> (hash of the system prompt so identical system prompts share a base KV slot)
   • "default"

---

2. Proxy Session Lifecycle

When a request arrives at ollama-proxy.py:

with session_lock:
    if session_id == current_session:
        proceed

    # 1. Persist previous session slot
    if current_session and current_session != "default":
        POST /slots/0?action=save {"filename": f"{current_session}.bin"}

    # 2. Restore new session or start fresh
    if os.path.exists(f"/var/cache/llama-slots/{session_id}.bin"):
        POST /slots/0?action=restore {"filename": f"{session_id}.bin"}
    else:
        POST /slots/0?action=erase

    current_session = session_id

---

3. Session Management Endpoints

Exposed on ollama-proxy (port 11434):

• GET /api/sessions/current: Returns active session_id and slot file status.
• GET /api/sessions: Lists all saved session .bin slot files with file sizes and timestamps.
• POST /api/sessions/clear: Forces erase on slot 0 and resets active session to "default".
• DELETE /api/sessions/<id>: Deletes the persisted .bin cache file for a specific session.