# Session-Managed KV Cache for llama-server + ollama-proxy

## Goal

Make the multi-GPU llama.cpp backend (exposed via `ollama-proxy.py`) correctly manage KV cache **per session**, so that:

1. Different users / agents / OpenWebUI chats do not pollute each other’s context.
2. Large shared system prompts (2–30k tokens) used by Maki, OpenWebUI, and other tools are reused instead of being re-prefilled every time.
3. Session state **survives reboots** (disk-backed).

Frontends:
- OpenWebUI
- Maki (https://github.com/wmantly/maki)

Backend stack:
- `llama-server` (port 8080, `--parallel 1`, `--slot-save-path /var/cache/llama-slots`, `--cache-ram 16384`)
- `ollama-proxy.py` (port 11434) – Ollama / OpenAI / Anthropic compatible surface with automatic session affinity

---

## Architecture

```text
OpenWebUI  ─┐
            ├──► ollama-proxy.py (port 11434) ──► llama-server (port 8080)
Maki       ─┘         │
                      └── session_id → /var/cache/llama-slots/<id>.bin
```

- Single slot (`--parallel 1`) for maximum context and dedicated tensor parallelism across all 3 GPUs.
- Proxy owns session affinity and decides when to save / restore / erase the slot.
- Disk persistence via `--slot-save-path /var/cache/llama-slots`.
- Hot prefix reuse via `--cache-ram 16384` + `cache_prompt: true`.

---

## 1. Session Identity Extraction

The proxy extracts a stable `session_id` on every request according to this priority:

1. **HTTP Headers**:
   - `X-Session-Id`
   - `X-Conversation-Id`
   - `Session-Id`
   - `Conversation-Id`
2. **JSON Body Fields**:
   - `session_id`
   - `conversation_id`
   - `chat_id`
   - `id` (when structured as a chat identifier)
3. **Fallback**:
   - `sys-<sha256[:16]>` (hash of the system prompt so identical system prompts share a base KV slot)
   - `"default"`

---

## 2. Proxy Session Lifecycle

When a request arrives at `ollama-proxy.py`:

```text
with session_lock:
    if session_id == current_session:
        proceed

    # 1. Persist previous session slot
    if current_session and current_session != "default":
        POST /slots/0?action=save {"filename": f"{current_session}.bin"}

    # 2. Restore new session or start fresh
    if os.path.exists(f"/var/cache/llama-slots/{session_id}.bin"):
        POST /slots/0?action=restore {"filename": f"{session_id}.bin"}
    else:
        POST /slots/0?action=erase

    current_session = session_id
```

---

## 3. Session Management Endpoints

Exposed on `ollama-proxy` (`port 11434`):

* `GET /api/sessions/current`: Returns active `session_id` and slot file status.
* `GET /api/sessions`: Lists all saved session `.bin` slot files with file sizes and timestamps.
* `POST /api/sessions/clear`: Forces erase on slot 0 and resets active session to `"default"`.
* `DELETE /api/sessions/<id>`: Deletes the persisted `.bin` cache file for a specific session.