# Theta Suite Multi-Site Architecture & Replication Specification

**Specification Version**: `1.0.0`  
**Status**: Draft Architecture Proposal  
**Target Suite Version**: `v1.50.0+`  
**Repository**: [`theta-suite`](https://github.com/theta42/theta-suite)

---

## Executive Summary

This specification defines the multi-site replication, fault tolerance, and site isolation architecture for the `theta-suite` ecosystem (`sso-manager-node`, `openbao`, `openldap`, `theta-proxy`, `jump-host`, and `theta-agent`).

The architecture follows a **Symmetric Deployment with Explicit Master Control and Autonomous Site Nodes**. Every site runs an identical software stack. At any given time, one site is designated as the **Master (Control Plane)**, while all other sites operate as **Autonomous Spoke Nodes**.

If a WAN partition or outage disconnects a Spoke site from the Master, **no automatic failover is attempted** (preventing split-brain data corruption). Instead, the Spoke node remains in **Spoke Mode**—allowing local user OAuth logins, local SSH jumps, local web application proxying, local secret lookups, and local agent telemetry/controls to continue operating **100% autonomously without WAN dependency**.

---

## 1. High-Level Architecture Diagram

```mermaid
flowchart TB
    subgraph MasterSite["Master Site (HQ / Control Plane)"]
        ssoM["sso-manager-node (Master Read/Write)"]
        ldapM["OpenLDAP (MMR Master Node 1)"]
        baoM["OpenBao (Master Secrets)"]
        proxyM["theta-proxy (HQ Web Gateway)"]
        jumpM["jump-host (HQ SSH Gateway)"]
        agentM["theta-agent (HQ Local Agents)"]
    end

    subgraph SpokeSiteB["Spoke Site B (Chicago)"]
        ssoB["sso-manager-node (Spoke Read-Only Catalog)"]
        ldapB["OpenLDAP (MMR Node 2 / Read-Only Replica)"]
        baoB["OpenBao (Spoke Local Secret Replica)"]
        proxyB["theta-proxy (Site B Local Web Gateway)"]
        jumpB["jump-host (Site B SSH Gateway - Filtered)"]
        agentB["theta-agent (Site B Local Agents)"]
    end

    subgraph SpokeSiteC["Spoke Site C (Austin)"]
        ssoC["sso-manager-node (Spoke Read-Only Catalog)"]
        ldapC["OpenLDAP (MMR Node 3 / Read-Only Replica)"]
        baoC["OpenBao (Spoke Local Secret Replica)"]
        proxyC["theta-proxy (Site C Local Web Gateway)"]
        jumpC["jump-host (Site C SSH Gateway - Filtered)"]
        agentC["theta-agent (Site C Local Agents)"]
    end

    ssoM <==>|"1. SSE/HTTP Catalog Sync"| ssoB
    ssoM <==>|"1. SSE/HTTP Catalog Sync"| ssoC
    ldapM <==>|"2. OpenLDAP MMR syncrepl"| ldapB
    ldapM <==>|"2. OpenLDAP MMR syncrepl"| ldapC
    baoM -.->|"3. Secret Version Replicator"| baoB
    baoM -.->|"3. Secret Version Replicator"| baoC
```

---

## 2. Core Architectural Guarantees

| Design Aspect | Architectural Choice | System Rationale |
| :--- | :--- | :--- |
| **Deployment Symmetry** | Identical Stack everywhere | Every site runs the same Docker Compose stack & code base. |
| **Catalog Authority** | Explicit Master Designation | Single write target for `inventory.sqlite` catalog mutations. |
| **Failover Control** | **Human `god_admin` Action Only** | Zero automatic failover; eliminates split-brain risks over WAN. |
| **Isolated Autonomy** | 100% Operational Locally | OAuth logins, secrets reads, SSH jumps, and agents work offline. |
| **Write Usability** | Transparent API Proxying | Operators can issue catalog writes from any connected SSO UI. |
| **Audit Trails** | Non-Canonical Async Log Shipping | Local site logs don't block operations; flushes to Master when online. |

---

## 3. Subsystem Implementation & Data Flow

### 3.1 Directory Catalog (`inventory.sqlite`)
* **Master Site**: Holds primary write authority over directory resources (sites, hosts, services, edges, access groups).
* **Spoke Sites**: Maintain a read-only SQLite catalog replica (`inventory.sqlite`).
* **Realtime Replication**: When a resource is modified on Master, Master broadcasts an SSE / WebSocket event (`POST /api/directory-admin/sync/catalog-event`). Spoke nodes apply the update locally in real time.
* **Transparent Write Proxying**: When an operator accesses `https://sso.site-b.example.com` and performs an edit:
  - If Site B is **connected to Master**: Site B proxies `POST /api/directory-admin/*` to Master. Master applies the edit and broadcasts the change.
  - If Site B is **isolated from Master**: The UI displays a warning banner:
    > *"Master site offline. Directory catalog edits are temporarily paused until WAN connection to Master is restored."*

### 3.2 User Identity & Credentials (`OpenLDAP`)
* **Replication**: OpenLDAP `slapd` daemons across all sites run in **N-Way Multi-Master Replication (MMR)** or Provider/Consumer `syncrepl` mode over LDAPS (`ldaps://sso-master:636`).
* **Sub-Millisecond Auth**: Linux PAM/SSSD, sudo rules, and user SSH public keys hit `ldaps://localhost:636` at each site.
* **WAN Outage Impact**: **Zero.** Local servers authenticate users against local OpenLDAP replicas with zero latency.

### 3.3 OAuth 2.0 / OIDC Authentication
* **Local Issuer Endpoint**: Every site runs a local OIDC issuer (`https://sso.site-b.example.com`).
* **Isolated Login Flow**:
  1. User accesses `https://app.site-b.example.com`.
  2. Spoke Proxy redirects to `https://sso.site-b.example.com/oauth/authorize`.
  3. Spoke SSO authenticates the user against the local OpenLDAP replica.
  4. Spoke SSO issues an OIDC JWT signed by the replicated site private key.
  5. Spoke Proxy verifies the JWT against Spoke SSO JWKS (`/.well-known/jwks.json`) and grants access.
* **WAN Outage Impact**: **Zero.** Full login and access token issuance continue working during WAN isolation.

### 3.4 Reverse Proxy (`theta-proxy`)
* **Site-Local Scope**: Each site's `theta-proxy` manages its own domain routes, upstream backends, and TLS certificates (`secret/proxy/conf` in local OpenBao, ACME / Let's Encrypt certificates).
* **No Replication Required**: Proxy routes and TLS certs are independent per site. They are not stored in global SSO catalog tables and are **never locked during master outages**.

### 3.5 SSH Jump Host (`jump-host`)
* **Site-Filtered Target Menus**: Each `jump-host` container is passed its local site identity (`SITE_SLUG=site-b-chicago`).
* **Target Filter Query**:
  ```sql
  SELECT * FROM resources 
  WHERE (kind = 'host' OR kind = 'service') 
    AND (site_slug = 'site-b-chicago' OR parent_site_id = 'site-b-id');
  ```
* **Isolation Resilience**: When a user SSHs to `jump.site-b.example.com:2222`, Jump Host reads the local catalog copy and local OpenLDAP replica. Users reach Site B target servers with zero WAN dependency.

### 3.6 Theta Agent WebSocket Hubs (`theta-agent`)
* **Site-Local WS Hub**: Machines at Site B connect their `theta-agent` daemon to **Site B's local SSO node** (`wss://sso.site-b.example.com/api/agent/ws`).
* **Local Telemetry & Desktop Controls**: Real-time memory, CPU, disk partitions, logged-in users, and desktop controls (lock session, display off, logout, reboot) operate 100% locally at Site B.
* **HQ Aggregation**: When WAN is connected, Spoke SSO nodes stream telemetry summaries to Master SSO for global dashboard viewing.

### 3.7 Non-Canonical Audit & Activity Logging
* **Local Site Storage**: OAuth logins, SSH session events, proxy access logs, and agent execution events are written to local site log buffers (`inventory.sqlite` audit table or local log files).
* **Asynchronous Shipping**: A background log worker flushes log batches to Master via `POST /api/directory-admin/audit/ingest` when WAN is connected.
* **Zero Catalog Side Effects**: Audit log events never mutate resource definitions or block catalog transactions.

---

## 4. Human `god_admin` Master Re-assignment Flow

Automatic failover across WAN is explicitly disabled to prevent split-brain. Master promotion requires a human **`god_admin`** user:

```mermaid
sequenceDiagram
    autonumber
    actor Admin as god_admin Operator
    participant Spoke as Site B SSO Node
    participant Master as Site A Master Node (Offline)

    Note over Spoke: Master site A goes offline (WAN outage)
    Spoke->>Spoke: Retain SPOKE Mode (Catalog Read-Only)
    Note over Spoke: Local OAuth, Secrets & Agents stay 100% Active

    Admin->>Spoke: Access UI / CLI & trigger "Promote to Master"
    Spoke-->>Admin: Prompt Confirmation & Split-Brain Warning
    Admin->>Spoke: Confirm Promotion
    Spoke->>Spoke: Set ROLE = MASTER, isMaster = true
    Spoke->>Spoke: Enable Catalog Write Engine & OpenBao Master Sync
    Spoke-->>Admin: Site B is now active Master
```

---

## 5. Site Configuration Schema (`/config/sso-secrets.js`)

```javascript
module.exports = {
  stack: {
    siteName: 'site-b-chicago',
    siteSlug: 'site-b-chicago',
    role: 'spoke', // 'master' or 'spoke'
    masterUrl: 'https://sso.site-a.example.com',
    localUrl: 'https://sso.site-b.example.com',
    ldapsHost: 'sso-manager',
  },
  replication: {
    syncIntervalMs: 5000,
    catalogSyncPath: '/api/directory-admin/sync/catalog',
    auditIngestPath: '/api/directory-admin/audit/ingest',
  }
};
```

---

## 6. Implementation Phasing Plan

1. **Phase 1 (v1.50.0)**: Add `site.role` configuration, catalog read-only enforcement on Spokes, transparent write proxying, and `SITE_SLUG` filtering for `jump-host`.
2. **Phase 2 (v1.51.0)**: Implement local OIDC JWT issuance on Spokes with JWKS cross-site validation and background secret version mirror worker.
3. **Phase 3 (v1.52.0)**: Add Spoke-to-Master non-canonical audit log shipping worker and UI `god_admin` Master promotion workflow.

---

*Document generated and committed to codebase under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*