From 85a822eb36d91929c0be6a027f2ed8d571075cbf Mon Sep 17 00:00:00 2001 From: William Mantly Date: Sat, 8 Aug 2026 20:35:38 -0400 Subject: [PATCH] docs: add multi-site architecture and replication specification (#181) --- docs/MULTI_SITE_SPEC.md | 181 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 docs/MULTI_SITE_SPEC.md diff --git a/docs/MULTI_SITE_SPEC.md b/docs/MULTI_SITE_SPEC.md new file mode 100644 index 0000000..87dcb05 --- /dev/null +++ b/docs/MULTI_SITE_SPEC.md @@ -0,0 +1,181 @@ +# Theta Suite Multi-Site Architecture & Replication Specification + +**Specification Version**: `1.0.0` +**Status**: Draft Architecture Proposal +**Target Suite Version**: `v1.50.0+` +**Repository**: [`theta-suite`](https://github.com/theta42/theta-suite) + +--- + +## Executive Summary + +This specification defines the multi-site replication, fault tolerance, and site isolation architecture for the `theta-suite` ecosystem (`sso-manager-node`, `openbao`, `openldap`, `theta-proxy`, `jump-host`, and `theta-agent`). + +The architecture follows a **Symmetric Deployment with Explicit Master Control and Autonomous Site Nodes**. Every site runs an identical software stack. At any given time, one site is designated as the **Master (Control Plane)**, while all other sites operate as **Autonomous Spoke Nodes**. + +If a WAN partition or outage disconnects a Spoke site from the Master, **no automatic failover is attempted** (preventing split-brain data corruption). Instead, the Spoke node remains in **Spoke Mode**—allowing local user OAuth logins, local SSH jumps, local web application proxying, local secret lookups, and local agent telemetry/controls to continue operating **100% autonomously without WAN dependency**. + +--- + +## 1. High-Level Architecture Diagram + +```mermaid +flowchart TB + subgraph MasterSite["Master Site (HQ / Control Plane)"] + ssoM["sso-manager-node (Master Read/Write)"] + ldapM["OpenLDAP (MMR Master Node 1)"] + baoM["OpenBao (Master Secrets)"] + proxyM["theta-proxy (HQ Web Gateway)"] + jumpM["jump-host (HQ SSH Gateway)"] + agentM["theta-agent (HQ Local Agents)"] + end + + subgraph SpokeSiteB["Spoke Site B (Chicago)"] + ssoB["sso-manager-node (Spoke Read-Only Catalog)"] + ldapB["OpenLDAP (MMR Node 2 / Read-Only Replica)"] + baoB["OpenBao (Spoke Local Secret Replica)"] + proxyB["theta-proxy (Site B Local Web Gateway)"] + jumpB["jump-host (Site B SSH Gateway - Filtered)"] + agentB["theta-agent (Site B Local Agents)"] + end + + subgraph SpokeSiteC["Spoke Site C (Austin)"] + ssoC["sso-manager-node (Spoke Read-Only Catalog)"] + ldapC["OpenLDAP (MMR Node 3 / Read-Only Replica)"] + baoC["OpenBao (Spoke Local Secret Replica)"] + proxyC["theta-proxy (Site C Local Web Gateway)"] + jumpC["jump-host (Site C SSH Gateway - Filtered)"] + agentC["theta-agent (Site C Local Agents)"] + end + + ssoM <==>|"1. SSE/HTTP Catalog Sync"| ssoB + ssoM <==>|"1. SSE/HTTP Catalog Sync"| ssoC + ldapM <==>|"2. OpenLDAP MMR syncrepl"| ldapB + ldapM <==>|"2. OpenLDAP MMR syncrepl"| ldapC + baoM -.->|"3. Secret Version Replicator"| baoB + baoM -.->|"3. Secret Version Replicator"| baoC +``` + +--- + +## 2. Core Architectural Guarantees + +| Design Aspect | Architectural Choice | System Rationale | +| :--- | :--- | :--- | +| **Deployment Symmetry** | Identical Stack everywhere | Every site runs the same Docker Compose stack & code base. | +| **Catalog Authority** | Explicit Master Designation | Single write target for `inventory.sqlite` catalog mutations. | +| **Failover Control** | **Human `god_admin` Action Only** | Zero automatic failover; eliminates split-brain risks over WAN. | +| **Isolated Autonomy** | 100% Operational Locally | OAuth logins, secrets reads, SSH jumps, and agents work offline. | +| **Write Usability** | Transparent API Proxying | Operators can issue catalog writes from any connected SSO UI. | +| **Audit Trails** | Non-Canonical Async Log Shipping | Local site logs don't block operations; flushes to Master when online. | + +--- + +## 3. Subsystem Implementation & Data Flow + +### 3.1 Directory Catalog (`inventory.sqlite`) +* **Master Site**: Holds primary write authority over directory resources (sites, hosts, services, edges, access groups). +* **Spoke Sites**: Maintain a read-only SQLite catalog replica (`inventory.sqlite`). +* **Realtime Replication**: When a resource is modified on Master, Master broadcasts an SSE / WebSocket event (`POST /api/directory-admin/sync/catalog-event`). Spoke nodes apply the update locally in real time. +* **Transparent Write Proxying**: When an operator accesses `https://sso.site-b.example.com` and performs an edit: + - If Site B is **connected to Master**: Site B proxies `POST /api/directory-admin/*` to Master. Master applies the edit and broadcasts the change. + - If Site B is **isolated from Master**: The UI displays a warning banner: + > *"Master site offline. Directory catalog edits are temporarily paused until WAN connection to Master is restored."* + +### 3.2 User Identity & Credentials (`OpenLDAP`) +* **Replication**: OpenLDAP `slapd` daemons across all sites run in **N-Way Multi-Master Replication (MMR)** or Provider/Consumer `syncrepl` mode over LDAPS (`ldaps://sso-master:636`). +* **Sub-Millisecond Auth**: Linux PAM/SSSD, sudo rules, and user SSH public keys hit `ldaps://localhost:636` at each site. +* **WAN Outage Impact**: **Zero.** Local servers authenticate users against local OpenLDAP replicas with zero latency. + +### 3.3 OAuth 2.0 / OIDC Authentication +* **Local Issuer Endpoint**: Every site runs a local OIDC issuer (`https://sso.site-b.example.com`). +* **Isolated Login Flow**: + 1. User accesses `https://app.site-b.example.com`. + 2. Spoke Proxy redirects to `https://sso.site-b.example.com/oauth/authorize`. + 3. Spoke SSO authenticates the user against the local OpenLDAP replica. + 4. Spoke SSO issues an OIDC JWT signed by the replicated site private key. + 5. Spoke Proxy verifies the JWT against Spoke SSO JWKS (`/.well-known/jwks.json`) and grants access. +* **WAN Outage Impact**: **Zero.** Full login and access token issuance continue working during WAN isolation. + +### 3.4 Reverse Proxy (`theta-proxy`) +* **Site-Local Scope**: Each site's `theta-proxy` manages its own domain routes, upstream backends, and TLS certificates (`secret/proxy/conf` in local OpenBao, ACME / Let's Encrypt certificates). +* **No Replication Required**: Proxy routes and TLS certs are independent per site. They are not stored in global SSO catalog tables and are **never locked during master outages**. + +### 3.5 SSH Jump Host (`jump-host`) +* **Site-Filtered Target Menus**: Each `jump-host` container is passed its local site identity (`SITE_SLUG=site-b-chicago`). +* **Target Filter Query**: + ```sql + SELECT * FROM resources + WHERE (kind = 'host' OR kind = 'service') + AND (site_slug = 'site-b-chicago' OR parent_site_id = 'site-b-id'); + ``` +* **Isolation Resilience**: When a user SSHs to `jump.site-b.example.com:2222`, Jump Host reads the local catalog copy and local OpenLDAP replica. Users reach Site B target servers with zero WAN dependency. + +### 3.6 Theta Agent WebSocket Hubs (`theta-agent`) +* **Site-Local WS Hub**: Machines at Site B connect their `theta-agent` daemon to **Site B's local SSO node** (`wss://sso.site-b.example.com/api/agent/ws`). +* **Local Telemetry & Desktop Controls**: Real-time memory, CPU, disk partitions, logged-in users, and desktop controls (lock session, display off, logout, reboot) operate 100% locally at Site B. +* **HQ Aggregation**: When WAN is connected, Spoke SSO nodes stream telemetry summaries to Master SSO for global dashboard viewing. + +### 3.7 Non-Canonical Audit & Activity Logging +* **Local Site Storage**: OAuth logins, SSH session events, proxy access logs, and agent execution events are written to local site log buffers (`inventory.sqlite` audit table or local log files). +* **Asynchronous Shipping**: A background log worker flushes log batches to Master via `POST /api/directory-admin/audit/ingest` when WAN is connected. +* **Zero Catalog Side Effects**: Audit log events never mutate resource definitions or block catalog transactions. + +--- + +## 4. Human `god_admin` Master Re-assignment Flow + +Automatic failover across WAN is explicitly disabled to prevent split-brain. Master promotion requires a human **`god_admin`** user: + +```mermaid +sequenceDiagram + autonumber + actor Admin as god_admin Operator + participant Spoke as Site B SSO Node + participant Master as Site A Master Node (Offline) + + Note over Spoke: Master site A goes offline (WAN outage) + Spoke->>Spoke: Retain SPOKE Mode (Catalog Read-Only) + Note over Spoke: Local OAuth, Secrets & Agents stay 100% Active + + Admin->>Spoke: Access UI / CLI & trigger "Promote to Master" + Spoke-->>Admin: Prompt Confirmation & Split-Brain Warning + Admin->>Spoke: Confirm Promotion + Spoke->>Spoke: Set ROLE = MASTER, isMaster = true + Spoke->>Spoke: Enable Catalog Write Engine & OpenBao Master Sync + Spoke-->>Admin: Site B is now active Master +``` + +--- + +## 5. Site Configuration Schema (`/config/sso-secrets.js`) + +```javascript +module.exports = { + stack: { + siteName: 'site-b-chicago', + siteSlug: 'site-b-chicago', + role: 'spoke', // 'master' or 'spoke' + masterUrl: 'https://sso.site-a.example.com', + localUrl: 'https://sso.site-b.example.com', + ldapsHost: 'sso-manager', + }, + replication: { + syncIntervalMs: 5000, + catalogSyncPath: '/api/directory-admin/sync/catalog', + auditIngestPath: '/api/directory-admin/audit/ingest', + } +}; +``` + +--- + +## 6. Implementation Phasing Plan + +1. **Phase 1 (v1.50.0)**: Add `site.role` configuration, catalog read-only enforcement on Spokes, transparent write proxying, and `SITE_SLUG` filtering for `jump-host`. +2. **Phase 2 (v1.51.0)**: Implement local OIDC JWT issuance on Spokes with JWKS cross-site validation and background secret version mirror worker. +3. **Phase 3 (v1.52.0)**: Add Spoke-to-Master non-canonical audit log shipping worker and UI `god_admin` Master promotion workflow. + +--- + +*Document generated and committed to codebase under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*