docs: add multi-site architecture and replication specification (#181)
This commit is contained in:
GitHub
@@ -0,0 +1,181 @@
|
||||
# Theta Suite Multi-Site Architecture & Replication Specification
|
||||
|
||||
**Specification Version**: `1.0.0`
|
||||
**Status**: Draft Architecture Proposal
|
||||
**Target Suite Version**: `v1.50.0+`
|
||||
**Repository**: [`theta-suite`](https://github.com/theta42/theta-suite)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This specification defines the multi-site replication, fault tolerance, and site isolation architecture for the `theta-suite` ecosystem (`sso-manager-node`, `openbao`, `openldap`, `theta-proxy`, `jump-host`, and `theta-agent`).
|
||||
|
||||
The architecture follows a **Symmetric Deployment with Explicit Master Control and Autonomous Site Nodes**. Every site runs an identical software stack. At any given time, one site is designated as the **Master (Control Plane)**, while all other sites operate as **Autonomous Spoke Nodes**.
|
||||
|
||||
If a WAN partition or outage disconnects a Spoke site from the Master, **no automatic failover is attempted** (preventing split-brain data corruption). Instead, the Spoke node remains in **Spoke Mode**—allowing local user OAuth logins, local SSH jumps, local web application proxying, local secret lookups, and local agent telemetry/controls to continue operating **100% autonomously without WAN dependency**.
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph MasterSite["Master Site (HQ / Control Plane)"]
|
||||
ssoM["sso-manager-node (Master Read/Write)"]
|
||||
ldapM["OpenLDAP (MMR Master Node 1)"]
|
||||
baoM["OpenBao (Master Secrets)"]
|
||||
proxyM["theta-proxy (HQ Web Gateway)"]
|
||||
jumpM["jump-host (HQ SSH Gateway)"]
|
||||
agentM["theta-agent (HQ Local Agents)"]
|
||||
end
|
||||
|
||||
subgraph SpokeSiteB["Spoke Site B (Chicago)"]
|
||||
ssoB["sso-manager-node (Spoke Read-Only Catalog)"]
|
||||
ldapB["OpenLDAP (MMR Node 2 / Read-Only Replica)"]
|
||||
baoB["OpenBao (Spoke Local Secret Replica)"]
|
||||
proxyB["theta-proxy (Site B Local Web Gateway)"]
|
||||
jumpB["jump-host (Site B SSH Gateway - Filtered)"]
|
||||
agentB["theta-agent (Site B Local Agents)"]
|
||||
end
|
||||
|
||||
subgraph SpokeSiteC["Spoke Site C (Austin)"]
|
||||
ssoC["sso-manager-node (Spoke Read-Only Catalog)"]
|
||||
ldapC["OpenLDAP (MMR Node 3 / Read-Only Replica)"]
|
||||
baoC["OpenBao (Spoke Local Secret Replica)"]
|
||||
proxyC["theta-proxy (Site C Local Web Gateway)"]
|
||||
jumpC["jump-host (Site C SSH Gateway - Filtered)"]
|
||||
agentC["theta-agent (Site C Local Agents)"]
|
||||
end
|
||||
|
||||
ssoM <==>|"1. SSE/HTTP Catalog Sync"| ssoB
|
||||
ssoM <==>|"1. SSE/HTTP Catalog Sync"| ssoC
|
||||
ldapM <==>|"2. OpenLDAP MMR syncrepl"| ldapB
|
||||
ldapM <==>|"2. OpenLDAP MMR syncrepl"| ldapC
|
||||
baoM -.->|"3. Secret Version Replicator"| baoB
|
||||
baoM -.->|"3. Secret Version Replicator"| baoC
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Core Architectural Guarantees
|
||||
|
||||
| Design Aspect | Architectural Choice | System Rationale |
|
||||
| :--- | :--- | :--- |
|
||||
| **Deployment Symmetry** | Identical Stack everywhere | Every site runs the same Docker Compose stack & code base. |
|
||||
| **Catalog Authority** | Explicit Master Designation | Single write target for `inventory.sqlite` catalog mutations. |
|
||||
| **Failover Control** | **Human `god_admin` Action Only** | Zero automatic failover; eliminates split-brain risks over WAN. |
|
||||
| **Isolated Autonomy** | 100% Operational Locally | OAuth logins, secrets reads, SSH jumps, and agents work offline. |
|
||||
| **Write Usability** | Transparent API Proxying | Operators can issue catalog writes from any connected SSO UI. |
|
||||
| **Audit Trails** | Non-Canonical Async Log Shipping | Local site logs don't block operations; flushes to Master when online. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Subsystem Implementation & Data Flow
|
||||
|
||||
### 3.1 Directory Catalog (`inventory.sqlite`)
|
||||
* **Master Site**: Holds primary write authority over directory resources (sites, hosts, services, edges, access groups).
|
||||
* **Spoke Sites**: Maintain a read-only SQLite catalog replica (`inventory.sqlite`).
|
||||
* **Realtime Replication**: When a resource is modified on Master, Master broadcasts an SSE / WebSocket event (`POST /api/directory-admin/sync/catalog-event`). Spoke nodes apply the update locally in real time.
|
||||
* **Transparent Write Proxying**: When an operator accesses `https://sso.site-b.example.com` and performs an edit:
|
||||
- If Site B is **connected to Master**: Site B proxies `POST /api/directory-admin/*` to Master. Master applies the edit and broadcasts the change.
|
||||
- If Site B is **isolated from Master**: The UI displays a warning banner:
|
||||
> *"Master site offline. Directory catalog edits are temporarily paused until WAN connection to Master is restored."*
|
||||
|
||||
### 3.2 User Identity & Credentials (`OpenLDAP`)
|
||||
* **Replication**: OpenLDAP `slapd` daemons across all sites run in **N-Way Multi-Master Replication (MMR)** or Provider/Consumer `syncrepl` mode over LDAPS (`ldaps://sso-master:636`).
|
||||
* **Sub-Millisecond Auth**: Linux PAM/SSSD, sudo rules, and user SSH public keys hit `ldaps://localhost:636` at each site.
|
||||
* **WAN Outage Impact**: **Zero.** Local servers authenticate users against local OpenLDAP replicas with zero latency.
|
||||
|
||||
### 3.3 OAuth 2.0 / OIDC Authentication
|
||||
* **Local Issuer Endpoint**: Every site runs a local OIDC issuer (`https://sso.site-b.example.com`).
|
||||
* **Isolated Login Flow**:
|
||||
1. User accesses `https://app.site-b.example.com`.
|
||||
2. Spoke Proxy redirects to `https://sso.site-b.example.com/oauth/authorize`.
|
||||
3. Spoke SSO authenticates the user against the local OpenLDAP replica.
|
||||
4. Spoke SSO issues an OIDC JWT signed by the replicated site private key.
|
||||
5. Spoke Proxy verifies the JWT against Spoke SSO JWKS (`/.well-known/jwks.json`) and grants access.
|
||||
* **WAN Outage Impact**: **Zero.** Full login and access token issuance continue working during WAN isolation.
|
||||
|
||||
### 3.4 Reverse Proxy (`theta-proxy`)
|
||||
* **Site-Local Scope**: Each site's `theta-proxy` manages its own domain routes, upstream backends, and TLS certificates (`secret/proxy/conf` in local OpenBao, ACME / Let's Encrypt certificates).
|
||||
* **No Replication Required**: Proxy routes and TLS certs are independent per site. They are not stored in global SSO catalog tables and are **never locked during master outages**.
|
||||
|
||||
### 3.5 SSH Jump Host (`jump-host`)
|
||||
* **Site-Filtered Target Menus**: Each `jump-host` container is passed its local site identity (`SITE_SLUG=site-b-chicago`).
|
||||
* **Target Filter Query**:
|
||||
```sql
|
||||
SELECT * FROM resources
|
||||
WHERE (kind = 'host' OR kind = 'service')
|
||||
AND (site_slug = 'site-b-chicago' OR parent_site_id = 'site-b-id');
|
||||
```
|
||||
* **Isolation Resilience**: When a user SSHs to `jump.site-b.example.com:2222`, Jump Host reads the local catalog copy and local OpenLDAP replica. Users reach Site B target servers with zero WAN dependency.
|
||||
|
||||
### 3.6 Theta Agent WebSocket Hubs (`theta-agent`)
|
||||
* **Site-Local WS Hub**: Machines at Site B connect their `theta-agent` daemon to **Site B's local SSO node** (`wss://sso.site-b.example.com/api/agent/ws`).
|
||||
* **Local Telemetry & Desktop Controls**: Real-time memory, CPU, disk partitions, logged-in users, and desktop controls (lock session, display off, logout, reboot) operate 100% locally at Site B.
|
||||
* **HQ Aggregation**: When WAN is connected, Spoke SSO nodes stream telemetry summaries to Master SSO for global dashboard viewing.
|
||||
|
||||
### 3.7 Non-Canonical Audit & Activity Logging
|
||||
* **Local Site Storage**: OAuth logins, SSH session events, proxy access logs, and agent execution events are written to local site log buffers (`inventory.sqlite` audit table or local log files).
|
||||
* **Asynchronous Shipping**: A background log worker flushes log batches to Master via `POST /api/directory-admin/audit/ingest` when WAN is connected.
|
||||
* **Zero Catalog Side Effects**: Audit log events never mutate resource definitions or block catalog transactions.
|
||||
|
||||
---
|
||||
|
||||
## 4. Human `god_admin` Master Re-assignment Flow
|
||||
|
||||
Automatic failover across WAN is explicitly disabled to prevent split-brain. Master promotion requires a human **`god_admin`** user:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
actor Admin as god_admin Operator
|
||||
participant Spoke as Site B SSO Node
|
||||
participant Master as Site A Master Node (Offline)
|
||||
|
||||
Note over Spoke: Master site A goes offline (WAN outage)
|
||||
Spoke->>Spoke: Retain SPOKE Mode (Catalog Read-Only)
|
||||
Note over Spoke: Local OAuth, Secrets & Agents stay 100% Active
|
||||
|
||||
Admin->>Spoke: Access UI / CLI & trigger "Promote to Master"
|
||||
Spoke-->>Admin: Prompt Confirmation & Split-Brain Warning
|
||||
Admin->>Spoke: Confirm Promotion
|
||||
Spoke->>Spoke: Set ROLE = MASTER, isMaster = true
|
||||
Spoke->>Spoke: Enable Catalog Write Engine & OpenBao Master Sync
|
||||
Spoke-->>Admin: Site B is now active Master
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Site Configuration Schema (`/config/sso-secrets.js`)
|
||||
|
||||
```javascript
|
||||
module.exports = {
|
||||
stack: {
|
||||
siteName: 'site-b-chicago',
|
||||
siteSlug: 'site-b-chicago',
|
||||
role: 'spoke', // 'master' or 'spoke'
|
||||
masterUrl: 'https://sso.site-a.example.com',
|
||||
localUrl: 'https://sso.site-b.example.com',
|
||||
ldapsHost: 'sso-manager',
|
||||
},
|
||||
replication: {
|
||||
syncIntervalMs: 5000,
|
||||
catalogSyncPath: '/api/directory-admin/sync/catalog',
|
||||
auditIngestPath: '/api/directory-admin/audit/ingest',
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation Phasing Plan
|
||||
|
||||
1. **Phase 1 (v1.50.0)**: Add `site.role` configuration, catalog read-only enforcement on Spokes, transparent write proxying, and `SITE_SLUG` filtering for `jump-host`.
|
||||
2. **Phase 2 (v1.51.0)**: Implement local OIDC JWT issuance on Spokes with JWKS cross-site validation and background secret version mirror worker.
|
||||
3. **Phase 3 (v1.52.0)**: Add Spoke-to-Master non-canonical audit log shipping worker and UI `god_admin` Master promotion workflow.
|
||||
|
||||
---
|
||||
|
||||
*Document generated and committed to codebase under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*
|
||||
Reference in New Issue
Block a user