# Theta Suite Multi-Site Architecture & VPN Specification **Specification Version**: `2.0.0` **Status**: Final Architecture Specification **Target Suite Version**: `v1.50.0+` **Repository**: [`theta-suite`](https://github.com/theta42/theta-suite) --- ## Executive Summary This specification defines the multi-site replication, fault tolerance, autonomous site isolation, and integrated WireGuard mesh network architecture for the `theta-suite` ecosystem. The architecture enforces **Deployment Symmetry**, **Explicit Master Control via `god_admin` Authority**, **`theta-gateway` Mesh & SSH Integration**, and **Zero-WAN Dependency for Local Operations**. --- ## 1. High-Level System Architecture ```mermaid flowchart TB subgraph ControlPlane["Master Site (Site 10.1 - HQ / Control Plane)"] ssoM["sso-manager-node (Master Read/Write)"] ldapM["OpenLDAP (MMR Master Node 1)"] baoM["OpenBao (Master Secrets Engine)"] proxyM["theta-proxy (HQ Web Gateway)"] gateM["theta-gateway (HQ SSH & WireGuard Mesh Gate)"] agentM["theta-agent (HQ Local Agents)"] end subgraph SiteB["Spoke Site 10.2 (Staten Island LAN)"] ssoB["sso-manager-node (Spoke Read-Only Catalog)"] ldapB["OpenLDAP (MMR Node 2 / Read-Only Replica)"] baoB["OpenBao (Spoke Local Secret Replica)"] proxyB["theta-proxy (Site 10.2 Local Web Gateway)"] gateB["theta-gateway (Site 10.2 SSH & WG Mesh Gate)"] agentB["theta-agent (Site 10.2 Local Agents)"] end subgraph SiteNL["Spoke Site 10.5 (Netherlands Offshore Exit)"] ssoNL["sso-manager-node (Spoke Read-Only Catalog)"] ldapNL["OpenLDAP (MMR Node 5 / Read-Only Replica)"] baoNL["OpenBao (Spoke Local Secret Replica)"] proxyNL["theta-proxy (Site 10.5 Local Web Gateway)"] gateNL["theta-gateway (Site 10.5 Offshore Exit Gate)"] agentNL["theta-agent (Site 10.5 Local Agents)"] end gateM <==>|"WireGuard Encrypted Mesh Tunnel (172.24.0.x)"| gateB gateM <==>|"WireGuard Encrypted Mesh Tunnel (172.24.0.x)"| gateNL gateB <==>|"WireGuard Direct Tunnel"| gateNL ssoM <==>|"HTTP SSE Catalog Change Events"| ssoB ldapM <==>|"OpenLDAP MMR syncrepl (ldaps://636)"| ldapB baoM -.->|"Secret Version Replicator"| baoB ``` --- ## 2. Component Core Roles ### 2.1 `theta-gateway` (Merged SSH Jump & WireGuard Mesh Gateway) The container previously named `jump-host` is officially rebranded and expanded to **`theta-gateway`**. It acts as the single security & routing gateway for each site: 1. **SSH Jump Host (Port 2222)**: Handles interactive terminal jumps, user identity verification via local OpenLDAP, and dynamic SSH key injection. 2. **Site-Aware SSH Target Filtering**: `theta-gateway` filters target host/service selection menus strictly to resources assigned to that local site (`SITE_SLUG`). 3. **WireGuard Mesh Router (Port 51820 / 51871)**: Handles tunnel termination (`wg0`), peer key management, keepalives, and inter-site routing. 4. **NETMAP & Policy Routing Engine**: Applies `iptables` NETMAP shadow subnet translations, dynamic return path masquerading, and policy exit routes (`table offshore`, `table us_vps`). ### 2.2 `sso-manager-node` (Control Plane & Local Issuer) * **Master Role (`isMaster = true`)**: Holds single write authority for directory resources in `inventory.sqlite`. Processes write requests proxied from Spokes. * **Spoke Role (`isMaster = false`)**: Runs in Read-Only Catalog mode. Performs local OIDC JWT token issuance for local web apps via local OpenLDAP authentication. ### 2.3 `theta-proxy` (Site-Local Web Gateway) * **Local Route Scope**: Manages local web application reverse proxying and TLS certificates (ACME / Let's Encrypt / local certs) 100% locally. * **Zero Locking**: Proxy route definitions and TLS certs are not locked during Master WAN outages. ### 2.4 `theta-agent` (Site-Local Agent Hub) * **Local WS Connection**: Connects to the local site's `sso-manager-node` WebSocket (`wss://sso.site-b.example.com/api/agent/ws`). * **Local Autonomy**: Real-time telemetry, memory/CPU metrics, disk usage, active logged-in users (`who`), and desktop control commands (lock, display off, logout, reboot) operate 100% locally. --- ## 3. Explicit Master Control & Human `god_admin` Authority To guarantee **0% split-brain risk**, automatic failover across WAN is explicitly disabled: ``` WAN OUTAGE DETECTED │ ▼ Spoke Node Unconditionally Retains SPOKE Mode (Read-Only Catalog / Full Local Operations) │ ▼ Requires Human god_admin Promotion Action (Explicit Confirmation Modal in SSO UI / CLI) ``` 1. **Unreachable Master Behavior**: If a Spoke node loses WAN connection to the Master, it **unconditionally remains in Spoke Mode**. 2. **Human Re-assignment**: Changing or promoting a Master node requires an explicit action by an authenticated **`god_admin`** user via the SSO UI or `theta-suite-admin promote-master` CLI. --- ## 4. Integrated WireGuard Mesh & Key Provisioning Protocol ### 4.1 Automatic Key Exchange & Peer Discovery 1. When a new `theta-gateway` boots or joins a site via Join Key: - It generates a Curve25519 keypair and registers its public key, listen port, and public endpoint with `sso-manager-node` (`POST /api/mesh/gateway/register`). 2. `sso-manager-node` calculates the site index (`Site 10.x`), assigns mesh IPs (`172.24.0.x`), and broadcasts updated peer definitions to all active `theta-gateway` instances. 3. Each `theta-gateway` updates its running WireGuard interface (`wg0`) dynamically via `wgctrl` / `iptables` without dropping existing connections. ### 4.2 NETMAP Shadow Subnet Addressing (`10..168.0/24`) To prevent IP collisions when multiple sites use default `192.168.1.0/24` physical LANs, `theta-gateway` automatically enables **NETMAP Shadow Subnets by default**: ```bash # NETMAP: Shadow network (10..168.x) to physical LAN (192.168.1.x) PostUp = iptables -t nat -A PREROUTING -i wg0 -d 10..168.0/24 -j NETMAP --to 192.168.1.0/24 PostUp = iptables -t nat -A POSTROUTING -o wg0 -s 192.168.1.0/24 -j NETMAP --to 10..168.0/24 PostUp = ip route add local 10..168.0/24 dev lo ``` * **Effect**: A server at Site 10.2 (`192.168.1.50`) can reach a server at Site 10.4 (`192.168.1.50`) by pinging `10.4.168.50`. Neither site needs to modify router DHCP or local subnets! ### 4.3 Policy Exit Routing & Return Path SOURCENAT * **Custom Route Tables**: `theta-gateway` supports selective outbound exit tables (`table offshore`, `table us_vps`) based on IP ranges (`ip rule add from 10.x.254.0/24 lookup offshore`). * **Dynamic Return Path SOURCENAT**: Exit nodes (e.g. Netherlands `10.5`) apply source-NAT masquerading for incoming tunnel traffic to guarantee symmetric return path routing: ```bash PostUp = iptables -t nat -A POSTROUTING -o wg0 ! -s 172.24.0.0/13 -j MASQUERADE ``` --- ## 5. Roaming Admin Access (QR Codes & Optional Tailscale) 1. **Native WireGuard Client Profiles**: - `sso-manager-node` includes a built-in **Client Profile Generator** in the UI. - Admins can generate a mobile/laptop profile, displaying a **QR code** for immediate scan into the official WireGuard app on iOS/Android or a downloadable `wg0.conf` for laptops. 2. **Optional Tailscale / Headscale Connector**: - For roaming devices in environments where WireGuard UDP ports are blocked, `theta-gateway` supports an optional `tailscale` / `headscale` sidecar integration. --- ## 6. Non-Canonical Audit Logging * **Local Activity Storage**: OAuth logins, SSH session events, proxy access logs, and agent execution events write to local site audit tables without blocking local operations. * **Asynchronous Log Worker**: A background worker flushes log batches to Master via `POST /api/directory-admin/audit/ingest` when WAN is online. --- ## Appendix A: Production Reference WireGuard Topology Config ### Site 10.2 (Staten Island LAN Node) Gateway Reference (`wg0.conf`) ```ini [Interface] Address = 172.24.0.2/32 PrivateKey = ListenPort = 51820 Table = off # Mesh Subnet Routes PostUp = ip route add 10.0.0.0/8 dev %i PostUp = ip route add 172.24.0.0/13 dev %i # Policy Routing Exits PostUp = ip route add default via 10.5.0.1 dev %i table offshore PostUp = ip route add default via 172.24.0.1 dev %i table us_vps PostUp = ip rule add from 10.2.254.0/24 lookup offshore PostUp = ip rule add from 10.2.253.0/24 lookup main preference 1000 # NETMAP Shadow Network (10.2.168.x -> 192.168.1.x) PostUp = iptables -t nat -A PREROUTING -i %i -d 10.2.168.0/24 -j NETMAP --to 192.168.1.0/24 PostUp = iptables -t nat -A POSTROUTING -o %i -s 192.168.1.0/24 -j NETMAP --to 10.2.168.0/24 PostUp = ip route add local 10.2.168.0/24 dev lo # Forwarding & NAT PostUp = iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o %i -j MASQUERADE PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE PostUp = iptables -A FORWARD -i %i -o eth0 -j ACCEPT PostUp = iptables -A FORWARD -i eth0 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT # System Kernel Options PostUp = sysctl -w net.ipv4.ip_forward=1 PostUp = sysctl -w net.ipv4.conf.all.rp_filter=0 PostUp = sysctl -w net.ipv4.conf.eth0.rp_filter=0 PostUp = sysctl -w net.ipv4.conf.%i.rp_filter=0 # --- PEERS --- [Peer] # Site 10.1: US Hub / VPS Exit PublicKey = QZCvR3N1CdUabC2xWfc1lmYKHfSiXYs1UoVINIMftws= Endpoint = gg-si1.wgnode.com:51820 AllowedIPs = 172.24.0.0/16, 10.0.0.0/8, 0.0.0.0/0 PersistentKeepalive = 25 [Peer] # Site 10.5: Netherlands Offshore Exit Node PublicKey = MlF6h3YI1MIvOlgyNozCMoa/rICoLNtc7r/pseKiHQQ= Endpoint = nl-alexhost.wgnode.com:51871 AllowedIPs = 172.24.0.5/32, 10.5.0.0/16, 0.0.0.0/0 PersistentKeepalive = 25 ``` ### Site 10.5 (Netherlands Exit Node) Gateway Reference (`wg0.conf`) ```ini [Interface] Address = 172.24.0.5/32 PrivateKey = ListenPort = 51871 PostUp = ip addr add 10.5.0.1/16 dev %i PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE # Dynamic Return Path Masquerading (SOURCENAT) PostUp = iptables -t nat -A POSTROUTING -o %i ! -s 172.24.0.0/13 -j MASQUERADE PostUp = sysctl -w net.ipv4.ip_forward=1 [Peer] # Site 10.2: Staten Island LAN PublicKey = AsS7aikCUrXpdfSvwFnMs0yUaoQ7ZCkoUVOmNdl7NS8= AllowedIPs = 172.24.0.2/32, 10.2.0.0/16 [Peer] # Site 10.1: US Hub VPS PublicKey = QZCvR3N1CdUabC2xWfc1lmYKHfSiXYs1UoVINIMftws= AllowedIPs = 172.24.0.1/32, 10.1.0.0/8 ``` --- *Final Architecture Specification committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*