# Theta Suite Multi-Site Architecture & VPN Specification **Specification Version**: `2.1.0` **Status**: Target architecture / roadmap. **A simpler v1 is already shipped** — see the box below before reading further. **Target Suite Version**: `v1.50.0+` **Repository**: [`theta-suite`](https://github.com/theta42/theta-suite) > ## Shipped today (theta-directory v2.2.0–v2.3.0, theta-suite v2.2.0) > A spoke joins a master by pulling a **one-time directory export** (LDAP LDIF + resource catalog) over a **site join key**, entirely over the existing HTTPS API — no WireGuard mesh, no live replication, no OpenBao secret sync. It's simpler than everything below and it's real, tested, and released. Read [`sso-manager-node/docs/site-join.md`](https://github.com/theta42/theta-directory/blob/master/docs/site-join.md) first; it documents exactly what exists: > - `POST /api/site/join-keys`, `/api/site/export`, `/api/site/ping`, `/api/site/join` — mint a key on the master, pull-and-adopt on the spoke. > - Fresh-install-only (no merging into a populated directory). > - Spoke is read-only post-join (writes 403 toward the master); role persists in `/config/site.json`, survives restarts. > - `setup.env`'s `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY` wire it into first-run `setup.sh`. > > Everything below this point is the **larger target architecture** this session designed (WireGuard mesh, live fire-and-forget replication, identical-directory signing, no-inbound relay, mDNS local-discovery) — none of it is built, and **none of it is required** for the shipped v1 above to work. Treat it as where multi-site could grow next (continuous sync instead of one-time adoption, true site-to-site networking, spokes with no inbound path at all), not as a description of current behavior. Don't let this document's detail imply more is built than actually is — check the status table at the bottom, or better, check `git log`/the linked doc, before trusting either. Design scale: a handful of sites (dozen max, 254 hard ceiling — see §4), a few hundred users/hosts total. This is a deliberate, small, trusted-operator deployment, not a hyperscale/adversarial-tenant one — several decisions below (fire-and-forget replication, identical directories) trade blast-radius for simplicity *because* the scale allows it. Don't generalize these choices past that scale without re-deriving them. --- ## 1. High-Level System Architecture ```mermaid flowchart TB subgraph ControlPlane["Master Site (write authority)"] ssoM["sso-manager-node (isMaster=true)"] ldapM["OpenLDAP (MMR write node)"] baoM["OpenBao (local, replication source)"] proxyM["theta-proxy"] gateM["theta-gateway"] agentM["theta-agent"] end subgraph SiteB["Spoke — inbound (has a public IP)"] ssoB["sso-manager-node (isMaster=false)"] ldapB["OpenLDAP (MMR read replica)"] baoB["OpenBao (local replica)"] proxyB["theta-proxy — serves this site's public traffic directly"] gateB["theta-gateway"] agentB["theta-agent"] end subgraph SiteC["Spoke — no inbound (CGNAT)"] ssoC["sso-manager-node (isMaster=false)"] ldapC["OpenLDAP (MMR read replica)"] baoC["OpenBao (local replica)"] proxyC["theta-proxy — LAN-local traffic only"] gateC["theta-gateway"] agentC["theta-agent"] end gateM <==>|"WireGuard mesh tunnel"| gateB gateM <==>|"WireGuard mesh tunnel"| gateC ssoM -.->|"fire-and-forget push: catalog + secrets + signing key"| ssoB ssoM -.->|"fire-and-forget push"| ssoC ldapM <==>|"OpenLDAP MMR syncrepl"| ldapB ldapM <==>|"OpenLDAP MMR syncrepl"| ldapC proxyM -->|"TLS-terminate + relay (no direct path exists)"| gateM gateM ==>|"WG tunnel"| gateC ``` --- ## 2. Every Directory Is Identical Master and every spoke run the **same LDAP data, the same OpenBao secrets, and the same agent-signing key**. Hitting any site's `sso-manager-node` for read/auth purposes is equivalent to hitting any other. The only asymmetry is **write authority** (§3). This is a deliberate tradeoff, not a default: it means compromising *any single spoke* — including the smallest, least-secured one — grants an attacker the same agent-command authority (`update_binary`, `arbitrary_bash`, service control) as compromising the master, because every site holds the same Ed25519 signing key (`sso-manager-node/nodejs/utils/agent_keys.js`). Accepted here because the deployment scale is small and trusted. Do not extend this pattern to a larger/adversarial-tenant deployment without revisiting it. Consequence: `theta-agent` needs **no change** to support multi-site — it already does TOFU pairing against a single trusted key (`websocket.go:341-351`), and since that key is identical everywhere, any site's `sso-manager-node` can validly sign a command for any agent, anywhere, without agents needing a keyring. ### 2.1 What Replicates, and How | Data | Mechanism | Direction | |---|---|---| | LDAP (users, groups) | OpenLDAP MMR syncrepl | master (write) → spokes (read-only) | | OpenBao secrets (incl. agent-signing key at `secret/agent/signing-key`) | **New**: custom replicator (OpenBao has no built-in multi-site replication — Performance Replication is Vault-Enterprise-only, confirmed absent from OpenBao as of this writing) | master (write) → spokes (read-only) | | Directory catalog (Resources: hosts, apps, sites) | Existing catalog change events | master (write) → spokes (read-only) | | Audit log | Async batch worker, already speced (§6) | spokes → master | ### 2.2 Replication Delivery: Fire-and-Forget Master is the sole writer (§3), so there is exactly one producer per data type — no conflict resolution, no consensus, no vector clocks needed. On every write, master pushes the change to all connected spokes **concurrently** (not sequentially — spokes are independent WG peers, none blocks on another) and does **not** wait for acks. A spoke that's offline queues nothing on the master's side; on reconnect, the spoke pulls (or master replays) missed versions. This is a deliberate choice over "wait for all spokes to ack": with a dozen spokes, concurrent push completes in low hundreds of milliseconds on the happy path, but *waiting* for acks makes every write's latency bounded by the slowest/offline spoke — reintroducing the split-brain-adjacent stall that §3's explicit-promotion design exists to avoid. Never make a master write block on spoke reachability. --- ## 3. Explicit Master Control & Human `god_admin` Authority Automatic failover across WAN is explicitly disabled — 0% split-brain risk by design: ``` WAN OUTAGE DETECTED │ ▼ Spoke Node Unconditionally Retains SPOKE Mode │ ▼ Requires Human god_admin Promotion Action ``` 1. **Unreachable master**: a spoke that loses the master unconditionally stays a spoke. No auto-election. 2. **Promotion is a single coordinated action, not two steps**: `POST /api/directory-admin/site-promote` (god_admin-gated) calls out to the *current* master over the WG tunnel and demotes it as part of the same operation — there's never a window with two masters. (Requires the old master to be reachable; if it isn't, that's an operator-visible failure to resolve manually, not a silent partial-promotion.) 3. Because every directory is identical (§2), promotion carries **no agent re-keying cost** — this was the main risk in earlier drafts of this design and is now moot. 4. Site state (name, slug, `isMaster`, `masterUrl`, `wanConnected`) lives on the site's own `kind:'site'` Resource (`metadata.multiSite`), not in server memory — it must survive restarts and be visible via the same directory API as everything else. --- ## 4. `spoke.env` vs `setup.env` A spoke shares almost none of `setup.env`'s concerns (it doesn't mint LDAP admin/JWT/service-account secrets — those arrive via replication, §2) so it gets its own, much shorter file: ``` CFG_DOMAIN=theta42.com # REQUIRED, must match the master's exactly — this is the shared LDAP base DN (dc=theta42,dc=com). Never per-site. CFG_SITE_NAME=staten-island # this site's name/slug CFG_SPOKE_INBOUND=false # true: this site has a public IP and serves its own traffic directly (standalone-style). false: no inbound path exists; master relays (§5). CFG_PUBLIC_DOMAIN= # only used when CFG_SPOKE_INBOUND=true — this site's own domain, own DNS, own ACME cert, independent of the master's domain. CFG_JOIN_TOKEN= # one-time token from the master, used for WG mesh auto-registration (§4.1) and initial catalog/secret pull. CFG_MASTER_ENDPOINT= # master's WG endpoint (host:port) to join through. ``` `CFG_DOMAIN` is the identity namespace (LDAP DN) and must be identical across every site — MMR replicas cannot diverge on base DN. `CFG_PUBLIC_DOMAIN` is a *web-hostname* concern, unrelated to LDAP, and only exists at all for inbound spokes. ### 4.1 WireGuard Mesh Auto-Registration 1. A new `theta-gateway` boots with `CFG_JOIN_TOKEN` + `CFG_MASTER_ENDPOINT`, generates its Curve25519 keypair, and calls `POST /api/mesh/gateway/register` on the master over an initial bootstrap tunnel. 2. Master assigns the next free **site index** (one octet, used identically in both `172.24..0/16` and `10..0.0/16` per the reference topology in Appendix A) and returns full mesh peer config. 3. **Site index ceiling is 254** (0 and 255 excluded) — a hard technical limit of this addressing scheme, not an arbitrary cap. Real deployments target a dozen or fewer; no need to cap lower than the real ceiling. 4. Each `theta-gateway` applies the new peer set to its running `wg0` via `wgctrl` without dropping existing connections. --- ## 5. Inbound vs. No-Inbound Spokes Whether a spoke has a public IP determines everything about how its traffic reaches the outside world — these are two distinct, documented operating modes, not a single universal mechanism. ### 5.1 Inbound Spoke (`CFG_SPOKE_INBOUND=true`) Behaves like a standalone install. Own `CFG_PUBLIC_DOMAIN`, own DNS pointed at its own public IP, own ACME cert. `theta-proxy` and `theta-gateway` serve public web + SSH traffic directly — no relay involved. The only WAN-facing traffic to the master is replication (§2) and audit shipping (§6). ### 5.2 No-Inbound Spoke (`CFG_SPOKE_INBOUND=false`) No public IP exists, so *any* external access must go through the master: 1. Master mints a public hostname for the spoke's services (e.g. `sso-{slug}.{master's public domain}`) and creates the corresponding `theta-proxy` route (already dynamic/DB-backed — `proxy/nodejs/models/host.js` — no new plumbing needed there). 2. Master **terminates TLS** for that hostname and relays to the spoke over the WG tunnel — both `theta-proxy` (any site-hosted web app) and `theta-gateway` (SSH jump) traffic relay this way, not just SSO. 3. Terminating at the master (rather than SNI passthrough) is fine here specifically because master↔spoke already rides an encrypted WG tunnel — there's no unencrypted hop being introduced. ### 5.3 Local-Direct Resolution (Skip the Relay On-LAN) A client physically on a no-inbound spoke's LAN would otherwise hairpin out to the master and back to reach its own local site. Solved via **mDNS local-service-discovery**, not directory-side network topology: 1. The spoke's `theta-gateway`/`theta-proxy` announces itself on the local segment via mDNS (`_theta-suite._tcp.local`, TXT records: site slug, public hostnames it fronts, local IP). 2. `theta-agent`, when a config flag (`preferLocalDiscoveredDirectory` or similar — see the agent-side spec, Appendix B) is enabled, listens for this announcement and overrides local resolution for matching hostnames to the discovered local IP. 3. No match (off-site, or flag disabled) → normal public DNS → master relay. Multicast is link-local by nature, so "on-site or not" needs no explicit detection logic — presence/absence of the announcement *is* the signal. This also solves roaming-admin access (§ formerly "5", folded in here) for free: same laptop, same flag, local-fast-path at the office and relay-path everywhere else. 4. **Hard rule**: mDNS is unauthenticated on a LAN. It may only ever change *where* the agent connects, never *whether* it trusts what answers — TLS/hostname validation against the redirected IP must stay intact, so a spoofed rogue announcement produces a TLS failure, not a silent MITM. This piece needs Windows/Mac-specific implementation and testing that can't be done from this (Linux) environment — see Appendix B for the standalone spec handed off for that work. --- ## 6. Non-Canonical Audit Logging Unchanged from prior draft: OAuth logins, SSH session events, proxy access, and agent execution events write to local site audit tables without blocking on WAN. An async worker flushes batches to master via `POST /api/directory-admin/audit/ingest` when reachable. --- ## Appendix A: Production Reference WireGuard Topology Config ### Site 10.2 (Staten Island LAN Node) Gateway Reference (`wg0.conf`) ```ini [Interface] Address = 172.24.0.2/32 PrivateKey = ListenPort = 51820 Table = off # Mesh Subnet Routes PostUp = ip route add 10.0.0.0/8 dev %i PostUp = ip route add 172.24.0.0/13 dev %i # Policy Routing Exits PostUp = ip route add default via 10.5.0.1 dev %i table offshore PostUp = ip route add default via 172.24.0.1 dev %i table us_vps PostUp = ip rule add from 10.2.254.0/24 lookup offshore PostUp = ip rule add from 10.2.253.0/24 lookup main preference 1000 # NETMAP Shadow Network (10.2.168.x -> 192.168.1.x) PostUp = iptables -t nat -A PREROUTING -i %i -d 10.2.168.0/24 -j NETMAP --to 192.168.1.0/24 PostUp = iptables -t nat -A POSTROUTING -o %i -s 192.168.1.0/24 -j NETMAP --to 10.2.168.0/24 PostUp = ip route add local 10.2.168.0/24 dev lo # Forwarding & NAT PostUp = iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o %i -j MASQUERADE PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE PostUp = iptables -A FORWARD -i %i -o eth0 -j ACCEPT PostUp = iptables -A FORWARD -i eth0 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT # System Kernel Options PostUp = sysctl -w net.ipv4.ip_forward=1 PostUp = sysctl -w net.ipv4.conf.all.rp_filter=0 PostUp = sysctl -w net.ipv4.conf.eth0.rp_filter=0 PostUp = sysctl -w net.ipv4.conf.%i.rp_filter=0 # --- PEERS --- [Peer] # Site 10.1: US Hub / VPS Exit PublicKey = QZCvR3N1CdUabC2xWfc1lmYKHfSiXYs1UoVINIMftws= Endpoint = gg-si1.wgnode.com:51820 AllowedIPs = 172.24.0.0/16, 10.0.0.0/8, 0.0.0.0/0 PersistentKeepalive = 25 [Peer] # Site 10.5: Netherlands Offshore Exit Node PublicKey = MlF6h3YI1MIvOlgyNozCMoa/rICoLNtc7r/pseKiHQQ= Endpoint = nl-alexhost.wgnode.com:51871 AllowedIPs = 172.24.0.5/32, 10.5.0.0/16, 0.0.0.0/0 PersistentKeepalive = 25 ``` ### Site 10.5 (Netherlands Exit Node) Gateway Reference (`wg0.conf`) ```ini [Interface] Address = 172.24.0.5/32 PrivateKey = ListenPort = 51871 PostUp = ip addr add 10.5.0.1/16 dev %i PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE # Dynamic Return Path Masquerading (SOURCENAT) PostUp = iptables -t nat -A POSTROUTING -o %i ! -s 172.24.0.0/13 -j MASQUERADE PostUp = sysctl -w net.ipv4.ip_forward=1 [Peer] # Site 10.2: Staten Island LAN PublicKey = AsS7aikCUrXpdfSvwFnMs0yUaoQ7ZCkoUVOmNdl7NS8= AllowedIPs = 172.24.0.2/32, 10.2.0.0/16 [Peer] # Site 10.1: US Hub VPS PublicKey = QZCvR3N1CdUabC2xWfc1lmYKHfSiXYs1UoVINIMftws= AllowedIPs = 172.24.0.1/32, 10.1.0.0/8 ``` --- ## Appendix B: Agent-Side Work See [`AGENT_LOCAL_DISCOVERY_SPEC.md`](./AGENT_LOCAL_DISCOVERY_SPEC.md) — split out because it needs Windows/Mac implementation and testing that a Linux-only dev environment cannot meaningfully do. That doc is the handoff: it specifies behavior precisely enough to implement and test independently, without needing to re-derive the reasoning in this file. --- ## Status of This Spec vs. Code (as of this revision) | Piece | Status | |---|---| | Site role persisted (not in-memory) | **Shipped** — `/config/site.json` on `sso-manager-node`, survives restarts (v2.2.0) | | Join key issuance + one-time directory adoption | **Shipped** — `/api/site/join-keys`, `/api/site/export`, `/api/site/join`, fresh-install-gated (v2.2.0–v2.3.0) | | Spoke read-only enforcement | **Shipped** — directory-write routes 403 toward the master once joined (v2.3.0) | | WAN health check | **Shipped** — `/api/site/ping`, live in the Master Site modal (v2.2.0–v2.3.0) | | `setup.env` / `setup.sh` join wiring | **Shipped** — `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY`, `bootstrap/site-join.js` (theta-suite v2.2.0) | | Continuous/live replication (vs. one-time export-on-join) | Not built — today's join is a snapshot; a site that drifts after joining doesn't re-sync | | WireGuard site-to-site mesh (real tunnels, not just HTTPS) | Not built — join happens over whatever network path already reaches the master's HTTPS API | | Identical-directory signing key / OpenBao secret replication | Not built | | No-inbound-spoke relay (master proxies a spoke with no public IP) | Not built — today's join requires the spoke to reach the master's API, and vice versa for export; a spoke with zero inbound *and* zero outbound path to the master can't join at all yet | | mDNS local-discovery | Not built — speced for handoff, see Appendix B (still applicable regardless of which replication mechanism eventually lands) | *Committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*