From 0367c335422e8f56c04a2bef51b0e0c9cc49da02 Mon Sep 17 00:00:00 2001 From: William Mantly Date: Mon, 10 Aug 2026 15:38:39 -0400 Subject: [PATCH] docs(multi-site): reconcile spec with shipped v1, add mDNS agent handoff spec MULTI_SITE_SPEC.md described a WireGuard-mesh + live-replication design as if unbuilt-but-planned; meanwhile theta-directory v2.2.0-v2.3.0 (rolled up in theta-suite v2.2.0) already shipped a simpler, real join mechanism (one-time LDIF/catalog export over a site join key, read-only spoke enforcement, setup.sh wiring) that this doc didn't mention at all. Added a callout pointing at docs/site-join.md as the actual current behavior, and corrected the status table so it no longer implies unbuilt features are implemented. Also adds AGENT_LOCAL_DISCOVERY_SPEC.md, a standalone handoff spec for the mDNS "prefer local discovered directory" optimization -- confirmed not implemented anywhere in theta-agent. Needs Windows/Mac-native investigation this environment can't do; written so it can be picked up independently. --- docs/AGENT_LOCAL_DISCOVERY_SPEC.md | 53 +++++++ docs/MULTI_SITE_SPEC.md | 214 +++++++++++++++++------------ 2 files changed, 180 insertions(+), 87 deletions(-) create mode 100644 docs/AGENT_LOCAL_DISCOVERY_SPEC.md diff --git a/docs/AGENT_LOCAL_DISCOVERY_SPEC.md b/docs/AGENT_LOCAL_DISCOVERY_SPEC.md new file mode 100644 index 0000000..82942fd --- /dev/null +++ b/docs/AGENT_LOCAL_DISCOVERY_SPEC.md @@ -0,0 +1,53 @@ +# theta-agent: Local-Discovery Spec (mDNS "prefer local directory") + +**Audience**: implementer on Windows/Mac (this was authored on Linux; Windows/Mac-specific network and hosts-file behavior needs to be built and tested there, not assumed here). +**Repo**: `theta-agent` (Go). Signing/config mechanism referenced below: `websocket.go`, `config.go`. +**Parent doc**: [`MULTI_SITE_SPEC.md`](./MULTI_SITE_SPEC.md) §5.3 — read that section for the "why," this doc is the "what," precisely enough to implement without re-deriving the reasoning. + +## Problem + +A no-inbound spoke site's public hostname (e.g. `sso-staten-island.theta42.com`) resolves, from the internet, to the master's IP, which relays over WireGuard to the spoke. A device physically on that spoke's LAN resolving the same hostname takes the same path — out to the master, back over the tunnel — even though the real service is a few feet away. This is wasted latency, not a correctness bug, but it's the kind of thing users notice. + +## What to Build + +### 1. Announcer (gateway/proxy side — may already be scoped elsewhere, confirm before duplicating) +The spoke's `theta-gateway` or `theta-proxy` periodically advertises itself via mDNS on the local segment: +- Service type: `_theta-suite._tcp.local` +- TXT records: `site=`, `hosts=` +- Advertised address: the service's own local LAN IP + +### 2. Listener + Override (this doc's actual scope — theta-agent) +- New config field in `agent.yml`, e.g. `prefer_local_discovered_directory: bool` (default `false` — opt-in, not automatic, since it changes name resolution behavior on the host). +- When `true`, the agent runs an mDNS browser for `_theta-suite._tcp.local` in the background. +- On receiving an announcement whose `hosts` TXT list includes a hostname the agent cares about (at minimum: the hostname the agent itself is currently configured to connect to for its WS connection), the agent installs a **local override** redirecting that hostname to the discovered local IP. +- No announcement seen (agent off-site, or flag disabled) → no override installed, normal DNS resolution applies. Nothing else about the agent's behavior changes in this case. +- If a previously-discovered site's announcement stops being seen (TTL expiry / agent moved networks), the override must be **removed**, not left stale. Don't let a laptop that left the office keep resolving the old office hostname to a now-unreachable LAN IP. + +### 3. Override Mechanism — Platform-Specific, Needs Real Investigation + +This is the part that most needs Windows/Mac-native work; do not assume the Linux/Unix approach ports directly: + +- **Windows**: hosts file lives at `%SystemRoot%\System32\drivers\etc\hosts`; writing to it requires elevation, and Windows caches DNS results independently (`ipconfig /flushdns` needed after an edit, or the change won't take effect immediately — verify whether theta-agent already runs elevated on Windows, since if it doesn't, this whole approach may need a different mechanism, e.g. a local proxy/resolver instead of hosts-file edits). +- **macOS**: hosts file at `/etc/hosts`, also requires root; macOS's mDNSResponder/DNS caching behavior differs from Windows and Linux (`dscacheutil -flushcache; killall -HUP mDNSResponder` territory) — confirm whether an installed hosts entry is actually honored promptly, or whether the built-in mDNSResponder needs to be told directly instead of fighting it with a hosts-file edit (macOS already *has* native mDNS support baked into resolution — it may be simpler/more idiomatic there to register via the OS's own Bonjour APIs rather than hand-roll hosts-file mutation). +- **Linux**: `/etc/hosts`, requires root, comparatively straightforward, `systemd-resolved` caching considerations may apply depending on distro. + +Given the platform divergence, seriously consider whether a **local stub resolver** (theta-agent listens on `127.0.0.1:`, answers matching hostnames from its own discovery cache, forwards everything else upstream — with the OS's DNS pointed at it only for the duration this feature is active) is actually simpler and more uniform across all three platforms than hosts-file mutation, despite the extra moving part. Recommend evaluating both before committing to an implementation; this doc intentionally doesn't prescribe one, since that call needs platform testing this environment can't do. + +## Hard Security Rule (non-negotiable, applies regardless of mechanism chosen) + +mDNS is **unauthenticated** on a local network — anyone on the same LAN segment can broadcast a spoofed announcement. This feature may only ever change **where** the agent connects (which IP a hostname resolves to). It must **never** change **whether** the agent trusts what answers there. Concretely: +- TLS certificate validation and hostname verification against the redirected IP must remain fully enforced — no exceptions, no "local network so it's fine" carve-out. +- A spoofed rogue announcement pointing a hostname at an attacker's local IP should produce a TLS handshake failure (cert won't match), not a silent connection. If your chosen mechanism has *any* code path where local-discovery bypasses or weakens cert checking, that's a bug, not an optimization — fix it before shipping. + +## Out of Scope for This Piece + +- The announcer side's exact library/implementation on the gateway/proxy (Linux — can be built where this spec was authored, not blocked on Windows/Mac). +- Anything about the master-relay mechanism itself (§5.2 of the parent spec) — this doc is purely the "skip the relay when local" optimization layered on top of it. + +## Definition of Done + +- Flag exists, defaults off. +- Enabling it on a machine physically on a spoke's LAN measurably routes traffic to the local spoke instead of through the master relay (verify via a network capture or the proxy's own access logs on each side, not just "it feels faster"). +- Leaving that LAN (or disabling the flag) reliably reverts to normal resolution — no stale overrides. +- A test with a spoofed/rogue mDNS announcement (a second, non-legitimate advertiser) results in a TLS failure, not a successful connection to the impostor. +- Behavior verified on both Windows and macOS, not just Linux. diff --git a/docs/MULTI_SITE_SPEC.md b/docs/MULTI_SITE_SPEC.md index 4c9d650..7005bc0 100644 --- a/docs/MULTI_SITE_SPEC.md +++ b/docs/MULTI_SITE_SPEC.md @@ -1,17 +1,20 @@ # Theta Suite Multi-Site Architecture & VPN Specification -**Specification Version**: `2.0.0` -**Status**: Final Architecture Specification -**Target Suite Version**: `v1.50.0+` +**Specification Version**: `2.1.0` +**Status**: Target architecture / roadmap. **A simpler v1 is already shipped** — see the box below before reading further. +**Target Suite Version**: `v1.50.0+` **Repository**: [`theta-suite`](https://github.com/theta42/theta-suite) ---- +> ## Shipped today (theta-directory v2.2.0–v2.3.0, theta-suite v2.2.0) +> A spoke joins a master by pulling a **one-time directory export** (LDAP LDIF + resource catalog) over a **site join key**, entirely over the existing HTTPS API — no WireGuard mesh, no live replication, no OpenBao secret sync. It's simpler than everything below and it's real, tested, and released. Read [`sso-manager-node/docs/site-join.md`](https://github.com/theta42/theta-directory/blob/master/docs/site-join.md) first; it documents exactly what exists: +> - `POST /api/site/join-keys`, `/api/site/export`, `/api/site/ping`, `/api/site/join` — mint a key on the master, pull-and-adopt on the spoke. +> - Fresh-install-only (no merging into a populated directory). +> - Spoke is read-only post-join (writes 403 toward the master); role persists in `/config/site.json`, survives restarts. +> - `setup.env`'s `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY` wire it into first-run `setup.sh`. +> +> Everything below this point is the **larger target architecture** this session designed (WireGuard mesh, live fire-and-forget replication, identical-directory signing, no-inbound relay, mDNS local-discovery) — none of it is built, and **none of it is required** for the shipped v1 above to work. Treat it as where multi-site could grow next (continuous sync instead of one-time adoption, true site-to-site networking, spokes with no inbound path at all), not as a description of current behavior. Don't let this document's detail imply more is built than actually is — check the status table at the bottom, or better, check `git log`/the linked doc, before trusting either. -## Executive Summary - -This specification defines the multi-site replication, fault tolerance, autonomous site isolation, and integrated WireGuard mesh network architecture for the `theta-suite` ecosystem. - -The architecture enforces **Deployment Symmetry**, **Explicit Master Control via `god_admin` Authority**, **`theta-gateway` Mesh & SSH Integration**, and **Zero-WAN Dependency for Local Operations**. +Design scale: a handful of sites (dozen max, 254 hard ceiling — see §4), a few hundred users/hosts total. This is a deliberate, small, trusted-operator deployment, not a hyperscale/adversarial-tenant one — several decisions below (fire-and-forget replication, identical directories) trade blast-radius for simplicity *because* the scale allows it. Don't generalize these choices past that scale without re-deriving them. --- @@ -19,131 +22,147 @@ The architecture enforces **Deployment Symmetry**, **Explicit Master Control via ```mermaid flowchart TB - subgraph ControlPlane["Master Site (Site 10.1 - HQ / Control Plane)"] - ssoM["sso-manager-node (Master Read/Write)"] - ldapM["OpenLDAP (MMR Master Node 1)"] - baoM["OpenBao (Master Secrets Engine)"] - proxyM["theta-proxy (HQ Web Gateway)"] - gateM["theta-gateway (HQ SSH & WireGuard Mesh Gate)"] - agentM["theta-agent (HQ Local Agents)"] + subgraph ControlPlane["Master Site (write authority)"] + ssoM["sso-manager-node (isMaster=true)"] + ldapM["OpenLDAP (MMR write node)"] + baoM["OpenBao (local, replication source)"] + proxyM["theta-proxy"] + gateM["theta-gateway"] + agentM["theta-agent"] end - subgraph SiteB["Spoke Site 10.2 (Staten Island LAN)"] - ssoB["sso-manager-node (Spoke Read-Only Catalog)"] - ldapB["OpenLDAP (MMR Node 2 / Read-Only Replica)"] - baoB["OpenBao (Spoke Local Secret Replica)"] - proxyB["theta-proxy (Site 10.2 Local Web Gateway)"] - gateB["theta-gateway (Site 10.2 SSH & WG Mesh Gate)"] - agentB["theta-agent (Site 10.2 Local Agents)"] + subgraph SiteB["Spoke — inbound (has a public IP)"] + ssoB["sso-manager-node (isMaster=false)"] + ldapB["OpenLDAP (MMR read replica)"] + baoB["OpenBao (local replica)"] + proxyB["theta-proxy — serves this site's public traffic directly"] + gateB["theta-gateway"] + agentB["theta-agent"] end - subgraph SiteNL["Spoke Site 10.5 (Netherlands Offshore Exit)"] - ssoNL["sso-manager-node (Spoke Read-Only Catalog)"] - ldapNL["OpenLDAP (MMR Node 5 / Read-Only Replica)"] - baoNL["OpenBao (Spoke Local Secret Replica)"] - proxyNL["theta-proxy (Site 10.5 Local Web Gateway)"] - gateNL["theta-gateway (Site 10.5 Offshore Exit Gate)"] - agentNL["theta-agent (Site 10.5 Local Agents)"] + subgraph SiteC["Spoke — no inbound (CGNAT)"] + ssoC["sso-manager-node (isMaster=false)"] + ldapC["OpenLDAP (MMR read replica)"] + baoC["OpenBao (local replica)"] + proxyC["theta-proxy — LAN-local traffic only"] + gateC["theta-gateway"] + agentC["theta-agent"] end - gateM <==>|"WireGuard Encrypted Mesh Tunnel (172.24.0.x)"| gateB - gateM <==>|"WireGuard Encrypted Mesh Tunnel (172.24.0.x)"| gateNL - gateB <==>|"WireGuard Direct Tunnel"| gateNL + gateM <==>|"WireGuard mesh tunnel"| gateB + gateM <==>|"WireGuard mesh tunnel"| gateC - ssoM <==>|"HTTP SSE Catalog Change Events"| ssoB - ldapM <==>|"OpenLDAP MMR syncrepl (ldaps://636)"| ldapB - baoM -.->|"Secret Version Replicator"| baoB + ssoM -.->|"fire-and-forget push: catalog + secrets + signing key"| ssoB + ssoM -.->|"fire-and-forget push"| ssoC + ldapM <==>|"OpenLDAP MMR syncrepl"| ldapB + ldapM <==>|"OpenLDAP MMR syncrepl"| ldapC + + proxyM -->|"TLS-terminate + relay (no direct path exists)"| gateM + gateM ==>|"WG tunnel"| gateC ``` --- -## 2. Component Core Roles +## 2. Every Directory Is Identical -### 2.1 `theta-gateway` (Merged SSH Jump & WireGuard Mesh Gateway) -The container previously named `jump-host` is officially rebranded and expanded to **`theta-gateway`**. It acts as the single security & routing gateway for each site: -1. **SSH Jump Host (Port 2222)**: Handles interactive terminal jumps, user identity verification via local OpenLDAP, and dynamic SSH key injection. -2. **Site-Aware SSH Target Filtering**: `theta-gateway` filters target host/service selection menus strictly to resources assigned to that local site (`SITE_SLUG`). -3. **WireGuard Mesh Router (Port 51820 / 51871)**: Handles tunnel termination (`wg0`), peer key management, keepalives, and inter-site routing. -4. **NETMAP & Policy Routing Engine**: Applies `iptables` NETMAP shadow subnet translations, dynamic return path masquerading, and policy exit routes (`table offshore`, `table us_vps`). +Master and every spoke run the **same LDAP data, the same OpenBao secrets, and the same agent-signing key**. Hitting any site's `sso-manager-node` for read/auth purposes is equivalent to hitting any other. The only asymmetry is **write authority** (§3). -### 2.2 `sso-manager-node` (Control Plane & Local Issuer) -* **Master Role (`isMaster = true`)**: Holds single write authority for directory resources in `inventory.sqlite`. Processes write requests proxied from Spokes. -* **Spoke Role (`isMaster = false`)**: Runs in Read-Only Catalog mode. Performs local OIDC JWT token issuance for local web apps via local OpenLDAP authentication. +This is a deliberate tradeoff, not a default: it means compromising *any single spoke* — including the smallest, least-secured one — grants an attacker the same agent-command authority (`update_binary`, `arbitrary_bash`, service control) as compromising the master, because every site holds the same Ed25519 signing key (`sso-manager-node/nodejs/utils/agent_keys.js`). Accepted here because the deployment scale is small and trusted. Do not extend this pattern to a larger/adversarial-tenant deployment without revisiting it. -### 2.3 `theta-proxy` (Site-Local Web Gateway) -* **Local Route Scope**: Manages local web application reverse proxying and TLS certificates (ACME / Let's Encrypt / local certs) 100% locally. -* **Zero Locking**: Proxy route definitions and TLS certs are not locked during Master WAN outages. +Consequence: `theta-agent` needs **no change** to support multi-site — it already does TOFU pairing against a single trusted key (`websocket.go:341-351`), and since that key is identical everywhere, any site's `sso-manager-node` can validly sign a command for any agent, anywhere, without agents needing a keyring. -### 2.4 `theta-agent` (Site-Local Agent Hub) -* **Local WS Connection**: Connects to the local site's `sso-manager-node` WebSocket (`wss://sso.site-b.example.com/api/agent/ws`). -* **Local Autonomy**: Real-time telemetry, memory/CPU metrics, disk usage, active logged-in users (`who`), and desktop control commands (lock, display off, logout, reboot) operate 100% locally. +### 2.1 What Replicates, and How + +| Data | Mechanism | Direction | +|---|---|---| +| LDAP (users, groups) | OpenLDAP MMR syncrepl | master (write) → spokes (read-only) | +| OpenBao secrets (incl. agent-signing key at `secret/agent/signing-key`) | **New**: custom replicator (OpenBao has no built-in multi-site replication — Performance Replication is Vault-Enterprise-only, confirmed absent from OpenBao as of this writing) | master (write) → spokes (read-only) | +| Directory catalog (Resources: hosts, apps, sites) | Existing catalog change events | master (write) → spokes (read-only) | +| Audit log | Async batch worker, already speced (§6) | spokes → master | + +### 2.2 Replication Delivery: Fire-and-Forget + +Master is the sole writer (§3), so there is exactly one producer per data type — no conflict resolution, no consensus, no vector clocks needed. On every write, master pushes the change to all connected spokes **concurrently** (not sequentially — spokes are independent WG peers, none blocks on another) and does **not** wait for acks. A spoke that's offline queues nothing on the master's side; on reconnect, the spoke pulls (or master replays) missed versions. + +This is a deliberate choice over "wait for all spokes to ack": with a dozen spokes, concurrent push completes in low hundreds of milliseconds on the happy path, but *waiting* for acks makes every write's latency bounded by the slowest/offline spoke — reintroducing the split-brain-adjacent stall that §3's explicit-promotion design exists to avoid. Never make a master write block on spoke reachability. --- ## 3. Explicit Master Control & Human `god_admin` Authority -To guarantee **0% split-brain risk**, automatic failover across WAN is explicitly disabled: +Automatic failover across WAN is explicitly disabled — 0% split-brain risk by design: ``` WAN OUTAGE DETECTED │ ▼ Spoke Node Unconditionally Retains SPOKE Mode - (Read-Only Catalog / Full Local Operations) │ ▼ Requires Human god_admin Promotion Action - (Explicit Confirmation Modal in SSO UI / CLI) ``` -1. **Unreachable Master Behavior**: If a Spoke node loses WAN connection to the Master, it **unconditionally remains in Spoke Mode**. -2. **Human Re-assignment**: Changing or promoting a Master node requires an explicit action by an authenticated **`god_admin`** user via the SSO UI or `theta-suite-admin promote-master` CLI. +1. **Unreachable master**: a spoke that loses the master unconditionally stays a spoke. No auto-election. +2. **Promotion is a single coordinated action, not two steps**: `POST /api/directory-admin/site-promote` (god_admin-gated) calls out to the *current* master over the WG tunnel and demotes it as part of the same operation — there's never a window with two masters. (Requires the old master to be reachable; if it isn't, that's an operator-visible failure to resolve manually, not a silent partial-promotion.) +3. Because every directory is identical (§2), promotion carries **no agent re-keying cost** — this was the main risk in earlier drafts of this design and is now moot. +4. Site state (name, slug, `isMaster`, `masterUrl`, `wanConnected`) lives on the site's own `kind:'site'` Resource (`metadata.multiSite`), not in server memory — it must survive restarts and be visible via the same directory API as everything else. --- -## 4. Integrated WireGuard Mesh & Key Provisioning Protocol +## 4. `spoke.env` vs `setup.env` -### 4.1 Automatic Key Exchange & Peer Discovery -1. When a new `theta-gateway` boots or joins a site via Join Key: - - It generates a Curve25519 keypair and registers its public key, listen port, and public endpoint with `sso-manager-node` (`POST /api/mesh/gateway/register`). -2. `sso-manager-node` calculates the site index (`Site 10.x`), assigns mesh IPs (`172.24.0.x`), and broadcasts updated peer definitions to all active `theta-gateway` instances. -3. Each `theta-gateway` updates its running WireGuard interface (`wg0`) dynamically via `wgctrl` / `iptables` without dropping existing connections. +A spoke shares almost none of `setup.env`'s concerns (it doesn't mint LDAP admin/JWT/service-account secrets — those arrive via replication, §2) so it gets its own, much shorter file: -### 4.2 NETMAP Shadow Subnet Addressing (`10..168.0/24`) -To prevent IP collisions when multiple sites use default `192.168.1.0/24` physical LANs, `theta-gateway` automatically enables **NETMAP Shadow Subnets by default**: - -```bash -# NETMAP: Shadow network (10..168.x) to physical LAN (192.168.1.x) -PostUp = iptables -t nat -A PREROUTING -i wg0 -d 10..168.0/24 -j NETMAP --to 192.168.1.0/24 -PostUp = iptables -t nat -A POSTROUTING -o wg0 -s 192.168.1.0/24 -j NETMAP --to 10..168.0/24 -PostUp = ip route add local 10..168.0/24 dev lo +``` +CFG_DOMAIN=theta42.com # REQUIRED, must match the master's exactly — this is the shared LDAP base DN (dc=theta42,dc=com). Never per-site. +CFG_SITE_NAME=staten-island # this site's name/slug +CFG_SPOKE_INBOUND=false # true: this site has a public IP and serves its own traffic directly (standalone-style). false: no inbound path exists; master relays (§5). +CFG_PUBLIC_DOMAIN= # only used when CFG_SPOKE_INBOUND=true — this site's own domain, own DNS, own ACME cert, independent of the master's domain. +CFG_JOIN_TOKEN= # one-time token from the master, used for WG mesh auto-registration (§4.1) and initial catalog/secret pull. +CFG_MASTER_ENDPOINT= # master's WG endpoint (host:port) to join through. ``` -* **Effect**: A server at Site 10.2 (`192.168.1.50`) can reach a server at Site 10.4 (`192.168.1.50`) by pinging `10.4.168.50`. Neither site needs to modify router DHCP or local subnets! +`CFG_DOMAIN` is the identity namespace (LDAP DN) and must be identical across every site — MMR replicas cannot diverge on base DN. `CFG_PUBLIC_DOMAIN` is a *web-hostname* concern, unrelated to LDAP, and only exists at all for inbound spokes. -### 4.3 Policy Exit Routing & Return Path SOURCENAT -* **Custom Route Tables**: `theta-gateway` supports selective outbound exit tables (`table offshore`, `table us_vps`) based on IP ranges (`ip rule add from 10.x.254.0/24 lookup offshore`). -* **Dynamic Return Path SOURCENAT**: Exit nodes (e.g. Netherlands `10.5`) apply source-NAT masquerading for incoming tunnel traffic to guarantee symmetric return path routing: - ```bash - PostUp = iptables -t nat -A POSTROUTING -o wg0 ! -s 172.24.0.0/13 -j MASQUERADE - ``` +### 4.1 WireGuard Mesh Auto-Registration + +1. A new `theta-gateway` boots with `CFG_JOIN_TOKEN` + `CFG_MASTER_ENDPOINT`, generates its Curve25519 keypair, and calls `POST /api/mesh/gateway/register` on the master over an initial bootstrap tunnel. +2. Master assigns the next free **site index** (one octet, used identically in both `172.24..0/16` and `10..0.0/16` per the reference topology in Appendix A) and returns full mesh peer config. +3. **Site index ceiling is 254** (0 and 255 excluded) — a hard technical limit of this addressing scheme, not an arbitrary cap. Real deployments target a dozen or fewer; no need to cap lower than the real ceiling. +4. Each `theta-gateway` applies the new peer set to its running `wg0` via `wgctrl` without dropping existing connections. --- -## 5. Roaming Admin Access (QR Codes & Optional Tailscale) +## 5. Inbound vs. No-Inbound Spokes -1. **Native WireGuard Client Profiles**: - - `sso-manager-node` includes a built-in **Client Profile Generator** in the UI. - - Admins can generate a mobile/laptop profile, displaying a **QR code** for immediate scan into the official WireGuard app on iOS/Android or a downloadable `wg0.conf` for laptops. -2. **Optional Tailscale / Headscale Connector**: - - For roaming devices in environments where WireGuard UDP ports are blocked, `theta-gateway` supports an optional `tailscale` / `headscale` sidecar integration. +Whether a spoke has a public IP determines everything about how its traffic reaches the outside world — these are two distinct, documented operating modes, not a single universal mechanism. + +### 5.1 Inbound Spoke (`CFG_SPOKE_INBOUND=true`) +Behaves like a standalone install. Own `CFG_PUBLIC_DOMAIN`, own DNS pointed at its own public IP, own ACME cert. `theta-proxy` and `theta-gateway` serve public web + SSH traffic directly — no relay involved. The only WAN-facing traffic to the master is replication (§2) and audit shipping (§6). + +### 5.2 No-Inbound Spoke (`CFG_SPOKE_INBOUND=false`) +No public IP exists, so *any* external access must go through the master: + +1. Master mints a public hostname for the spoke's services (e.g. `sso-{slug}.{master's public domain}`) and creates the corresponding `theta-proxy` route (already dynamic/DB-backed — `proxy/nodejs/models/host.js` — no new plumbing needed there). +2. Master **terminates TLS** for that hostname and relays to the spoke over the WG tunnel — both `theta-proxy` (any site-hosted web app) and `theta-gateway` (SSH jump) traffic relay this way, not just SSO. +3. Terminating at the master (rather than SNI passthrough) is fine here specifically because master↔spoke already rides an encrypted WG tunnel — there's no unencrypted hop being introduced. + +### 5.3 Local-Direct Resolution (Skip the Relay On-LAN) + +A client physically on a no-inbound spoke's LAN would otherwise hairpin out to the master and back to reach its own local site. Solved via **mDNS local-service-discovery**, not directory-side network topology: + +1. The spoke's `theta-gateway`/`theta-proxy` announces itself on the local segment via mDNS (`_theta-suite._tcp.local`, TXT records: site slug, public hostnames it fronts, local IP). +2. `theta-agent`, when a config flag (`preferLocalDiscoveredDirectory` or similar — see the agent-side spec, Appendix B) is enabled, listens for this announcement and overrides local resolution for matching hostnames to the discovered local IP. +3. No match (off-site, or flag disabled) → normal public DNS → master relay. Multicast is link-local by nature, so "on-site or not" needs no explicit detection logic — presence/absence of the announcement *is* the signal. This also solves roaming-admin access (§ formerly "5", folded in here) for free: same laptop, same flag, local-fast-path at the office and relay-path everywhere else. +4. **Hard rule**: mDNS is unauthenticated on a LAN. It may only ever change *where* the agent connects, never *whether* it trusts what answers — TLS/hostname validation against the redirected IP must stay intact, so a spoofed rogue announcement produces a TLS failure, not a silent MITM. + +This piece needs Windows/Mac-specific implementation and testing that can't be done from this (Linux) environment — see Appendix B for the standalone spec handed off for that work. --- ## 6. Non-Canonical Audit Logging -* **Local Activity Storage**: OAuth logins, SSH session events, proxy access logs, and agent execution events write to local site audit tables without blocking local operations. -* **Asynchronous Log Worker**: A background worker flushes log batches to Master via `POST /api/directory-admin/audit/ingest` when WAN is online. +Unchanged from prior draft: OAuth logins, SSH session events, proxy access, and agent execution events write to local site audit tables without blocking on WAN. An async worker flushes batches to master via `POST /api/directory-admin/audit/ingest` when reachable. --- @@ -226,4 +245,25 @@ AllowedIPs = 172.24.0.1/32, 10.1.0.0/8 --- -*Final Architecture Specification committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).* +## Appendix B: Agent-Side Work + +See [`AGENT_LOCAL_DISCOVERY_SPEC.md`](./AGENT_LOCAL_DISCOVERY_SPEC.md) — split out because it needs Windows/Mac implementation and testing that a Linux-only dev environment cannot meaningfully do. That doc is the handoff: it specifies behavior precisely enough to implement and test independently, without needing to re-derive the reasoning in this file. + +--- + +## Status of This Spec vs. Code (as of this revision) + +| Piece | Status | +|---|---| +| Site role persisted (not in-memory) | **Shipped** — `/config/site.json` on `sso-manager-node`, survives restarts (v2.2.0) | +| Join key issuance + one-time directory adoption | **Shipped** — `/api/site/join-keys`, `/api/site/export`, `/api/site/join`, fresh-install-gated (v2.2.0–v2.3.0) | +| Spoke read-only enforcement | **Shipped** — directory-write routes 403 toward the master once joined (v2.3.0) | +| WAN health check | **Shipped** — `/api/site/ping`, live in the Master Site modal (v2.2.0–v2.3.0) | +| `setup.env` / `setup.sh` join wiring | **Shipped** — `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY`, `bootstrap/site-join.js` (theta-suite v2.2.0) | +| Continuous/live replication (vs. one-time export-on-join) | Not built — today's join is a snapshot; a site that drifts after joining doesn't re-sync | +| WireGuard site-to-site mesh (real tunnels, not just HTTPS) | Not built — join happens over whatever network path already reaches the master's HTTPS API | +| Identical-directory signing key / OpenBao secret replication | Not built | +| No-inbound-spoke relay (master proxies a spoke with no public IP) | Not built — today's join requires the spoke to reach the master's API, and vice versa for export; a spoke with zero inbound *and* zero outbound path to the master can't join at all yet | +| mDNS local-discovery | Not built — speced for handoff, see Appendix B (still applicable regardless of which replication mechanism eventually lands) | + +*Committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*