Merge pull request #204 from theta42/release-v2.3.0

release(v2.3.0): live replication + gateway-to-gateway WireGuard mesh

CHANGELOG.md

@@ -9,6 +9,46 @@ orchestration code; see each submodule's own `CHANGELOG.md`
 [jump-host](https://github.com/theta42/jump-host/blob/master/CHANGELOG.md))
 for what changed inside the apps it composes.
 
+## [v2.3.0] - 2026-08-10
+
+Rolls up **theta-directory v2.4.0**, **jump-host v2.1.0**, **theta-agent v2.1.2**. Live catalog replication and a real gateway-to-gateway WireGuard mesh land in the same pass — multi-site directory sync stops being a one-time snapshot, and site-to-site networking becomes real infrastructure instead of a documented-but-unbuilt design. See `docs/MULTI_SITE_SPEC.md` for the full architecture and an explicit TODO list of what's still open (Windows/macOS mDNS, routing directory traffic over the mesh, `theta-proxy` no-inbound relay automation).
+
+### theta-directory v2.4.0
+
+#### Added
+- **Live catalog replication.** A spoke now stays in sync after joining instead of only getting a one-time snapshot: it registers its own endpoint with the master at join time (`POST /api/site/spokes`, Bearer the site join key), and every successful master catalog write fires a fire-and-forget push (`utils/site_replicate.js`) at every registered spoke, concurrently — one unreachable spoke never blocks or delays delivery to another. The spoke's `POST /api/site/resync` handler re-runs the same tested export-pull-and-import path used at join time rather than applying a partial diff.
+- **Identical-directory agent-signing key.** `POST /api/site/export` now best-effort includes the master's agent-signing key; a spoke adopts it via `agent_keys.adopt()` on both join and every resync, so any site's `sso-manager-node` can validly sign a command for any agent enrolled at any other site (a deliberate blast-radius tradeoff for this deployment's small, trusted scale — see `docs/MULTI_SITE_SPEC.md` §2).
+- **Coordinated master promotion.** `POST /api/directory-admin/site-promote` now demotes the previous master as part of the same action (mints it a fresh join key, calls its new `POST /api/site/demote`) instead of leaving a manual two-step gap where two nodes could both believe they're master. Best-effort: an unreachable old master never blocks the local promotion — the response's `handoff` field reports what happened.
+- **Master Site modal UI**: new "Live Replication" (spoke) / "Registered Spokes" (master) status rows; the join form gained a "this site's own reachable URL" field wired to `selfUrl`, which the join API already supported but the UI never sent; the promote button's success toast now reports the actual handoff result.
+
+#### Fixed
+- **`site-promote`'s god_admin check was dead on arrival** — it read `req.user.groups`, a field nothing in the codebase ever populates, so the check silently evaluated to an empty array on every request. Promotion returned 403 for every user, including a real god_admin, since it shipped in v2.0.0. Only surfaced by live two-container testing, not by inspection.
+- **The read-only write-gate blocked `site-promote` on a spoke** before its handler could run — the one mutating request a spoke must be able to make to itself.
+- **`GET /api/site/config` was returning live credentials** (`masterJoinKey`, `replicationPushToken`) directly to the browser on every admin session. Replaced with boolean derivatives.
+
+### jump-host v2.1.0
+
+#### Added
+- **Gateway-to-gateway WireGuard mesh** (`routes/mesh.js`) — real site-to-site tunnels between theta-gateway instances, distinct from the existing roaming-client/exit-node WireGuard feature. Join-token bootstrap, mesh-index addressing (172.24.\<idx\>.0/16 + 10.\<idx\>.0.0/16).
+- **In-kernel WireGuard with a userspace fallback** (`utils/wg_iface.js`) — prefers `ip link add type wireguard`, falls back to `wireguard-go` when the kernel module isn't available.
+- **mDNS local-discovery announcer** (`services/mdns_announce.js`) — advertises which public hostnames this site fronts so a `theta-agent` on the same LAN segment can skip the relay/WAN path.
+- **Mesh UI** (`/mesh`) — gateway identity, join-token minting, remote-join form, meshed-gateways table.
+
+Verified with real two-container tests: an actual encrypted WireGuard tunnel passing ICMP traffic end to end (0% loss), and the mDNS announce/discover/apply/revert cycle over real multicast. Two real bugs found and fixed: `wg set ... allowed-ips` doesn't add a kernel route (a real handshake completed with zero routing until `setPeer()` was fixed to add it); mDNS's default IPv6 query aborting the entire lookup after a valid IPv4 response had already arrived.
+
+### theta-agent v2.1.2
+
+#### Added
+- **Linux mDNS local-discovery** (`local_discovery.go`, `hosts_override.go`) — opt-in via `prefer_local_directory`; skips the relay/WAN path when a local `theta-gateway`/`theta-proxy` announces it fronts this agent's `server_url` host. Never touches TLS/certificate validation — only changes where the agent connects, never whether it trusts what answers.
+
+#### Fixed (found via live two-container testing over real multicast)
+- `mdns.Lookup()`'s default IPv6 query aborted the whole lookup — discarding an already-valid IPv4 response — when IPv6 wasn't available. Fixed by disabling IPv6 querying explicitly.
+- Hosts-file writes used write-tmp-then-rename; `/etc/hosts` is frequently a bind mount (every container runtime does this) and `rename()` onto one fails with `EBUSY`. Switched to truncate-and-rewrite in place.
+
+Windows/macOS local-discovery remain unbuilt — see `docs/AGENT_LOCAL_DISCOVERY_SPEC.md`.
+
+Also backfills v2.1.0/v2.1.1 changelog entries (Windows agent, WireGuard client, installer, CI) in theta-agent's own CHANGELOG.md, which were tagged and released earlier but never documented there.
+
 ## [v2.2.0] - 2026-08-10
 
 Multi-site join is now end-to-end: theta-directory can adopt an existing

bootstrap/site-join.js

@@ -5,7 +5,13 @@
  * setup.env sets CFG_MASTER_DIRECTORY_URL + CFG_MASTER_DIRECTORY_JOIN_KEY:
  *
  *   docker compose exec sso-manager node /bootstrap/site-join.js \
- *       https://sso.master.example.com stj_9f2e...
+ *       https://sso.master.example.com stj_9f2e... https://sso.this-site.example.com
+ *
+ * The third argument (selfUrl, optional) is this site's own public SSO host
+ * (setup.sh passes https://$CFG_SSO_HOST) -- without it the join still
+ * succeeds, it just registers for one-time adoption only: the master has no
+ * way to reach this spoke to push live replication resync pings at it (see
+ * theta-directory's docs/site-join.md and utils/site_replicate.js).
  *
  * Self-contained (Node built-ins + global fetch), same rule as bootstrap.js —
  * it does NOT require the SSO's internal models. It logs in as the bootstrap
@@ -26,6 +32,7 @@ const SSO_INTERNAL    = 'http://localhost:3001';
 
 const masterUrl = process.argv[2];
 const joinKey   = process.argv[3];
+const selfUrl   = process.argv[4] || '';
 
 function log(msg) { console.error('[site-join] ' + msg); }
 
@@ -52,7 +59,7 @@ async function main() {
   const res = await fetch(`${SSO_INTERNAL}/api/site/join`, {
     method: 'POST',
     headers: { 'auth-token': token, 'Content-Type': 'application/json' },
-    body: JSON.stringify({ masterUrl, joinKey }),
+    body: JSON.stringify({ masterUrl, joinKey, ...(selfUrl ? { selfUrl } : {}) }),
   });
   const text = await res.text().catch(() => '');
   let data = null;
@@ -68,11 +75,13 @@ async function main() {
   }
 
   log(`Joined master site ${masterUrl} as ${data.siteSlug || '?'}`);
+  log(`Live replication: ${(data.replication && data.replication.note) || 'unknown'}`);
   console.log([
     `JOINED=yes`,
     `SITE_SLUG=${data.siteSlug || ''}`,
     `RESOURCES=${(data.resources && data.resources.created) || 0}`,
-    `LDAP=${(data.ldap && data.ldap.note) || ''}`
+    `LDAP=${(data.ldap && data.ldap.note) || ''}`,
+    `LIVE_REPLICATION=${(data.replication && data.replication.live) ? 'yes' : 'no'}`
   ].join(' '));
 }
 

docs/AGENT_LOCAL_DISCOVERY_SPEC.md

@@ -0,0 +1,53 @@
+# theta-agent: Local-Discovery Spec (mDNS "prefer local directory")
+
+**Audience**: implementer on Windows/Mac (this was authored on Linux; Windows/Mac-specific network and hosts-file behavior needs to be built and tested there, not assumed here).
+**Repo**: `theta-agent` (Go). Signing/config mechanism referenced below: `websocket.go`, `config.go`.
+**Parent doc**: [`MULTI_SITE_SPEC.md`](./MULTI_SITE_SPEC.md) §5.3 — read that section for the "why," this doc is the "what," precisely enough to implement without re-deriving the reasoning.
+
+## Problem
+
+A no-inbound spoke site's public hostname (e.g. `sso-staten-island.theta42.com`) resolves, from the internet, to the master's IP, which relays over WireGuard to the spoke. A device physically on that spoke's LAN resolving the same hostname takes the same path — out to the master, back over the tunnel — even though the real service is a few feet away. This is wasted latency, not a correctness bug, but it's the kind of thing users notice.
+
+## What to Build
+
+### 1. Announcer (gateway/proxy side — may already be scoped elsewhere, confirm before duplicating)
+The spoke's `theta-gateway` or `theta-proxy` periodically advertises itself via mDNS on the local segment:
+- Service type: `_theta-suite._tcp.local`
+- TXT records: `site=<slug>`, `hosts=<comma-separated list of public hostnames this site fronts>`
+- Advertised address: the service's own local LAN IP
+
+### 2. Listener + Override (this doc's actual scope — theta-agent)
+- New config field in `agent.yml`, e.g. `prefer_local_discovered_directory: bool` (default `false` — opt-in, not automatic, since it changes name resolution behavior on the host).
+- When `true`, the agent runs an mDNS browser for `_theta-suite._tcp.local` in the background.
+- On receiving an announcement whose `hosts` TXT list includes a hostname the agent cares about (at minimum: the hostname the agent itself is currently configured to connect to for its WS connection), the agent installs a **local override** redirecting that hostname to the discovered local IP.
+- No announcement seen (agent off-site, or flag disabled) → no override installed, normal DNS resolution applies. Nothing else about the agent's behavior changes in this case.
+- If a previously-discovered site's announcement stops being seen (TTL expiry / agent moved networks), the override must be **removed**, not left stale. Don't let a laptop that left the office keep resolving the old office hostname to a now-unreachable LAN IP.
+
+### 3. Override Mechanism — Platform-Specific, Needs Real Investigation
+
+This is the part that most needs Windows/Mac-native work; do not assume the Linux/Unix approach ports directly:
+
+- **Windows**: hosts file lives at `%SystemRoot%\System32\drivers\etc\hosts`; writing to it requires elevation, and Windows caches DNS results independently (`ipconfig /flushdns` needed after an edit, or the change won't take effect immediately — verify whether theta-agent already runs elevated on Windows, since if it doesn't, this whole approach may need a different mechanism, e.g. a local proxy/resolver instead of hosts-file edits).
+- **macOS**: hosts file at `/etc/hosts`, also requires root; macOS's mDNSResponder/DNS caching behavior differs from Windows and Linux (`dscacheutil -flushcache; killall -HUP mDNSResponder` territory) — confirm whether an installed hosts entry is actually honored promptly, or whether the built-in mDNSResponder needs to be told directly instead of fighting it with a hosts-file edit (macOS already *has* native mDNS support baked into resolution — it may be simpler/more idiomatic there to register via the OS's own Bonjour APIs rather than hand-roll hosts-file mutation).
+- **Linux**: `/etc/hosts`, requires root, comparatively straightforward, `systemd-resolved` caching considerations may apply depending on distro.
+
+Given the platform divergence, seriously consider whether a **local stub resolver** (theta-agent listens on `127.0.0.1:<port>`, answers matching hostnames from its own discovery cache, forwards everything else upstream — with the OS's DNS pointed at it only for the duration this feature is active) is actually simpler and more uniform across all three platforms than hosts-file mutation, despite the extra moving part. Recommend evaluating both before committing to an implementation; this doc intentionally doesn't prescribe one, since that call needs platform testing this environment can't do.
+
+## Hard Security Rule (non-negotiable, applies regardless of mechanism chosen)
+
+mDNS is **unauthenticated** on a local network — anyone on the same LAN segment can broadcast a spoofed announcement. This feature may only ever change **where** the agent connects (which IP a hostname resolves to). It must **never** change **whether** the agent trusts what answers there. Concretely:
+- TLS certificate validation and hostname verification against the redirected IP must remain fully enforced — no exceptions, no "local network so it's fine" carve-out.
+- A spoofed rogue announcement pointing a hostname at an attacker's local IP should produce a TLS handshake failure (cert won't match), not a silent connection. If your chosen mechanism has *any* code path where local-discovery bypasses or weakens cert checking, that's a bug, not an optimization — fix it before shipping.
+
+## Out of Scope for This Piece
+
+- The announcer side's exact library/implementation on the gateway/proxy (Linux — can be built where this spec was authored, not blocked on Windows/Mac).
+- Anything about the master-relay mechanism itself (§5.2 of the parent spec) — this doc is purely the "skip the relay when local" optimization layered on top of it.
+
+## Definition of Done
+
+- Flag exists, defaults off.
+- Enabling it on a machine physically on a spoke's LAN measurably routes traffic to the local spoke instead of through the master relay (verify via a network capture or the proxy's own access logs on each side, not just "it feels faster").
+- Leaving that LAN (or disabling the flag) reliably reverts to normal resolution — no stale overrides.
+- A test with a spoofed/rogue mDNS announcement (a second, non-legitimate advertiser) results in a TLS failure, not a successful connection to the impostor.
+- Behavior verified on both Windows and macOS, not just Linux.

docs/MULTI_SITE_SPEC.md

@@ -1,17 +1,17 @@
 # Theta Suite Multi-Site Architecture & VPN Specification
 
-**Specification Version**: `2.0.0`  
-**Status**: Final Architecture Specification  
+**Specification Version**: `2.2.0`
+**Status**: Mostly shipped. Read the status table at the bottom before trusting any section's detail as current behavior — this document accumulated across several build passes and earlier sections describe things that were aspirational when written and real by the time later sections were added.
 **Target Suite Version**: `v1.50.0+`
 **Repository**: [`theta-suite`](https://github.com/theta42/theta-suite)
 
----
+> ## Shipped today
+> - **Join, live replication, promotion** (`sso-manager-node`): a spoke joins via a one-time export over a site join key (`POST /api/site/join-keys` / `/export` / `/join`), then registers its own endpoint so the master can push live resync pings on every catalog write — no longer a one-time snapshot. Promotion (`POST /api/directory-admin/site-promote`) coordinates a real handoff, demoting the old master as one action. Identical agent-signing keys ride the same export/resync path. Read [`sso-manager-node/docs/site-join.md`](https://github.com/theta42/theta-directory/blob/master/docs/site-join.md) and `directory_spec.md` §11 for the endpoint-level detail.
+> - **Gateway-to-gateway WireGuard mesh** (`theta-gateway`): real site-to-site tunnels via `POST /api/mesh/register`/`/join`, kernel WireGuard with a userspace `wireguard-go` fallback. Verified with an actual two-container encrypted tunnel passing traffic, not a mock.
+> - **Not yet connected to each other**: the mesh is a transport layer that exists on its own; `sso-manager-node`'s HTTPS-based join/replicate calls don't route over it yet. That wiring, plus the no-inbound relay it would enable (mechanism verified, automation not built — see status table), is the next layer.
+> - **mDNS local-discovery, Linux**: shipped and verified end-to-end — `theta-gateway` announces (`services/mdns_announce.js`), `theta-agent` discovers and applies a hosts-file override, cleanly reverts when the announcement disappears. Windows/macOS remain unbuilt — see the TODO list.
 
-## Executive Summary
-
-This specification defines the multi-site replication, fault tolerance, autonomous site isolation, and integrated WireGuard mesh network architecture for the `theta-suite` ecosystem.
-
-The architecture enforces **Deployment Symmetry**, **Explicit Master Control via `god_admin` Authority**, **`theta-gateway` Mesh & SSH Integration**, and **Zero-WAN Dependency for Local Operations**.
+Design scale: a handful of sites (dozen max, 254 hard ceiling — see §4), a few hundred users/hosts total. This is a deliberate, small, trusted-operator deployment, not a hyperscale/adversarial-tenant one — several decisions below (fire-and-forget replication, identical directories) trade blast-radius for simplicity *because* the scale allows it. Don't generalize these choices past that scale without re-deriving them.
 
 ---
 
@@ -19,131 +19,147 @@ The architecture enforces **Deployment Symmetry**, **Explicit Master Control via
 
 ```mermaid
 flowchart TB
-    subgraph ControlPlane["Master Site (Site 10.1 - HQ / Control Plane)"]
-        ssoM["sso-manager-node (Master Read/Write)"]
-        ldapM["OpenLDAP (MMR Master Node 1)"]
-        baoM["OpenBao (Master Secrets Engine)"]
-        proxyM["theta-proxy (HQ Web Gateway)"]
-        gateM["theta-gateway (HQ SSH & WireGuard Mesh Gate)"]
-        agentM["theta-agent (HQ Local Agents)"]
+    subgraph ControlPlane["Master Site (write authority)"]
+        ssoM["sso-manager-node (isMaster=true)"]
+        ldapM["OpenLDAP (MMR write node)"]
+        baoM["OpenBao (local, replication source)"]
+        proxyM["theta-proxy"]
+        gateM["theta-gateway"]
+        agentM["theta-agent"]
     end
 
-    subgraph SiteB["Spoke Site 10.2 (Staten Island LAN)"]
-        ssoB["sso-manager-node (Spoke Read-Only Catalog)"]
-        ldapB["OpenLDAP (MMR Node 2 / Read-Only Replica)"]
-        baoB["OpenBao (Spoke Local Secret Replica)"]
-        proxyB["theta-proxy (Site 10.2 Local Web Gateway)"]
-        gateB["theta-gateway (Site 10.2 SSH & WG Mesh Gate)"]
-        agentB["theta-agent (Site 10.2 Local Agents)"]
+    subgraph SiteB["Spoke — inbound (has a public IP)"]
+        ssoB["sso-manager-node (isMaster=false)"]
+        ldapB["OpenLDAP (MMR read replica)"]
+        baoB["OpenBao (local replica)"]
+        proxyB["theta-proxy — serves this site's public traffic directly"]
+        gateB["theta-gateway"]
+        agentB["theta-agent"]
     end
 
-    subgraph SiteNL["Spoke Site 10.5 (Netherlands Offshore Exit)"]
-        ssoNL["sso-manager-node (Spoke Read-Only Catalog)"]
-        ldapNL["OpenLDAP (MMR Node 5 / Read-Only Replica)"]
-        baoNL["OpenBao (Spoke Local Secret Replica)"]
-        proxyNL["theta-proxy (Site 10.5 Local Web Gateway)"]
-        gateNL["theta-gateway (Site 10.5 Offshore Exit Gate)"]
-        agentNL["theta-agent (Site 10.5 Local Agents)"]
+    subgraph SiteC["Spoke — no inbound (CGNAT)"]
+        ssoC["sso-manager-node (isMaster=false)"]
+        ldapC["OpenLDAP (MMR read replica)"]
+        baoC["OpenBao (local replica)"]
+        proxyC["theta-proxy — LAN-local traffic only"]
+        gateC["theta-gateway"]
+        agentC["theta-agent"]
     end
 
-    gateM <==>|"WireGuard Encrypted Mesh Tunnel (172.24.0.x)"| gateB
-    gateM <==>|"WireGuard Encrypted Mesh Tunnel (172.24.0.x)"| gateNL
-    gateB <==>|"WireGuard Direct Tunnel"| gateNL
+    gateM <==>|"WireGuard mesh tunnel"| gateB
+    gateM <==>|"WireGuard mesh tunnel"| gateC
 
-    ssoM <==>|"HTTP SSE Catalog Change Events"| ssoB
-    ldapM <==>|"OpenLDAP MMR syncrepl (ldaps://636)"| ldapB
-    baoM -.->|"Secret Version Replicator"| baoB
+    ssoM -.->|"fire-and-forget push: catalog + secrets + signing key"| ssoB
+    ssoM -.->|"fire-and-forget push"| ssoC
+    ldapM <==>|"OpenLDAP MMR syncrepl"| ldapB
+    ldapM <==>|"OpenLDAP MMR syncrepl"| ldapC
+
+    proxyM -->|"TLS-terminate + relay (no direct path exists)"| gateM
+    gateM ==>|"WG tunnel"| gateC
 ```
 
 ---
 
-## 2. Component Core Roles
+## 2. Every Directory Is Identical
 
-### 2.1 `theta-gateway` (Merged SSH Jump & WireGuard Mesh Gateway)
-The container previously named `jump-host` is officially rebranded and expanded to **`theta-gateway`**. It acts as the single security & routing gateway for each site:
-1. **SSH Jump Host (Port 2222)**: Handles interactive terminal jumps, user identity verification via local OpenLDAP, and dynamic SSH key injection.
-2. **Site-Aware SSH Target Filtering**: `theta-gateway` filters target host/service selection menus strictly to resources assigned to that local site (`SITE_SLUG`).
-3. **WireGuard Mesh Router (Port 51820 / 51871)**: Handles tunnel termination (`wg0`), peer key management, keepalives, and inter-site routing.
-4. **NETMAP & Policy Routing Engine**: Applies `iptables` NETMAP shadow subnet translations, dynamic return path masquerading, and policy exit routes (`table offshore`, `table us_vps`).
+Master and every spoke run the **same LDAP data, the same OpenBao secrets, and the same agent-signing key**. Hitting any site's `sso-manager-node` for read/auth purposes is equivalent to hitting any other. The only asymmetry is **write authority** (§3).
 
-### 2.2 `sso-manager-node` (Control Plane & Local Issuer)
-* **Master Role (`isMaster = true`)**: Holds single write authority for directory resources in `inventory.sqlite`. Processes write requests proxied from Spokes.
-* **Spoke Role (`isMaster = false`)**: Runs in Read-Only Catalog mode. Performs local OIDC JWT token issuance for local web apps via local OpenLDAP authentication.
+This is a deliberate tradeoff, not a default: it means compromising *any single spoke* — including the smallest, least-secured one — grants an attacker the same agent-command authority (`update_binary`, `arbitrary_bash`, service control) as compromising the master, because every site holds the same Ed25519 signing key (`sso-manager-node/nodejs/utils/agent_keys.js`). Accepted here because the deployment scale is small and trusted. Do not extend this pattern to a larger/adversarial-tenant deployment without revisiting it.
 
-### 2.3 `theta-proxy` (Site-Local Web Gateway)
-* **Local Route Scope**: Manages local web application reverse proxying and TLS certificates (ACME / Let's Encrypt / local certs) 100% locally.
-* **Zero Locking**: Proxy route definitions and TLS certs are not locked during Master WAN outages.
+Consequence: `theta-agent` needs **no change** to support multi-site — it already does TOFU pairing against a single trusted key (`websocket.go:341-351`), and since that key is identical everywhere, any site's `sso-manager-node` can validly sign a command for any agent, anywhere, without agents needing a keyring.
 
-### 2.4 `theta-agent` (Site-Local Agent Hub)
-* **Local WS Connection**: Connects to the local site's `sso-manager-node` WebSocket (`wss://sso.site-b.example.com/api/agent/ws`).
-* **Local Autonomy**: Real-time telemetry, memory/CPU metrics, disk usage, active logged-in users (`who`), and desktop control commands (lock, display off, logout, reboot) operate 100% locally.
+### 2.1 What Replicates, and How
+
+| Data | Mechanism | Direction |
+|---|---|---|
+| LDAP (users, groups) | OpenLDAP MMR syncrepl | master (write) → spokes (read-only) |
+| OpenBao secrets (incl. agent-signing key at `secret/agent/signing-key`) | **New**: custom replicator (OpenBao has no built-in multi-site replication — Performance Replication is Vault-Enterprise-only, confirmed absent from OpenBao as of this writing) | master (write) → spokes (read-only) |
+| Directory catalog (Resources: hosts, apps, sites) | Existing catalog change events | master (write) → spokes (read-only) |
+| Audit log | Async batch worker, already speced (§6) | spokes → master |
+
+### 2.2 Replication Delivery: Fire-and-Forget
+
+Master is the sole writer (§3), so there is exactly one producer per data type — no conflict resolution, no consensus, no vector clocks needed. On every write, master pushes the change to all connected spokes **concurrently** (not sequentially — spokes are independent WG peers, none blocks on another) and does **not** wait for acks. A spoke that's offline queues nothing on the master's side; on reconnect, the spoke pulls (or master replays) missed versions.
+
+This is a deliberate choice over "wait for all spokes to ack": with a dozen spokes, concurrent push completes in low hundreds of milliseconds on the happy path, but *waiting* for acks makes every write's latency bounded by the slowest/offline spoke — reintroducing the split-brain-adjacent stall that §3's explicit-promotion design exists to avoid. Never make a master write block on spoke reachability.
 
 ---
 
 ## 3. Explicit Master Control & Human `god_admin` Authority
 
-To guarantee **0% split-brain risk**, automatic failover across WAN is explicitly disabled:
+Automatic failover across WAN is explicitly disabled — 0% split-brain risk by design:
 
 ```
                           WAN OUTAGE DETECTED
                                    │
                                    ▼
              Spoke Node Unconditionally Retains SPOKE Mode
-             (Read-Only Catalog / Full Local Operations)
                                    │
                                    ▼
              Requires Human god_admin Promotion Action
-             (Explicit Confirmation Modal in SSO UI / CLI)
 ```
 
-1. **Unreachable Master Behavior**: If a Spoke node loses WAN connection to the Master, it **unconditionally remains in Spoke Mode**.
-2. **Human Re-assignment**: Changing or promoting a Master node requires an explicit action by an authenticated **`god_admin`** user via the SSO UI or `theta-suite-admin promote-master` CLI.
+1. **Unreachable master**: a spoke that loses the master unconditionally stays a spoke. No auto-election.
+2. **Promotion is a single coordinated action, not two steps**: `POST /api/directory-admin/site-promote` (god_admin-gated) calls out to the *current* master over the WG tunnel and demotes it as part of the same operation — there's never a window with two masters. (Requires the old master to be reachable; if it isn't, that's an operator-visible failure to resolve manually, not a silent partial-promotion.)
+3. Because every directory is identical (§2), promotion carries **no agent re-keying cost** — this was the main risk in earlier drafts of this design and is now moot.
+4. Site state (name, slug, `isMaster`, `masterUrl`, `wanConnected`) lives on the site's own `kind:'site'` Resource (`metadata.multiSite`), not in server memory — it must survive restarts and be visible via the same directory API as everything else.
 
 ---
 
-## 4. Integrated WireGuard Mesh & Key Provisioning Protocol
+## 4. `spoke.env` vs `setup.env`
 
-### 4.1 Automatic Key Exchange & Peer Discovery
-1. When a new `theta-gateway` boots or joins a site via Join Key:
-   - It generates a Curve25519 keypair and registers its public key, listen port, and public endpoint with `sso-manager-node` (`POST /api/mesh/gateway/register`).
-2. `sso-manager-node` calculates the site index (`Site 10.x`), assigns mesh IPs (`172.24.0.x`), and broadcasts updated peer definitions to all active `theta-gateway` instances.
-3. Each `theta-gateway` updates its running WireGuard interface (`wg0`) dynamically via `wgctrl` / `iptables` without dropping existing connections.
+A spoke shares almost none of `setup.env`'s concerns (it doesn't mint LDAP admin/JWT/service-account secrets — those arrive via replication, §2) so it gets its own, much shorter file:
 
-### 4.2 NETMAP Shadow Subnet Addressing (`10.<site_id>.168.0/24`)
-To prevent IP collisions when multiple sites use default `192.168.1.0/24` physical LANs, `theta-gateway` automatically enables **NETMAP Shadow Subnets by default**:
-
-```bash
-# NETMAP: Shadow network (10.<site_id>.168.x) to physical LAN (192.168.1.x)
-PostUp = iptables -t nat -A PREROUTING -i wg0 -d 10.<site_id>.168.0/24 -j NETMAP --to 192.168.1.0/24
-PostUp = iptables -t nat -A POSTROUTING -o wg0 -s 192.168.1.0/24 -j NETMAP --to 10.<site_id>.168.0/24
-PostUp = ip route add local 10.<site_id>.168.0/24 dev lo
+```
+CFG_DOMAIN=theta42.com          # REQUIRED, must match the master's exactly — this is the shared LDAP base DN (dc=theta42,dc=com). Never per-site.
+CFG_SITE_NAME=staten-island     # this site's name/slug
+CFG_SPOKE_INBOUND=false         # true: this site has a public IP and serves its own traffic directly (standalone-style). false: no inbound path exists; master relays (§5).
+CFG_PUBLIC_DOMAIN=              # only used when CFG_SPOKE_INBOUND=true — this site's own domain, own DNS, own ACME cert, independent of the master's domain.
+CFG_JOIN_TOKEN=                 # one-time token from the master, used for WG mesh auto-registration (§4.1) and initial catalog/secret pull.
+CFG_MASTER_ENDPOINT=            # master's WG endpoint (host:port) to join through.
 ```
 
-* **Effect**: A server at Site 10.2 (`192.168.1.50`) can reach a server at Site 10.4 (`192.168.1.50`) by pinging `10.4.168.50`. Neither site needs to modify router DHCP or local subnets!
+`CFG_DOMAIN` is the identity namespace (LDAP DN) and must be identical across every site — MMR replicas cannot diverge on base DN. `CFG_PUBLIC_DOMAIN` is a *web-hostname* concern, unrelated to LDAP, and only exists at all for inbound spokes.
 
-### 4.3 Policy Exit Routing & Return Path SOURCENAT
-* **Custom Route Tables**: `theta-gateway` supports selective outbound exit tables (`table offshore`, `table us_vps`) based on IP ranges (`ip rule add from 10.x.254.0/24 lookup offshore`).
-* **Dynamic Return Path SOURCENAT**: Exit nodes (e.g. Netherlands `10.5`) apply source-NAT masquerading for incoming tunnel traffic to guarantee symmetric return path routing:
-  ```bash
-  PostUp = iptables -t nat -A POSTROUTING -o wg0 ! -s 172.24.0.0/13 -j MASQUERADE
-  ```
+### 4.1 WireGuard Mesh Auto-Registration
+
+1. A new `theta-gateway` boots with `CFG_JOIN_TOKEN` + `CFG_MASTER_ENDPOINT`, generates its Curve25519 keypair, and calls `POST /api/mesh/gateway/register` on the master over an initial bootstrap tunnel.
+2. Master assigns the next free **site index** (one octet, used identically in both `172.24.<site>.0/16` and `10.<site>.0.0/16` per the reference topology in Appendix A) and returns full mesh peer config.
+3. **Site index ceiling is 254** (0 and 255 excluded) — a hard technical limit of this addressing scheme, not an arbitrary cap. Real deployments target a dozen or fewer; no need to cap lower than the real ceiling.
+4. Each `theta-gateway` applies the new peer set to its running `wg0` via `wgctrl` without dropping existing connections.
 
 ---
 
-## 5. Roaming Admin Access (QR Codes & Optional Tailscale)
+## 5. Inbound vs. No-Inbound Spokes
 
-1. **Native WireGuard Client Profiles**:
-   - `sso-manager-node` includes a built-in **Client Profile Generator** in the UI.
-   - Admins can generate a mobile/laptop profile, displaying a **QR code** for immediate scan into the official WireGuard app on iOS/Android or a downloadable `wg0.conf` for laptops.
-2. **Optional Tailscale / Headscale Connector**:
-   - For roaming devices in environments where WireGuard UDP ports are blocked, `theta-gateway` supports an optional `tailscale` / `headscale` sidecar integration.
+Whether a spoke has a public IP determines everything about how its traffic reaches the outside world — these are two distinct, documented operating modes, not a single universal mechanism.
+
+### 5.1 Inbound Spoke (`CFG_SPOKE_INBOUND=true`)
+Behaves like a standalone install. Own `CFG_PUBLIC_DOMAIN`, own DNS pointed at its own public IP, own ACME cert. `theta-proxy` and `theta-gateway` serve public web + SSH traffic directly — no relay involved. The only WAN-facing traffic to the master is replication (§2) and audit shipping (§6).
+
+### 5.2 No-Inbound Spoke (`CFG_SPOKE_INBOUND=false`)
+No public IP exists, so *any* external access must go through the master:
+
+1. Master mints a public hostname for the spoke's services (e.g. `sso-{slug}.{master's public domain}`) and creates the corresponding `theta-proxy` route (already dynamic/DB-backed — `proxy/nodejs/models/host.js` — no new plumbing needed there).
+2. Master **terminates TLS** for that hostname and relays to the spoke over the WG tunnel — both `theta-proxy` (any site-hosted web app) and `theta-gateway` (SSH jump) traffic relay this way, not just SSO.
+3. Terminating at the master (rather than SNI passthrough) is fine here specifically because master↔spoke already rides an encrypted WG tunnel — there's no unencrypted hop being introduced.
+
+### 5.3 Local-Direct Resolution (Skip the Relay On-LAN)
+
+A client physically on a no-inbound spoke's LAN would otherwise hairpin out to the master and back to reach its own local site. Solved via **mDNS local-service-discovery**, not directory-side network topology:
+
+1. The spoke's `theta-gateway`/`theta-proxy` announces itself on the local segment via mDNS (`_theta-suite._tcp.local`, TXT records: site slug, public hostnames it fronts, local IP).
+2. `theta-agent`, when a config flag (`preferLocalDiscoveredDirectory` or similar — see the agent-side spec, Appendix B) is enabled, listens for this announcement and overrides local resolution for matching hostnames to the discovered local IP.
+3. No match (off-site, or flag disabled) → normal public DNS → master relay. Multicast is link-local by nature, so "on-site or not" needs no explicit detection logic — presence/absence of the announcement *is* the signal. This also solves roaming-admin access (§ formerly "5", folded in here) for free: same laptop, same flag, local-fast-path at the office and relay-path everywhere else.
+4. **Hard rule**: mDNS is unauthenticated on a LAN. It may only ever change *where* the agent connects, never *whether* it trusts what answers — TLS/hostname validation against the redirected IP must stay intact, so a spoofed rogue announcement produces a TLS failure, not a silent MITM.
+
+This piece needs Windows/Mac-specific implementation and testing that can't be done from this (Linux) environment — see Appendix B for the standalone spec handed off for that work.
 
 ---
 
 ## 6. Non-Canonical Audit Logging
 
-* **Local Activity Storage**: OAuth logins, SSH session events, proxy access logs, and agent execution events write to local site audit tables without blocking local operations.
-* **Asynchronous Log Worker**: A background worker flushes log batches to Master via `POST /api/directory-admin/audit/ingest` when WAN is online.
+Unchanged from prior draft: OAuth logins, SSH session events, proxy access, and agent execution events write to local site audit tables without blocking on WAN. An async worker flushes batches to master via `POST /api/directory-admin/audit/ingest` when reachable.
 
 ---
 
@@ -226,4 +242,36 @@ AllowedIPs = 172.24.0.1/32, 10.1.0.0/8
 
 ---
 
-*Final Architecture Specification committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*
+## Appendix B: Agent-Side Work
+
+See [`AGENT_LOCAL_DISCOVERY_SPEC.md`](./AGENT_LOCAL_DISCOVERY_SPEC.md) — split out because it needs Windows/Mac implementation and testing that a Linux-only dev environment cannot meaningfully do. That doc is the handoff: it specifies behavior precisely enough to implement and test independently, without needing to re-derive the reasoning in this file.
+
+---
+
+## Status of This Spec vs. Code (as of this revision)
+
+| Piece | Status |
+|---|---|
+| Site role persisted (not in-memory) | **Shipped** — `/config/site.json` on `sso-manager-node`, survives restarts (v2.2.0) |
+| Join key issuance + one-time directory adoption | **Shipped** — `/api/site/join-keys`, `/api/site/export`, `/api/site/join`, fresh-install-gated (v2.2.0–v2.3.0) |
+| Spoke read-only enforcement | **Shipped** — directory-write routes 403 toward the master once joined (v2.3.0) |
+| WAN health check | **Shipped** — `/api/site/ping`, live in the Master Site modal (v2.2.0–v2.3.0) |
+| `setup.env` / `setup.sh` join wiring | **Shipped** — `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY`, `bootstrap/site-join.js` (theta-suite v2.2.0) |
+| Continuous/live replication (vs. one-time export-on-join) | **Shipped** (`sso-manager-node`) — a spoke registers its own endpoint at join time (`POST /api/site/spokes`), and every successful master catalog write fires a fire-and-forget push (`utils/site_replicate.js`) at every registered spoke, which re-pulls a fresh export. Verified end-to-end in `docker-compose.multisite-e2e.yml`. |
+| Identical-directory signing key | **Shipped** — `POST /api/site/export` includes the master's agent-signing key; a spoke adopts it via `agent_keys.adopt()` on join and every resync. OpenBao secret replication *beyond* this one key is still not built. |
+| Coordinated master promotion (demote the old master as one action) | **Shipped** — `POST /api/site/demote` + `site-promote`'s handoff logic. Fixed two real pre-existing bugs while wiring this in: `site-promote`'s god_admin check read a `req.user.groups` field nothing ever populated (permanently 403'd for everyone), and the read-only write-gate 403'd `site-promote` itself before the handler could run. |
+| WireGuard gateway-to-gateway mesh (`theta-gateway`) | **Shipped** — `POST /api/mesh/register`/`/join` (join-token bootstrap), `utils/wg_iface.js` (kernel WireGuard, falls back to userspace `wireguard-go`). Verified with a real two-container test: actual encrypted tunnel, real ICMP traffic across it, 0% loss. This is the mesh transport layer only — nothing in `sso-manager-node`'s replication yet routes traffic *over* it; today's site-to-site HTTPS calls (join/export/resync) still go over whatever network path already reaches the target, same as before this layer existed. |
+| No-inbound-spoke relay (master proxies a spoke with no public IP) | **Mechanism verified, automation not built.** Confirmed with a standalone test (not `theta-proxy`'s actual Lua/Redis engine, which needs its own dedicated pass to wire safely): a spoke with zero published ports, reachable only via its WG mesh IP, served a request that an external client sent to the master's public port — the master terminated the connection and relayed over the tunnel. So the underlying idea works; what's missing is `theta-proxy` automatically creating that relay route when a no-inbound spoke registers (needs a real service-to-service credential between `sso-manager-node` and `theta-proxy`/`theta-gateway` that doesn't exist yet — a new integration, not a small wiring task), and today's HTTPS-based join/replicate still requires the spoke to reach the master's API directly (and vice versa for export), so a spoke with zero inbound *and* zero outbound path still can't join at all. |
+| mDNS local-discovery (Linux) | **Shipped** — `theta-gateway` announces (`services/mdns_announce.js`, opt-in via `THETA_LOCAL_DISCOVERY_HOSTS`), `theta-agent` discovers and applies a hosts-file override (`local_discovery.go`, opt-in via `prefer_local_directory`). Verified end-to-end with real containers over real multicast: announce → discover → apply → clean revert on disappearance, all confirmed. Caught two real bugs along the way (`mdns.Lookup()`'s IPv6 query aborting the whole lookup even after a valid IPv4 response arrived; `rename()` failing with EBUSY over a bind-mounted `/etc/hosts`, common in every container runtime) — see the commit messages in `theta-agent`. |
+| mDNS local-discovery (Windows, macOS) | Not built — needs platform-native testing this environment can't do (hosts-file vs. stub-resolver tradeoff, elevation, DNS-cache behavior per OS — see Appendix B §3). This is now the **only unbuilt piece** of the original design. |
+
+### TODO — what's actually left, in rough dependency order
+
+1. **mDNS local-discovery, Windows + macOS** — needs platform-native testing this Linux environment cannot do (hosts-file vs. stub-resolver tradeoff, elevation, DNS-cache quirks per OS — see Appendix B §3). Blocked on a Windows/Mac dev environment, not on design. The Linux side (announcer + agent listener) is done and verified — this is the only remaining piece of the original design with no Linux-buildable path forward.
+2. **Route `sso-manager-node`'s HTTPS traffic (join/export/resync) over the WireGuard mesh** instead of the open internet, now that the mesh exists as its own transport layer. Currently the two subsystems don't know about each other.
+3. **`theta-proxy` automation for the no-inbound relay** — mechanism is verified (see status table), but nothing creates the relay route automatically when a no-inbound spoke registers. Needs a new service-to-service credential between `sso-manager-node` and `theta-proxy`/`theta-gateway` — a real design decision (who mints it, what it authorizes), not just wiring.
+4. **OpenBao secret replication beyond the one agent-signing key** — LDAP admin creds, JWT secret, other per-deployment secrets that currently differ per site.
+5. **`theta-proxy`/`theta-gateway` service-to-service auth model in general** — items 2 and 3 both need it; worth designing once rather than inventing a credential per integration.
+6. **Mesh peer removal cleanup** — `wg_iface.removePeer()` doesn't remove the kernel routes `setPeer()` adds (flagged in code, not yet exercised because nothing removes a mesh peer today).
+
+*Committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*

setup.env.example

@@ -55,7 +55,12 @@ CFG_DOMAIN=example.com
 # -> Mint key). Honored ONLY on a first-run bring-up (before ./config/ exists),
 # so it can never merge an already-populated directory; re-runs ignore it.
 # The spoke adopts the master's users/groups/resources and persists its spoke
-# role in ./config/site.json (isMaster=false, masterUrl, siteSlug).
+# role in ./config/site.json (isMaster=false, masterUrl, siteSlug). It also
+# registers itself with the master (using this site's own CFG_SSO_HOST) so
+# future catalog changes on the master get pushed here live instead of this
+# being a one-time snapshot -- the master must be able to reach THIS site's
+# CFG_SSO_HOST for that part to work; if it can't (this site has no inbound
+# path), the join still succeeds, it just never receives live updates.
 #CFG_MASTER_DIRECTORY_URL=https://sso.master.example.com
 #CFG_MASTER_DIRECTORY_JOIN_KEY=stj_9f2e...
 

setup.sh

@@ -1161,8 +1161,12 @@ fi
 # joined reports "already a spoke" and setup continues.
 if [[ -n "${CFG_MASTER_DIRECTORY_URL:-}" && -n "${CFG_MASTER_DIRECTORY_JOIN_KEY:-}" ]]; then
 	info "Joining master site ${CFG_MASTER_DIRECTORY_URL} (CFG_MASTER_DIRECTORY_*)..."
+	# selfUrl (https://$CFG_SSO_HOST, already derived above) registers this
+	# spoke for LIVE replication -- without it the join still succeeds, but
+	# the master has no way to reach this spoke to push resync pings, so it
+	# only ever gets the one-time snapshot from the moment it joined.
 	if ! "${COMPOSE[@]}" exec -T sso-manager node /bootstrap/site-join.js \
-			"$CFG_MASTER_DIRECTORY_URL" "$CFG_MASTER_DIRECTORY_JOIN_KEY"; then
+			"$CFG_MASTER_DIRECTORY_URL" "$CFG_MASTER_DIRECTORY_JOIN_KEY" "https://$CFG_SSO_HOST"; then
 		die "site join failed — check the master URL + site join key (mint one on the master's Site Join Keys card)."
 	fi
 else