Merge pull request #216 from theta42/release-v2.7.0

release(v2.7.0): auto-configured OpenLDAP multi-master replication

.gitignore

@@ -19,6 +19,11 @@ proxy.env
 # per-deployment and is not committed.
 setup.env
 
+# spoke.env — same rule as setup.env, but for the join-a-cluster vars split
+# out for clarity (spoke.env.example IS committed). Holds a real site join
+# key once filled in.
+spoke.env
+
 # Backup artifacts (hold secrets — the whole user directory + Redis dumps)
 *.rdb
 *.ldif

CHANGELOG.md

@@ -9,6 +9,150 @@ orchestration code; see each submodule's own `CHANGELOG.md`
 [jump-host](https://github.com/theta42/jump-host/blob/master/CHANGELOG.md))
 for what changed inside the apps it composes.
 
+## [v2.7.0] - 2026-08-11
+
+Rolls up **sso-manager-node v2.7.0**. Closes the last "operator hand-sets
+this" item on the multi-site TODO: OpenLDAP N-way multi-master replication
+now configures itself.
+
+### theta-suite orchestration
+- **`bootstrap/site-ldap-register.js`**: runs on every `setup.sh` invocation
+  (master and spoke), fetches this node's auto-assigned `LDAP_SERVER_ID` +
+  current peer list from sso-manager-node's new endpoints, and restarts
+  `sso-manager` only when the computed config actually changed.
+- `CFG_LDAP_MMR_MANUAL=true` skips the automatic step for a topology outside
+  this cluster the script can't derive on its own -- otherwise it always
+  runs (every fresh install starts as a master) and would overwrite
+  hand-set values.
+- `setup.env.example`'s old `#LDAP_SERVER_ID=1`/`#LDAP_REPLICATION_HOSTS=`
+  manual-config prompt is gone for the common case.
+
+### sso-manager-node v2.7.0
+- `SiteSpoke.ldapServerId` auto-assigned at registration (same pattern as
+  jump-host's mesh index); each site's LDAP URL derived from its
+  already-known HTTP(S) endpoint. New `GET /api/site/ldap-peers`
+  (spoke-facing) and `GET /directory-admin/ldap-replication-config`
+  (master-local). Verified against real running containers.
+
+## [v2.6.0] - 2026-08-11
+
+Rolls up **sso-manager-node v2.6.0** and **jump-host v2.1.1** (already
+current). Fixes several real bugs found on a live deployment: theta-agent
+never got its `server_url` written, `theta-agent update` 404'd because
+setup.sh installed a stale committed binary, the Directory's site slug and
+gateway count were both wrong, and duplicate group rows accumulated on
+repeated resource promotion.
+
+### theta-suite orchestration
+- **`server_url` now gets written into `/etc/theta42/agent.yml`.** setup.sh's
+  theta-agent install step `sed`'d in `join_key` but never touched
+  `server_url`, so it kept `agent.yml.example`'s literal placeholder forever.
+  Self-heals an already-installed `agent.yml` too (never touches
+  `join_key`/`auth_token`).
+- **`theta-agent update` no longer 404s.** setup.sh now downloads the current
+  release binary from GitHub instead of trusting one committed in the
+  theta-agent submodule checkout, which predated an upstream fix and could
+  never self-update out of the bug. Matches theta-agent's own `install.sh`.
+  The stale committed binaries were removed from the theta-agent repo.
+- **`SITE_SLUG` auto-derived from `CFG_SITE_NAME`.** Nothing ever set it
+  before, so a fresh master always showed the app's own literal
+  "site-default" fallback.
+- **`PROXY_INTERNAL_URL`/`JUMP_INTERNAL_URL` wired into `docker-compose.yml`.**
+  Both the no-inbound relay automation and the new real gateway-mesh count
+  existed in sso-manager-node's code but were completely unreachable in
+  every real deployment -- neither env var was ever actually set.
+- **`spoke.env.example`** -- a dedicated file for the join-a-cluster vars,
+  split out of `setup.env` for clarity (which still has every option).
+  Layered on top of `setup.env` when present. Also adds `CFG_PUBLIC_DOMAIN`
+  (documented in the spec but never wired into setup.sh before).
+
+### sso-manager-node v2.6.0
+- Fixed duplicate access/admin groups accumulating on repeated resource
+  promotion (three independent copies of the same missing-existence-check
+  bug).
+- `GET /api/directory-admin/resources` no longer runs a full LDAP group
+  self-heal fan-out on every list -- moved to write-time, with an explicit
+  `POST /resources/heal-groups` for backfill.
+- Recovers an nmap scan that completed successfully despite a benign stderr
+  warning nmap itself prints (`node-nmap` treated it as a fatal failure).
+- The Multi-Site modal's gateway count now queries jump-host's real mesh
+  registry instead of an unrelated WireGuard subsystem.
+
+## [v2.5.0] - 2026-08-10
+
+Rolls up **sso-manager-node v2.5.0** and **jump-host v2.1.1** — closes the
+last gap in no-inbound relay automation (the mechanism existed at the API
+level but nothing in the real operator bring-up flow could reach it) and
+fixes two real bugs found live-testing it.
+
+### theta-suite orchestration
+- **`bootstrap/site-relay-register.js`** + `CFG_SPOKE_NO_INBOUND`/
+  `CFG_SPOKE_PUBLIC_HOST` (`setup.env.example`): a no-inbound spoke's
+  `setup.sh` run now discovers its own jump-host's WireGuard mesh IP and
+  registers it with the master on every invocation (idempotent no-op until
+  the two jump-hosts are actually meshed — that peering stays a deliberate
+  manual step, same as minting/pasting a site join key).
+- `docs/MULTI_SITE_SPEC.md` and the published `docs/jump-host/mesh.md` page
+  updated — both still described this as "designed but not automated" after
+  the API-level work had already shipped in sso-manager-node/jump-host.
+
+### sso-manager-node v2.5.0
+- **No-inbound relay automation reachable from the real join flow.**
+  `POST /api/site/join` now forwards `noInbound`/`meshIp`/`publicHost`
+  through to `POST /api/site/spokes`, which drives `utils/proxy_client.js`
+  to auto-create/update the relay route on the master's `theta-proxy` (a new
+  self-service `prx_...` API token client — reuses `theta-proxy`'s existing
+  token system, not a new credential type). Verified against a real running
+  `theta-proxy` container.
+- **Replication traffic prefers the mesh.** `utils/site_replicate.js`'s
+  fire-and-forget resync push tries a registered spoke's `meshIp` first,
+  falling back to its public endpoint on failure.
+
+### jump-host v2.1.1
+- **`GET /api/mesh/self`** — this gateway's own mesh IP, for local scripts
+  (gated by any self-service API token, not a full admin session).
+- **Fixed: `/api/mesh/register` was unreachable via HTTP.** A route-mounting
+  order bug meant every `/api/mesh/*` request hit an admin-session gate
+  before `routes/mesh.js` ever ran, so a real gateway-to-gateway mesh join
+  always 401'd. Found live-testing `/self` with two real containers.
+- **Fixed: the initiating side of a mesh join never recorded its own
+  identity** — `GET /api/mesh/self` and the mesh UI's own-entry handling
+  silently saw nothing on whichever gateway called `/join` (only the
+  receiving side of `/register` persisted a self-entry). Verified with two
+  real meshed containers: both sides now report their own correct mesh IP.
+- **Mesh peer removal now cleans up its kernel routes** (`wg_iface.removePeer()`)
+  — verified live: routes present after `setPeer`, gone after `removePeer`.
+
+## [v2.4.0] - 2026-08-10
+
+Rolls up **theta-agent v2.2.0** — mDNS local-discovery is now Windows-capable,
+closing the last gap in the original multi-site design on the agent side.
+
+### theta-agent v2.2.0
+- **Windows local-discovery**: the hosts override now runs on Windows
+  (`%SystemRoot%\System32\drivers\etc\hosts`, CRLF-aware, `ipconfig /flushdns`
+  after every change). Reachable because the agent runs as a SYSTEM service, so
+  the elevation question in the spec resolved in our favor. The Windows CI leg
+  now runs the real Windows write path instead of skipping.
+- **Local route pinning** (`local_route*.go`): the hosts override only fixes
+  *name resolution*; the packet path is the routing table's job. If the WireGuard
+  mesh tunnel is up with `AllowedIPs` covering the LAN subnet (or a full-tunnel
+  `0.0.0.0/0`), the tunnel route would swallow the direct connection to the
+  discovered LAN IP. Discovery now pins a `/32` host route via the owning local
+  interface (`route.exe add ... metric 1` on Windows, `ip route replace` on
+  Linux) and drops it on revert — closing a real gap in the shipped Linux path.
+- **Prompt reconnect**: an apply/revert signals the WebSocket loop, which
+  reconnects immediately instead of waiting out its 5s backoff.
+- **Installer version fix**: the setup.exe previously hardcoded `2.1.0` in its
+  file name and version resources no matter the tag; it now derives the version
+  from the git tag.
+
+### docs
+- `docs/MULTI_SITE_SPEC.md` status table updated: Windows local-discovery marked
+  shipped; macOS remains the one unbuilt piece (hosts override compiles on
+  darwin but needs `dscacheutil -flushcache` + real hardware testing, being done
+  on a macOS VM).
+
 ## [v2.3.0] - 2026-08-10
 
 Rolls up **theta-directory v2.4.0**, **jump-host v2.1.0**, **theta-agent v2.1.2**. Live catalog replication and a real gateway-to-gateway WireGuard mesh land in the same pass — multi-site directory sync stops being a one-time snapshot, and site-to-site networking becomes real infrastructure instead of a documented-but-unbuilt design. See `docs/MULTI_SITE_SPEC.md` for the full architecture and an explicit TODO list of what's still open (Windows/macOS mDNS, routing directory traffic over the mesh, `theta-proxy` no-inbound relay automation).

bootstrap/site-ldap-register.js

@@ -0,0 +1,139 @@
+#!/usr/bin/env node
+/*
+ * theta-suite site-ldap-register — runs inside the sso-manager container on
+ * every setup.sh run (both master and spoke) to keep OpenLDAP N-way
+ * multi-master replication config (docs/replication.md) in sync without an
+ * operator hand-maintaining LDAP_SERVER_ID/LDAP_REPLICATION_HOSTS.
+ *
+ * The master assigns each spoke a unique LDAP_SERVER_ID at join time (same
+ * mechanism as jump-host's WireGuard mesh index) and derives every site's
+ * LDAP URL from its already-known HTTPS endpoint -- see sso-manager-node's
+ * GET /api/site/ldap-peers (spoke-facing) and
+ * GET /directory-admin/ldap-replication-config (master-local).
+ *
+ * This script fetches whichever of those two applies to this node's role,
+ * and writes the result to /config/ldap-replication.env (KEY=VALUE, the
+ * same shape setup.env/spoke.env use) if it changed since last run. setup.sh
+ * sources that file before starting sso-manager on every invocation, and
+ * restarts the container when this script reports a change -- OpenLDAP's
+ * static slapd.conf is only read at process start, so a config change needs
+ * a restart to take effect; there's no live push, which is why this has to
+ * be re-run periodically (every setup.sh invocation) rather than working
+ * once at join time and never again, especially on the MASTER, whose peer
+ * list changes every time a new spoke joins.
+ *
+ *   docker compose exec sso-manager node /bootstrap/site-ldap-register.js <selfUrl>
+ *
+ * Self-contained (Node built-ins + global fetch), same rule as
+ * bootstrap.js/site-join.js -- does NOT require the SSO's internal models.
+ *
+ * Output (stdout, KEY=VALUE for setup.sh): LDAP_CONFIG_CHANGED=<yes|no>,
+ * LDAP_SERVER_ID=<n>, LDAP_REPLICATION_HOSTS=<space-separated, may be empty>.
+ * Progress logs go to stderr.
+ */
+'use strict';
+
+const fs = require('fs');
+
+const SITE_CONFIG = '/config/site.json';
+const LDAP_CONFIG_FILE = '/config/ldap-replication.env';
+const SSO_INTERNAL = 'http://localhost:3001';
+
+const selfUrl = process.argv[2];
+
+function log(msg) { console.error('[site-ldap-register] ' + msg); }
+
+function readPersisted() {
+  if (!fs.existsSync(LDAP_CONFIG_FILE)) return { LDAP_SERVER_ID: '', LDAP_REPLICATION_HOSTS: '' };
+  const out = { LDAP_SERVER_ID: '', LDAP_REPLICATION_HOSTS: '' };
+  for (const line of fs.readFileSync(LDAP_CONFIG_FILE, 'utf8').split('\n')) {
+    const m = line.match(/^([A-Z_]+)=(.*)$/);
+    if (m && m[1] in out) out[m[1]] = m[2];
+  }
+  return out;
+}
+
+async function fetchMasterConfig() {
+  const sso = require('/config/sso-secrets.js');
+  const adminUid = (sso.bootstrap && sso.bootstrap.adminUid) || 'admin';
+  const adminPass = (sso.bootstrap && sso.bootstrap.adminPass) || '';
+
+  const loginRes = await fetch(`${SSO_INTERNAL}/api/auth/login`, {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({ uid: adminUid, password: adminPass }),
+  });
+  if (!loginRes.ok) throw new Error(`local admin login failed (${loginRes.status}): ${await loginRes.text().catch(() => '')}`);
+  const { token } = await loginRes.json();
+  if (!token) throw new Error('local admin login returned no token');
+
+  const cfgRes = await fetch(`${SSO_INTERNAL}/api/directory-admin/ldap-replication-config`, {
+    headers: { 'auth-token': token },
+  });
+  if (!cfgRes.ok) throw new Error(`ldap-replication-config failed (${cfgRes.status}): ${await cfgRes.text().catch(() => '')}`);
+  return cfgRes.json();
+}
+
+async function fetchSpokeConfig(site, selfUrl) {
+  const url = `${site.masterUrl.replace(/\/+$/, '')}/api/site/ldap-peers?endpoint=${encodeURIComponent(selfUrl)}`;
+  const res = await fetch(url, { headers: { Authorization: 'Bearer ' + site.masterJoinKey } });
+  const text = await res.text().catch(() => '');
+  let data = null;
+  try { data = JSON.parse(text); } catch (e) { /* not JSON */ }
+  if (!res.ok) {
+    if (res.status === 404) {
+      log('This site is not registered as a spoke on the master yet (join with selfUrl, or re-run site-relay-register.js). Skipping.');
+      return null;
+    }
+    throw new Error(`ldap-peers failed (${res.status}): ${(data && data.message) || text}`);
+  }
+  return data;
+}
+
+async function main() {
+  if (!fs.existsSync(SITE_CONFIG)) {
+    log('No /config/site.json yet. Skipping.');
+    console.log('LDAP_CONFIG_CHANGED=no');
+    return;
+  }
+  const site = JSON.parse(fs.readFileSync(SITE_CONFIG, 'utf8'));
+
+  let result;
+  if (site.isMaster) {
+    result = await fetchMasterConfig();
+  } else {
+    if (!site.masterUrl || !site.masterJoinKey) {
+      log('Spoke role but missing masterUrl/masterJoinKey. Skipping.');
+      console.log('LDAP_CONFIG_CHANGED=no');
+      return;
+    }
+    if (!selfUrl) throw new Error('usage: node /bootstrap/site-ldap-register.js <selfUrl> (required for a spoke)');
+    result = await fetchSpokeConfig(site, selfUrl);
+    if (!result) {
+      console.log('LDAP_CONFIG_CHANGED=no');
+      return;
+    }
+  }
+
+  const serverId = String(result.ldapServerId || '');
+  const hosts = (result.peers || []).map((p) => p.ldapHost).filter(Boolean).join(' ');
+
+  const before = readPersisted();
+  const changed = before.LDAP_SERVER_ID !== serverId || before.LDAP_REPLICATION_HOSTS !== hosts;
+
+  if (changed) {
+    fs.writeFileSync(LDAP_CONFIG_FILE, `LDAP_SERVER_ID=${serverId}\nLDAP_REPLICATION_HOSTS=${hosts}\n`);
+    log(`Replication config changed -- ServerID ${serverId}, ${(result.peers || []).length} peer(s). Wrote ${LDAP_CONFIG_FILE}.`);
+  } else {
+    log(`Replication config unchanged -- ServerID ${serverId}, ${(result.peers || []).length} peer(s).`);
+  }
+
+  console.log(`LDAP_CONFIG_CHANGED=${changed ? 'yes' : 'no'}`);
+  console.log(`LDAP_SERVER_ID=${serverId}`);
+  console.log(`LDAP_REPLICATION_HOSTS=${hosts}`);
+}
+
+main().catch((e) => {
+  console.error('[site-ldap-register] FAILED: ' + e.message);
+  process.exit(1);
+});

bootstrap/site-relay-register.js

@@ -0,0 +1,128 @@
+#!/usr/bin/env node
+/*
+ * theta-suite site-relay-register — runs inside the sso-manager container
+ * (same pattern as site-join.js) to finish no-inbound relay automation for a
+ * spoke with no public IP (MULTI_SITE_SPEC.md §5.2).
+ *
+ * site-join.js's initial join can't supply a mesh IP: this site's jump-host
+ * isn't meshed to the master's yet at that point (mesh peering is a manual,
+ * out-of-band action on both jump-hosts -- mint a join token on the master's
+ * jump-host, paste it into this site's jump-host "Join a mesh" UI action --
+ * the same reason the site join key itself is minted/pasted by hand rather
+ * than automated). This script is the follow-up: run it (setup.sh does, on
+ * every run, when CFG_SPOKE_NO_INBOUND is set) once meshing is done, and it
+ * discovers this jump-host's mesh IP and registers it with the master so
+ * theta-proxy there can auto-create the relay route (see sso-manager-node's
+ * utils/proxy_client.js). Safe to run before meshing completes -- reports
+ * "not meshed yet" and exits 0 so a re-run later just picks it up.
+ *
+ *   docker compose exec sso-manager node /bootstrap/site-relay-register.js \
+ *       https://sso.this-site.example.com sso-branch2.master-domain.example.com
+ *
+ * Self-contained (Node built-ins + global fetch), same rule as bootstrap.js
+ * and site-join.js -- it does NOT require the SSO's internal models. It
+ * reads this node's own spoke role from /config/site.json (written by
+ * site-join.js) and logs into the LOCAL jump-host as its bootstrap-minted
+ * local admin (/config/jump-secrets.js) to call jump-host's own
+ * GET /api/mesh/self.
+ *
+ * Output (stdout, KEY=VALUE for setup.sh): RELAY=<registered|not-meshed|not-a-spoke|skipped>.
+ * Progress logs go to stderr.
+ */
+'use strict';
+
+const fs = require('fs');
+
+const SITE_CONFIG = '/config/site.json';
+const JUMP_SECRETS = '/config/jump-secrets.js';
+const JUMP_INTERNAL = 'http://jump-host:3002';
+
+const selfUrl = process.argv[2];
+const publicHost = process.argv[3];
+
+function log(msg) { console.error('[site-relay-register] ' + msg); }
+
+async function main() {
+  if (!selfUrl || !publicHost) {
+    throw new Error('usage: node /bootstrap/site-relay-register.js <selfUrl> <publicHost>');
+  }
+
+  if (!fs.existsSync(SITE_CONFIG)) {
+    log('No /config/site.json yet — this node has not joined a master. Nothing to do.');
+    console.log('RELAY=not-a-spoke');
+    return;
+  }
+  const site = JSON.parse(fs.readFileSync(SITE_CONFIG, 'utf8'));
+  if (site.isMaster || !site.masterUrl || !site.masterJoinKey) {
+    log('Not a joined spoke (missing masterUrl/masterJoinKey, or this is a master). Nothing to do.');
+    console.log('RELAY=not-a-spoke');
+    return;
+  }
+
+  if (!fs.existsSync(JUMP_SECRETS)) {
+    log('No /config/jump-secrets.js — jump-host has not been provisioned yet. Skipping.');
+    console.log('RELAY=skipped');
+    return;
+  }
+  const jumpSecrets = require(JUMP_SECRETS);
+  const jumpAdminUser = (jumpSecrets.auth && jumpSecrets.auth.adminUsers && jumpSecrets.auth.adminUsers[0]) || 'jumpadmin';
+  const jumpAdminPass = (jumpSecrets.auth && jumpSecrets.auth.localAdminPass) || '';
+  if (!jumpAdminPass) {
+    log('jump-secrets.js has no local admin password. Skipping.');
+    console.log('RELAY=skipped');
+    return;
+  }
+
+  const loginRes = await fetch(`${JUMP_INTERNAL}/api/auth/login`, {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    // jump-host's login route (@simpleworkjs/oidc-client's shared router)
+    // expects `username`, not `uid` -- unlike sso-manager-node's own
+    // /api/auth/login (see site-join.js). Confirmed against a real running
+    // jump-host container; `uid` here just silently 401s.
+    body: JSON.stringify({ username: jumpAdminUser, password: jumpAdminPass }),
+  });
+  if (!loginRes.ok) {
+    throw new Error(`jump-host admin login failed (${loginRes.status}): ${await loginRes.text().catch(() => '')}`);
+  }
+  const { token: jumpToken } = await loginRes.json();
+  if (!jumpToken) throw new Error('jump-host login returned no token');
+
+  const selfRes = await fetch(`${JUMP_INTERNAL}/api/mesh/self`, { headers: { 'auth-token': jumpToken } });
+  if (!selfRes.ok) {
+    throw new Error(`jump-host mesh self-lookup failed (${selfRes.status}): ${await selfRes.text().catch(() => '')}`);
+  }
+  const selfData = await selfRes.json();
+  if (!selfData.meshIp) {
+    log('jump-host is not meshed yet (no mesh IP assigned). Mesh-join it first (jump-host UI), then re-run setup.sh.');
+    console.log('RELAY=not-meshed');
+    return;
+  }
+  log(`Discovered mesh IP ${selfData.meshIp}. Registering with ${site.masterUrl}...`);
+
+  const regRes = await fetch(`${site.masterUrl.replace(/\/+$/, '')}/api/site/spokes`, {
+    method: 'POST',
+    headers: { Authorization: 'Bearer ' + site.masterJoinKey, 'Content-Type': 'application/json' },
+    body: JSON.stringify({
+      endpoint: selfUrl,
+      siteSlug: site.siteSlug || '',
+      noInbound: true,
+      meshIp: selfData.meshIp,
+      publicHost,
+    }),
+  });
+  const text = await regRes.text().catch(() => '');
+  let data = null;
+  try { data = JSON.parse(text); } catch (e) { /* not JSON */ }
+  if (!regRes.ok) {
+    throw new Error(`relay registration failed (${regRes.status}): ${(data && data.message) || text}`);
+  }
+
+  log(`Relay: ${(data.relay && data.relay.note) || 'registered'}`);
+  console.log('RELAY=registered');
+}
+
+main().catch((e) => {
+  console.error('[site-relay-register] FAILED: ' + e.message);
+  process.exit(1);
+});

docker-compose.yml

@@ -71,8 +71,19 @@ services:
       # setup.sh (policy sso-broker) — NOT the root token.
       - NODE_ENV=production
       - NODE_PORT=3001
+      # Only a first-run default (site_config.js's envDefaults()) -- a real
+      # join/promote persists its own value to /config/site.json afterward,
+      # which always wins. Derived by setup.sh from CFG_SITE_NAME.
+      - SITE_SLUG=${SITE_SLUG:-}
       - LDAP_SERVER_ID=${LDAP_SERVER_ID:-}
       - LDAP_REPLICATION_HOSTS=${LDAP_REPLICATION_HOSTS:-}
+      # utils/proxy_client.js (no-inbound relay automation) and
+      # utils/jump_client.js (real mesh-gateway count on the Multi-Site
+      # modal) both no-op/skip without these -- neither was ever actually
+      # wired into the compose environment before, so both features were
+      # unreachable in every real deployment despite existing in code.
+      - PROXY_INTERNAL_URL=http://proxy:3000
+      - JUMP_INTERNAL_URL=http://jump-host:3002
       - VAULT_ADDR=http://openbao:8200
       - VAULT_TOKEN=${SSO_VAULT_TOKEN:-}
       # Optional upstream HTTP(S) proxy for outbound calls (SMTP, etc.) at

docs/MULTI_SITE_SPEC.md

@@ -8,8 +8,8 @@
 > ## Shipped today
 > - **Join, live replication, promotion** (`sso-manager-node`): a spoke joins via a one-time export over a site join key (`POST /api/site/join-keys` / `/export` / `/join`), then registers its own endpoint so the master can push live resync pings on every catalog write — no longer a one-time snapshot. Promotion (`POST /api/directory-admin/site-promote`) coordinates a real handoff, demoting the old master as one action. Identical agent-signing keys ride the same export/resync path. Read [`sso-manager-node/docs/site-join.md`](https://github.com/theta42/theta-directory/blob/master/docs/site-join.md) and `directory_spec.md` §11 for the endpoint-level detail.
 > - **Gateway-to-gateway WireGuard mesh** (`theta-gateway`): real site-to-site tunnels via `POST /api/mesh/register`/`/join`, kernel WireGuard with a userspace `wireguard-go` fallback. Verified with an actual two-container encrypted tunnel passing traffic, not a mock.
-> - **Not yet connected to each other**: the mesh is a transport layer that exists on its own; `sso-manager-node`'s HTTPS-based join/replicate calls don't route over it yet. That wiring, plus the no-inbound relay it would enable (mechanism verified, automation not built — see status table), is the next layer.
-> - **mDNS local-discovery, Linux**: shipped and verified end-to-end — `theta-gateway` announces (`services/mdns_announce.js`), `theta-agent` discovers and applies a hosts-file override, cleanly reverts when the announcement disappears. Windows/macOS remain unbuilt — see the TODO list.
+> - **Cross-component routing + no-inbound relay automation**: `sso-manager-node`'s replication traffic now prefers a spoke's mesh IP over the open internet when one is on file (`utils/site_replicate.js`), and a no-inbound spoke's join (`POST /api/site/join` → `/api/site/spokes`) can carry `noInbound`/`meshIp`/`publicHost`, which drives `utils/proxy_client.js` to auto-create the relay route on the master's `theta-proxy` via its existing self-service API token system (reused, not a new credential type). The one piece that stays a manual, out-of-band step is the mesh peering itself (mint a join token on one jump-host, paste it into the other's "Join a mesh" UI) — `theta-suite`'s `bootstrap/site-relay-register.js` (`CFG_SPOKE_NO_INBOUND`/`CFG_SPOKE_PUBLIC_HOST`) picks up from there on the next `setup.sh` run.
+> - **mDNS local-discovery (Linux + Windows)**: shipped and verified — `theta-gateway` announces (`services/mdns_announce.js`), `theta-agent` discovers and applies a hosts-file override, cleanly reverts when the announcement disappears. Linux was verified end-to-end over real multicast; Windows shipped in `theta-agent` v2.2.0 (CRLF-aware hosts override, `ipconfig /flushdns`, and a /32 host-route pin so the WireGuard tunnel can't swallow the direct LAN path). macOS still needs real testing — see the TODO note.
 
 Design scale: a handful of sites (dozen max, 254 hard ceiling — see §4), a few hundred users/hosts total. This is a deliberate, small, trusted-operator deployment, not a hyperscale/adversarial-tenant one — several decisions below (fire-and-forget replication, identical directories) trade blast-radius for simplicity *because* the scale allows it. Don't generalize these choices past that scale without re-deriving them.
 
@@ -256,22 +256,23 @@ See [`AGENT_LOCAL_DISCOVERY_SPEC.md`](./AGENT_LOCAL_DISCOVERY_SPEC.md) — split
 | Join key issuance + one-time directory adoption | **Shipped** — `/api/site/join-keys`, `/api/site/export`, `/api/site/join`, fresh-install-gated (v2.2.0–v2.3.0) |
 | Spoke read-only enforcement | **Shipped** — directory-write routes 403 toward the master once joined (v2.3.0) |
 | WAN health check | **Shipped** — `/api/site/ping`, live in the Master Site modal (v2.2.0–v2.3.0) |
-| `setup.env` / `setup.sh` join wiring | **Shipped** — `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY`, `bootstrap/site-join.js` (theta-suite v2.2.0) |
+| `setup.env` / `setup.sh` join wiring | **Shipped** — `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY`, `bootstrap/site-join.js` (theta-suite v2.2.0). Also readable from a dedicated `spoke.env` (`spoke.env.example`, layered on top of `setup.env`) for operators who want join-a-cluster config kept separate from the rest of first-run setup. |
 | Continuous/live replication (vs. one-time export-on-join) | **Shipped** (`sso-manager-node`) — a spoke registers its own endpoint at join time (`POST /api/site/spokes`), and every successful master catalog write fires a fire-and-forget push (`utils/site_replicate.js`) at every registered spoke, which re-pulls a fresh export. Verified end-to-end in `docker-compose.multisite-e2e.yml`. |
 | Identical-directory signing key | **Shipped** — `POST /api/site/export` includes the master's agent-signing key; a spoke adopts it via `agent_keys.adopt()` on join and every resync. OpenBao secret replication *beyond* this one key is still not built. |
+| OpenLDAP N-way multi-master replication auto-config | **Shipped** — the master auto-assigns each spoke a unique `LDAP_SERVER_ID` at registration (`SiteSpoke.ldapServerId`, same pattern as jump-host's mesh index) and derives every site's LDAP URL from its already-known HTTPS endpoint; `theta-suite`'s `bootstrap/site-ldap-register.js` applies it, re-checked on every `setup.sh` run since the peer list grows as spokes join. Verified against real running containers. Known gap: the master's own config only updates when ITS `setup.sh` is re-run, not live the moment a new spoke joins (see `docs/replication.md`). |
 | Coordinated master promotion (demote the old master as one action) | **Shipped** — `POST /api/site/demote` + `site-promote`'s handoff logic. Fixed two real pre-existing bugs while wiring this in: `site-promote`'s god_admin check read a `req.user.groups` field nothing ever populated (permanently 403'd for everyone), and the read-only write-gate 403'd `site-promote` itself before the handler could run. |
-| WireGuard gateway-to-gateway mesh (`theta-gateway`) | **Shipped** — `POST /api/mesh/register`/`/join` (join-token bootstrap), `utils/wg_iface.js` (kernel WireGuard, falls back to userspace `wireguard-go`). Verified with a real two-container test: actual encrypted tunnel, real ICMP traffic across it, 0% loss. This is the mesh transport layer only — nothing in `sso-manager-node`'s replication yet routes traffic *over* it; today's site-to-site HTTPS calls (join/export/resync) still go over whatever network path already reaches the target, same as before this layer existed. |
-| No-inbound-spoke relay (master proxies a spoke with no public IP) | **Mechanism verified, automation not built.** Confirmed with a standalone test (not `theta-proxy`'s actual Lua/Redis engine, which needs its own dedicated pass to wire safely): a spoke with zero published ports, reachable only via its WG mesh IP, served a request that an external client sent to the master's public port — the master terminated the connection and relayed over the tunnel. So the underlying idea works; what's missing is `theta-proxy` automatically creating that relay route when a no-inbound spoke registers (needs a real service-to-service credential between `sso-manager-node` and `theta-proxy`/`theta-gateway` that doesn't exist yet — a new integration, not a small wiring task), and today's HTTPS-based join/replicate still requires the spoke to reach the master's API directly (and vice versa for export), so a spoke with zero inbound *and* zero outbound path still can't join at all. |
+| WireGuard gateway-to-gateway mesh (`theta-gateway`) | **Shipped** — `POST /api/mesh/register`/`/join` (join-token bootstrap), `utils/wg_iface.js` (kernel WireGuard, falls back to userspace `wireguard-go`). Verified with a real two-container test: actual encrypted tunnel, real ICMP traffic across it, 0% loss. `wg_iface.removePeer()` also cleans up the kernel routes `setPeer()` added (verified live: routes present after `setPeer`, gone after `removePeer`, own local route untouched), and `DELETE /api/mesh/gateways/:id` exposes it from the mesh UI. |
+| Cross-component routing (replication over the mesh) | **Shipped** — `utils/site_replicate.js` tries a registered spoke's `meshIp` first (falling back to its public `endpoint` on failure) when pushing resync pings; a spoke with no `meshIp` on file behaves exactly as before. |
+| No-inbound-spoke relay (master proxies a spoke with no public IP) | **Shipped at the API/automation layer, wired into the real bootstrap flow.** `POST /api/site/join`/`/api/site/spokes` accept `noInbound`/`meshIp`/`publicHost` and call `utils/proxy_client.js`, which mints/reuses a `theta-proxy` self-service API token (`prx_...`, OpenBao `secret/integrations/theta-proxy`) and calls the proxy's real Host API to create or update the relay route — verified against a real running `theta-proxy` container (`GET /api/host/:item`'s actual `{item, results: {...}}` response shape, not the flat shape first assumed). `theta-suite`'s `bootstrap/site-relay-register.js` + `CFG_SPOKE_NO_INBOUND`/`CFG_SPOKE_PUBLIC_HOST` (`setup.env.example`) drive it from the operator-facing bring-up flow, re-run automatically on every `setup.sh` invocation until the jump-host mesh IP is discoverable. What's still a manual step, deliberately: the gateway-to-gateway mesh *peering* itself (mint a join token on one jump-host, paste it into the other's UI) — same pattern as minting/pasting a site join key, not something an unattended script should do blind. A spoke with zero inbound *and* zero outbound path still can't join at all (join/export still need the spoke to reach the master's API directly). |
 | mDNS local-discovery (Linux) | **Shipped** — `theta-gateway` announces (`services/mdns_announce.js`, opt-in via `THETA_LOCAL_DISCOVERY_HOSTS`), `theta-agent` discovers and applies a hosts-file override (`local_discovery.go`, opt-in via `prefer_local_directory`). Verified end-to-end with real containers over real multicast: announce → discover → apply → clean revert on disappearance, all confirmed. Caught two real bugs along the way (`mdns.Lookup()`'s IPv6 query aborting the whole lookup even after a valid IPv4 response arrived; `rename()` failing with EBUSY over a bind-mounted `/etc/hosts`, common in every container runtime) — see the commit messages in `theta-agent`. |
-| mDNS local-discovery (Windows, macOS) | Not built — needs platform-native testing this environment can't do (hosts-file vs. stub-resolver tradeoff, elevation, DNS-cache behavior per OS — see Appendix B §3). This is now the **only unbuilt piece** of the original design. |
+| mDNS local-discovery (Windows) | **Shipped** — `theta-agent` v2.2.0: Windows hosts override (`%SystemRoot%\System32\drivers\etc\hosts`, CRLF-aware, `ipconfig /flushdns` after each change — reachable because the agent runs as a SYSTEM service, so the elevation question resolved in our favor), plus a /32 host-route pin via the owning local interface (`route.exe add ... metric 1`) so the WireGuard mesh tunnel can't swallow the direct LAN path, and a prompt WS reconnect on apply/revert. Tests run the real Windows write path on the Windows CI leg. |
+| mDNS local-discovery (macOS) | Not built — the hosts override compiles on darwin via the shared unix path, but macOS still needs `dscacheutil -flushcache` and real hardware testing (mDNSResponder behavior, hosts-file vs. native Bonjour — see Appendix B §3). Being built on a real macOS VM. |
 
-### TODO — what's actually left, in rough dependency order
+### TODO — what's actually left
 
-1. **mDNS local-discovery, Windows + macOS** — needs platform-native testing this Linux environment cannot do (hosts-file vs. stub-resolver tradeoff, elevation, DNS-cache quirks per OS — see Appendix B §3). Blocked on a Windows/Mac dev environment, not on design. The Linux side (announcer + agent listener) is done and verified — this is the only remaining piece of the original design with no Linux-buildable path forward.
-2. **Route `sso-manager-node`'s HTTPS traffic (join/export/resync) over the WireGuard mesh** instead of the open internet, now that the mesh exists as its own transport layer. Currently the two subsystems don't know about each other.
-3. **`theta-proxy` automation for the no-inbound relay** — mechanism is verified (see status table), but nothing creates the relay route automatically when a no-inbound spoke registers. Needs a new service-to-service credential between `sso-manager-node` and `theta-proxy`/`theta-gateway` — a real design decision (who mints it, what it authorizes), not just wiring.
-4. **OpenBao secret replication beyond the one agent-signing key** — LDAP admin creds, JWT secret, other per-deployment secrets that currently differ per site.
-5. **`theta-proxy`/`theta-gateway` service-to-service auth model in general** — items 2 and 3 both need it; worth designing once rather than inventing a credential per integration.
-6. **Mesh peer removal cleanup** — `wg_iface.removePeer()` doesn't remove the kernel routes `setPeer()` adds (flagged in code, not yet exercised because nothing removes a mesh peer today).
+1. **Full secret replication** — only the agent-signing key is replicated today. LDAP admin credentials, JWT secrets, and other per-deployment secrets still differ per site, which complicates full disaster recovery. **Paused pending a real-deployment question independent of the code**: this repo's own `conf/secrets.js` was found to contain committed real credentials during this work (LDAP bind, SMTP, VoIP.ms) — see the git-remediation note elsewhere in this repo's history. Building a feature that copies live secrets to additional sites shouldn't proceed until provider-side rotation of those specific credentials is confirmed done; the mechanism itself (generic secret sync, never touching those particular values) can still be designed without that answer.
+2. Service-to-service auth, cross-component routing, no-inbound relay automation, and mesh peer cleanup (the four items formerly listed here) are **done** — see the status table above. What remains genuinely open in that area is documented there inline (mesh peering stays a manual step by design; zero-inbound-and-zero-outbound spokes still can't join).
+
+**mDNS local-discovery, macOS** is deliberately not listed above: the Linux and Windows sides are shipped and verified (`theta-agent` v2.2.0), and macOS is being built on a real macOS VM where the darwin-specific behavior (mDNSResponder/DNS-cache) can actually be tested. Check `theta-agent`'s recent history before assuming it's still open.
 
 *Committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*

docs/index.md

@@ -10,8 +10,9 @@ Theta Suite is your one-line solution to replacing fragmented, hard-to-wire
 authentication setups with a unified security stack. It wires together OIDC
 authentication, LDAP user directories, automated host enrollment, and
 centralized secret management in a single command. It eliminates the manual
-configuration friction so you get secure access, auditability, and multi-site
-replication running in seconds.
+configuration friction so you get secure access, auditability, and
+[multi-site](sso/multi-site.html) replication when you need more than one
+location.
 
 ##  Who This Is For
 * **Self-Hosters & Homelab Engineers:** Anyone running local bare metal,
@@ -24,8 +25,8 @@ replication running in seconds.
   lock-in.
 * **DevOps & Systems Operators:** Engineers who value idempotent, single-command
   deployments (`./setup.sh`) and need a production-grade baseline supporting
-  zero-trust proxying, SSH jump-host access control, and multi-site replication
-  out of the box.
+  zero-trust proxying, SSH jump-host access control, and
+  [multi-site](sso/multi-site.html) replication out of the box.
 
 ## Screenshots
 

docs/jump-host/index.md

@@ -84,7 +84,7 @@ Theta Gateway answers both from your directory:
   inventory, not a static list
 - **Per-user key injection** — no downstream changes, no key distribution
 - **Shell, exec, and SFTP** bridging
-- **WireGuard mesh routing** — cross-site network access alongside SSH
+- **[WireGuard mesh routing](mesh.html)** — cross-site network access alongside SSH
 - **Web UI + HTTP API** for auditing and metrics — active sessions, a searchable
   audit log, per-user/per-host counters
 - **Full audit trail** — who, target, method, result, bytes, duration, and the

docs/jump-host/mesh.md

@@ -0,0 +1,64 @@
+---
+layout: default
+title: Gateway Mesh
+---
+
+# Gateway Mesh
+
+Theta Gateway can mesh with other Theta Gateway instances over real
+site-to-site WireGuard tunnels — separate from its [SSH jump
+host](connecting.html) role, and separate from the roaming-client/exit-node
+WireGuard feature (individual peer configs for laptops/phones). This is
+gateway-to-gateway: two sites' networks reaching each other directly.
+
+## Why and when to use this
+
+- **Direct site-to-site networking**, not just SSH. Once two gateways are
+  meshed, hosts behind each can reach each other over the tunnel using the
+  mesh addressing scheme below — not limited to jumping through SSH.
+- **No manual WireGuard config.** Meshing is a join-token exchange; both
+  sides come out with a live, working peer entry for each other
+  automatically.
+- **Works without a kernel WireGuard module.** Prefers in-kernel WireGuard,
+  falls back to the userspace `wireguard-go` implementation automatically —
+  useful for older kernels, some container/cloud images, or hosts where the
+  kernel module isn't available.
+
+## How it works
+
+1. On the gateway you want others to join, mint a join token: **Mesh** page
+   → **Mint a Join Token**. It's single-use and expires in 15 minutes.
+2. On the new gateway, use **Join a Remote Gateway's Mesh**: paste the other
+   gateway's URL and the token.
+3. Both sides now have a live WireGuard peer for each other. The **Meshed
+   Gateways** table shows every peer, its assigned mesh subnet, and when it
+   was last seen.
+
+Each gateway is assigned a **mesh index** (an integer 1–254) the first time
+it either mints a token or is registered by another gateway. That index
+determines its subnet: `172.24.<index>.0/24` for the mesh tunnel itself, plus
+`10.<index>.0.0/16` reserved for that site's own local network — 254 sites is
+the hard ceiling this addressing scheme supports.
+
+## Requirements
+
+- Both gateways need a reachable endpoint (host:port) for the WireGuard
+  handshake — typically the same public host the SSH/web ports are already
+  on, with UDP 51820 reachable.
+- `NET_ADMIN` capability (or equivalent) on the container/host running the
+  gateway, to create the WireGuard interface.
+
+## Connected to directory sync
+
+[Theta Directory's multi-site join](../sso/multi-site.html) (catalog + LDAP
+replication between a master and its spokes) prefers this mesh once it's up:
+a spoke that's registered a mesh IP gets its live resync pushes routed over
+the tunnel instead of the open internet, falling back to its public endpoint
+if the mesh path fails. A spoke with no public IP at all can also register as
+no-inbound (`CFG_SPOKE_NO_INBOUND` in `theta-suite`'s `setup.env`) so the
+master auto-creates a relay route through its own `theta-proxy` — the master
+terminates TLS for that spoke's hostname and relays over this mesh. The mesh
+peering itself (this page) stays a manual step on both sides; directory join
+and relay registration pick up from there. See the [architecture
+spec](https://github.com/theta42/theta-suite/blob/master/docs/MULTI_SITE_SPEC.md)
+for the full detail.

docs/sso/index.md

@@ -45,7 +45,8 @@ stack with one command.
 - **Direct LDAP binds** — anything that binds LDAP directly (Linux hosts
   via PAM/SSSD, Gitea, Emby, …) uses LDAPS/StartTLS against the same
   directory.
-- **Geo-Location Scaling** — built-in support for N-Way Multi-Master OpenLDAP [replication](replication.html) across physical sites.
+- **[Multi-Site](multi-site.html)** — one master site, any number of read-only spokes that join with a single key and stay live-synced, with god_admin-gated promotion if the master goes down for good.
+- **Geo-Location Scaling** — built-in support for N-Way Multi-Master OpenLDAP [replication](replication.html) across physical sites (a different, lower-level mechanism — see [Multi-Site](multi-site.html) for how the two compare).
 - **[Directory & Inventory](directory.html)** — map sites, hosts, and services as a graph with rich metadata (IP/MAC, OS/kernel, ports, git repos), auto-provisioned access groups, and automatic registration from theta-suite's agents and discovery plugins. Drives directory-aware tools like the [SSH jump host](../jump-host/).
 - **Subtype metrics & lifecycle drivers** — telemetry, log streaming, and remote control for resources tagged with a `subType` (`systemd`, `docker`, `proxmox`, `wireguard`, `postgresql`, `redis`, `k8s`, …).
 - **OpenBao-backed secrets** — per-resource and per-user secrets with explicit upward inheritance (`Resource → Host → Cluster → Site`).

docs/sso/multi-site.md

@@ -0,0 +1,114 @@
+---
+layout: default
+title: Multi-Site (Master/Spoke Join)
+---
+
+# Multi-Site (Master/Spoke Join)
+
+If you run more than one physical site, Theta Directory can run one site as
+the **master** (single write authority for the shared catalog) and any
+number of **spokes** — read-only replicas that stay in sync automatically and
+run local authentication with zero WAN dependency.
+
+This is a different, higher-level mechanism than [raw LDAP N-way
+replication](replication.html) — see [How this relates to LDAP
+replication](#how-this-relates-to-ldap-replication) below if you're deciding
+between the two.
+
+## Why and when to use this
+
+- **Zero-touch spoke setup.** One join key, one URL, and a spoke adopts the
+  whole directory (users, groups, resource catalog) in one step — no manual
+  `syncrepl` configuration.
+- **Single write authority, no split-brain.** Only the master accepts
+  directory writes. A spoke that loses WAN connectivity keeps working for
+  local reads/auth and unconditionally stays read-only — it never silently
+  promotes itself. Changing which site is master always requires an explicit,
+  authenticated action by a `god_admin`.
+- **Stays in sync, not just a one-time copy.** Once joined, a spoke keeps
+  receiving live updates whenever the master's catalog changes — you don't
+  re-run the join to pick up new hosts/apps/users.
+
+## How it works
+
+1. **On the master**, an admin mints a **site join key** (Directory → the
+   Master Site modal → **Site Join Keys** → Mint key). It's shown once,
+   stored hashed, and revocable.
+2. **On the spoke** (must be a fresh install — no users beyond the bootstrap
+   admin, no enrolled agents), either:
+   - Paste the master's URL and the join key into the Master Site modal's
+     **Join an Existing Site** form, or
+   - Set `CFG_MASTER_DIRECTORY_URL` / `CFG_MASTER_DIRECTORY_JOIN_KEY` before
+     the first `./setup.sh` run -- either in `setup.env` (which has every
+     option), or in a dedicated `spoke.env` (`cp spoke.env.example spoke.env`)
+     if you'd rather keep join-a-cluster config separate from the rest of the
+     stack's setup. Both are read; `spoke.env`'s values win on a conflict.
+     No public IP on this site at all? `spoke.env.example` also covers the
+     no-inbound relay vars (`CFG_SPOKE_NO_INBOUND`/`CFG_SPOKE_PUBLIC_HOST`).
+3. The spoke pulls the master's full export (LDAP tree, resource catalog,
+   agent-signing key) and adopts it, then registers its own reachable URL
+   with the master so it can receive live updates going forward.
+4. From then on, every change to the master's catalog pushes to every
+   registered spoke automatically. A spoke's own directory-write requests are
+   rejected with a `403` pointing at the master — writes always go there.
+
+### Promoting a spoke to master
+
+If the master site goes down for good (or you're relocating write
+authority), a `god_admin` can promote any spoke from its own Master Site
+modal. Promotion is one coordinated action: it demotes the previous master as
+part of the same request (best-effort — an unreachable old master never
+blocks the promotion, since that's exactly the scenario this exists for), and
+every other spoke gets pointed at the new master automatically.
+
+## What replicates
+
+| Data | How |
+|---|---|
+| LDAP (users, groups) | Full export on join; live push on every master change |
+| Resource catalog (hosts, apps, sites) | Same |
+| Agent-signing key | Same — every site can validly sign a command for any agent enrolled at *any* site |
+
+The agent-signing key being identical everywhere is a deliberate tradeoff for
+small, trusted deployments (a handful of sites, not hundreds) — it means
+compromising the least-secured spoke has the same agent-command blast radius
+as compromising the master. If that tradeoff doesn't fit your deployment,
+don't rely on this mechanism as-is.
+
+Secrets *beyond* the agent-signing key (LDAP admin password, JWT secret, and
+so on) are **not** currently synced — each site still generates its own.
+
+## Requirements and current limits
+
+- Both sites need a network path to each other's HTTP(S) API — the master to
+  pull an export from, the spoke to push replication updates back to. A site
+  with no inbound path at all (e.g. behind CGNAT) can't join yet on its own;
+  a relay mechanism for that case is designed but not automated (see the
+  [architecture spec](https://github.com/theta42/theta-suite/blob/master/docs/MULTI_SITE_SPEC.md)
+  for the current status).
+- Joining only ever happens on a **fresh install**. There's no way to merge
+  an already-populated directory into a master's — re-provision the host
+  first.
+
+## How this relates to LDAP replication
+
+[N-way LDAP replication](replication.html) is a *lower-level*, different
+mechanism: every site runs a fully independent, fully writable `slapd`, wired
+together with raw `syncrepl` environment variables, and there's no concept of
+a master or a managed join. It predates this feature and is still there for
+deployments that specifically want every site independently writable.
+
+Multi-site join (this page) is the opposite design: one write authority, a
+managed onboarding flow, and automatic ongoing sync — closer to what most
+"add a second office" or "add a home-lab spoke" setups actually want. **Don't
+combine the two** — pick one per deployment.
+
+## See also
+
+- Full architecture and current implementation status:
+  [`MULTI_SITE_SPEC.md`](https://github.com/theta42/theta-suite/blob/master/docs/MULTI_SITE_SPEC.md)
+  in the `theta-suite` repo.
+- Endpoint-level detail: [`docs/site-join.md`](https://github.com/theta42/theta-directory/blob/master/docs/site-join.md)
+  in the `theta-directory` repo.
+- Site-to-site networking (WireGuard mesh between gateways, independent of
+  directory sync): [Theta Gateway → Mesh](../jump-host/mesh.html).

setup.env.example

@@ -61,9 +61,47 @@ CFG_DOMAIN=example.com
 # being a one-time snapshot -- the master must be able to reach THIS site's
 # CFG_SSO_HOST for that part to work; if it can't (this site has no inbound
 # path), the join still succeeds, it just never receives live updates.
+#
+# All of these (and the no-inbound relay pair below) also live in their own
+# spoke.env.example, if you'd rather keep join-a-cluster config in a
+# dedicated file instead of here -- both are read, spoke.env's values win.
 #CFG_MASTER_DIRECTORY_URL=https://sso.master.example.com
 #CFG_MASTER_DIRECTORY_JOIN_KEY=stj_9f2e...
 
+# This site's own public web domain, independent of CFG_DOMAIN above (the
+# shared LDAP identity namespace, which must be identical across every site).
+# Optional -- only meaningful for an inbound spoke/standalone site that wants
+# its own domain rather than sharing the master's.
+#CFG_PUBLIC_DOMAIN=branch2.example.com
+
+# No public IP at all (CGNAT, etc.)? The master can still reach this spoke by
+# relaying over the gateway-to-gateway WireGuard mesh instead of the open
+# internet (MULTI_SITE_SPEC.md §5.2) -- but the mesh peering itself is a
+# manual, out-of-band step on BOTH jump-hosts (mint a mesh join token on the
+# master's jump-host, paste it into this site's jump-host "Join a mesh" UI
+# action) that can't run unattended inside this script. Once that's done, set
+# these two and re-run setup.sh: it discovers this jump-host's assigned mesh
+# IP (GET /api/mesh/self) and registers it with the master, which then
+# auto-creates the relay route on its own theta-proxy. Safe to leave set
+# before meshing -- setup.sh just reports "not meshed yet" and skips until a
+# later re-run finds the mesh IP.
+#CFG_SPOKE_NO_INBOUND=true
+#CFG_SPOKE_PUBLIC_HOST=sso-branch2.master-domain.example.com
+
+# Two service-to-service integrations the Directory uses (both reuse each
+# app's existing self-service API token system -- see MULTI_SITE_SPEC.md's
+# "service-to-service auth" note -- not a new credential type each):
+#   - No-inbound relay automation (above) needs a theta-proxy API token so
+#     sso-manager can create/update the relay Host route on its own.
+#   - The Multi-Site modal's real gateway-mesh count needs a jump-host API
+#     token (minted by a jump-admin user) to read GET /api/mesh/gateways.
+# Neither is required for the rest of the stack to work -- both features
+# just report "not configured" until you mint a token in each app's own web
+# UI (Settings -> API Tokens) and store it in OpenBao, from inside the
+# sso-manager container (VAULT_ADDR/VAULT_TOKEN are already set there):
+#   docker compose exec sso-manager node -e "require('@simpleworkjs/bao-conf').set('integrations/theta-proxy', {token: 'prx_...'})"
+#   docker compose exec sso-manager node -e "require('@simpleworkjs/bao-conf').set('integrations/theta-jump', {token: 'jmp_...'})"
+
 # ── Optional outbound HTTP(S) proxy ──────────────────────────────────────────
 # For an isolated/offline/corporate-network test host that only reaches the
 # internet through an upstream HTTP proxy — NOT the theta42 "proxy" app.
@@ -114,14 +152,20 @@ CFG_DOMAIN=example.com
 #CFG_THETA_AGENT_FULL_CONTROL=1
 
 # ── Geo-Location Scaling (N-Way Multi-Master LDAP) ───────────────────────────
-# If deploying this stack across multiple physical sites to provide local HA
-# for directory services, you can enable N-Way Multi-Master OpenLDAP replication.
-# This requires assigning a unique ID to each site and listing the LDAPS URLs
-# of all OTHER sites in the cluster.
+# If you're joining a directory cluster (CFG_MASTER_DIRECTORY_URL/spoke.env
+# above), N-Way Multi-Master OpenLDAP replication is configured for you
+# automatically -- setup.sh's bootstrap/site-ldap-register.js asks the master
+# for a unique LDAP_SERVER_ID and the current list of every other site's LDAP
+# URL on every run (see docs/replication.md), restarting sso-manager only
+# when that config actually changed. Nothing to set here for the common case.
 #
-# Each site MUST have a unique LDAP_SERVER_ID (e.g. 1, 2, 3).
-# LDAP_REPLICATION_HOSTS is a space-separated list of the other sites' LDAP URLs.
-# Example for Site 1:
+# Have a manually-coordinated LDAP MMR topology this script can't derive on
+# its own (e.g. peers outside this theta-suite cluster)? Set
+# CFG_LDAP_MMR_MANUAL=true to skip the automatic step entirely and set
+# LDAP_SERVER_ID/LDAP_REPLICATION_HOSTS directly -- without this, the
+# automatic step runs on every deployment (every fresh install starts as a
+# master) and will overwrite them.
+#CFG_LDAP_MMR_MANUAL=true
 #LDAP_SERVER_ID=1
 #LDAP_REPLICATION_HOSTS="ldaps://sso.site2.com:636 ldaps://sso.site3.com:636"
 # ── Proxy HTTP/HTTPS Defaults ────────────────────────────────────────────────

setup.sh

@@ -200,6 +200,10 @@ fi
 # later steps can use it. The authoritative CFG_* for secrets are still
 # resolved in ensure_config; this is only the hostname override.
 [[ -f ./setup.env ]] && parse_kv_file ./setup.env
+# spoke.env (optional, see spoke.env.example): the join-a-cluster vars split
+# out of setup.env for clarity, layered on top so its values win over any
+# same-named ones in setup.env. Same first-run-only rule as setup.env below.
+[[ -f ./spoke.env ]] && parse_kv_file ./spoke.env
 export CFG_JUMP_HOST
 CFG_CREATE_ALL_HTTP="${CFG_CREATE_ALL_HTTP:-0}"
 export CFG_CREATE_ALL_HTTP
@@ -467,6 +471,10 @@ BAOEOF
 		info "Reading domain/hosts from ./setup.env ..."
 		parse_kv_file ./setup.env
 	fi
+	if [[ -f ./spoke.env ]]; then
+		info "Reading multi-site join config from ./spoke.env ..."
+		parse_kv_file ./spoke.env
+	fi
 
 	# Bind the CFG_* vars to empty where setup.env / the environment didn't set
 	# them, so the .env migration's `${LDAP_X:-$CFG_X}` defaults below don't trip
@@ -540,9 +548,25 @@ BAOEOF
 	[[ -n "$CFG_DOMAIN" ]] \
 		|| die "First run: 'cp setup.env.example setup.env', set CFG_DOMAIN to your domain (e.g. example.com), then re-run ./setup.sh"
 	CFG_BASE_DN="${CFG_BASE_DN:-$(dn_from_domain "$CFG_DOMAIN")}"
-	CFG_SSO_HOST="${CFG_SSO_HOST:-sso.$CFG_DOMAIN}"
-	CFG_PROXY_HOST="${CFG_PROXY_HOST:-proxy.$CFG_DOMAIN}"
+	# CFG_PUBLIC_DOMAIN (MULTI_SITE_SPEC.md §4): an inbound spoke's own public
+	# web domain, independent of CFG_DOMAIN. CFG_DOMAIN is the LDAP identity
+	# namespace and MUST be identical across every site (MMR replicas can't
+	# diverge on base DN) -- CFG_PUBLIC_DOMAIN only changes where the web
+	# hostnames point, never the DN. Unset (the default): behaves exactly as
+	# before, hostnames derive from CFG_DOMAIN like any standalone install.
+	CFG_SSO_HOST="${CFG_SSO_HOST:-sso.${CFG_PUBLIC_DOMAIN:-$CFG_DOMAIN}}"
+	CFG_PROXY_HOST="${CFG_PROXY_HOST:-proxy.${CFG_PUBLIC_DOMAIN:-$CFG_DOMAIN}}"
 	CFG_SITE_NAME="${CFG_SITE_NAME:-local}"
+	# Multi-site identity (site_config.js's `siteSlug`, shown on the Directory's
+	# Multi-Site modal) -- without this it's never set anywhere and every fresh
+	# master shows the module's own literal fallback, "site-default", forever.
+	# Derived from CFG_SITE_NAME with the same slugify rule bootstrap.js uses
+	# for the site Resource's own slug (site_$(slugify), underscore prefix --
+	# this is hyphenated to match site_config.js's own "site-default" format).
+	# site.json overrides this after first bring-up (join/promote write real
+	# values there), so this only ever matters for a fresh install.
+	SITE_SLUG="site-$(echo "$CFG_SITE_NAME" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-+|-+$//g')"
+	export SITE_SLUG
 	CFG_ORG="${CFG_ORG:-Theta Directory}"
 	CFG_ADMIN_UID="${CFG_ADMIN_UID:-admin}"
 	CFG_ADMIN_EMAIL="${CFG_ADMIN_EMAIL:-admin@$CFG_PROXY_HOST}"
@@ -1036,6 +1060,15 @@ info "Starting bao-renewer (service-token renewal sidecar)..."
 SSO_GIT_COMMIT="$(git -C sso-manager-node rev-parse --short HEAD 2>/dev/null || echo unknown)"
 export SSO_GIT_COMMIT
 env_upsert SSO_GIT_COMMIT "$SSO_GIT_COMMIT"
+# OpenLDAP multi-master replication (docs/replication.md, auto-configured --
+# see step 7e below and bootstrap/site-ldap-register.js): pick up whatever
+# LDAP_SERVER_ID/LDAP_REPLICATION_HOSTS a PRIOR run already computed, so a
+# restart doesn't silently drop back to standalone (no LDAP_SERVER_ID env at
+# all). A truly fresh install has no file yet -- that's fine, it just starts
+# standalone until step 7e computes and applies real values. Skipped under
+# CFG_LDAP_MMR_MANUAL=true so a manually hand-set LDAP_SERVER_ID/
+# LDAP_REPLICATION_HOSTS in setup.env isn't clobbered by a stale auto file.
+[[ "${CFG_LDAP_MMR_MANUAL:-false}" != "true" && -f "$CONFIG_DIR/ldap-replication.env" ]] && parse_kv_file "$CONFIG_DIR/ldap-replication.env"
 info "Building + starting sso-manager (first run builds the image; this takes a while)..."
 "${COMPOSE[@]}" up -d --build sso-manager
 
@@ -1279,6 +1312,59 @@ NODEEOF
 )
 echo "$JUMP_HOSTS_OUT" | sed 's/^/[setup] /'
 
+# ── 7b2. No-inbound relay registration (first-run *and* every re-run) ─────────
+# CFG_SPOKE_NO_INBOUND: this site has no public IP, so the master relays to it
+# over the gateway-to-gateway WireGuard mesh (MULTI_SITE_SPEC.md §5.2). The
+# mesh peering itself is a manual, out-of-band step on both jump-hosts (mint a
+# join token on the master's jump-host, paste it into this site's jump-host
+# "Join a mesh" UI action) -- it can't run unattended here, and it commonly
+# happens AFTER this first setup.sh run finishes. So this step runs on every
+# invocation, not just first-run: it discovers this jump-host's mesh IP and
+# (re-)registers it with the master, and is a no-op until meshing is done.
+if [[ "${CFG_SPOKE_NO_INBOUND:-false}" == "true" ]]; then
+	if [[ -z "${CFG_SPOKE_PUBLIC_HOST:-}" ]]; then
+		warn "CFG_SPOKE_NO_INBOUND=true but CFG_SPOKE_PUBLIC_HOST is unset — skipping relay registration."
+	else
+		info "Checking no-inbound relay registration (CFG_SPOKE_PUBLIC_HOST=${CFG_SPOKE_PUBLIC_HOST})..."
+		"${COMPOSE[@]}" exec -T sso-manager node /bootstrap/site-relay-register.js \
+			"https://$CFG_SSO_HOST" "$CFG_SPOKE_PUBLIC_HOST" || warn "relay registration did not complete — check: ${COMPOSE[*]} exec sso-manager node /bootstrap/site-relay-register.js https://$CFG_SSO_HOST $CFG_SPOKE_PUBLIC_HOST"
+	fi
+fi
+
+# ── 7e. OpenLDAP multi-master replication auto-config (every run) ────────────
+# docs/replication.md: the master assigns each spoke a unique LDAP_SERVER_ID
+# and derives every site's LDAP URL automatically (bootstrap/
+# site-ldap-register.js) instead of an operator hand-maintaining
+# LDAP_SERVER_ID/LDAP_REPLICATION_HOSTS. Runs on every invocation -- both
+# master (its peer list grows as spokes join) and spoke -- and restarts
+# sso-manager only when the computed config actually changed, since
+# OpenLDAP's static slapd.conf is only read at process start.
+#
+# CFG_LDAP_MMR_MANUAL=true skips this entirely -- every fresh install starts
+# as a master, so without this escape hatch an operator's own hand-set
+# LDAP_SERVER_ID/LDAP_REPLICATION_HOSTS (a topology outside this theta-suite
+# cluster this script can't derive) would get silently overwritten.
+if [[ "${CFG_LDAP_MMR_MANUAL:-false}" == "true" ]]; then
+	info "CFG_LDAP_MMR_MANUAL=true — skipping automatic LDAP replication config."
+	LDAP_REG_OUT=""
+else
+LDAP_REG_OUT=$("${COMPOSE[@]}" exec -T sso-manager node /bootstrap/site-ldap-register.js "https://$CFG_SSO_HOST" 2>&1) || warn "LDAP replication config check failed — check: ${COMPOSE[*]} exec sso-manager node /bootstrap/site-ldap-register.js https://$CFG_SSO_HOST"
+echo "$LDAP_REG_OUT" | sed 's/^/[setup] /'
+fi
+if echo "$LDAP_REG_OUT" | grep -q '^LDAP_CONFIG_CHANGED=yes'; then
+	info "LDAP replication config changed — restarting sso-manager to apply it..."
+	[[ -f "$CONFIG_DIR/ldap-replication.env" ]] && parse_kv_file "$CONFIG_DIR/ldap-replication.env"
+	"${COMPOSE[@]}" up -d --force-recreate sso-manager
+	info "Waiting for sso-manager to be healthy again..."
+	for i in $(seq 1 60); do
+		if docker exec sso-manager wget -q -O- http://localhost:3001/health >/dev/null 2>&1; then
+			info "sso-manager is healthy."; break
+		fi
+		if (( i == 60 )); then warn "sso-manager did not become healthy in 120s after the LDAP config restart. Check: ${COMPOSE[*]} logs sso-manager"; break; fi
+		sleep 2
+	done
+fi
+
 # ── 7c. Install theta-agent on the host ──────────────────────────────────────
 # Controlled by CFG_THETA_AGENT_ENABLE (default: 1 = enabled)
 CFG_THETA_AGENT_ENABLE="${CFG_THETA_AGENT_ENABLE:-1}"
@@ -1286,16 +1372,28 @@ if [[ "$CFG_THETA_AGENT_ENABLE" == "1" ]]; then
 	info "Setting up theta-agent on the host..."
 	(
 		cd theta-agent || exit 0
-		# Install the prebuilt binary that ships in the theta-agent submodule (the
-		# repo's own install.sh uses the same release binary). We do NOT build from
-		# source here: a previous `go build -o theta-agent main.go websocket.go
-		# config.go` omitted executor.go/telemetry.go, failed to compile, and was
-		# silently skipped, so the agent was never installed.
-		if [[ ! -f "theta-agent-linux-amd64" ]]; then
-			warn "Prebuilt theta-agent-linux-amd64 missing from the theta-agent submodule. Skipping theta-agent installation."
+		# Download the current release binary from GitHub rather than trusting a
+		# binary committed in the submodule checkout. A committed binary drifts:
+		# theta-agent's own `theta-agent update` moved to pulling from GitHub
+		# Releases (DESIGN-WINDOWS.md §9, "nothing binary lives in the repos")
+		# once, but this script kept installing the stale binary that shipped
+		# with an old submodule pin, which still pointed `update` at a dead
+		# SSO /resources/ URL that never existed server-side -- so an agent
+		# installed this way could never even self-update out of the bug. We
+		# do NOT build from source here either: a previous `go build -o
+		# theta-agent main.go websocket.go config.go` omitted
+		# executor.go/telemetry.go, failed to compile, and was silently
+		# skipped, so the agent was never installed.
+		AGENT_BIN_URL="https://github.com/theta42/theta-agent/releases/latest/download/theta-agent-linux-amd64"
+		AGENT_BIN_TMP="$(mktemp)"
+		info "  Downloading latest theta-agent-linux-amd64 release binary..."
+		if ! curl -fsSL -o "$AGENT_BIN_TMP" "$AGENT_BIN_URL" || [[ ! -s "$AGENT_BIN_TMP" ]]; then
+			rm -f "$AGENT_BIN_TMP"
+			warn "Could not download theta-agent-linux-amd64 from $AGENT_BIN_URL. Skipping theta-agent installation."
 		else
-			info "  Installing prebuilt theta-agent binary..."
-			if [[ -x "theta-agent-linux-amd64" ]]; then
+			chmod +x "$AGENT_BIN_TMP"
+			info "  Installing theta-agent binary..."
+			if true; then
 				# The agent binary reads /etc/theta42/agent.yml (theta-agent/main.go).
 				sudo mkdir -p /etc/theta42
 				if [[ ! -f /etc/theta42/agent.yml ]]; then
@@ -1326,7 +1424,12 @@ if [[ "$CFG_THETA_AGENT_ENABLE" == "1" ]]; then
 					else
 						warn "No agent join key available — /etc/theta42/agent.yml has no credential and the agent will not connect."
 					fi
-					# We want to connect to either https or http depending on CFG_CREATE_ALL_HTTP
+					# We want to connect to either https or http depending on CFG_CREATE_ALL_HTTP.
+					# Without this, agent.yml keeps agent.yml.example's literal
+					# "https://sso.example.com" placeholder forever -- nothing
+					# else in this block ever touched server_url, only join_key.
+					AGENT_SCHEME="https"; [[ "${CFG_CREATE_ALL_HTTP:-0}" == "1" ]] && AGENT_SCHEME="http"
+					sudo sed -i "s|^server_url:.*|server_url: \"${AGENT_SCHEME}://${CFG_SSO_HOST}\"|" /etc/theta42/agent.yml
 					sudo getent group theta-secrets >/dev/null 2>&1 || sudo groupadd -r theta-secrets 2>/dev/null || true
 					sudo getent group theta >/dev/null 2>&1 || sudo groupadd -r theta 2>/dev/null || true
 					SECRETS_GRP="root"
@@ -1334,12 +1437,25 @@ if [[ "$CFG_THETA_AGENT_ENABLE" == "1" ]]; then
 					sudo chown -R "root:$SECRETS_GRP" /etc/theta42 2>/dev/null || true
 					sudo chmod 750 /etc/theta42
 					sudo chmod 640 /etc/theta42/agent.yml
+				else
+					# Self-heal an already-installed agent.yml that predates the
+					# server_url fix above -- it would otherwise keep whatever
+					# placeholder/stale host it was first installed with
+					# forever, since nothing else in this script ever revisits
+					# an existing agent.yml. Never touches join_key/auth_token:
+					# those may since have been rewritten by the agent itself
+					# with real issued credentials.
+					AGENT_SCHEME="https"; [[ "${CFG_CREATE_ALL_HTTP:-0}" == "1" ]] && AGENT_SCHEME="http"
+					if sudo grep -q '^server_url:' /etc/theta42/agent.yml; then
+						sudo sed -i "s|^server_url:.*|server_url: \"${AGENT_SCHEME}://${CFG_SSO_HOST}\"|" /etc/theta42/agent.yml
+					fi
 				fi
 				# Stop a running agent before overwriting its binary (cp into a
 				# running executable fails with "Text file busy" on a re-install).
 				sudo systemctl stop theta-agent.service 2>/dev/null || true
-				sudo cp theta-agent-linux-amd64 /usr/local/bin/theta-agent
+				sudo cp "$AGENT_BIN_TMP" /usr/local/bin/theta-agent
 				sudo chmod +x /usr/local/bin/theta-agent
+				rm -f "$AGENT_BIN_TMP"
 
 				# Install desktop tray companion if available
 				TRAY_SRC="dist/theta-agent-tray-linux-amd64"

spoke.env.example

@@ -0,0 +1,54 @@
+# ─────────────────────────────────────────────────────────────────────────────
+# spoke.env — join this stack to an existing Theta Directory as a read-only
+# spoke, instead of seeding a fresh master (MULTI_SITE_SPEC.md).
+#
+# This is the ONE place the join-a-cluster vars live -- split out of
+# setup.env.example (which still has every option, including these, for a
+# single-file bring-up) purely for clarity: standing up a spoke is a distinct
+# operation from configuring a fresh install, so it gets its own small file
+# instead of being buried among unrelated options. Set what you need here;
+# everything else (domain, admin creds, SMTP, ...) still comes from setup.env
+# as normal -- copy setup.env.example too and fill in CFG_DOMAIN there first.
+#
+# Same first-run-only rule as setup.env: read once (layered on top of
+# setup.env, so a var set in both places takes this file's value), then
+# ignored once ./config/ exists -- an already-running directory can never be
+# merged into a master's this way. The one exception is the no-inbound relay
+# vars at the bottom, which setup.sh re-checks on every run (see their
+# comment) since mesh peering usually finishes after the first bring-up.
+#
+#   cp setup.env.example setup.env   # if you haven't already -- set CFG_DOMAIN
+#   cp spoke.env.example spoke.env
+#   $EDITOR spoke.env                # set CFG_MASTER_DIRECTORY_URL + _JOIN_KEY below
+#   ./setup.sh
+#
+# Copying this file to spoke.env (gitignored) keeps your join key out of git.
+# ─────────────────────────────────────────────────────────────────────────────
+
+# The master's URL and a site join key. Mint a key on the master:
+# Directory -> the Master Site modal -> Site Join Keys -> Mint key.
+# Both required to join; if either is unset this stack seeds a fresh master
+# instead (setup.env.example's normal behavior).
+CFG_MASTER_DIRECTORY_URL=https://sso.master.example.com
+CFG_MASTER_DIRECTORY_JOIN_KEY=stj_9f2e...
+
+# This spoke's own public web domain, if it needs one independent of the
+# master's (an inbound spoke serving its own traffic directly -- see
+# CFG_SPOKE_NO_INBOUND below for the opposite case). Optional: CFG_DOMAIN
+# (in setup.env) is the shared LDAP identity namespace and must be identical
+# across every site in the cluster -- this only changes where THIS site's own
+# web hostnames (sso.*, proxy.*) point, never the LDAP base DN.
+#CFG_PUBLIC_DOMAIN=branch2.example.com
+
+# No public IP at all (CGNAT, etc.)? The master can still reach this spoke by
+# relaying over the gateway-to-gateway WireGuard mesh instead of the open
+# internet (MULTI_SITE_SPEC.md §5.2) -- but the mesh peering itself is a
+# manual, out-of-band step on BOTH jump-hosts (mint a mesh join token on the
+# master's jump-host, paste it into this site's jump-host "Join a mesh" UI
+# action) that can't run unattended inside this script. Once that's done, set
+# these two and re-run setup.sh: it discovers this jump-host's assigned mesh
+# IP and registers it with the master, which then auto-creates the relay
+# route on its own theta-proxy. Safe to leave set before meshing -- setup.sh
+# just reports "not meshed yet" and skips until a later re-run finds the IP.
+#CFG_SPOKE_NO_INBOUND=true
+#CFG_SPOKE_PUBLIC_HOST=sso-branch2.master-domain.example.com