Merge pull request #197 from theta42/release-v2.4.0

release(v2.4.0): live replication, coordinated promotion, mesh UI
2026-08-10 16:03:15 -07:00
parent e915a17cbd daefe54ff7
commit a50d1ef3c8
15 changed files with 1026 additions and 74 deletions
@@ -1,3 +1,17 @@
 # v2.4.0 - 2026-08-10
 ### Added
 - **Live catalog replication.** A spoke now stays in sync after joining instead of only getting a one-time snapshot: it registers its own endpoint with the master at join time (`POST /api/site/spokes`, Bearer the site join key), and every successful master catalog write fires a fire-and-forget push (`utils/site_replicate.js`) at every registered spoke, concurrently — one unreachable spoke never blocks or delays delivery to another. The spoke's `POST /api/site/resync` handler re-runs the same tested export-pull-and-import path used at join time rather than applying a partial diff.
 - **Identical-directory agent-signing key.** `POST /api/site/export` now best-effort includes the master's agent-signing key; a spoke adopts it via `agent_keys.adopt()` on both join and every resync, so any site's `sso-manager-node` can validly sign a command for any agent enrolled at any other site (a deliberate blast-radius tradeoff for this deployment's small, trusted scale — see `theta-suite`'s `docs/MULTI_SITE_SPEC.md` §2).
 - **Coordinated master promotion.** `POST /api/directory-admin/site-promote` now demotes the previous master as part of the same action (mints it a fresh join key, calls its new `POST /api/site/demote`) instead of leaving a manual two-step gap where two nodes could both believe they're master. Best-effort: an unreachable old master (the WAN-outage scenario this control exists for) never blocks the local promotion — the response's `handoff` field reports what happened.
 - **Master Site modal UI**: new "Live Replication" (spoke) / "Registered Spokes" (master) status rows; the join form gained a "this site's own reachable URL" field (prefilled from the browser origin) wired to the `selfUrl` the join API already supported but the UI never sent — a UI-driven join previously never registered for live replication, only the `setup.sh` bootstrap path did; the promote button's success toast now reports the actual handoff result.
 - New `nodejs/models/site_spoke.js` (registered spokes + their push tokens) and `docker-compose.multisite-e2e.yml` + `test/multisite_join_e2e.js` (real two-container master+spoke regression test covering join, live replication, promotion, and demotion end to end).
 ### Fixed
 - **`site-promote`'s god_admin check was dead on arrival.** It read `req.user.groups`, a field nothing in the codebase ever populates (every other admin gate resolves membership live via `permission.byGroup()`/`Group.list(user.dn)`, which also handles nested-group membership) — the check silently evaluated to an empty array on every request, so promotion returned 403 for every user, including a real god_admin, since it shipped in v2.0.0. Only surfaced by the live e2e test, not by inspection.
 - **The read-only write-gate blocked `site-promote` on a spoke before its handler could run** — the one mutating request a spoke must be able to make to itself. Exempted `/site-promote` from the gate.
 - **`GET /api/site/config` was returning `masterJoinKey` and `replicationPushToken`** — live credentials — directly in the JSON response to any admin session. Replaced with boolean derivatives (`hasMasterJoinKey`, `liveReplication`).
 # v2.3.0 - 2026-08-10
 ### Added
@@ -50,6 +50,8 @@ COPY test/seed-test-user.sh /usr/local/bin/seed-test-user
 RUN chmod +x /usr/local/bin/seed-test-user
 # End-to-end LDAP tunnel test client (docker-compose.e2e.yml)
 COPY test/tunnel_e2e.js ./test/tunnel_e2e.js
 # End-to-end multi-site join test client (docker-compose.multisite-e2e.yml)
 COPY test/multisite_join_e2e.js ./test/multisite_join_e2e.js
 # Default command: seed the test user, then run the test suite
 CMD ["sh", "-c", "seed-test-user && npm test"]
@@ -399,3 +399,7 @@ The Directory incorporates a **4-tier Driver Resolution Engine** (`services/driv
 - `POST /api/directory-admin/resources/:id/driver-action` — Execute management action (`{ action, params }`)
 - `GET /api/directory-admin/resources/:id/driver-logs` — Tail log output (`?lines=100`)
 ## 11. Multi-Site
 This directory can run across multiple sites (one **master** with write authority, any number of **spoke** read-only replicas that stay live-synced after joining), coordinate master promotion, and share the agent-signing key across sites. Full design and operational detail: [`docs/site-join.md`](docs/site-join.md) and, at the suite level, `theta-suite`'s `docs/MULTI_SITE_SPEC.md`.
@@ -0,0 +1,69 @@
 # End-to-end test of the real, shipped multi-site join flow (docs/site-join.md).
 #
 # Spins up two full all-in-one instances (app + bundled slapd each, like
 # docker-compose.repl-test.yml) — "master" and "spoke" — plus a client that
 # drives the actual HTTP API a human/operator would use: mint a site join key
 # on master, join from spoke, verify the spoke adopted the catalog, went
 # read-only, and reports live WAN health.
 #
 #   docker compose -f docker-compose.multisite-e2e.yml up --build --abort-on-container-exit
 #   # exit code 0 = MULTISITE E2E PASS
 #
 # slapcat (used by POST /api/site/export) only sees the LDAP data of the
 # container it runs in, so this MUST use the all-in-one image (master and
 # spoke each carry their own slapd) — the split ldap+redis+app harness used
 # by docker-compose.test.yml/e2e.yml won't exercise export/join at all.
 services:
  master:
    build:
      context: .
      dockerfile: Dockerfile.openldap
    container_name: multisite_e2e_master
    environment:
      - LDAP_BASE_DN=dc=master,dc=test
      - LDAP_ADMIN_PASS=secret
      - ORG_NAME=E2E Master
      - app_oauth__jwtSecret=e2e-multisite-master-jwt-secret
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"]
      interval: 2s
      timeout: 3s
      retries: 40
      start_period: 5s
  spoke:
    build:
      context: .
      dockerfile: Dockerfile.openldap
    container_name: multisite_e2e_spoke
    environment:
      - LDAP_BASE_DN=dc=spoke,dc=test
      - LDAP_ADMIN_PASS=secret
      - ORG_NAME=E2E Spoke
      - app_oauth__jwtSecret=e2e-multisite-spoke-jwt-secret
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"]
      interval: 2s
      timeout: 3s
      retries: 40
      start_period: 5s
  client:
    build:
      context: .
      dockerfile: Dockerfile.test-runner
    command: ["sh", "-c", "node test/multisite_join_e2e.js"]
    environment:
      - MASTER_URL=http://master:3001
      - SPOKE_URL=http://spoke:3001
      - MASTER_LDAP_HOST=master
      - MASTER_BASE_DN=dc=master,dc=test
      - SPOKE_LDAP_HOST=spoke
      - SPOKE_BASE_DN=dc=spoke,dc=test
      - LDAP_ADMIN_PASS=secret
    depends_on:
      master:
        condition: service_healthy
      spoke:
        condition: service_healthy
@@ -6,9 +6,7 @@ copy for local latency and autonomy (see the root `MULTI_SITE_SPEC.md` for the
 full architecture). This page covers the server endpoints that make a spoke
 "join" an existing master.
-> Status: **server endpoints + UI + setup.sh wiring.** A fresh bring-up can
+> Status: **server endpoints + UI + setup.sh wiring, live replication, coordinated promotion.** A fresh bring-up can adopt a master directory via the Directory UI or via `setup.env`; a joined spoke is read-only with live WAN health, stays in sync after joining (not just a one-time snapshot), and can be promoted to master with the old master demoted as part of the same action.
 > adopt a master directory via the Directory UI or via `setup.env`, and a
 > joined spoke is read-only with live WAN health.
 ## The flow
@@ -19,13 +17,58 @@ full architecture). This page covers the server endpoints that make a spoke
   - **setup.sh**: set `CFG_MASTER_DIRECTORY_URL` + `CFG_MASTER_DIRECTORY_JOIN_KEY`
     in `setup.env` before the first run.
 3. The spoke pulls the master's directory export (LDAP tree + resource
-   catalog), imports it, and persists its own spoke role
+   catalog + agent-signing key), imports it, and persists its own spoke role
   (`isMaster: false`, `masterUrl`, `siteSlug`) in `/config/site.json`.
 4. If the spoke also knows its own reachable URL (`selfUrl` — `setup.sh` passes
   `https://$CFG_SSO_HOST` automatically), it registers itself with the master
   (`POST /api/site/spokes`) so the master can push live updates back to it
   afterward — see **Live replication** below. Without `selfUrl` the join still
   succeeds; the spoke just stays a one-time snapshot.
 Joining is allowed only on a **fresh install** (no users beyond the bootstrap
 admin, no enrolled agents) — the join endpoint enforces this, so a populated
 directory can never be merged into a master's.
 ## Live replication (not a one-time snapshot)
 A registered spoke stays in sync: every successful catalog write on the
 master fires a fire-and-forget push (`utils/site_replicate.js`) at every
 registered spoke, concurrently — one unreachable spoke never blocks or delays
 delivery to another. The spoke's `POST /api/site/resync` handler (called by
 that push) re-runs the same export-pull-and-import logic used at join time,
 so there's exactly one tested code path for "make my catalog match the
 master's," not a separate diff-application mechanism.
 The agent-signing key travels the same path: `POST /api/site/export`
 best-effort includes it, and the spoke adopts it via `agent_keys.adopt()` on
 both join and every resync. Every site holding the same signing key means any
 site's `sso-manager-node` can validly sign a command for any agent enrolled
 at any other site — a deliberate tradeoff (see `MULTI_SITE_SPEC.md` §2)
 accepted for this deployment's small, trusted scale. Don't extend this
 pattern to a larger/adversarial-tenant deployment without revisiting it.
 ## Coordinated master promotion
 `POST /api/directory-admin/site-promote` (`god_admin` only) promotes this
 node to master as **one coordinated action**, not a manual two-step
 demote-then-promote:
 1. If this node currently has a master on file, it mints a fresh join key and
   calls that master's `POST /api/site/demote` (authenticated with the join
   key this node already holds), handing over the new key so the demoted node
   can keep talking to the new master afterward.
 2. This step is **best-effort** — an unreachable old master (the WAN-outage
   scenario this whole control exists for) never blocks the local promotion.
   The response's `handoff` field reports what happened
   (`"previous master demoted"`, an HTTP failure, or "unreachable, promoted
   locally anyway") so the operator can reconcile it manually if needed.
 3. Every known spoke gets a fire-and-forget `master-promoted` resync ping so
   they pick up the new master on their next sync.
 The Master Site modal's **Promote to Master** button surfaces the `handoff`
 result in a toast so the operator sees immediately whether the old master was
 actually reached.
 ## Endpoints
 | Method | Path | Purpose |
@@ -35,9 +78,13 @@ directory can never be merged into a master's.
 | `POST` | `/api/site/join-keys/:id/revoke` | Stop it accepting new joins |
 | `DELETE` | `/api/site/join-keys/:id` | Remove it |
 | `GET` | `/api/site/config` | Current role (isMaster, masterUrl, siteSlug) |
-| `POST` | `/api/site/export` | Master directory export (Bearer `stj_` key) |
+| `POST` | `/api/site/export` | Master directory export incl. agent-signing key (Bearer `stj_` key) |
 | `POST` | `/api/site/ping` | Lightweight master reachability probe (Bearer `stj_` key) |
-| `POST` | `/api/site/join` | Adopt a master directory (admin session) |
+| `POST` | `/api/site/join` | Adopt a master directory + register for live replication (admin session) |
 | `POST` | `/api/site/spokes` | Register a spoke's endpoint for live replication (Bearer `stj_` key, called by the spoke right after join) |
 | `POST` | `/api/site/resync` | Re-pull the master's export (Bearer the spoke's own `pushToken`, called by the master's fire-and-forget push) |
 | `POST` | `/api/site/demote` | Step down to spoke of a new master (Bearer `stj_` key, called by the newly-promoted node) |
 | `POST` | `/api/directory-admin/site-promote` | Promote this node to master, coordinating demotion of the old one (`god_admin` session) |
 ## Behavior after joining (spoke)
@@ -74,3 +121,25 @@ already joined reports "already a spoke" and setup continues (idempotent).
  gated on both sides.
 - The join key is stored on the spoke only so it can reach the master for WAN
  health (and, in a later layer, write-proxy).
 - `pushToken` (the credential a spoke stores so it can recognize a legitimate
  resync push from its master) is minted fresh per spoke registration and, by
  design, kept in retrievable form on the master — unlike a join key, it's a
  credential the master must keep *presenting*, not just verifying, so it
  can't be one-way hashed. Compare `models/site_spoke.js`'s doc comment for
  why that's the correct tradeoff, not an oversight.
 - Every site sharing one agent-signing key (see **Live replication** above)
  means a compromised spoke — including the smallest, least-secured one — has
  the same agent-command authority as the master. Accepted for this
  deployment's scale; see `MULTI_SITE_SPEC.md` §2 before reusing this pattern
  somewhere that assumption doesn't hold.
 ## Not yet built
 - Traffic between sites (join/export/resync) still goes over the open
  network path that already reaches the target — it does not route over the
  WireGuard mesh `theta-gateway` can now establish (see `MULTI_SITE_SPEC.md`).
 - A no-inbound spoke (no public IP at all) still can't join — the mechanism
  for a master to relay through the mesh to such a spoke is verified as
  working, but nothing automates creating that route yet.
 - OpenBao secret replication covers only the agent-signing key; LDAP admin
  creds, JWT secret, and other per-deployment secrets aren't synced.
@@ -22,6 +22,7 @@ const { SharedSecretGrant } = require('./shared_secret_grant');
 const { VaultAppToken } = require('./vault_app_token');
 const { Agent, AgentJoinKey } = require('./agent');
 const { SiteJoinKey } = require('./site_join_key');
 const { SiteSpoke } = require('./site_spoke');
 async function initORM() {
  const ormConf = conf.orm || {
    dialect: 'sqlite',
@@ -36,7 +37,7 @@ async function initORM() {
      conf: { orm: ormConf },
      models: [
        Resource, ResourceEdge, ResourceGroup, AccessRequest, Webhook, PluginInstance,
-        SharedSecret, SharedSecretGrant, VaultAppToken, Agent, AgentJoinKey, SiteJoinKey,
+        SharedSecret, SharedSecretGrant, VaultAppToken, Agent, AgentJoinKey, SiteJoinKey, SiteSpoke,
        Token, AuthToken, InviteToken, ImpersonationToken, PasswordResetToken, OtpToken, ServiceToken
      ]
    });
@@ -0,0 +1,42 @@
 'use strict';
 const crypto = require('crypto');
 const { Model } = require('@simpleworkjs/orm');
 // A spoke known to THIS node while it's acting as master — the registry that
 // makes live replication possible. A spoke registers itself here (POST
 // /api/site/spokes, authenticated by the same join key it used to join)
 // right after adopting the master's export, handing over its own reachable
 // endpoint. In return it's issued a `pushToken`: a shared secret the master
 // then presents on every future POST <spoke endpoint>/api/site/resync call.
 //
 // This is a DIFFERENT credential direction than SiteJoinKey: a join key is
 // presented TO the master and only ever needs to be verified (so it's stored
 // hashed, like a password). pushToken is presented BY the master, repeatedly,
 // so it has to be retrievable here -- there is no getting around storing it
 // in plaintext on the master, the same way Webhook.secret is (see
 // services/webhook_emitter.js) for the same reason (an HMAC/bearer credential
 // the sender must keep re-presenting, not a one-time secret only ever
 // verified).
 class SiteSpoke extends Model {
 	static generatePushToken() {
 		return crypto.randomBytes(24).toString('base64url');
 	}
 	static fields = {
 		id: { type: 'uuid', primaryKey: true },
 		endpoint: { type: 'string', isRequired: true, unique: true },
 		siteSlug: { type: 'string' },
 		pushToken: { type: 'string', isRequired: true },
 		created_on: { type: 'integer' },
 		last_seen_on: { type: 'integer' }
 	};
 	toPublic() {
 		const data = this.toJSON ? this.toJSON() : { ...this };
 		delete data.pushToken;
 		return data;
 	}
 }
 module.exports = { SiteSpoke };
@@ -1,6 +1,6 @@
 {
  "name": "t42-theta-directory",
-  "version": "2.3.0",
+  "version": "2.4.0",
  "description": "A very simple LDAP management and SSO system",
  "author": [
    {
@@ -11,7 +11,7 @@
  "scripts": {
    "start": "node ./bin/www",
    "dev": "npx nodemon --ignore public/ ./bin/www",
-    "test": "NODE_ENV=test jest tests/groups.test.js tests/subtypes.test.js tests/site_join.test.js tests/site_config.test.js --forceExit"
+    "test": "NODE_ENV=test jest tests/groups.test.js tests/subtypes.test.js tests/site_join.test.js tests/site_config.test.js tests/site_replicate.test.js --forceExit"
  },
  "jest": {
    "testEnvironment": "node",
@@ -2,6 +2,7 @@
 const router = require('express').Router();
 const permission = require('../utils/permission');
 const { Resource, ResourceEdge, ResourceGroup } = require('../models/resource');
 const { SiteJoinKey } = require('../models/site_join_key');
 const { Group } = require('../models/group_ldap');
 const { User } = require('../models/user_ldap');
 const { cnFromDn } = require('../utils/user_groups');
@@ -9,6 +10,7 @@ const { projectResources } = require('@simpleworkjs/directory-schema');
 const SUPER_ADMIN_GROUP = permission.SUPER_ADMIN_GROUP;
 const groups = require('../utils/groups');
 const meshReplicate = require('../utils/site_replicate');
 // Make `childCn` a member of `parentCn`, i.e. everyone in the child is
 // transitively in the parent. Idempotent and non-fatal: "already a member" is
@@ -252,19 +254,38 @@ router.get('/resources', async (req, res, next) => {
  } catch (err) { next(err); }
 });
-// ── Spoke read-only enforcement ─────────────────────────────────────────────
+// ── Spoke read-only enforcement + live replication trigger ──────────────────
 // On a joined spoke the catalog is a copy of the master's; directory writes
 // must go to the master (MULTI_SITE_SPEC.md — spoke = read-only catalog). Any
 // mutating request below this point is rejected on a spoke with a pointer to
 // the master. (site-status / site-promote live AFTER this middleware and are
 // not directory writes.)
 //
 // On the MASTER, a successful mutation here fires a fire-and-forget resync
 // push (utils/site_replicate.js) at every registered spoke, so the shipped
 // join flow's one-time snapshot doesn't go stale the moment the catalog
 // changes. Fires on res.on('finish') (after the response is actually sent,
 // status known) rather than before the handler runs, so a write that fails
 // validation never triggers a pointless replication round-trip.
 // /site-promote is deliberately exempt below: it's the ONE mutating request a
 // spoke must be able to make to itself (that's the entire point -- a spoke
 // promoting itself to master). Without this exemption the gate 403s the
 // promotion request before it ever reaches the handler, since this
 // middleware is registered ahead of router.post('/site-promote', ...) later
 // in the file and Express matches router.use() against every path.
 router.use((req, res, next) => {
-  if (['POST', 'PUT', 'DELETE', 'PATCH'].includes(req.method)) {
+  const mutating = ['POST', 'PUT', 'DELETE', 'PATCH'].includes(req.method);
  if (mutating && req.path !== '/site-promote') {
    const cfg = siteConfig.get();
    if (!cfg.isMaster) {
      const hint = cfg.masterUrl ? ' Directory writes must go to the master at ' + cfg.masterUrl + '.' : '';
      return res.status(403).json({ status: 'error', message: 'This node is a spoke (read-only catalog).' + hint });
    }
    res.on('finish', () => {
      if (res.statusCode >= 200 && res.statusCode < 300) {
        meshReplicate.replicateToSpokes(`${req.method} ${req.path}`);
      }
    });
  }
  next();
 });
@@ -906,6 +927,7 @@ router.post('/discovered/merge', async (req, res, next) => {
 const siteConfig = require('../utils/site_config');
 const { siteIsFresh } = require('../utils/site_join');
 const { Agent } = require('../models/agent');
 const { SiteSpoke } = require('../models/site_spoke');
 // probeMasterHealth checks whether this (spoke) node can reach its master over
 // the site join key. The master's /api/site/ping is deliberately lightweight.
@@ -941,6 +963,13 @@ router.get('/site-status', async (req, res, next) => {
    if (cfg.isMaster) {
      canJoin = await siteIsFresh({ User, Agent }).catch(() => false);
    }
    // registeredSpokesCount (master) / liveReplication (spoke): surfaces
    // whether live replication is actually wired up, not just whether the
    // join itself succeeded -- a spoke that joined without `selfUrl` (e.g.
    // via an older bootstrap, or the UI form before it grew the field) is
    // fully joined but silently stuck on the one-time snapshot, which was
    // otherwise invisible anywhere in the UI.
    const registeredSpokesCount = cfg.isMaster ? await SiteSpoke.list().then(l => l.length).catch(() => 0) : 0;
    res.json({
      status: 'ok',
      config: {
@@ -949,7 +978,9 @@ router.get('/site-status', async (req, res, next) => {
        siteSlug: cfg.siteSlug,
        wanConnected,
        siteMode: cfg.isMaster ? 'master' : 'spoke',
-        canJoin
+        canJoin,
        liveReplication: !cfg.isMaster ? !!cfg.replicationPushToken : undefined,
        registeredSpokesCount
      },
      sitesCount: sites.length,
      sites: sites.map(s => ({ id: s.id, name: s.name, slug: s.slug })),
@@ -960,21 +991,67 @@ router.get('/site-status', async (req, res, next) => {
 router.post('/site-promote', async (req, res, next) => {
  try {
-    // Check god_admin privileges
+    // god_admin privilege check. This used to read req.user.groups, which
-    const userGroups = req.user && req.user.groups ? req.user.groups : [];
+    // nothing in the codebase ever populates -- User.get() (what
-    const isGodAdmin = userGroups.includes('god_admin') || userGroups.includes(SUPER_ADMIN_GROUP);
+    // Auth.checkToken returns as req.user) has no .groups field; every other
    // admin gate in this app resolves membership live via
    // permission.byGroup()/Group.list(user.dn), which also correctly
    // resolves NESTED group membership (a user who is god_admin via a nested
    // group, not just direct membership). The old check silently evaluated
    // to an empty array for every request, making this endpoint
    // unreachable for ANY user -- caught by the multi-site e2e promotion
    // test (docker-compose.multisite-e2e.yml), not by inspection.
    const isGodAdmin = await permission.byGroup(req.user, [SUPER_ADMIN_GROUP]).catch(() => false);
    if (!isGodAdmin) {
      return res.status(403).json({ status: 'error', message: 'Master promotion requires explicit god_admin authority' });
    }
-    siteConfig.save({ isMaster: true, masterUrl: '' });
+    // MULTI_SITE_SPEC.md §3.2: promotion is ONE coordinated action, never a
    // manual two-step "demote the old one first" — if we currently know a
    // master (we were a spoke), hand it off before flipping ourselves. This
    // is best-effort: an unreachable old master (the whole point of the
    // WAN-outage promotion scenario §3 describes) must never block a
    // god_admin's local promotion, it's just reported so the operator can
    // reconcile it manually.
    const beforeCfg = siteConfig.get();
    let handoffNote = 'no previous master on file (already master, or fresh install)';
    if (!beforeCfg.isMaster && beforeCfg.masterUrl && beforeCfg.masterJoinKey) {
      try {
        const { raw: freshKey } = await SiteJoinKey.issue({
          label: 'promotion-handoff-' + new Date().toISOString().slice(0, 10),
          createdBy: req.user ? req.user.uid : 'admin'
        });
        const controller = new AbortController();
        const timer = setTimeout(() => controller.abort(), 15000);
        let resp;
        try {
          resp = await fetch(beforeCfg.masterUrl + '/api/site/demote', {
            method: 'POST',
            headers: { Authorization: 'Bearer ' + beforeCfg.masterJoinKey, 'Content-Type': 'application/json' },
            body: JSON.stringify({ newMasterUrl: (req.body && req.body.selfUrl) || '', newJoinKey: freshKey }),
            signal: controller.signal
          });
        } finally { clearTimeout(timer); }
        handoffNote = resp.ok ? 'previous master demoted' : ('previous master demote failed: HTTP ' + resp.status);
      } catch (e) {
        handoffNote = 'previous master unreachable (' + e.message + ') — promoted locally anyway; reconcile it manually once it\'s back';
      }
    }
-    console.log(`[MULTI-SITE] Node promoted to MASTER by user ${req.user ? req.user.uid : 'admin'}`);
+    siteConfig.save({ isMaster: true, masterUrl: '', masterJoinKey: undefined });
    console.log(`[MULTI-SITE] Node promoted to MASTER by user ${req.user ? req.user.uid : 'admin'} (handoff: ${handoffNote})`);
    // Fire-and-forget: let every known spoke know a new master exists so
    // their next resync targets it. (They'll also learn this the hard way if
    // their old-master resync calls start failing, but this speeds it up.)
    meshReplicate.replicateToSpokes('master-promoted');
    const cfg = siteConfig.get();
    res.json({
      status: 'ok',
      message: 'Node successfully promoted to Master Site',
      handoff: handoffNote,
      config: {
        isMaster: true,
        masterUrl: '',
@@ -14,6 +14,7 @@
 // calls it with a join key), so it is defined BEFORE the auth middleware.
 const express = require('express');
 const crypto = require('crypto');
 const { execFile } = require('child_process');
 const { promisify } = require('util');
 const os = require('os');
@@ -25,10 +26,13 @@ const permission = require('../utils/permission');
 const conf = require('@simpleworkjs/conf');
 const { Resource, ResourceEdge } = require('../models/resource');
 const { SiteJoinKey } = require('../models/site_join_key');
 const { SiteSpoke } = require('../models/site_spoke');
 const { replicateToSpokes } = require('../utils/site_replicate');
 const User = require('../models/user');
 const { Agent } = require('../models/agent');
 const siteConfig = require('../utils/site_config');
 const { importDirectory, ldapAddArgs, baseDnFrom, siteIsFresh } = require('../utils/site_join');
 const agentKeys = require('../utils/agent_keys');
 const execFileAsync = promisify(execFile);
 const router = express.Router();
@@ -63,10 +67,16 @@ router.post('/export', async (req, res, next) => {
    const key = await SiteJoinKey.authenticate(rawKey);
    if (!key) return res.status(401).json({ status: 'error', message: 'invalid or revoked site join key' });
-    const [ldif, resources, edges] = await Promise.all([
+    const [ldif, resources, edges, signingKey] = await Promise.all([
      slurpLdif(),
      Resource.list(),
-      ResourceEdge.list()
+      ResourceEdge.list(),
      // Best-effort: a master with no OpenBao reachable (or no key generated
      // yet) still exports successfully -- signingKey is just omitted, and
      // the spoke keeps whatever key (if any) it already has. Identical
      // signing keys across sites is a nice-to-have on top of the join
      // working at all, never a reason to fail the join.
      agentKeys.load().then((k) => k && { privateKeyPem: k.privateKeyPem, publicKeyPem: k.publicKeyPem }).catch(() => null)
    ]);
    await key.update({ use_count: (key.use_count || 0) + 1, last_used_on: Math.floor(Date.now() / 1000) }).catch(() => {});
@@ -77,7 +87,8 @@ router.post('/export', async (req, res, next) => {
      baseDn: baseDnFrom(conf),
      ldif,
      resources: (resources || []).map(r => (r.toJSON ? r.toJSON() : r)),
-      edges: (edges || []).map(e => (e.toJSON ? e.toJSON() : e))
+      edges: (edges || []).map(e => (e.toJSON ? e.toJSON() : e)),
      ...(signingKey ? { signingKey } : {})
    });
  } catch (e) { next(e); }
 });
@@ -95,6 +106,104 @@ router.post('/ping', async (req, res, next) => {
  } catch (e) { next(e); }
 });
 // ── Spoke registration (MASTER side, Bearer site-join-key; no admin session)
 // A spoke calls this right after adopting a join, handing over its own
 // reachable endpoint so the master can push live-replication resync pings to
 // it later (see utils/site_replicate.js). Idempotent on endpoint: calling it
 // again (e.g. a spoke re-registering after its own restart) returns the same
 // pushToken rather than minting a new one, so the spoke doesn't need to
 // re-learn a credential it already has.
 router.post('/spokes', async (req, res, next) => {
  try {
    const auth = req.headers.authorization || '';
    const rawKey = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';
    const key = await SiteJoinKey.authenticate(rawKey);
    if (!key) return res.status(401).json({ status: 'error', message: 'invalid or revoked site join key' });
    const { endpoint, siteSlug } = req.body || {};
    if (!endpoint || !/^https?:\/\//.test(endpoint)) {
      return res.status(400).json({ status: 'error', message: 'a valid http(s) endpoint is required' });
    }
    const now = Math.floor(Date.now() / 1000);
    let spoke = (await SiteSpoke.list({ where: { endpoint } }))[0];
    if (spoke) {
      await spoke.update({ siteSlug: siteSlug || spoke.siteSlug, last_seen_on: now });
    } else {
      spoke = await SiteSpoke.create({
        id: crypto.randomUUID(),
        endpoint,
        siteSlug: siteSlug || null,
        pushToken: SiteSpoke.generatePushToken(),
        created_on: now,
        last_seen_on: now
      });
    }
    logAudit('spoke_registered', { endpoint, siteSlug: spoke.siteSlug });
    res.json({ status: 'ok', pushToken: spoke.pushToken });
  } catch (e) { next(e); }
 });
 // ── Resync (SPOKE side, Bearer pushToken; no admin session) ─────────────────
 // The receiving end of utils/site_replicate.js's fire-and-forget push: the
 // master pings this when its catalog changes. Deliberately just
 // re-runs the same export-pull + import this node already did at join time
 // (adoptFromMaster below) rather than applying a partial diff -- one tested
 // code path for "make my catalog match the master's," not two.
 router.post('/resync', async (req, res, next) => {
  try {
    const cfg = siteConfig.get();
    if (cfg.isMaster) return res.status(400).json({ status: 'error', message: 'this node is master; resync is a spoke-only operation' });
    const auth = req.headers.authorization || '';
    const presented = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';
    if (!cfg.replicationPushToken || presented !== cfg.replicationPushToken) {
      return res.status(401).json({ status: 'error', message: 'invalid resync push token' });
    }
    if (!cfg.masterUrl || !cfg.masterJoinKey) {
      return res.status(409).json({ status: 'error', message: 'no master join credentials on file' });
    }
    const imp = await adoptFromMaster({ masterUrl: cfg.masterUrl, joinKey: cfg.masterJoinKey });
    logAudit('resynced', { reason: (req.body && req.body.reason) || 'unspecified', resourcesCreated: imp.created, resourcesUpdated: imp.updated });
    res.json({ status: 'ok', resources: { created: imp.created, updated: imp.updated, edges: imp.edgeCount } });
  } catch (e) { next(e); }
 });
 // ── Demote (called on the OLD master; Bearer site-join-key; no admin session)
 // MULTI_SITE_SPEC.md §3.2: promoting a spoke must be a single coordinated
 // action, never a two-step "hope nobody's master for a while" gap. The node
 // being promoted calls this on whatever it currently believes is master,
 // using the join-key credential it already holds from when it joined --
 // authenticating "demote me" is exactly the same trust relationship as
 // authenticating "let me pull an export," so no new credential type is
 // needed for THIS direction. (The new master's future ability to push
 // replication/resync to the newly-demoted node is a separate credential --
 // newJoinKey below -- since that's the master->spoke direction, same as
 // every other spoke registration.)
 router.post('/demote', async (req, res, next) => {
  try {
    const auth = req.headers.authorization || '';
    const rawKey = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';
    const key = await SiteJoinKey.authenticate(rawKey);
    if (!key) return res.status(401).json({ status: 'error', message: 'invalid or revoked site join key' });
    const cfg = siteConfig.get();
    if (!cfg.isMaster) {
      return res.status(400).json({ status: 'error', message: 'this node is already a spoke' });
    }
    const { newMasterUrl, newJoinKey } = req.body || {};
    if (!newMasterUrl || !newJoinKey) {
      return res.status(400).json({ status: 'error', message: 'newMasterUrl and newJoinKey are required' });
    }
    const base = String(newMasterUrl).replace(/\/+$/, '');
    siteConfig.save({ isMaster: false, masterUrl: base, masterJoinKey: newJoinKey });
    logAudit('demoted', { demotedBy: key.keyPrefix, newMasterUrl: base });
    res.json({ status: 'ok', message: 'Demoted to spoke of ' + base });
  } catch (e) { next(e); }
 });
 // ── Everything below requires an admin session ──────────────────────────────
 router.use(middleware.auth);
 router.use(async (req, res, next) => {
@@ -110,9 +219,23 @@ router.use(async (req, res, next) => {
 });
 // Current multi-site role (master/spoke, site slug, master URL).
 // Never sent to the client: masterJoinKey and replicationPushToken are live
 // credentials, not display data. Callers get boolean derivatives instead
 // (hasMasterJoinKey, liveReplication) -- enough to render UI state without
 // putting a secret in a browser response.
 router.get('/config', async (req, res, next) => {
-  try { res.json({ status: 'ok', config: siteConfig.get() }); }
+  try {
-  catch (e) { next(e); }
+    const cfg = siteConfig.get();
    const { masterJoinKey, replicationPushToken, ...safe } = cfg;
    res.json({
      status: 'ok',
      config: {
        ...safe,
        hasMasterJoinKey: !!masterJoinKey,
        liveReplication: !!replicationPushToken
      }
    });
  } catch (e) { next(e); }
 });
 // ── Site join key management (admin) ────────────────────────────────────────
@@ -161,9 +284,77 @@ router.delete('/join-keys/:id', async (req, res, next) => {
 // role. Only valid on a node that is currently the master (i.e. a fresh
 // bring-up that has not joined anything yet) — see setup.sh wiring for the
 // pre-seed timing (this pass is server endpoints only).
 // Shared by /join (first adoption) and /resync (live-replication re-pull):
 // fetch the master's export and apply it locally (catalog + LDAP). Throws on
 // any failure that should surface as a 502 to the caller.
 async function adoptFromMaster({ masterUrl, joinKey }) {
  const base = String(masterUrl).replace(/\/+$/, '');
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), 30000);
  let resp;
  try {
    resp = await fetch(base + '/api/site/export', {
      method: 'POST',
      headers: { Authorization: 'Bearer ' + joinKey, 'Content-Type': 'application/json' },
      body: '{}',
      signal: controller.signal
    });
  } finally { clearTimeout(timer); }
  if (!resp.ok) {
    const text = (await resp.text().catch(() => '')).slice(0, 200);
    const err = new Error('master export failed: HTTP ' + resp.status + ' ' + text);
    err.httpStatus = 502;
    throw err;
  }
  const exportData = await resp.json();
  if (!exportData || exportData.status !== 'ok' || !exportData.ldif) {
    const err = new Error('master export returned no directory');
    err.httpStatus = 502;
    throw err;
  }
  // 1. Adopt the resource catalog.
  const imp = await importDirectory({ Resource, ResourceEdge, exportData });
  // 2. Adopt the LDAP tree. The spoke keeps its own cn=admin / base DN;
  //    ldapadd -c skips existing entries, so users/groups come from master.
  let ldapNote = 'imported';
  try {
    const adminDn = conf.ldap && conf.ldap.bindDN;
    // The admin credential for the local slapd (read from config at runtime —
    // never hardcoded; named without the literal "password" keyword so secret
    // scanners don't false-positive on a variable assignment).
    const ldapCred = conf.ldap && conf.ldap.bindPassword;
    const ldifFile = path.join(os.tmpdir(), 'theta-site-join.ldif');
    fs.writeFileSync(ldifFile, exportData.ldif, 'utf8');
    const argv = ldapAddArgs({ bindDN: adminDn, ldapCred, ldifFile, ldapUrl: conf.ldap && conf.ldap.url });
    await execFileAsync(argv[0], argv.slice(1), { maxBuffer: 4 * 1024 * 1024, timeout: 120000 });
    fs.unlink(ldifFile).catch(() => {});
  } catch (e) {
    ldapNote = 'skipped/failed: ' + e.message;
  }
  // 3. Adopt the master's agent-signing key, if it sent one (MULTI_SITE_SPEC.md
  //    §2 -- identical directories). Best-effort: OpenBao being unreachable
  //    here shouldn't fail a join/resync any more than it would on a
  //    standalone install.
  let signingKeyNote = 'not provided by master';
  if (exportData.signingKey) {
    try {
      await agentKeys.adopt(exportData.signingKey);
      signingKeyNote = 'adopted';
    } catch (e) {
      signingKeyNote = 'failed: ' + e.message;
    }
  }
  return { imp, ldapNote, signingKeyNote, exportData, base };
 }
 router.post('/join', async (req, res, next) => {
  try {
-    const { masterUrl, joinKey } = req.body || {};
+    const { masterUrl, joinKey, selfUrl } = req.body || {};
    if (!masterUrl || !joinKey) {
      return res.status(400).json({ status: 'error', message: 'masterUrl and joinKey are required' });
    }
@@ -181,54 +372,52 @@ router.post('/join', async (req, res, next) => {
      });
    }
-    const base = String(masterUrl).replace(/\/+$/, '');
+    let adopted;
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), 30000);
    let resp;
    try {
-      resp = await fetch(base + '/api/site/export', {
+      adopted = await adoptFromMaster({ masterUrl, joinKey });
        method: 'POST',
        headers: { Authorization: 'Bearer ' + joinKey, 'Content-Type': 'application/json' },
        body: '{}',
        signal: controller.signal
      });
    } finally { clearTimeout(timer); }
    if (!resp.ok) {
      const text = (await resp.text().catch(() => '')).slice(0, 200);
      return res.status(502).json({ status: 'error', message: 'master export failed: HTTP ' + resp.status + ' ' + text });
    }
    const exportData = await resp.json();
    if (!exportData || exportData.status !== 'ok' || !exportData.ldif) {
      return res.status(502).json({ status: 'error', message: 'master export returned no directory' });
    }
    // 1. Adopt the resource catalog.
    const imp = await importDirectory({ Resource, ResourceEdge, exportData });
    // 2. Adopt the LDAP tree. The spoke keeps its own cn=admin / base DN;
    //    ldapadd -c skips existing entries, so users/groups come from master.
    let ldapNote = 'imported';
    try {
      const adminDn = conf.ldap && conf.ldap.bindDN;
      // The admin credential for the local slapd (read from config at runtime —
      // never hardcoded; named without the literal "password" keyword so secret
      // scanners don't false-positive on a variable assignment).
      const ldapCred = conf.ldap && conf.ldap.bindPassword;
      const ldifFile = path.join(os.tmpdir(), 'theta-site-join.ldif');
      fs.writeFileSync(ldifFile, exportData.ldif, 'utf8');
      const argv = ldapAddArgs({ bindDN: adminDn, ldapCred, ldifFile, ldapUrl: conf.ldap && conf.ldap.url });
      await execFileAsync(argv[0], argv.slice(1), { maxBuffer: 4 * 1024 * 1024, timeout: 120000 });
      fs.unlink(ldifFile).catch(() => {});
    } catch (e) {
-      ldapNote = 'skipped/failed: ' + e.message;
+      return res.status(e.httpStatus || 502).json({ status: 'error', message: e.message });
    }
    const { imp, ldapNote, signingKeyNote, exportData, base } = adopted;
    // 3. Register with the master for live replication, if this node knows
    //    its own reachable endpoint (selfUrl -- see setup.env's
    //    CFG_SELF_DIRECTORY_URL). Best-effort: a spoke that can't/won't
    //    register still joins successfully, it just won't receive live
    //    resync pushes (falls back to being exactly today's one-time
    //    snapshot for that spoke, not a hard failure).
    let replicationPushToken = null;
    let replicationNote = 'not registered (no selfUrl given)';
    if (selfUrl) {
      try {
        const regResp = await fetch(base + '/api/site/spokes', {
          method: 'POST',
          headers: { Authorization: 'Bearer ' + joinKey, 'Content-Type': 'application/json' },
          body: JSON.stringify({ endpoint: selfUrl, siteSlug: exportData.siteSlug || cfg.siteSlug })
        });
        if (regResp.ok) {
          const regBody = await regResp.json();
          replicationPushToken = regBody.pushToken;
          replicationNote = 'registered for live replication';
        } else {
          replicationNote = 'registration failed: HTTP ' + regResp.status;
        }
      } catch (e) {
        replicationNote = 'registration failed: ' + e.message;
      }
    }
-    // 3. Persist the spoke role (survives restarts). The join key is kept so
+    // 4. Persist the spoke role (survives restarts). The join key is kept so
-    //    the spoke can run WAN-health checks (and, in a later layer, proxy
+    //    the spoke can run WAN-health checks against the master; the push
-    //    writes) against the master — it is a spoke-to-master credential, not
+    //    token (if registration succeeded) is what authenticates the
-    //    a shared secret.
+    //    master's future resync pushes back to THIS node.
-    siteConfig.save({ isMaster: false, masterUrl: base, siteSlug: exportData.siteSlug || cfg.siteSlug, masterJoinKey: joinKey });
+    siteConfig.save({
      isMaster: false,
      masterUrl: base,
      siteSlug: exportData.siteSlug || cfg.siteSlug,
      masterJoinKey: joinKey,
      ...(replicationPushToken ? { replicationPushToken } : {})
    });
    logAudit('joined', {
      actor: req.user.uid,
@@ -237,7 +426,9 @@ router.post('/join', async (req, res, next) => {
      resourcesCreated: imp.created,
      resourcesUpdated: imp.updated,
      edges: imp.edgeCount,
-      ldap: ldapNote
+      ldap: ldapNote,
      signingKey: signingKeyNote,
      replication: replicationNote
    });
    res.json({
@@ -245,7 +436,9 @@ router.post('/join', async (req, res, next) => {
      message: 'Joined master site ' + base,
      siteSlug: exportData.siteSlug || cfg.siteSlug,
      resources: { created: imp.created, updated: imp.updated, edges: imp.edgeCount },
-      ldap: { note: ldapNote }
+      ldap: { note: ldapNote },
      signingKey: { note: signingKeyNote },
      replication: { note: replicationNote, live: !!replicationPushToken }
    });
  } catch (e) { next(e); }
 });
@@ -0,0 +1,90 @@
 'use strict';
 // In-memory stand-in for the SiteSpoke ORM model.
 let spokeStore;
 function makeSpokeMock() {
  spokeStore = [];
  return {
    list: jest.fn(async () => [...spokeStore]),
    _seed(rows) { spokeStore.push(...rows); }
  };
 }
 let mockFetchCalls = [];
 let mockFetchImpl = async () => ({ ok: true, status: 200 });
 describe('site_replicate', () => {
  let siteReplicate;
  let SiteSpoke;
  let originalFetch;
  beforeEach(() => {
    jest.resetModules();
    mockFetchCalls = [];
    mockFetchImpl = async () => ({ ok: true, status: 200 });
    jest.doMock('../models/site_spoke', () => ({ SiteSpoke: makeSpokeMock() }));
    siteReplicate = require('../utils/site_replicate');
    SiteSpoke = require('../models/site_spoke').SiteSpoke;
    // site_replicate.js uses the global fetch (Node 18+ built-in), not
    // node-fetch -- stub that directly.
    originalFetch = global.fetch;
    global.fetch = (...args) => { mockFetchCalls.push(args); return mockFetchImpl(...args); };
  });
  afterEach(() => {
    global.fetch = originalFetch;
  });
  test('pushes to every known spoke concurrently with its own pushToken', async () => {
    SiteSpoke._seed([
      { endpoint: 'https://spoke-a.example.com', pushToken: 'token-a' },
      { endpoint: 'https://spoke-b.example.com', pushToken: 'token-b' }
    ]);
    await siteReplicate.replicateToSpokes('catalog-changed');
    await new Promise((r) => setImmediate(r));
    expect(mockFetchCalls.length).toBe(2);
    const urls = mockFetchCalls.map((c) => c[0]).sort();
    expect(urls).toEqual(['https://spoke-a.example.com/api/site/resync', 'https://spoke-b.example.com/api/site/resync']);
    const [, optsA] = mockFetchCalls.find((c) => c[0].includes('spoke-a'));
    expect(optsA.headers.Authorization).toBe('Bearer token-a');
    expect(JSON.parse(optsA.body).reason).toBe('catalog-changed');
  });
  test('no known spokes: resolves cleanly, no fetch calls', async () => {
    await siteReplicate.replicateToSpokes('catalog-changed');
    await new Promise((r) => setImmediate(r));
    expect(mockFetchCalls.length).toBe(0);
  });
  test('one spoke failing does not prevent delivery to another', async () => {
    SiteSpoke._seed([
      { endpoint: 'https://dead-spoke.example.com', pushToken: 'token-dead' },
      { endpoint: 'https://live-spoke.example.com', pushToken: 'token-live' }
    ]);
    mockFetchImpl = async (url) => {
      if (url.includes('dead-spoke')) throw new Error('connection refused');
      return { ok: true, status: 200 };
    };
    await expect(siteReplicate.replicateToSpokes('event')).resolves.toBeUndefined();
    await new Promise((r) => setImmediate(r));
    expect(mockFetchCalls.length).toBe(2);
  });
  test('a non-2xx response from a spoke does not throw out of replicateToSpokes', async () => {
    SiteSpoke._seed([{ endpoint: 'https://spoke-a.example.com', pushToken: 'token-a' }]);
    mockFetchImpl = async () => ({ ok: false, status: 500 });
    await expect(siteReplicate.replicateToSpokes('event')).resolves.toBeUndefined();
  });
  test('SiteSpoke.list() throwing does not propagate to the caller', async () => {
    SiteSpoke.list = jest.fn(async () => { throw new Error('db unavailable'); });
    await expect(siteReplicate.replicateToSpokes('event')).resolves.toBeUndefined();
    expect(mockFetchCalls.length).toBe(0);
  });
 });
@@ -90,10 +90,34 @@ function status() {
 	return { available: !!cached, error: loadError };
 }
 // Overwrite the stored key with material handed over by a master (multi-site
 // "identical directories" — MULTI_SITE_SPEC.md §2). Every site sharing one
 // signing key is what lets any site's sso-manager validly sign a command for
 // an agent enrolled at any other site, at the accepted cost that compromising
 // ANY one site's OpenBao is equivalent to compromising all of them for agent
 // command authority. That tradeoff was deliberately accepted for this
 // deployment's scale (a handful of trusted sites) -- do not call this to sync
 // keys across a boundary where sites don't trust each other equally.
 //
 // Idempotent: adopting the same key material twice (e.g. on every resync
 // ping) is a no-op past the first call.
 async function adopt({ privateKeyPem, publicKeyPem }) {
 	if (!privateKeyPem || !publicKeyPem) throw new Error('adopt() requires both privateKeyPem and publicKeyPem');
 	if (cached && cached.privateKeyPem === privateKeyPem && cached.publicKeyPem === publicKeyPem) {
 		return cached; // already holding this exact key -- nothing to do
 	}
 	const material = { privateKeyPem, publicKeyPem };
 	await baoConf.set(PATH, material);
 	cached = { ...material, publicKeyBase64: rawPublicKeyBase64(publicKeyPem) };
 	loadError = null;
 	console.log('[agent_keys] adopted signing key from master (multi-site identical-directory sync)');
 	return cached;
 }
 // Test seam: drop the in-process cache.
 function _reset() {
 	cached = null;
 	loadError = null;
 }
-module.exports = { load, status, rawPublicKeyBase64, _reset, PATH };
+module.exports = { load, status, adopt, rawPublicKeyBase64, _reset, PATH };
@@ -0,0 +1,57 @@
 'use strict';
 // Live replication push -- the piece the shipped v1 join flow doesn't have on
 // its own (join is a one-time export/import snapshot; nothing kept a spoke in
 // sync afterward). This fires a lightweight "something changed, re-pull" ping
 // at every spoke registered in SiteSpoke, concurrently, fire-and-forget: never
 // awaited by its caller, and one unreachable spoke never delays or blocks
 // another. See MULTI_SITE_SPEC.md §2.2 for why this must never become a
 // blocking design (a write must never stall on spoke reachability).
 //
 // Deliberately a PUSH-A-SIGNAL / PULL-A-SNAPSHOT design, not a push-a-diff
 // design: the receiving spoke reacts by calling the master's already-shipped,
 // already-tested POST /api/site/export + importDirectory() path again (see
 // routes/api_site.js's /resync handler), rather than this module inventing a
 // second, parallel way to represent "what changed." Fewer moving parts, and
 // no risk of a diff payload and a full export ever disagreeing.
 const { SiteSpoke } = require('../models/site_spoke');
 const RESYNC_TIMEOUT_MS = 8000;
 function replicateToSpokes(reason) {
 	return (async () => {
 		let spokes;
 		try {
 			spokes = await SiteSpoke.list();
 		} catch (err) {
 			console.error('[site-replicate] failed to list known spokes:', err.message);
 			return;
 		}
 		for (const spoke of spokes) {
 			// Not awaited -- every spoke is pushed to concurrently.
 			pingOne(spoke, reason).catch((err) => {
 				console.error(`[site-replicate] resync ping to ${spoke.endpoint} failed:`, err.message);
 			});
 		}
 	})();
 }
 async function pingOne(spoke, reason) {
 	const url = String(spoke.endpoint).replace(/\/+$/, '') + '/api/site/resync';
 	const controller = new AbortController();
 	const timer = setTimeout(() => controller.abort(), RESYNC_TIMEOUT_MS);
 	try {
 		const resp = await fetch(url, {
 			method: 'POST',
 			headers: { Authorization: 'Bearer ' + spoke.pushToken, 'Content-Type': 'application/json' },
 			body: JSON.stringify({ reason: reason || 'catalog-changed' }),
 			signal: controller.signal
 		});
 		if (!resp.ok) throw new Error('status ' + resp.status);
 	} finally {
 		clearTimeout(timer);
 	}
 }
 module.exports = { replicateToSpokes };
@@ -3339,6 +3339,10 @@
              '<tr><th>WAN Sync Health:</th><td>' + (res.config.wanConnected === false
                ? '<span class="badge bg-danger"><i class="fa-solid fa-xmark me-1"></i> Offline / Disconnected</span>'
                : '<span class="badge bg-success"><i class="fa-solid fa-check me-1"></i> Online / Operational</span>') + '</td></tr>' +
              (!isMaster ? '<tr><th>Live Replication:</th><td>' + (cfg.liveReplication
                ? '<span class="badge bg-success"><i class="fa-solid fa-bolt me-1"></i> Live (catalog updates push automatically)</span>'
                : '<span class="badge bg-warning text-dark"><i class="fa-solid fa-triangle-exclamation me-1"></i> Snapshot only (re-join to register for live updates)</span>') + '</td></tr>' : '') +
              (isMaster ? '<tr><th>Registered Spokes:</th><td><span class="badge bg-success">' + (cfg.registeredSpokesCount || 0) + ' receiving live updates</span></td></tr>' : '') +
              '<tr><th>Registered Sites:</th><td><span class="badge bg-primary">' + (res.sitesCount || 0) + ' sites</span></td></tr>' +
              '<tr><th>Theta Gateways:</th><td><span class="badge bg-dark">' + (res.gatewaysCount || 0) + ' active gateways</span></td></tr>' +
            '</table>' +
@@ -3359,6 +3363,10 @@
              '<div class="col-md-7"><input type="text" id="site-join-url" class="form-control form-control-sm" placeholder="Master Directory URL (e.g. https://sso.master.example.com)"></div>' +
              '<div class="col-md-5"><input type="text" id="site-join-key" class="form-control form-control-sm font-monospace" placeholder="Site join key (stj_...)"></div>' +
            '</div>' +
            '<div class="mt-2">' +
              '<label class="form-label small mb-1">This site\'s own reachable URL <span class="text-muted">(so the master can push live updates here — leave blank to only get a one-time snapshot)</span></label>' +
              '<input type="text" id="site-join-self-url" class="form-control form-control-sm" value="' + esc(window.location.origin) + '">' +
            '</div>' +
            '<button class="btn btn-sm btn-primary mt-2" onclick="joinCurrentSiteToMaster()"><i class="fa-solid fa-link me-1"></i> Join Site</button>' +
          '</div>' +
        '</div>';
@@ -3408,8 +3416,9 @@
    if (!confirmed) return;
    try {
-      const res = await app.api.post('directory-admin/site-promote', {});
+      const res = await app.api.post('directory-admin/site-promote', { selfUrl: window.location.origin });
-      app.messages.toast(res.message || 'Node promoted to Master Site', 'success');
+      const handoffOk = res.handoff === 'previous master demoted' || /no previous master/.test(res.handoff || '');
      app.messages.toast((res.message || 'Node promoted to Master Site') + ' — ' + (res.handoff || ''), handoffOk ? 'success' : 'warning');
      app.modal.close();
      refreshSiteStatus();
    } catch (e) {
@@ -3421,6 +3430,7 @@
  async function joinCurrentSiteToMaster() {
    const masterUrl = ($('#site-join-url').val() || '').trim();
    const joinKey = ($('#site-join-key').val() || '').trim();
    const selfUrl = ($('#site-join-self-url').val() || '').trim();
    if (!masterUrl || !joinKey) {
      return app.messages.toast('Enter the master Directory URL and a site join key', 'warning');
    }
@@ -3431,8 +3441,11 @@
    if (!confirmed) return;
    try {
-      const res = await app.api.post('site/join', { masterUrl, joinKey });
+      const res = await app.api.post('site/join', { masterUrl, joinKey, ...(selfUrl ? { selfUrl } : {}) });
-      app.messages.toast(res.message || 'Joined master site', 'success');
+      const replicationNote = res.replication && res.replication.live
        ? ' Live replication is active.'
        : ' Snapshot only — this site will not receive live updates (' + ((res.replication && res.replication.note) || 'no selfUrl given') + ').';
      app.messages.toast((res.message || 'Joined master site') + replicationNote, res.replication && res.replication.live ? 'success' : 'warning');
      app.modal.close();
      refreshSiteStatus();
    } catch (e) {
@@ -0,0 +1,297 @@
 'use strict';
 // End-to-end test of the real, shipped multi-site join flow (docs/site-join.md).
 // Drives the actual HTTP API two humans (a master admin + a spoke admin)
 // would use: seed an admin on each side, mint a site join key on master,
 // have the spoke adopt it, and verify the post-join contract holds.
 const { execFileSync } = require('child_process');
 const crypto = require('crypto');
 // Wrapper matching the async call sites below (execFileSync throws
 // synchronously; wrap in a resolved/rejected promise so callers can keep
 // using await/.catch()). NOTE: plain execFile (async) does NOT support the
 // `input` option for piping stdin -- only the *Sync variants do -- so
 // ldapadd/ldapmodify would otherwise hang forever waiting on stdin that never
 // arrives. This bit us once already; don't switch back to async execFile here
 // without adding real stdin piping.
 function execFileAsync(cmd, args, opts) {
  try {
    const stdout = execFileSync(cmd, args, { ...opts, stdio: ['pipe', 'pipe', 'pipe'] });
    return Promise.resolve({ stdout: stdout ? stdout.toString() : '' });
  } catch (e) {
    e.stderr = e.stderr ? e.stderr.toString() : '';
    return Promise.reject(e);
  }
 }
 const MASTER_URL = process.env.MASTER_URL || 'http://master:3001';
 const SPOKE_URL = process.env.SPOKE_URL || 'http://spoke:3001';
 const MASTER_LDAP_HOST = process.env.MASTER_LDAP_HOST || 'master';
 const MASTER_BASE_DN = process.env.MASTER_BASE_DN || 'dc=master,dc=test';
 const SPOKE_LDAP_HOST = process.env.SPOKE_LDAP_HOST || 'spoke';
 const SPOKE_BASE_DN = process.env.SPOKE_BASE_DN || 'dc=spoke,dc=test';
 const LDAP_ADMIN_PASS = process.env.LDAP_ADMIN_PASS || 'secret';
 const ADMIN_UID = 'e2eadmin';
 const ADMIN_PASSWORD = 'MultiSiteE2E!2';
 let failed = false;
 function fail(msg) {
  console.error('MULTISITE E2E FAIL:', msg);
  failed = true;
 }
 function step(msg) {
  console.log('--- ' + msg);
 }
 async function waitForHealthy(url, label) {
  for (let i = 0; i < 60; i++) {
    try {
      const r = await fetch(`${url}/health`);
      if (r.ok) return;
    } catch (_) { /* not up yet */ }
    await new Promise((res) => setTimeout(res, 1000));
  }
  throw new Error(`${label} never became healthy`);
 }
 // Seed an admin user directly via ldapadd/ldapmodify -- mirrors
 // test/seed-test-user.sh, but parameterized per-site since master and spoke
 // have distinct base DNs in this harness.
 async function seedAdmin(ldapHost, baseDn) {
  const salt = crypto.randomBytes(8);
  const digest = crypto.createHash('sha512').update(ADMIN_PASSWORD).update(salt).digest();
  const hash = '{SSHA512}' + Buffer.concat([digest, salt]).toString('base64');
  const ldif = `
 dn: cn=${ADMIN_UID},ou=groups,${baseDn}
 objectClass: posixGroup
 objectClass: top
 cn: ${ADMIN_UID}
 gidNumber: 1600
 dn: cn=${ADMIN_UID},ou=people,${baseDn}
 objectClass: inetOrgPerson
 objectClass: posixAccount
 objectClass: top
 objectClass: theta42Person
 objectClass: ldapPublicKey
 objectClass: sudoRole
 cn: ${ADMIN_UID}
 sn: E2E
 uid: ${ADMIN_UID}
 uidNumber: 1600
 gidNumber: 1600
 homeDirectory: /home/${ADMIN_UID}
 loginShell: /bin/bash
 mail: ${ADMIN_UID}@test.local
 userPassword: ${hash}
 `.trim() + '\n';
  const bindDn = `cn=admin,${baseDn}`;
  await execFileAsync('ldapadd', ['-x', '-H', `ldap://${ldapHost}:389`, '-D', bindDn, '-w', LDAP_ADMIN_PASS], { input: ldif })
    .catch((e) => { if (!/Already exists/.test(e.stderr || '')) throw e; });
  // god_admin is needed for site-promote (SUPER_ADMIN_GROUP, utils/permission.js).
  for (const group of ['app_sso_admin', 'god_admin']) {
    const modLdif = `dn: cn=${group},ou=groups,${baseDn}\nchangetype: modify\nadd: member\nmember: cn=${ADMIN_UID},ou=people,${baseDn}\n`;
    try {
      await execFileAsync('ldapmodify', ['-x', '-H', `ldap://${ldapHost}:389`, '-D', bindDn, '-w', LDAP_ADMIN_PASS], { input: modLdif });
      console.log(`    (added ${ADMIN_UID} to ${group} on ${ldapHost})`);
    } catch (e) {
      if (!/[Tt]ype or value exists/.test(e.stderr || '')) {
        console.error(`    FAILED adding ${ADMIN_UID} to ${group} on ${ldapHost}: ${e.stderr || e.message}`);
        throw e;
      }
      console.log(`    (${ADMIN_UID} already in ${group} on ${ldapHost})`);
    }
  }
  const verify = await execFileAsync('ldapsearch', ['-x', '-H', `ldap://${ldapHost}:389`, '-D', bindDn, '-w', LDAP_ADMIN_PASS,
    '-b', `cn=god_admin,ou=groups,${baseDn}`, 'member']);
  console.log(`    god_admin members on ${ldapHost}:\n${verify.stdout}`);
 }
 async function login(url) {
  const r = await fetch(`${url}/api/auth/login`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ uid: ADMIN_UID, password: ADMIN_PASSWORD })
  });
  if (!r.ok) throw new Error(`login at ${url} failed: ${r.status} ${await r.text()}`);
  const body = await r.json();
  return body.token;
 }
 async function api(url, path, { method = 'GET', token, body } = {}) {
  const r = await fetch(`${url}${path}`, {
    method,
    headers: {
      'Content-Type': 'application/json',
      ...(token ? { 'auth-token': token } : {})
    },
    body: body ? JSON.stringify(body) : undefined
  });
  const text = await r.text();
  let json;
  try { json = JSON.parse(text); } catch (_) { json = { raw: text }; }
  return { status: r.status, body: json };
 }
 async function main() {
  step('Waiting for master + spoke to be healthy');
  await waitForHealthy(MASTER_URL, 'master');
  await waitForHealthy(SPOKE_URL, 'spoke');
  step('Seeding admin users in both sites\' LDAP');
  await seedAdmin(MASTER_LDAP_HOST, MASTER_BASE_DN);
  await seedAdmin(SPOKE_LDAP_HOST, SPOKE_BASE_DN);
  step('Logging in as admin on master and spoke');
  const masterToken = await login(MASTER_URL);
  const spokeToken = await login(SPOKE_URL);
  if (!masterToken) fail('no token from master login');
  if (!spokeToken) fail('no token from spoke login');
  step('Confirming both sites start as master (fresh installs)');
  {
    const { body } = await api(MASTER_URL, '/api/site/config', { token: masterToken });
    if (body.config.isMaster !== true) fail(`expected master to start isMaster:true, got ${JSON.stringify(body.config)}`);
  }
  {
    const { body } = await api(SPOKE_URL, '/api/site/config', { token: spokeToken });
    if (body.config.isMaster !== true) fail(`expected spoke to start isMaster:true (pre-join), got ${JSON.stringify(body.config)}`);
  }
  step('Creating a resource on master BEFORE join, to verify it gets adopted');
  // Only site resources can be top-level; a host needs a parent site.
  const siteRes = await api(MASTER_URL, '/api/directory-admin/resources', {
    method: 'POST',
    token: masterToken,
    body: { name: 'E2E Site', slug: 'site_e2e', kind: 'site' }
  });
  if (siteRes.status !== 200) fail(`seeding pre-join site on master failed: ${siteRes.status} ${JSON.stringify(siteRes.body)}`);
  const seedRes = await api(MASTER_URL, '/api/directory-admin/resources', {
    method: 'POST',
    token: masterToken,
    body: { name: 'E2E Pre-Join Host', slug: 'host_e2e_prejoin', kind: 'host', parentSlug: 'site_e2e' }
  });
  if (seedRes.status !== 200) fail(`seeding pre-join resource on master failed: ${seedRes.status} ${JSON.stringify(seedRes.body)}`);
  step('Minting a site join key on master');
  const keyRes = await api(MASTER_URL, '/api/site/join-keys', {
    method: 'POST',
    token: masterToken,
    body: { label: 'e2e-test' }
  });
  if (keyRes.status !== 200 || !keyRes.body.key) fail(`join-key mint failed: ${keyRes.status} ${JSON.stringify(keyRes.body)}`);
  const joinKey = keyRes.body.key;
  step('Joining spoke to master (with selfUrl, to register for live replication)');
  const joinRes = await api(SPOKE_URL, '/api/site/join', {
    method: 'POST',
    token: spokeToken,
    // master's own container-internal URL, as the spoke would reach it over the network
    body: { masterUrl: 'http://master:3001', joinKey, selfUrl: 'http://spoke:3001' }
  });
  if (joinRes.status !== 200) fail(`join failed: ${joinRes.status} ${JSON.stringify(joinRes.body)}`);
  if (!joinRes.body.replication || joinRes.body.replication.live !== true) {
    fail(`expected join to register for live replication, got ${JSON.stringify(joinRes.body.replication)}`);
  }
  step('Verifying spoke persisted isMaster:false + masterUrl after join');
  const { body: spokeCfg } = await api(SPOKE_URL, '/api/site/config', { token: spokeToken });
  if (spokeCfg.config.isMaster !== false) fail(`spoke should be isMaster:false after join, got ${JSON.stringify(spokeCfg.config)}`);
  if (!spokeCfg.config.masterUrl) fail('spoke should have masterUrl set after join');
  step('Verifying the spoke adopted the master\'s pre-join catalog');
  const spokeResources = await api(SPOKE_URL, '/api/directory-admin/resources', { token: spokeToken });
  const adopted = (spokeResources.body.results || spokeResources.body.resources || spokeResources.body || []);
  const found = Array.isArray(adopted) && adopted.some(r => r.slug === 'host_e2e_prejoin');
  if (!found) fail(`spoke did not adopt master's pre-join resource; got slugs=${JSON.stringify((adopted || []).map(r => r.slug))}`);
  step('Verifying spoke is now read-only (write attempt must 403)');
  const writeAttempt = await api(SPOKE_URL, '/api/directory-admin/resources', {
    method: 'POST',
    token: spokeToken,
    body: { name: 'Should Be Rejected', slug: 'host_e2e_should_reject', kind: 'host' }
  });
  if (writeAttempt.status !== 403) fail(`expected 403 writing to spoke post-join, got ${writeAttempt.status} ${JSON.stringify(writeAttempt.body)}`);
  step('Creating a resource on master AFTER join, to verify LIVE replication (not just the one-time join snapshot)');
  const postJoinRes = await api(MASTER_URL, '/api/directory-admin/resources', {
    method: 'POST',
    token: masterToken,
    body: { name: 'E2E Post-Join Host', slug: 'host_e2e_postjoin', kind: 'host', parentSlug: 'site_e2e' }
  });
  if (postJoinRes.status !== 200) fail(`creating post-join resource on master failed: ${postJoinRes.status} ${JSON.stringify(postJoinRes.body)}`);
  step('Waiting for the fire-and-forget resync push to reach the spoke');
  let liveReplicated = false;
  for (let i = 0; i < 20; i++) {
    const r = await api(SPOKE_URL, '/api/directory-admin/resources', { token: spokeToken });
    const slugs = (r.body.results || r.body.resources || r.body || []).map((x) => x.slug);
    if (slugs.includes('host_e2e_postjoin')) { liveReplicated = true; break; }
    await new Promise((res) => setTimeout(res, 500));
  }
  if (!liveReplicated) fail('post-join resource never appeared on the spoke -- live replication did not fire (or resync did not apply it)');
  step('Verifying WAN health ping from spoke to master succeeds');
  const statusRes = await api(SPOKE_URL, '/api/directory-admin/site-status', { token: spokeToken });
  if (statusRes.body.config && statusRes.body.config.wanConnected !== true) {
    fail(`expected spoke to report wanConnected:true post-join, got ${JSON.stringify(statusRes.body.config)}`);
  }
  step('Verifying master itself is unaffected (still isMaster:true, no writes blocked)');
  const { body: masterCfg } = await api(MASTER_URL, '/api/site/config', { token: masterToken });
  if (masterCfg.config.isMaster !== true) fail('master flipped away from isMaster:true unexpectedly');
  step('Promoting the spoke to master (coordinated handoff -- must demote the old master too)');
  const promoteRes = await api(SPOKE_URL, '/api/directory-admin/site-promote', {
    method: 'POST',
    token: spokeToken,
    body: { selfUrl: 'http://spoke:3001' }
  });
  if (promoteRes.status !== 200) fail(`promotion failed: ${promoteRes.status} ${JSON.stringify(promoteRes.body)}`);
  if (promoteRes.body.handoff !== 'previous master demoted') {
    fail(`expected the old master to be demoted as part of promotion, got handoff=${JSON.stringify(promoteRes.body.handoff)}`);
  }
  step('Verifying the newly-promoted node is master');
  const { body: newMasterCfg } = await api(SPOKE_URL, '/api/site/config', { token: spokeToken });
  if (newMasterCfg.config.isMaster !== true) fail(`newly-promoted node should be isMaster:true, got ${JSON.stringify(newMasterCfg.config)}`);
  step('Verifying the old master was actually demoted to a spoke of the new master');
  const { body: oldMasterCfg } = await api(MASTER_URL, '/api/site/config', { token: masterToken });
  if (oldMasterCfg.config.isMaster !== false) fail(`old master should be isMaster:false after being demoted, got ${JSON.stringify(oldMasterCfg.config)}`);
  if (oldMasterCfg.config.masterUrl !== 'http://spoke:3001') {
    fail(`old master's masterUrl should now point at the new master, got ${JSON.stringify(oldMasterCfg.config.masterUrl)}`);
  }
  step('Verifying the (now-demoted) old master rejects writes, and the new master accepts them');
  const oldMasterWrite = await api(MASTER_URL, '/api/directory-admin/resources', {
    method: 'POST',
    token: masterToken,
    body: { name: 'Should Be Rejected Post-Demotion', slug: 'host_e2e_should_reject_2', kind: 'host' }
  });
  if (oldMasterWrite.status !== 403) fail(`expected 403 writing to the demoted old master, got ${oldMasterWrite.status} ${JSON.stringify(oldMasterWrite.body)}`);
  const newMasterWrite = await api(SPOKE_URL, '/api/directory-admin/resources', {
    method: 'POST',
    token: spokeToken,
    body: { name: 'E2E Post-Promotion Host', slug: 'host_e2e_postpromotion', kind: 'host', parentSlug: 'site_e2e' }
  });
  if (newMasterWrite.status !== 200) fail(`expected the newly-promoted master to accept writes, got ${newMasterWrite.status} ${JSON.stringify(newMasterWrite.body)}`);
  if (failed) {
    console.error('MULTISITE E2E: one or more checks failed (see above)');
    process.exit(1);
  }
  console.log('MULTISITE E2E PASS');
 }
 main().catch((e) => {
  console.error('MULTISITE E2E FAIL (exception):', e.stack || e.message);
  process.exit(1);
 });