Merge pull request #210 from theta42/release-v2.8.0

release(v2.8.0): promotion/LDAP-orphan fix, site-slug unification, LDAP status UI

CHANGELOG.md

@@ -1,3 +1,12 @@
+# v2.8.0 - 2026-08-11
+
+### Fixed
+- **Promotion no longer orphans the demoted old master's LDAP replication.** Neither `/site-promote` nor `/demote` touched `SiteSpoke` -- the demoted old master got a fresh join key but no `SiteSpoke` entry on the new master (no `ldapServerId`, invisible to the peer list), and structurally could never self-heal via `/join` (refuses re-join for a node that's already a spoke). `/demote` now registers itself with the new master immediately, deriving its own endpoint from `stack.selfUrl`/`stack.ssoHost`. `/site-promote`'s response also now surfaces that the promoted node's own OpenLDAP ServerID needs a `setup.sh` re-run to actually apply.
+- **The Directory's site slug and the multi-site replication identity are unified.** These were two unrelated values that happened to share a name -- a deployment could show a real site name in the Directory catalog and the literal `site-default` fallback on the Multi-Site modal for the same node. `POST /resources` now syncs `site_config`'s `siteSlug` to match the moment this node's own site Resource is first created, for a still-default master only.
+
+### Added
+- **LDAP replication status + per-spoke detail on the Multi-Site modal.** New `utils/ldap_replication.js`'s `currentSlapdServerId()` reads the actual running ServerID from this node's own `slapd.conf` -- distinct from what the API currently advertises, which can genuinely disagree right after a promotion or a new spoke joining. `GET /directory-admin/site-status` now surfaces both plus a `stale` flag and a full spokes list (endpoint, assigned `ldapServerId`, relay path), not just an aggregate count.
+
 # v2.7.0 - 2026-08-11
 
 ### Added

docker-compose.multisite-e2e.yml

@@ -25,6 +25,11 @@ services:
       - LDAP_ADMIN_PASS=secret
       - ORG_NAME=E2E Master
       - app_oauth__jwtSecret=e2e-multisite-master-jwt-secret
+      # POST /demote's self-registration (routes/api_site.js) needs a real
+      # reachable endpoint for this container; stack.selfUrl overrides the
+      # normal https://<stack.ssoHost> derivation, which isn't reachable
+      # here (plain HTTP, no TLS/proxy in front, non-443 port).
+      - app_stack__selfUrl=http://master:3001
     healthcheck:
       test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"]
       interval: 2s
@@ -42,6 +47,7 @@ services:
       - LDAP_ADMIN_PASS=secret
       - ORG_NAME=E2E Spoke
       - app_oauth__jwtSecret=e2e-multisite-spoke-jwt-secret
+      - app_stack__selfUrl=http://spoke:3001
     healthcheck:
       test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"]
       interval: 2s

docs/site-join.md

@@ -135,11 +135,33 @@ already joined reports "already a spoke" and setup continues (idempotent).
 
 ## Not yet built
 
-- Traffic between sites (join/export/resync) still goes over the open
-  network path that already reaches the target — it does not route over the
-  WireGuard mesh `theta-gateway` can now establish (see `MULTI_SITE_SPEC.md`).
-- A no-inbound spoke (no public IP at all) still can't join — the mechanism
-  for a master to relay through the mesh to such a spoke is verified as
-  working, but nothing automates creating that route yet.
 - OpenBao secret replication covers only the agent-signing key; LDAP admin
   creds, JWT secret, and other per-deployment secrets aren't synced.
+- A promoted spoke's own OpenLDAP `ServerID` doesn't apply live -- `POST
+  /site-promote` starts advertising `1` for it immediately
+  (`GET /directory-admin/ldap-replication-config`), but nothing restarts
+  `slapd` with that value automatically (its static `slapd.conf` is only
+  read at process start). Re-run `setup.sh` on the newly-promoted node
+  promptly after promotion to actually apply it.
+- The master's own `LDAP_REPLICATION_HOSTS` peer list only recomputes on
+  its next `setup.sh` run, not live the instant a new spoke joins -- same
+  re-run-`setup.sh` caveat as above, just triggered by a join instead of a
+  promotion.
+
+## Shipped since the above was last stale
+
+- Traffic between sites (`utils/site_replicate.js`'s resync push) prefers a
+  registered spoke's WireGuard mesh IP over the open internet when one's on
+  file, falling back to the public endpoint on failure.
+- A no-inbound spoke (no public IP at all) CAN join: `noInbound`/`meshIp`/
+  `publicHost` on `POST /api/site/join` drive `utils/proxy_client.js`, which
+  auto-creates/updates the relay route on the master's own `theta-proxy`.
+  Mesh peering between the two jump-hosts is still a manual, one-time step
+  (see `theta-suite`'s `spoke.env.example` for the operator-facing side).
+  A spoke with zero inbound *and* zero outbound path still can't join --
+  the join itself needs to reach the master's API directly.
+- OpenLDAP N-way multi-master replication now auto-configures on join --
+  the master assigns each spoke a unique `LDAP_SERVER_ID` and derives every
+  site's `ldaps://` URL automatically (`GET /api/site/ldap-peers`,
+  `GET /directory-admin/ldap-replication-config`). See
+  `docs/replication.md`.

nodejs/package.json

@@ -1,6 +1,6 @@
 {
   "name": "t42-theta-directory",
-  "version": "2.7.0",
+  "version": "2.8.0",
   "description": "A very simple LDAP management and SSO system",
   "author": [
     {

nodejs/routes/api_directory_admin.js

@@ -365,6 +365,25 @@ router.post('/resources', async (req, res, next) => {
     const ancestorSite = await Resource.findAncestorSiteSlug(r.id);
     if (r.kind === 'site') {
       await ensureSiteGroups(r.slug, req.user.dn, r.name, r.id);
+      // Two previously-unrelated "site slug" concepts: this Resource's own
+      // slug (the Directory catalog's site container -- what every group
+      // name and the resource tree actually use) vs. site_config.js's
+      // siteSlug (the multi-site replication identity shown on the
+      // Multi-Site modal, sourced only from a separately-set SITE_SLUG env
+      // var). They coincidentally share the name "site slug" but nothing
+      // ever kept them in sync -- a real deployment could show "E2E Site"
+      // in the Directory tree and "site-default" on the Multi-Site modal
+      // for the exact same node. Sync them here, the moment this node's own
+      // site Resource is created (bootstrap.js's first call), so there's
+      // one real identity instead of two that can drift apart. Only for a
+      // still-default master: never overwrite a real multi-site identity a
+      // join/promote has already established, and a spoke's replication
+      // identity is the master's to assign, not this node's own resource
+      // creation to decide.
+      const cfg = siteConfig.get();
+      if (cfg.isMaster && cfg.siteSlug === 'site-default') {
+        siteConfig.save({ siteSlug: r.slug });
+      }
     } else if (gKind && ancestorSite) {
       await ensureSiteGroups(ancestorSite, req.user.dn, r.name); // backfill site tier if missing
       await provisionResourceGroups(r, gKind, ancestorSite, req.user.dn);
@@ -962,7 +981,7 @@ const siteConfig = require('../utils/site_config');
 const { siteIsFresh } = require('../utils/site_join');
 const { Agent } = require('../models/agent');
 const { SiteSpoke } = require('../models/site_spoke');
-const { ldapHostFor } = require('../utils/ldap_replication');
+const { ldapHostFor, currentSlapdServerId } = require('../utils/ldap_replication');
 
 // probeMasterHealth checks whether this (spoke) node can reach its master over
 // the site join key. The master's /api/site/ping is deliberately lightweight.
@@ -1002,13 +1021,29 @@ router.get('/site-status', async (req, res, next) => {
     // via an older bootstrap, or the UI form before it grew the field) is
     // fully joined but silently stuck on the one-time snapshot, which was
     // otherwise invisible anywhere in the UI.
-    const registeredSpokesCount = cfg.isMaster ? await SiteSpoke.list().then(l => l.length).catch(() => 0) : 0;
+    const allSpokes = cfg.isMaster ? await SiteSpoke.list().catch(() => []) : [];
+    const registeredSpokesCount = allSpokes.length;
     // Real gateway-to-gateway mesh peer count from jump-host's own registry
     // (utils/jump_client.js), not this app's unrelated WireGuard
     // roaming-client Resources. count is null (not 0) when the query
     // couldn't run at all -- the UI distinguishes "0 gateways" from "can't
     // tell" instead of showing a misleading zero.
     const gateways = await jumpClient.getGatewayCount();
+
+    // LDAP MMR status (docs/replication.md): configuredServerId is read
+    // straight from THIS node's own live slapd.conf; advertisedServerId is
+    // what GET /ldap-peers / /ldap-replication-config currently hand out for
+    // it. These can genuinely disagree -- a promotion or a newly-joined
+    // spoke changes the advertised value immediately, but OpenLDAP's static
+    // config only reloads at process start, so a mismatch means "re-run
+    // setup.sh here" rather than "something's broken". Only computed for the
+    // master (a spoke's advertised ID lives on the master, not locally, and
+    // querying it here would mean another WAN round-trip on every page load).
+    const configuredServerId = currentSlapdServerId();
+    const ldap = cfg.isMaster
+      ? { configuredServerId, advertisedServerId: 1, stale: configuredServerId !== null && configuredServerId !== 1, peersCount: allSpokes.filter(s => s.ldapServerId).length }
+      : { configuredServerId, advertisedServerId: null, stale: null, peersCount: null };
+
     res.json({
       status: 'ok',
       config: {
@@ -1024,7 +1059,16 @@ router.get('/site-status', async (req, res, next) => {
       sitesCount: sites.length,
       sites: sites.map(s => ({ id: s.id, name: s.name, slug: s.slug })),
       gatewaysCount: gateways.count,
-      gatewaysNote: gateways.note
+      gatewaysNote: gateways.note,
+      ldap,
+      // Per-spoke detail (master only) -- endpoint/siteSlug/noInbound/
+      // relayNote/ldapServerId, not just an aggregate count, so an operator
+      // can actually see what's registered instead of only "N spokes".
+      spokes: allSpokes.map(s => ({
+        siteSlug: s.siteSlug, endpoint: s.endpoint, noInbound: !!s.noInbound,
+        relayNote: s.relayNote || null, ldapServerId: s.ldapServerId || null,
+        lastSeenOn: s.last_seen_on || null
+      }))
     });
   } catch (err) { next(err); }
 });
@@ -1116,6 +1160,16 @@ router.post('/site-promote', async (req, res, next) => {
       status: 'ok',
       message: 'Node successfully promoted to Master Site',
       handoff: handoffNote,
+      // This node's own OpenLDAP ServerID stays whatever it was as a spoke
+      // (e.g. 2) until `setup.sh` is re-run here -- GET
+      // /ldap-replication-config will immediately start advertising 1 for
+      // this node (the master's reserved ID) since that's derived purely
+      // from cfg.isMaster, but nothing restarts slapd with the new value
+      // automatically (OpenLDAP's static slapd.conf is only read at process
+      // start, and this app has no safe way to restart its own container).
+      // Surfaced here + on the Multi-Site modal so an operator promoting a
+      // site knows to re-run setup.sh promptly, not just assume it's done.
+      ldapReplicationNote: 'Re-run setup.sh on this node to apply its new LDAP ServerID (1) and pick up the current spoke peer list -- OpenLDAP config only reloads at process start.',
       config: {
         isMaster: true,
         masterUrl: '',

nodejs/routes/api_site.js

@@ -260,7 +260,44 @@ router.post('/demote', async (req, res, next) => {
     const base = String(newMasterUrl).replace(/\/+$/, '');
     siteConfig.save({ isMaster: false, masterUrl: base, masterJoinKey: newJoinKey });
     logAudit('demoted', { demotedBy: key.keyPrefix, newMasterUrl: base });
-    res.json({ status: 'ok', message: 'Demoted to spoke of ' + base });
+
+    // Register with the new master immediately, the same way a real /join
+    // does (POST /spokes) -- without this, a demoted former master was
+    // orphaned: it had a masterJoinKey but no SiteSpoke entry on the new
+    // master (so no ldapServerId, no live replication push target), and
+    // structurally could never self-heal via /join (which refuses re-join
+    // for a node that's already a spoke, and requires a fresh install --
+    // neither true for a former master with real users/agents). Best-effort:
+    // failing to register here must not fail the demotion itself, same
+    // reasoning as a normal join's optional live-replication registration.
+    let registrationNote = 'not attempted (no stack.ssoHost/stack.selfUrl configured to register with)';
+    // stack.selfUrl is a full-URL override (scheme + port) for environments
+    // where "https://<ssoHost>" isn't the real reachable address -- the
+    // multisite e2e test harness (plain HTTP, docker-network hostnames,
+    // no TLS/proxy in front) is exactly that case; every real deployment
+    // just relies on the ssoHost derivation.
+    const selfUrl = (conf.stack && conf.stack.selfUrl) || (conf.stack && conf.stack.ssoHost && `https://${conf.stack.ssoHost}`);
+    if (selfUrl) {
+      try {
+        const regResp = await fetch(base + '/api/site/spokes', {
+          method: 'POST',
+          headers: { Authorization: 'Bearer ' + newJoinKey, 'Content-Type': 'application/json' },
+          body: JSON.stringify({ endpoint: selfUrl, siteSlug: cfg.siteSlug })
+        });
+        if (regResp.ok) {
+          const regBody = await regResp.json();
+          if (regBody.pushToken) siteConfig.save({ replicationPushToken: regBody.pushToken });
+          registrationNote = 'registered as a spoke of the new master';
+        } else {
+          registrationNote = 'registration failed: HTTP ' + regResp.status;
+        }
+      } catch (e) {
+        registrationNote = 'registration failed: ' + e.message;
+      }
+    }
+    logAudit('demoted_self_registered', { newMasterUrl: base, registrationNote });
+
+    res.json({ status: 'ok', message: 'Demoted to spoke of ' + base, registration: { note: registrationNote } });
   } catch (e) { next(e); }
 });
 

nodejs/utils/ldap_replication.js

@@ -1,5 +1,7 @@
 'use strict';
 
+const fs = require('fs');
+
 // OpenLDAP multi-master replication (docs/replication.md) config derivation,
 // shared between routes/api_site.js (the spoke-facing side: assigns a
 // ServerID at registration, serves GET /api/site/ldap-peers) and
@@ -41,4 +43,26 @@ function ldapHostFor(endpoint) {
 	}
 }
 
-module.exports = { MAX_LDAP_SERVER_ID, nextFreeLdapServerId, ldapHostFor };
+const SLAPD_CONF_PATH = process.env.SLAPD_CONF_PATH || '/etc/openldap/slapd.conf';
+
+// The ServerID this node's OpenLDAP is ACTUALLY running with right now, read
+// straight from slapd.conf (the same file docker-entrypoint.sh writes
+// `ServerID <n>` into). This can genuinely differ from what
+// GET /ldap-peers / /ldap-replication-config currently ADVERTISE for this
+// node -- OpenLDAP's static slapd.conf is only read at process start, so a
+// promotion or a new spoke joining doesn't retroactively change what's
+// already running until `setup.sh` restarts the container. Surfaced on the
+// Multi-Site modal so an operator can see "configured X, but slapd is still
+// running Y" instead of assuming replication is live because the API says so.
+function currentSlapdServerId() {
+	let contents;
+	try {
+		contents = fs.readFileSync(SLAPD_CONF_PATH, 'utf8');
+	} catch (e) {
+		return null;
+	}
+	const m = contents.match(/^ServerID\s+(\d+)/m);
+	return m ? Number(m[1]) : null;
+}
+
+module.exports = { MAX_LDAP_SERVER_ID, nextFreeLdapServerId, ldapHostFor, currentSlapdServerId };

nodejs/views/directory.ejs

@@ -3319,6 +3319,55 @@
     } catch (e) { console.error('Failed to fetch site status:', e); }
   }
 
+  // ldap: { configuredServerId, advertisedServerId, stale, peersCount } from
+  // GET /directory-admin/site-status (routes/api_directory_admin.js).
+  // configuredServerId is read from THIS node's live slapd.conf;
+  // advertisedServerId (master only) is what the API currently hands spokes.
+  // They can genuinely disagree right after a promotion or a new spoke
+  // joining -- OpenLDAP's static config only reloads at process start.
+  function renderLdapStatus(ldap) {
+    if (!ldap) return '<span class="text-muted">unknown</span>';
+    if (ldap.configuredServerId == null) {
+      return '<span class="badge bg-secondary"><i class="fa-solid fa-circle-minus me-1"></i> Not configured (standalone)</span>';
+    }
+    let html = '<span class="badge bg-dark">ServerID ' + esc(ldap.configuredServerId) + '</span>';
+    if (ldap.peersCount != null) {
+      html += ' <span class="badge bg-primary">' + ldap.peersCount + ' peer' + (ldap.peersCount === 1 ? '' : 's') + '</span>';
+    }
+    if (ldap.stale) {
+      html += ' <span class="badge bg-warning text-dark" title="This node now advertises ServerID ' + esc(ldap.advertisedServerId) +
+        ' (e.g. after a promotion), but slapd is still running with ' + esc(ldap.configuredServerId) +
+        ' -- re-run setup.sh here to apply it."><i class="fa-solid fa-triangle-exclamation me-1"></i> Needs setup.sh re-run</span>';
+    }
+    return html;
+  }
+
+  // spokes: [{siteSlug, endpoint, noInbound, relayNote, ldapServerId, lastSeenOn}]
+  // Per-spoke detail (master only) so an operator can see what's actually
+  // registered instead of only an aggregate count.
+  function renderSpokesTable(spokes) {
+    if (!spokes || !spokes.length) return '';
+    const rows = spokes.map(function(s) {
+      return '<tr>' +
+        '<td><code>' + esc(s.siteSlug || '?') + '</code></td>' +
+        '<td class="small">' + esc(s.endpoint) + '</td>' +
+        '<td>' + (s.ldapServerId != null ? '<span class="badge bg-dark">' + esc(s.ldapServerId) + '</span>' : '<span class="text-muted small">unassigned</span>') + '</td>' +
+        '<td>' + (s.noInbound
+          ? '<span class="badge bg-info text-dark" title="' + esc(s.relayNote || '') + '"><i class="fa-solid fa-diagram-project me-1"></i> Relayed</span>'
+          : '<span class="text-muted small">direct</span>') + '</td>' +
+      '</tr>';
+    }).join('');
+    return '<div class="card mt-3">' +
+      '<div class="card-header py-2 fw-bold small"><i class="fa-solid fa-diagram-project me-1"></i> Registered Spokes</div>' +
+      '<div class="card-body p-0">' +
+        '<table class="table table-sm table-hover mb-0">' +
+          '<thead><tr><th>Site</th><th>Endpoint</th><th>LDAP ServerID</th><th>Path</th></tr></thead>' +
+          '<tbody>' + rows + '</tbody>' +
+        '</table>' +
+      '</div>' +
+    '</div>';
+  }
+
   async function openSiteStatusModal() {
     try {
       const res = await app.api.get('directory-admin/site-status');
@@ -3351,12 +3400,14 @@
               '<tr><th>Theta Gateways:</th><td>' + (res.gatewaysCount == null
                 ? '<span class="badge bg-secondary" title="' + esc(res.gatewaysNote || 'not configured') + '"><i class="fa-solid fa-question me-1"></i> Unknown (jump-host integration not configured)</span>'
                 : '<span class="badge bg-dark">' + res.gatewaysCount + ' active gateway' + (res.gatewaysCount === 1 ? '' : 's') + '</span>') + '</td></tr>' +
+              '<tr><th>LDAP Replication (MMR):</th><td>' + renderLdapStatus(res.ldap) + '</td></tr>' +
             '</table>' +
           '</div>' +
         '</div>' +
         '<div class="alert alert-secondary small mb-3">' +
           '<i class="fa-solid fa-network-wired me-1"></i> <strong>WireGuard Gateway Mesh & NETMAP</strong>: Inter-site routing operates via <code>theta-gateway</code> subnets (<code>10.x.0.0/16</code>) with default NETMAP shadow translations (<code>10.x.168.0/24 &rarr; 192.168.1.0/24</code>).' +
-        '</div>';
+        '</div>' +
+        (isMaster ? renderSpokesTable(res.spokes) : '');
 
       // Fresh install (no users/resources yet): offer to JOIN an existing
       // master site instead of seeding a new directory.

test/multisite_join_e2e.js

@@ -172,6 +172,12 @@ async function main() {
   });
   if (siteRes.status !== 200) fail(`seeding pre-join site on master failed: ${siteRes.status} ${JSON.stringify(siteRes.body)}`);
 
+  step('Verifying the Directory site Resource\'s slug synced into the multi-site replication identity');
+  const { body: masterCfgAfterSite } = await api(MASTER_URL, '/api/site/config', { token: masterToken });
+  if (masterCfgAfterSite.config.siteSlug !== 'site_e2e') {
+    fail(`expected site_config's siteSlug to sync to the new site Resource's slug (site_e2e), got ${JSON.stringify(masterCfgAfterSite.config.siteSlug)}`);
+  }
+
   const seedRes = await api(MASTER_URL, '/api/directory-admin/resources', {
     method: 'POST',
     token: masterToken,
@@ -229,6 +235,19 @@ async function main() {
   const selfInOwnPeerList = (spokeLdapCfg.peers || []).some(p => p.ldapServerId === spokeLdapCfg.ldapServerId);
   if (selfInOwnPeerList) fail(`spoke's own peer list should not include itself, got ${JSON.stringify(spokeLdapCfg.peers)}`);
 
+  step('Verifying the master\'s own site-status surfaces LDAP status + per-spoke detail (Multi-Site modal data)');
+  const { body: masterStatus } = await api(MASTER_URL, '/api/directory-admin/site-status', { token: masterToken });
+  if (!masterStatus.ldap || masterStatus.ldap.advertisedServerId !== 1) {
+    fail(`master's site-status should report ldap.advertisedServerId 1, got ${JSON.stringify(masterStatus.ldap)}`);
+  }
+  if (masterStatus.ldap.peersCount !== 1) {
+    fail(`master's site-status should report exactly 1 LDAP peer (the spoke), got ${JSON.stringify(masterStatus.ldap)}`);
+  }
+  const statusSpokeEntry = (masterStatus.spokes || []).find(s => s.endpoint === 'http://spoke:3001');
+  if (!statusSpokeEntry || typeof statusSpokeEntry.ldapServerId !== 'number') {
+    fail(`master's site-status spokes list should include the spoke with an ldapServerId, got ${JSON.stringify(masterStatus.spokes)}`);
+  }
+
   step('Verifying the spoke adopted the master\'s pre-join catalog');
   const spokeResources = await api(SPOKE_URL, '/api/directory-admin/resources', { token: spokeToken });
   const adopted = (spokeResources.body.results || spokeResources.body.resources || spokeResources.body || []);
@@ -281,6 +300,16 @@ async function main() {
   if (promoteRes.body.handoff !== 'previous master demoted') {
     fail(`expected the old master to be demoted as part of promotion, got handoff=${JSON.stringify(promoteRes.body.handoff)}`);
   }
+  if (!promoteRes.body.ldapReplicationNote) {
+    fail('expected /site-promote to surface a note that this node\'s LDAP ServerID needs a setup.sh re-run to apply');
+  }
+
+  step('Verifying the demoted old master auto-registered itself as a real spoke of the new master (not orphaned)');
+  const { body: newMasterLdapCfg } = await api(SPOKE_URL, '/api/directory-admin/ldap-replication-config', { token: spokeToken });
+  const oldMasterAsPeer = (newMasterLdapCfg.peers || []).find(p => p.ldapHost === 'ldaps://master:636');
+  if (!oldMasterAsPeer || typeof oldMasterAsPeer.ldapServerId !== 'number') {
+    fail(`the demoted old master should appear as a registered peer with an assigned ldapServerId, got ${JSON.stringify(newMasterLdapCfg.peers)}`);
+  }
 
   step('Verifying the newly-promoted node is master');
   const { body: newMasterCfg } = await api(SPOKE_URL, '/api/site/config', { token: spokeToken });