Two previously-unrelated "site slug" concepts existed: the Directory
catalog's site Resource (site.slug, what group names and the resource
tree actually use -- e.g. "E2E Site" / site_e2e) vs. site_config.js's
siteSlug (the multi-site replication identity shown on the Multi-Site
modal's "Local Site Slug" row, sourced only from a separately-set
SITE_SLUG env var). Nothing ever kept them in sync -- a real
deployment could show a real site name in the Directory tree and the
literal "site-default" fallback on the Multi-Site modal for the exact
same node, which is exactly what a live demo of the modal surfaced.
Synced at the source: POST /resources now sets site_config's siteSlug
to match, the moment this node's own site Resource is first created
(bootstrap.js's initial call). Only for a still-default master --
never overwrites a real multi-site identity a join/promote has
already established, and never touches a spoke's identity (the
master's to assign via registration, not this node's own resource
creation to decide).
Verified against a real running container via
docker-compose.multisite-e2e.yml's existing site-creation step.
The modal previously showed zero LDAP replication status -- no
ServerID, no MMR active/inactive indicator, nothing -- and only
aggregate counts, never per-spoke detail (no-inbound flag, relay
note, assigned ldapServerId). An operator had no way to tell whether
replication was actually configured/working without SSHing in.
Added utils/ldap_replication.js's currentSlapdServerId(), which reads
the ACTUAL running ServerID straight from this node's own slapd.conf
-- distinct from what GET /ldap-peers / /ldap-replication-config
currently ADVERTISE for it, which can genuinely disagree right after a
promotion or a new spoke joining (OpenLDAP's static config only
reloads at process start). GET /directory-admin/site-status now
surfaces both plus a `stale` flag, and a full spokes list (not just a
count) with each one's endpoint, LDAP ServerID, and relay path.
directory.ejs renders this as an "LDAP Replication (MMR)" status row
(ServerID, peer count, a "needs setup.sh re-run" warning when stale)
and a Registered Spokes table.
Verified two ways: real running containers via
docker-compose.multisite-e2e.yml (new site-status assertions), and an
actual browser session against the promoted node -- screenshotted the
rendered modal showing the real registered spoke with its assigned
ldapServerId and the correctly-surfaced "not configured (standalone)"
MMR state (this test node never ran site-ldap-register.js against it,
so the mismatch itself is the expected, documented behavior).
Found while auditing the new LDAP MMR auto-config for gaps: neither
POST /site-promote nor POST /demote ever touched SiteSpoke. Two real
problems:
1. The demoted old master got a fresh masterJoinKey but was never
registered as a spoke of the new master -- no SiteSpoke row, no
ldapServerId, invisible to GET /ldap-peers's peer list. It also
structurally could not self-heal: POST /join refuses re-join for a
node that's already a spoke, and separately requires a fresh
install (siteIsFresh()) -- neither true for a former master with
real users/agents. Fixed: /demote now registers itself with the new
master immediately (POST /spokes), the same way a real join does,
deriving its own endpoint from stack.selfUrl (override) or
https://stack.ssoHost (the normal case).
2. The promoted node's live OpenLDAP ServerID doesn't change --
GET /ldap-replication-config starts advertising 1 for it
immediately (derived purely from cfg.isMaster), but nothing
restarts slapd with that value (OpenLDAP's static slapd.conf only
reloads at process start, and this app has no safe way to restart
its own container). Can't be fixed in-process; surfaced instead --
/site-promote's response now includes ldapReplicationNote telling
the operator to re-run setup.sh promptly.
Verified against real running containers (docker-compose.multisite-e2e.yml):
after promotion, the demoted old master correctly appears in the new
master's LDAP peer list with a real assigned ldapServerId.
OpenLDAP N-way multi-master replication (docs/replication.md) required
an operator to hand-set LDAP_SERVER_ID (unique per site) and
LDAP_REPLICATION_HOSTS (every OTHER site's LDAP URL, kept in sync by
hand across every node) -- real coordination work, and easy to get
wrong or let drift as sites are added.
Automates the coordination the master is already in a position to do:
- SiteSpoke gets ldapServerId, auto-assigned (next free from 2 upward,
1 reserved for the master) at registration and reused across
re-registrations -- same pattern as jump-host's mesh index.
- ldapHost is derived from each site's already-known HTTP(S) endpoint
(same hostname, port 636) rather than a separately-configured field
that could drift from it.
- New utils/ldap_replication.js (nextFreeLdapServerId, ldapHostFor),
shared between the spoke-facing GET /api/site/ldap-peers (Bearer
site join key, returns this caller's own ID + every peer) and the
master-local GET /directory-admin/ldap-replication-config (computes
its own config directly from SiteSpoke, no HTTP round-trip needed).
Verified against real running containers (docker-compose.multisite-e2e.yml):
after a real join, the master's computed config correctly includes the
spoke as a peer with an assigned ID, and the spoke's own fetched
config matches that ID and correctly excludes itself from its own
peer list.
Known limitation, documented in docs/replication.md: the master's own
LDAP_REPLICATION_HOSTS only gets recomputed when ITS setup.sh is
re-run (or an admin re-applies it directly) -- there's no live push to
an already-running master when a new spoke joins. A spoke's own config
is re-checked on every setup.sh run, which is the common/recurring
event; the master side is a documented manual step for now rather than
a live hot-reload (which would need OpenLDAP's dynamic cn=config
backend -- a bigger change, deliberately out of scope here to avoid
risking a live directory's LDAP replication on undertested config).
Two real bugs, both only surfaced by the live two-container e2e test
(docker-compose.multisite-e2e.yml), not by inspection:
1. POST /site-promote's god_admin check read req.user.groups -- a field
nothing in the codebase ever populates (Auth.checkToken returns
User.get(), which has no .groups; every other admin gate resolves
membership live via permission.byGroup()/Group.list(user.dn), which
also handles nested-group membership). The check silently evaluated to
an empty array on every request, so site-promote returned 403 for
every user, including a real god_admin -- unusable since it shipped in
v2.0.0. Fixed to use permission.byGroup(), the same pattern used
elsewhere in this file and in api_site.js.
2. The read-only write-gate middleware (api_directory_admin.js) is
registered before router.post('/site-promote', ...) later in the same
file, so on a spoke it 403'd every promotion attempt before the
handler ever ran -- the one mutating request a spoke must be able to
make to itself. Exempted /site-promote from the gate.
Added coordinated demotion (MULTI_SITE_SPEC.md §3.2 -- promotion as ONE
action, never a two-step gap with two masters): site-promote now calls
the previous master's new POST /api/site/demote (Bearer the join key it
already holds, handing over a freshly-minted key for the demoted node's
own future use) before flipping itself to master. Best-effort: an
unreachable old master never blocks a god_admin's local promotion (the
WAN-outage scenario is the entire reason this control exists), it's
just reported in the response for manual reconciliation.
e2e test extended to promote the spoke, verify the old master was
actually demoted (isMaster:false, masterUrl pointing at the new master),
and verify writes now succeed on the new master and 403 on the old one.
Full chain verified passing: join -> live replication -> promotion ->
demotion -> write authority follows the promotion.
The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's
catalog never updated after joining. This adds the two pieces that were
explicitly designed but missing:
- Live replication: a spoke registers its own endpoint with the master
right after joining (POST /api/site/spokes, Bearer join-key), receiving
a pushToken. Every successful catalog write on the master now fires a
fire-and-forget resync ping (utils/site_replicate.js) at every known
spoke, concurrently -- one unreachable spoke never blocks or delays
another (wired into the existing write-gate middleware in
api_directory_admin.js). The spoke's POST /api/site/resync handler
reuses the already-tested export+import path rather than applying a
partial diff.
- Identical directories: POST /api/site/export now best-effort includes
the master's agent-signing key; a spoke adopts it via agent_keys.adopt()
on both join and every resync, so every site's sso-manager can validly
sign a command for any agent enrolled anywhere -- the accepted tradeoff
discussed for this deployment's scale (blast radius for simplicity).
New SiteSpoke model tracks registered spokes (endpoint + pushToken);
registered it in models/index.js (a real bug the e2e test below caught --
SiteSpoke.list() 500'd with "Cannot read properties of null (reading
'adapter')" until the model was added to initORM's model list).
Verified end-to-end against docker-compose.multisite-e2e.yml: mint join
key -> join with selfUrl -> write a NEW resource on master post-join ->
poll the spoke -> it shows up within a few seconds via the resync push,
no manual re-join needed. MULTISITE E2E PASS.
Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one
failing spoke doesn't block another, empty-registry and list()-throws
edge cases).
docker-compose.multisite-e2e.yml boots two full all-in-one instances
(master + spoke, each with bundled slapd) and a client that drives the
actual HTTP API: seeds admins, mints a join key, joins the spoke, and
verifies the spoke persisted its role across restart, adopted the
pre-join catalog, is enforced read-only, and reports WAN health.
Verified passing locally. Existing docker-compose.test.yml/e2e.yml split
ldap+redis+app into separate containers, which doesn't work here --
POST /api/site/export runs slapcat in-process, so master and spoke each
need their own bundled slapd (Dockerfile.openldap), not a shared one.
wmantly