OpenLDAP N-way multi-master replication (docs/replication.md) required
an operator to hand-set LDAP_SERVER_ID (unique per site) and
LDAP_REPLICATION_HOSTS (every OTHER site's LDAP URL, kept in sync by
hand across every node) -- real coordination work, and easy to get
wrong or let drift as sites are added.
Automates the coordination the master is already in a position to do:
- SiteSpoke gets ldapServerId, auto-assigned (next free from 2 upward,
1 reserved for the master) at registration and reused across
re-registrations -- same pattern as jump-host's mesh index.
- ldapHost is derived from each site's already-known HTTP(S) endpoint
(same hostname, port 636) rather than a separately-configured field
that could drift from it.
- New utils/ldap_replication.js (nextFreeLdapServerId, ldapHostFor),
shared between the spoke-facing GET /api/site/ldap-peers (Bearer
site join key, returns this caller's own ID + every peer) and the
master-local GET /directory-admin/ldap-replication-config (computes
its own config directly from SiteSpoke, no HTTP round-trip needed).
Verified against real running containers (docker-compose.multisite-e2e.yml):
after a real join, the master's computed config correctly includes the
spoke as a peer with an assigned ID, and the spoke's own fetched
config matches that ID and correctly excludes itself from its own
peer list.
Known limitation, documented in docs/replication.md: the master's own
LDAP_REPLICATION_HOSTS only gets recomputed when ITS setup.sh is
re-run (or an admin re-applies it directly) -- there's no live push to
an already-running master when a new spoke joins. A spoke's own config
is re-checked on every setup.sh run, which is the common/recurring
event; the master side is a documented manual step for now rather than
a live hot-reload (which would need OpenLDAP's dynamic cn=config
backend -- a bigger change, deliberately out of scope here to avoid
risking a live directory's LDAP replication on undertested config).
Implements TODO items "service-to-service auth model" and "no-inbound
relay automation" together -- the first was never really "no credential
type exists", it was that nothing wired one of the credential types that
ALREADY exist (theta-proxy, jump-host, and this app each already have
their own self-service API token system, models/api_token.js) into an
actual inter-service call. This is that wiring, not a new invented
credential type.
- utils/proxy_client.js: ensureRelayRoute({host, ip, targetPort}) calls
theta-proxy's real Host API (GET/POST/PUT /api/host) using a `prx_...`
token an operator mints on theta-proxy and stores in OpenBao
(secret/integrations/theta-proxy), same pattern as agent_keys.js.
Idempotent and best-effort -- never fails the caller if the token/URL
isn't configured, since this is an enhancement on top of a working
join, not a join requirement.
- POST /api/site/spokes accepts optional noInbound/meshIp/publicHost
fields; when a spoke reports itself no-inbound, the master
best-effort creates/updates the matching relay route automatically.
SiteSpoke gained the fields + a relayNote for visibility.
Verified against a REAL running theta-proxy container (not mocked):
booted it standalone, logged in as the local admin, minted a real
`prx_` token via its actual API, and drove ensureRelayRoute() against
it for real. Caught a real bug doing this: GET /api/host/:item wraps
the record in `{item, results}`, not flat -- the mocked unit tests
(which I wrote first) all had the flat shape baked in and passed
cleanly, so this only surfaced against the real API. Fixed in both the
implementation and the unit tests' mocked response shape.
The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's
catalog never updated after joining. This adds the two pieces that were
explicitly designed but missing:
- Live replication: a spoke registers its own endpoint with the master
right after joining (POST /api/site/spokes, Bearer join-key), receiving
a pushToken. Every successful catalog write on the master now fires a
fire-and-forget resync ping (utils/site_replicate.js) at every known
spoke, concurrently -- one unreachable spoke never blocks or delays
another (wired into the existing write-gate middleware in
api_directory_admin.js). The spoke's POST /api/site/resync handler
reuses the already-tested export+import path rather than applying a
partial diff.
- Identical directories: POST /api/site/export now best-effort includes
the master's agent-signing key; a spoke adopts it via agent_keys.adopt()
on both join and every resync, so every site's sso-manager can validly
sign a command for any agent enrolled anywhere -- the accepted tradeoff
discussed for this deployment's scale (blast radius for simplicity).
New SiteSpoke model tracks registered spokes (endpoint + pushToken);
registered it in models/index.js (a real bug the e2e test below caught --
SiteSpoke.list() 500'd with "Cannot read properties of null (reading
'adapter')" until the model was added to initORM's model list).
Verified end-to-end against docker-compose.multisite-e2e.yml: mint join
key -> join with selfUrl -> write a NEW resource on master post-join ->
poll the spoke -> it shows up within a few seconds via the resync push,
no manual re-join needed. MULTISITE E2E PASS.
Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one
failing spoke doesn't block another, empty-registry and list()-throws
edge cases).
wmantly