feat(multi-site): live catalog replication + identical-directory signing key

The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's
catalog never updated after joining. This adds the two pieces that were
explicitly designed but missing:

- Live replication: a spoke registers its own endpoint with the master
  right after joining (POST /api/site/spokes, Bearer join-key), receiving
  a pushToken. Every successful catalog write on the master now fires a
  fire-and-forget resync ping (utils/site_replicate.js) at every known
  spoke, concurrently -- one unreachable spoke never blocks or delays
  another (wired into the existing write-gate middleware in
  api_directory_admin.js). The spoke's POST /api/site/resync handler
  reuses the already-tested export+import path rather than applying a
  partial diff.

- Identical directories: POST /api/site/export now best-effort includes
  the master's agent-signing key; a spoke adopts it via agent_keys.adopt()
  on both join and every resync, so every site's sso-manager can validly
  sign a command for any agent enrolled anywhere -- the accepted tradeoff
  discussed for this deployment's scale (blast radius for simplicity).

New SiteSpoke model tracks registered spokes (endpoint + pushToken);
registered it in models/index.js (a real bug the e2e test below caught --
SiteSpoke.list() 500'd with "Cannot read properties of null (reading
'adapter')" until the model was added to initORM's model list).

Verified end-to-end against docker-compose.multisite-e2e.yml: mint join
key -> join with selfUrl -> write a NEW resource on master post-join ->
poll the spoke -> it shows up within a few seconds via the resync push,
no manual re-join needed. MULTISITE E2E PASS.

Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one
failing spoke doesn't block another, empty-registry and list()-throws
edge cases).
This commit is contained in:
2026-08-10 16:34:38 -04:00
parent e5167729a8
commit d27763e556
9 changed files with 451 additions and 57 deletions
+16 -2
View File
@@ -9,6 +9,7 @@ const { projectResources } = require('@simpleworkjs/directory-schema');
const SUPER_ADMIN_GROUP = permission.SUPER_ADMIN_GROUP;
const groups = require('../utils/groups');
const meshReplicate = require('../utils/site_replicate');
// Make `childCn` a member of `parentCn`, i.e. everyone in the child is
// transitively in the parent. Idempotent and non-fatal: "already a member" is
@@ -252,19 +253,32 @@ router.get('/resources', async (req, res, next) => {
} catch (err) { next(err); }
});
// ── Spoke read-only enforcement ─────────────────────────────────────────────
// ── Spoke read-only enforcement + live replication trigger ──────────────────
// On a joined spoke the catalog is a copy of the master's; directory writes
// must go to the master (MULTI_SITE_SPEC.md — spoke = read-only catalog). Any
// mutating request below this point is rejected on a spoke with a pointer to
// the master. (site-status / site-promote live AFTER this middleware and are
// not directory writes.)
//
// On the MASTER, a successful mutation here fires a fire-and-forget resync
// push (utils/site_replicate.js) at every registered spoke, so the shipped
// join flow's one-time snapshot doesn't go stale the moment the catalog
// changes. Fires on res.on('finish') (after the response is actually sent,
// status known) rather than before the handler runs, so a write that fails
// validation never triggers a pointless replication round-trip.
router.use((req, res, next) => {
if (['POST', 'PUT', 'DELETE', 'PATCH'].includes(req.method)) {
const mutating = ['POST', 'PUT', 'DELETE', 'PATCH'].includes(req.method);
if (mutating) {
const cfg = siteConfig.get();
if (!cfg.isMaster) {
const hint = cfg.masterUrl ? ' Directory writes must go to the master at ' + cfg.masterUrl + '.' : '';
return res.status(403).json({ status: 'error', message: 'This node is a spoke (read-only catalog).' + hint });
}
res.on('finish', () => {
if (res.statusCode >= 200 && res.statusCode < 300) {
meshReplicate.replicateToSpokes(`${req.method} ${req.path}`);
}
});
}
next();
});