feat(multi-site): live catalog replication + identical-directory signing key

The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's
catalog never updated after joining. This adds the two pieces that were
explicitly designed but missing:

- Live replication: a spoke registers its own endpoint with the master
  right after joining (POST /api/site/spokes, Bearer join-key), receiving
  a pushToken. Every successful catalog write on the master now fires a
  fire-and-forget resync ping (utils/site_replicate.js) at every known
  spoke, concurrently -- one unreachable spoke never blocks or delays
  another (wired into the existing write-gate middleware in
  api_directory_admin.js). The spoke's POST /api/site/resync handler
  reuses the already-tested export+import path rather than applying a
  partial diff.

- Identical directories: POST /api/site/export now best-effort includes
  the master's agent-signing key; a spoke adopts it via agent_keys.adopt()
  on both join and every resync, so every site's sso-manager can validly
  sign a command for any agent enrolled anywhere -- the accepted tradeoff
  discussed for this deployment's scale (blast radius for simplicity).

New SiteSpoke model tracks registered spokes (endpoint + pushToken);
registered it in models/index.js (a real bug the e2e test below caught --
SiteSpoke.list() 500'd with "Cannot read properties of null (reading
'adapter')" until the model was added to initORM's model list).

Verified end-to-end against docker-compose.multisite-e2e.yml: mint join
key -> join with selfUrl -> write a NEW resource on master post-join ->
poll the spoke -> it shows up within a few seconds via the resync push,
no manual re-join needed. MULTISITE E2E PASS.

Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one
failing spoke doesn't block another, empty-registry and list()-throws
edge cases).
This commit is contained in:
2026-08-10 16:34:38 -04:00
parent e5167729a8
commit d27763e556
9 changed files with 451 additions and 57 deletions
+57
View File
@@ -0,0 +1,57 @@
'use strict';
// Live replication push -- the piece the shipped v1 join flow doesn't have on
// its own (join is a one-time export/import snapshot; nothing kept a spoke in
// sync afterward). This fires a lightweight "something changed, re-pull" ping
// at every spoke registered in SiteSpoke, concurrently, fire-and-forget: never
// awaited by its caller, and one unreachable spoke never delays or blocks
// another. See MULTI_SITE_SPEC.md §2.2 for why this must never become a
// blocking design (a write must never stall on spoke reachability).
//
// Deliberately a PUSH-A-SIGNAL / PULL-A-SNAPSHOT design, not a push-a-diff
// design: the receiving spoke reacts by calling the master's already-shipped,
// already-tested POST /api/site/export + importDirectory() path again (see
// routes/api_site.js's /resync handler), rather than this module inventing a
// second, parallel way to represent "what changed." Fewer moving parts, and
// no risk of a diff payload and a full export ever disagreeing.
const { SiteSpoke } = require('../models/site_spoke');
const RESYNC_TIMEOUT_MS = 8000;
function replicateToSpokes(reason) {
return (async () => {
let spokes;
try {
spokes = await SiteSpoke.list();
} catch (err) {
console.error('[site-replicate] failed to list known spokes:', err.message);
return;
}
for (const spoke of spokes) {
// Not awaited -- every spoke is pushed to concurrently.
pingOne(spoke, reason).catch((err) => {
console.error(`[site-replicate] resync ping to ${spoke.endpoint} failed:`, err.message);
});
}
})();
}
async function pingOne(spoke, reason) {
const url = String(spoke.endpoint).replace(/\/+$/, '') + '/api/site/resync';
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), RESYNC_TIMEOUT_MS);
try {
const resp = await fetch(url, {
method: 'POST',
headers: { Authorization: 'Bearer ' + spoke.pushToken, 'Content-Type': 'application/json' },
body: JSON.stringify({ reason: reason || 'catalog-changed' }),
signal: controller.signal
});
if (!resp.ok) throw new Error('status ' + resp.status);
} finally {
clearTimeout(timer);
}
}
module.exports = { replicateToSpokes };