feat(multi-site): live catalog replication + identical-directory signing key
The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's catalog never updated after joining. This adds the two pieces that were explicitly designed but missing: - Live replication: a spoke registers its own endpoint with the master right after joining (POST /api/site/spokes, Bearer join-key), receiving a pushToken. Every successful catalog write on the master now fires a fire-and-forget resync ping (utils/site_replicate.js) at every known spoke, concurrently -- one unreachable spoke never blocks or delays another (wired into the existing write-gate middleware in api_directory_admin.js). The spoke's POST /api/site/resync handler reuses the already-tested export+import path rather than applying a partial diff. - Identical directories: POST /api/site/export now best-effort includes the master's agent-signing key; a spoke adopts it via agent_keys.adopt() on both join and every resync, so every site's sso-manager can validly sign a command for any agent enrolled anywhere -- the accepted tradeoff discussed for this deployment's scale (blast radius for simplicity). New SiteSpoke model tracks registered spokes (endpoint + pushToken); registered it in models/index.js (a real bug the e2e test below caught -- SiteSpoke.list() 500'd with "Cannot read properties of null (reading 'adapter')" until the model was added to initORM's model list). Verified end-to-end against docker-compose.multisite-e2e.yml: mint join key -> join with selfUrl -> write a NEW resource on master post-join -> poll the spoke -> it shows up within a few seconds via the resync push, no manual re-join needed. MULTISITE E2E PASS. Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one failing spoke doesn't block another, empty-registry and list()-throws edge cases).
This commit is contained in:
@@ -175,14 +175,17 @@ async function main() {
|
||||
if (keyRes.status !== 200 || !keyRes.body.key) fail(`join-key mint failed: ${keyRes.status} ${JSON.stringify(keyRes.body)}`);
|
||||
const joinKey = keyRes.body.key;
|
||||
|
||||
step('Joining spoke to master');
|
||||
step('Joining spoke to master (with selfUrl, to register for live replication)');
|
||||
const joinRes = await api(SPOKE_URL, '/api/site/join', {
|
||||
method: 'POST',
|
||||
token: spokeToken,
|
||||
// master's own container-internal URL, as the spoke would reach it over the network
|
||||
body: { masterUrl: 'http://master:3001', joinKey }
|
||||
body: { masterUrl: 'http://master:3001', joinKey, selfUrl: 'http://spoke:3001' }
|
||||
});
|
||||
if (joinRes.status !== 200) fail(`join failed: ${joinRes.status} ${JSON.stringify(joinRes.body)}`);
|
||||
if (!joinRes.body.replication || joinRes.body.replication.live !== true) {
|
||||
fail(`expected join to register for live replication, got ${JSON.stringify(joinRes.body.replication)}`);
|
||||
}
|
||||
|
||||
step('Verifying spoke persisted isMaster:false + masterUrl after join');
|
||||
const { body: spokeCfg } = await api(SPOKE_URL, '/api/site/config', { token: spokeToken });
|
||||
@@ -203,6 +206,24 @@ async function main() {
|
||||
});
|
||||
if (writeAttempt.status !== 403) fail(`expected 403 writing to spoke post-join, got ${writeAttempt.status} ${JSON.stringify(writeAttempt.body)}`);
|
||||
|
||||
step('Creating a resource on master AFTER join, to verify LIVE replication (not just the one-time join snapshot)');
|
||||
const postJoinRes = await api(MASTER_URL, '/api/directory-admin/resources', {
|
||||
method: 'POST',
|
||||
token: masterToken,
|
||||
body: { name: 'E2E Post-Join Host', slug: 'host_e2e_postjoin', kind: 'host', parentSlug: 'site_e2e' }
|
||||
});
|
||||
if (postJoinRes.status !== 200) fail(`creating post-join resource on master failed: ${postJoinRes.status} ${JSON.stringify(postJoinRes.body)}`);
|
||||
|
||||
step('Waiting for the fire-and-forget resync push to reach the spoke');
|
||||
let liveReplicated = false;
|
||||
for (let i = 0; i < 20; i++) {
|
||||
const r = await api(SPOKE_URL, '/api/directory-admin/resources', { token: spokeToken });
|
||||
const slugs = (r.body.results || r.body.resources || r.body || []).map((x) => x.slug);
|
||||
if (slugs.includes('host_e2e_postjoin')) { liveReplicated = true; break; }
|
||||
await new Promise((res) => setTimeout(res, 500));
|
||||
}
|
||||
if (!liveReplicated) fail('post-join resource never appeared on the spoke -- live replication did not fire (or resync did not apply it)');
|
||||
|
||||
step('Verifying WAN health ping from spoke to master succeeds');
|
||||
const statusRes = await api(SPOKE_URL, '/api/directory-admin/site-status', { token: spokeToken });
|
||||
if (statusRes.body.config && statusRes.body.config.wanConnected !== true) {
|
||||
|
||||
Reference in New Issue
Block a user