feat(multi-site): live catalog replication + identical-directory signing key
The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's catalog never updated after joining. This adds the two pieces that were explicitly designed but missing: - Live replication: a spoke registers its own endpoint with the master right after joining (POST /api/site/spokes, Bearer join-key), receiving a pushToken. Every successful catalog write on the master now fires a fire-and-forget resync ping (utils/site_replicate.js) at every known spoke, concurrently -- one unreachable spoke never blocks or delays another (wired into the existing write-gate middleware in api_directory_admin.js). The spoke's POST /api/site/resync handler reuses the already-tested export+import path rather than applying a partial diff. - Identical directories: POST /api/site/export now best-effort includes the master's agent-signing key; a spoke adopts it via agent_keys.adopt() on both join and every resync, so every site's sso-manager can validly sign a command for any agent enrolled anywhere -- the accepted tradeoff discussed for this deployment's scale (blast radius for simplicity). New SiteSpoke model tracks registered spokes (endpoint + pushToken); registered it in models/index.js (a real bug the e2e test below caught -- SiteSpoke.list() 500'd with "Cannot read properties of null (reading 'adapter')" until the model was added to initORM's model list). Verified end-to-end against docker-compose.multisite-e2e.yml: mint join key -> join with selfUrl -> write a NEW resource on master post-join -> poll the spoke -> it shows up within a few seconds via the resync push, no manual re-join needed. MULTISITE E2E PASS. Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one failing spoke doesn't block another, empty-registry and list()-throws edge cases).
This commit is contained in:
@@ -90,10 +90,34 @@ function status() {
|
||||
return { available: !!cached, error: loadError };
|
||||
}
|
||||
|
||||
// Overwrite the stored key with material handed over by a master (multi-site
|
||||
// "identical directories" — MULTI_SITE_SPEC.md §2). Every site sharing one
|
||||
// signing key is what lets any site's sso-manager validly sign a command for
|
||||
// an agent enrolled at any other site, at the accepted cost that compromising
|
||||
// ANY one site's OpenBao is equivalent to compromising all of them for agent
|
||||
// command authority. That tradeoff was deliberately accepted for this
|
||||
// deployment's scale (a handful of trusted sites) -- do not call this to sync
|
||||
// keys across a boundary where sites don't trust each other equally.
|
||||
//
|
||||
// Idempotent: adopting the same key material twice (e.g. on every resync
|
||||
// ping) is a no-op past the first call.
|
||||
async function adopt({ privateKeyPem, publicKeyPem }) {
|
||||
if (!privateKeyPem || !publicKeyPem) throw new Error('adopt() requires both privateKeyPem and publicKeyPem');
|
||||
if (cached && cached.privateKeyPem === privateKeyPem && cached.publicKeyPem === publicKeyPem) {
|
||||
return cached; // already holding this exact key -- nothing to do
|
||||
}
|
||||
const material = { privateKeyPem, publicKeyPem };
|
||||
await baoConf.set(PATH, material);
|
||||
cached = { ...material, publicKeyBase64: rawPublicKeyBase64(publicKeyPem) };
|
||||
loadError = null;
|
||||
console.log('[agent_keys] adopted signing key from master (multi-site identical-directory sync)');
|
||||
return cached;
|
||||
}
|
||||
|
||||
// Test seam: drop the in-process cache.
|
||||
function _reset() {
|
||||
cached = null;
|
||||
loadError = null;
|
||||
}
|
||||
|
||||
module.exports = { load, status, rawPublicKeyBase64, _reset, PATH };
|
||||
module.exports = { load, status, adopt, rawPublicKeyBase64, _reset, PATH };
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
'use strict';
|
||||
|
||||
// Live replication push -- the piece the shipped v1 join flow doesn't have on
|
||||
// its own (join is a one-time export/import snapshot; nothing kept a spoke in
|
||||
// sync afterward). This fires a lightweight "something changed, re-pull" ping
|
||||
// at every spoke registered in SiteSpoke, concurrently, fire-and-forget: never
|
||||
// awaited by its caller, and one unreachable spoke never delays or blocks
|
||||
// another. See MULTI_SITE_SPEC.md §2.2 for why this must never become a
|
||||
// blocking design (a write must never stall on spoke reachability).
|
||||
//
|
||||
// Deliberately a PUSH-A-SIGNAL / PULL-A-SNAPSHOT design, not a push-a-diff
|
||||
// design: the receiving spoke reacts by calling the master's already-shipped,
|
||||
// already-tested POST /api/site/export + importDirectory() path again (see
|
||||
// routes/api_site.js's /resync handler), rather than this module inventing a
|
||||
// second, parallel way to represent "what changed." Fewer moving parts, and
|
||||
// no risk of a diff payload and a full export ever disagreeing.
|
||||
|
||||
const { SiteSpoke } = require('../models/site_spoke');
|
||||
|
||||
const RESYNC_TIMEOUT_MS = 8000;
|
||||
|
||||
function replicateToSpokes(reason) {
|
||||
return (async () => {
|
||||
let spokes;
|
||||
try {
|
||||
spokes = await SiteSpoke.list();
|
||||
} catch (err) {
|
||||
console.error('[site-replicate] failed to list known spokes:', err.message);
|
||||
return;
|
||||
}
|
||||
for (const spoke of spokes) {
|
||||
// Not awaited -- every spoke is pushed to concurrently.
|
||||
pingOne(spoke, reason).catch((err) => {
|
||||
console.error(`[site-replicate] resync ping to ${spoke.endpoint} failed:`, err.message);
|
||||
});
|
||||
}
|
||||
})();
|
||||
}
|
||||
|
||||
async function pingOne(spoke, reason) {
|
||||
const url = String(spoke.endpoint).replace(/\/+$/, '') + '/api/site/resync';
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), RESYNC_TIMEOUT_MS);
|
||||
try {
|
||||
const resp = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: 'Bearer ' + spoke.pushToken, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ reason: reason || 'catalog-changed' }),
|
||||
signal: controller.signal
|
||||
});
|
||||
if (!resp.ok) throw new Error('status ' + resp.status);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { replicateToSpokes };
|
||||
Reference in New Issue
Block a user