The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's
catalog never updated after joining. This adds the two pieces that were
explicitly designed but missing:
- Live replication: a spoke registers its own endpoint with the master
right after joining (POST /api/site/spokes, Bearer join-key), receiving
a pushToken. Every successful catalog write on the master now fires a
fire-and-forget resync ping (utils/site_replicate.js) at every known
spoke, concurrently -- one unreachable spoke never blocks or delays
another (wired into the existing write-gate middleware in
api_directory_admin.js). The spoke's POST /api/site/resync handler
reuses the already-tested export+import path rather than applying a
partial diff.
- Identical directories: POST /api/site/export now best-effort includes
the master's agent-signing key; a spoke adopts it via agent_keys.adopt()
on both join and every resync, so every site's sso-manager can validly
sign a command for any agent enrolled anywhere -- the accepted tradeoff
discussed for this deployment's scale (blast radius for simplicity).
New SiteSpoke model tracks registered spokes (endpoint + pushToken);
registered it in models/index.js (a real bug the e2e test below caught --
SiteSpoke.list() 500'd with "Cannot read properties of null (reading
'adapter')" until the model was added to initORM's model list).
Verified end-to-end against docker-compose.multisite-e2e.yml: mint join
key -> join with selfUrl -> write a NEW resource on master post-join ->
poll the spoke -> it shows up within a few seconds via the resync push,
no manual re-join needed. MULTISITE E2E PASS.
Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one
failing spoke doesn't block another, empty-registry and list()-throws
edge cases).
Server endpoints for joining a spoke to a master directory (MULTI_SITE_SPEC.md).
This pass is server-only; setup.sh wiring and the UI are the next layer.
- Site join keys (SiteJoinKey model, stj_ prefix): mint/revoke/delete/list,
hashed at rest, shown once — the same model as agent join keys.
- POST /api/site/export (master, Bearer stj_ key, no admin session): returns the
local LDAP tree (slapcat LDIF) + resource catalog + siteSlug + baseDn.
- POST /api/site/join (spoke, admin): { masterUrl, joinKey } pulls the master
export, imports resources (upsert by slug) + LDAP (ldapadd -c), and persists
the spoke role. Refused if already a spoke.
- Persisted site role: utils/site_config.js keeps isMaster/masterUrl/siteSlug in
/config/site.json (env seeds defaults); site-status/site-promote now use it.
- Unit tests (site_join, site_config) with in-memory stubs, wired into npm test.
- docs/site-join.md + docs router entry.
- Repairs the corrupted multi-site emojis (crown/bolt) in directory.ejs.
- .gitguardian.yml ignores the generic-password false positive on reading the
LDAP bind credential from runtime config (never a hardcoded secret).
JOIN KEYS
v1.29.0 required an admin to pre-register every machine before its agent
would be spoken to. The security model was right; the workflow was not --
installing the agent should be enough to add a host.
POST /api/agent/join-keys mints one credential an operator hands out. A
host presenting it is enrolled automatically and immediately issued its
OWN per-agent token plus the public key it must pin, delivered in the
config frame. The join key is a bootstrap credential, never the host's
identity, so one key stays convenient without becoming a fleet-wide
skeleton key: every host remains individually revocable.
DIRECTORY
Collapsing the tree did nothing. applyTreeCollapse found the caret with
`.tree-caret i` and returned early when absent -- Font Awesome's SVG mode
rewrites <i> to <svg>, so that selector matched nothing and the early
return skipped setting hideBelowDepth. State now lives on the caret
button and is rotated by CSS.
The Discovery Plugins delete button called deleteDiscoveryPlugin(), which
was never defined. The pane also had no .actionMessage, and confirmations
render into one -- without it the promise never settles, so an awaited
confirmation hangs forever and the action silently never happens.
Plugin instances can now be edited.
DISCOVERY
A fresh install presented its own five containers as unmanaged
discoveries. The Docker plugin now recognises the stack's compose project
and attaches each container to the service it implements. Container slugs
came from the container id, which changes on recreate, so every deploy
minted a new resource and orphaned the old one.
DOCS
/docs/discovery 404'd (no slug entry) and `agents` pointed at plugins.md,
leaving docs/agents.md unreachable. Adds docs/discovery.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SECURITY
/api/agent/ws authenticated nothing. There was no agent registry, so any
client reaching the SSO could register as a node, publish discovery and
telemetry into the admin view, and receive commands -- including a signed
arbitrary_bash -- addressed to a token it guessed. Tokens were generated
in the BROWSER and never recorded server-side, so there was nothing to
validate against and no way to revoke one.
Agents are now rows in a new Agent table, authenticated by SHA-256 token
hash before the connection is registered or the welcome payload is sent.
Tokens are minted by POST /api/agent/enroll and shown once. Revoke and
rotate drop the live socket immediately. All agent actions are audited.
The Ed25519 command-signing key was generated in the AgentManager
constructor, so it changed on every restart and the public_key pinned in
an agent's agent.yml stopped matching. It now lives in OpenBao at
secret/agent/signing-key; if it cannot be loaded the SSO refuses to send
high-risk commands rather than signing with a key no agent has seen.
DIRECTORY
Agents bind to a host resource instead of being matched by hostname, and
a bound agent's discovery is written onto that resource -- previously the
one source running ON the host contributed nothing to the directory.
The resource tree is collapsible, with state persisted per browser.
DISCOVERY
The Proxmox plugin zipped MACs and IPs from two flat lists by index,
attributing addresses to the wrong NIC on multi-NIC guests. NICs are now
keyed by MAC. Adds an endpoint resource parenting each node, sourceId/
vmid/node identity, container-interface filtering, node IP/MAC, and
offline-node handling.
The reconciler could make a resource its own parent, named hosts after
their MAC address, had a dead isIp() regex (\\. matches a backslash),
merged across kinds, and re-read the whole inventory per resource.
Dockerfile.test-runner never copied nodejs/plugins, so every plugin test
suite failed in CI as "Cannot find module".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- /api/vault proxy now injects X-Vault-Token: the proxy declared its request
hook with http-proxy-middleware v3 syntax (on: { proxyReq }), which the
installed HPM v2 silently ignores — so every vault call reached OpenBao
unauthenticated (the recurring 403). Rewritten as v2 onProxyReq.
- Header injection ordered before fixRequestBody (the body write flushes
headers; setting X-Vault-Token after it failed on every POST/PUT).
- initORM add-only schema heal: sequelize.sync() never ALTERs, so newer columns
(PluginInstance.lastLog) are now added via describeTable + addColumn.
- Long-lived external-app tokens via sso-app role (768h periodic); VaultAppToken
stores each app token's accessor and renews it at boot + every 6h; re-minting
revokes the previous token via its accessor.
- Wire-level tests for the vault proxy + app-token accessor lifecycle.
- package.json + lockfile bumped to 1.23.0.
Co-Authored-By: Claude <noreply@anthropic.com>
- vault_broker: always reconcile policy content before serving a cached
token (compare-and-skip), so stale stored policies can't cause a recurring
403 'permission denied'; policy content is parsed live by OpenBao, so edits
apply to existing tokens immediately.
- Shared secrets: publish to secret/shared/<owner>/<slug>; grant read to users
and apps by editing the grantee's policy content (live-applied). New
SharedSecret/SharedSecretGrant ORM models, /api/shared-secrets router, and a
Shared tab in the vault UI.
- package.json + lockfile bumped to 1.21.0 to match the tag.
Co-Authored-By: Claude <noreply@anthropic.com>
Generalize the half-built discovery plugins into a real plugin system: plugin
TYPES (the plugins/<category>/<type>.js modules with manifests) and loadable,
configurable, multi-copy plugin INSTANCES (PluginInstance ORM model) managed
from a dedicated /plugins page and /api/plugins API, with per-instance secrets
in OpenBao at secret/plugins/<id>/conf.
- plugin_registry.js: getTypes/getModule/splitConfig/mask + required-field helpers
- PluginInstance model (Sequelize): id/pluginType/category/name/slug(unique)/
enabled/cron/config(json, non-secret)/lastRun*; registered in models/index.js
- plugin_secrets.js: read/write/remove/mergeForRun over @simpleworkjs/bao-conf
- scheduler.js: schedules from the DB registry; per-instance stable BullMQ
JobScheduler ids (plugin:<id>) for load/unload; legacy migration from
conf.discovery.plugins on first boot (idempotent, empty-table-guarded)
- api_plugins.js (replaces routes/plugins.js): types/list/get/create/update/
secrets/test/load/unload/run/delete/runs; admin-gated; secrets always masked
- /plugins page (plugins.ejs) + nav; Agents & Scheduler tab removed from
/directory; /docs/agents aliased to /docs/plugins
- proxmox/unifi/nmap gained manifests (configSchema/validate/run alias)
- tests/plugins.test.js: registry unit + plugin_secrets (mocked bao-conf) +
PluginInstance model round-trip/unique-slug
- docs (plugins.md, vault.md, _config.yml, API.md) + 1.16.1 -> 1.17.0
Requires theta-suite >= v1.30.1 for the sso-broker secret/plugins/* grant;
fails-soft with a clear error if absent.
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the end-user half of the directory and adds nested LDAP groups.
The directory could describe the lab but could not tell anyone what they had
or how to reach it, and several of the paths meant to do so were silently
returning nothing:
- GET /api/discovery/me resolved groups from req.user.groups, which does not
exist (req.user carries memberOf), so it returned only isPublic resources
for every human caller -- "My Services" was blank for everyone. The same
read made isDirectoryAdmin() false for real admins.
- The portal's "Discover More Services" called the admin-gated endpoint and
swallowed the 403, so it never rendered for non-admins at all.
- Services reported no address, because /me had reimplemented getMyAccess
without its parent-walking resolution.
Adds the catalog at /, self-service access requests, and admin access
visibility (per-resource counts, and the reverse "what can this user reach").
Nested groups come in two halves. groupOfNames.member already accepts a group
DN, so nesting needs no schema -- what it needs is resolution, which no
released OpenLDAP performs. The all-in-one image therefore builds slapd from a
pinned master commit for the nestgroup overlay, and the app computes the
closure itself when pointed at a server without it. Both paths are covered.
member-values is deliberately left out of nestgroup-flags: it expands `member`
when reading a group, which destroys the distinction between "listed here" and
"reachable through a nested group" and is not recoverable afterwards.
Full suite green in both resolution modes: 215 passed, 2 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Use published @simpleworkjs/orm ^0.2.8 (fixes redis adapter write path)
and model-redis ^1.6.0 instead of a local file: link that broke docker
npm ci with a misleading "no lockfile" error.
- OtpToken.issue/verify: replace nonexistent find()/listDetail() with
list({where}).
- routes/auth.js: ImpersonationToken.listDetail() -> list({where}).
- routes/token.js: drop listDetail() call; 404 on missing token instead
of returning {results: null} with 200 (orm get() returns null, does
not throw like model-redis Table.get did).
- OAuthClient: Resource has no is_valid column, so every client read as
disabled and all /oauth/authorize requests 400'd — validity now lives
in metadata (absent = valid). Also generate a unique slug on create
(Resource.slug is required+unique) and use Resource.get() for lookup.
- User.login: 401 cleanly when neither uid nor username is supplied.
- models/index.js: log ORM init and surface init failures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Personal access tokens so scripts/CI can call the management API without a
browser session. Each logged-in user mints their own token; it authenticates as
the creator (carries their LDAP group permissions, re-resolved live), so the
existing permission.byGroup checks apply unchanged.
- models/api_token.js: new ApiToken model (sso_<id>_<secret> format; id is the
lookup key, secret bcrypt-hashed + isPrivate, shown once). add()/rotate()/
authenticate(); optional expires_at; best-effort last_used_on. No _ttl
(persists; lifetime via expires_at).
- routes/api_token.js: self-service CRUD (list/get/update/delete/rotate),
owner-scoped (created_by === req.user.uid, 403 otherwise).
- middleware/auth.js + models/auth.js: accept `Authorization: Bearer sso_...`
(precedence over the auth-token session header); checkApiToken collapses
every failure to one generic 401 (no existence/secret/expiry leak).
- views/api_tokens.ejs + routes/index.js (GET /api-tokens): self-service page
(forceLogin, no group gate) — create (token shown once), edit, rotate, revoke.
- views/top.ejs: "API Tokens" nav entry visible to all logged-in users.
- public/js/app.js: app.apiToken client module.
- DEPLOYMENT.md + docs/deployment.md: API tokens section.
Co-authored-by: Claude <noreply@anthropic.com>
wmantly