Found while auditing the new LDAP MMR auto-config for gaps: neither
POST /site-promote nor POST /demote ever touched SiteSpoke. Two real
problems:
1. The demoted old master got a fresh masterJoinKey but was never
registered as a spoke of the new master -- no SiteSpoke row, no
ldapServerId, invisible to GET /ldap-peers's peer list. It also
structurally could not self-heal: POST /join refuses re-join for a
node that's already a spoke, and separately requires a fresh
install (siteIsFresh()) -- neither true for a former master with
real users/agents. Fixed: /demote now registers itself with the new
master immediately (POST /spokes), the same way a real join does,
deriving its own endpoint from stack.selfUrl (override) or
https://stack.ssoHost (the normal case).
2. The promoted node's live OpenLDAP ServerID doesn't change --
GET /ldap-replication-config starts advertising 1 for it
immediately (derived purely from cfg.isMaster), but nothing
restarts slapd with that value (OpenLDAP's static slapd.conf only
reloads at process start, and this app has no safe way to restart
its own container). Can't be fixed in-process; surfaced instead --
/site-promote's response now includes ldapReplicationNote telling
the operator to re-run setup.sh promptly.
Verified against real running containers (docker-compose.multisite-e2e.yml):
after promotion, the demoted old master correctly appears in the new
master's LDAP peer list with a real assigned ldapServerId.
OpenLDAP N-way multi-master replication (docs/replication.md) required
an operator to hand-set LDAP_SERVER_ID (unique per site) and
LDAP_REPLICATION_HOSTS (every OTHER site's LDAP URL, kept in sync by
hand across every node) -- real coordination work, and easy to get
wrong or let drift as sites are added.
Automates the coordination the master is already in a position to do:
- SiteSpoke gets ldapServerId, auto-assigned (next free from 2 upward,
1 reserved for the master) at registration and reused across
re-registrations -- same pattern as jump-host's mesh index.
- ldapHost is derived from each site's already-known HTTP(S) endpoint
(same hostname, port 636) rather than a separately-configured field
that could drift from it.
- New utils/ldap_replication.js (nextFreeLdapServerId, ldapHostFor),
shared between the spoke-facing GET /api/site/ldap-peers (Bearer
site join key, returns this caller's own ID + every peer) and the
master-local GET /directory-admin/ldap-replication-config (computes
its own config directly from SiteSpoke, no HTTP round-trip needed).
Verified against real running containers (docker-compose.multisite-e2e.yml):
after a real join, the master's computed config correctly includes the
spoke as a peer with an assigned ID, and the spoke's own fetched
config matches that ID and correctly excludes itself from its own
peer list.
Known limitation, documented in docs/replication.md: the master's own
LDAP_REPLICATION_HOSTS only gets recomputed when ITS setup.sh is
re-run (or an admin re-applies it directly) -- there's no live push to
an already-running master when a new spoke joins. A spoke's own config
is re-checked on every setup.sh run, which is the common/recurring
event; the master side is a documented manual step for now rather than
a live hot-reload (which would need OpenLDAP's dynamic cn=config
backend -- a bigger change, deliberately out of scope here to avoid
risking a live directory's LDAP replication on undertested config).
"Theta Gateways: N active gateways" was counting this app's own
unrelated WireGuard roaming-client/exit-node Resources
(metadata.subType === 'wireguard') -- a completely different subsystem
from the gateway-to-gateway mesh the modal is actually about, and it
never queried jump-host's mesh registry at all (so it couldn't show
the local self-entry either, since there was nothing mesh-related
being counted in the first place).
Added utils/jump_client.js (same pattern as utils/proxy_client.js:
reuses jump-host's existing self-service jmp_ API token system rather
than inventing a new credential) to query jump-host's real
GET /api/mesh/gateways. Reports a null count (not misleading 0) when
the integration isn't configured/reachable, surfaced distinctly in the
UI. Also added help links to the published multi-site/mesh docs on the
modal.
Includes docs links + count only -- this session also discovered that
utils/proxy_client.js's PROXY_INTERNAL_URL, and now JUMP_INTERNAL_URL,
were never actually wired into theta-suite's docker-compose.yml, so
both service-to-service integrations were unreachable in every real
deployment despite existing in code (fixed in theta-suite separately).
Three independent copies of the same bug: routes/discovery.js's
POST /discovery/promote/:slug (the actual "Promote" button in the UI)
and services/discovery_reconciler.js's autoPromote path both called
ResourceGroup.create() directly with no existence check -- unlike
routes/api_directory_admin.js's own ensureResourceGroup, which already
carried a comment describing this exact "groups appear 3x" bug and
fixing it, just not everywhere it occurred. ResourceGroup has no DB
unique constraint on (resourceId, groupCn), so a resource promoted
more than once (retried UI click, or the same LXC discovered from
multiple Proxmox cluster nodes) silently accumulated duplicate
access/admin rows every time. Added ResourceGroup.ensure() (the
existing check-then-create pattern, now on the model) and switched all
three call sites to it. New regression test in tests/reconciler.test.js.
Also: GET /api/directory-admin/resources ran a full group-model
self-heal fan-out (ensureSiteGroups per site + provisionResourceGroups
per resource, each several sequential LDAP round-trips) unconditionally
on every single list -- confirmed via code read as the actual
bottleneck once a directory has more than a handful of resources, not
data volume. Moved healing to where resources actually change instead
(POST/PUT /resources, POST /discovery/promote/:slug -- PUT had none at
all before this), and added POST /resources/heal-groups as an explicit
on-demand equivalent for backfilling a directory seeded before this
change.
Closes the gap where all of this session's new server-side capability
(live replication, coordinated promotion, identical signing keys) had
no UI at all -- an operator using the Master Site modal had no way to
know any of it existed or was working.
- Master Site modal: new "Live Replication" row (spoke) shows whether
this join actually registered for live updates or is stuck on a
one-time snapshot; new "Registered Spokes" row (master) shows how
many spokes are receiving live pushes.
- Join form: new "this site's own reachable URL" field, prefilled from
window.location.origin, wired to the selfUrl the join API already
supported but the UI never sent -- a UI-driven join previously NEVER
registered for live replication, only the setup.sh bootstrap path did.
The success toast now reports whether live replication actually
activated, not just "joined".
- Promote button: success toast now surfaces the handoff result (old
master demoted / unreachable / no previous master), so the operator
sees immediately whether the coordinated demotion actually happened.
- GET /api/site/config no longer returns masterJoinKey or
replicationPushToken in the response -- found while wiring this up:
live credentials were being sent straight to the browser for every
admin session. Replaced with boolean derivatives
(hasMasterJoinKey, liveReplication).
- GET /api/directory-admin/site-status gained liveReplication (spoke)
and registeredSpokesCount (master) so the modal has something to render.
Verified by actually driving it in a real browser against a live
container (not just code review): logged in, opened the modal, saw the
new rows, minted a real join key end-to-end, no console errors.
docs/site-join.md rewritten to cover live replication, signing-key sync,
coordinated promotion/demote, and the new endpoints -- it previously only
described the v2.2.0-v2.3.0 one-time-snapshot behavior.
Two real bugs, both only surfaced by the live two-container e2e test
(docker-compose.multisite-e2e.yml), not by inspection:
1. POST /site-promote's god_admin check read req.user.groups -- a field
nothing in the codebase ever populates (Auth.checkToken returns
User.get(), which has no .groups; every other admin gate resolves
membership live via permission.byGroup()/Group.list(user.dn), which
also handles nested-group membership). The check silently evaluated to
an empty array on every request, so site-promote returned 403 for
every user, including a real god_admin -- unusable since it shipped in
v2.0.0. Fixed to use permission.byGroup(), the same pattern used
elsewhere in this file and in api_site.js.
2. The read-only write-gate middleware (api_directory_admin.js) is
registered before router.post('/site-promote', ...) later in the same
file, so on a spoke it 403'd every promotion attempt before the
handler ever ran -- the one mutating request a spoke must be able to
make to itself. Exempted /site-promote from the gate.
Added coordinated demotion (MULTI_SITE_SPEC.md §3.2 -- promotion as ONE
action, never a two-step gap with two masters): site-promote now calls
the previous master's new POST /api/site/demote (Bearer the join key it
already holds, handing over a freshly-minted key for the demoted node's
own future use) before flipping itself to master. Best-effort: an
unreachable old master never blocks a god_admin's local promotion (the
WAN-outage scenario is the entire reason this control exists), it's
just reported in the response for manual reconciliation.
e2e test extended to promote the spoke, verify the old master was
actually demoted (isMaster:false, masterUrl pointing at the new master),
and verify writes now succeed on the new master and 403 on the old one.
Full chain verified passing: join -> live replication -> promotion ->
demotion -> write authority follows the promotion.
The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's
catalog never updated after joining. This adds the two pieces that were
explicitly designed but missing:
- Live replication: a spoke registers its own endpoint with the master
right after joining (POST /api/site/spokes, Bearer join-key), receiving
a pushToken. Every successful catalog write on the master now fires a
fire-and-forget resync ping (utils/site_replicate.js) at every known
spoke, concurrently -- one unreachable spoke never blocks or delays
another (wired into the existing write-gate middleware in
api_directory_admin.js). The spoke's POST /api/site/resync handler
reuses the already-tested export+import path rather than applying a
partial diff.
- Identical directories: POST /api/site/export now best-effort includes
the master's agent-signing key; a spoke adopts it via agent_keys.adopt()
on both join and every resync, so every site's sso-manager can validly
sign a command for any agent enrolled anywhere -- the accepted tradeoff
discussed for this deployment's scale (blast radius for simplicity).
New SiteSpoke model tracks registered spokes (endpoint + pushToken);
registered it in models/index.js (a real bug the e2e test below caught --
SiteSpoke.list() 500'd with "Cannot read properties of null (reading
'adapter')" until the model was added to initORM's model list).
Verified end-to-end against docker-compose.multisite-e2e.yml: mint join
key -> join with selfUrl -> write a NEW resource on master post-join ->
poll the spoke -> it shows up within a few seconds via the resync push,
no manual re-join needed. MULTISITE E2E PASS.
Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one
failing spoke doesn't block another, empty-registry and list()-throws
edge cases).
Completes the multi-site join layer on top of the v2.2.0 endpoints:
- UI (Master Site modal): a fresh install (canJoin) gets a 'Join an Existing
Site' form (master URL + stj_ key); a master gets a 'Site Join Keys' manager
(mint/revoke/list, key shown once); WAN Sync Health now reflects a live probe.
- POST /api/site/ping (Bearer stj_ key, no admin session): lightweight master
reachability probe for WAN health (cheap vs /export).
- Spoke read-only: directory-write routes (resources/edges/groups/secrets/
grants/driver-action/discovered) reject with 403 pointing at the master.
- Fresh-install guard: /api/site/join refuses unless no users beyond the
bootstrap admin and no enrolled agents (siteIsFresh), and site-status exposes
canJoin so the UI only offers join on a genuinely fresh install. The
bootstrap's seeded default resources are NOT the signal (they always exist).
- The spoke stores the join key (masterJoinKey) in /config/site.json so WAN
health (and a future write-proxy) can reach the master.
- Tests: siteIsFresh cases in tests/site_join.test.js.
Server endpoints for joining a spoke to a master directory (MULTI_SITE_SPEC.md).
This pass is server-only; setup.sh wiring and the UI are the next layer.
- Site join keys (SiteJoinKey model, stj_ prefix): mint/revoke/delete/list,
hashed at rest, shown once — the same model as agent join keys.
- POST /api/site/export (master, Bearer stj_ key, no admin session): returns the
local LDAP tree (slapcat LDIF) + resource catalog + siteSlug + baseDn.
- POST /api/site/join (spoke, admin): { masterUrl, joinKey } pulls the master
export, imports resources (upsert by slug) + LDAP (ldapadd -c), and persists
the spoke role. Refused if already a spoke.
- Persisted site role: utils/site_config.js keeps isMaster/masterUrl/siteSlug in
/config/site.json (env seeds defaults); site-status/site-promote now use it.
- Unit tests (site_join, site_config) with in-memory stubs, wired into npm test.
- docs/site-join.md + docs router entry.
- Repairs the corrupted multi-site emojis (crown/bolt) in directory.ejs.
- .gitguardian.yml ignores the generic-password false positive on reading the
LDAP bind credential from runtime config (never a hardcoded secret).
- GET /api/directory-admin/resources let every kind:'host' resource through
regardless of promotion status, so "Auto-promote to Directory" unchecked on
a discovery plugin never kept unpromoted devices out of the Directory tab.
- GET /api/directory-admin/site-status queried the nonexistent Resource.subType
column instead of metadata.subType, throwing SequelizeDatabaseError.
- Added a "Show ignored" toggle to Discovered Inventory (off by default).
- Untracked nodejs/config/inventory.sqlite -- the app's default runtime DB,
not a fixture, committed by mistake across 13 prior releases.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(v2.0.0): bump version to 2.0.0, add Multi-Site Master badge, site-status API, and Master promotion UI
* fix(test): update test script to run unit tests without requiring live Redis socket
- group names match docs/GROUPS.md: {site}_{kind}_{name}_{level} (kind always present; services -> app kind); updated resolver + tests + access_request test
- site resource carries only god_admin + site-wide groups
- groups no longer appear 3x: idempotent ResourceGroup linking (self-heal was creating duplicates on every Directory load)
- /api/agent/* no longer 404s: REST router mounts unconditionally (was gated on the WS server)
- shared-secrets: slug regex allows underscores; GET list uses static pathFor (fixes 's.path is not a function')
- vault Apps tab: new GET /api/vault/apps + Minted apps list + purpose text; /docs/vault help link + docs cover Apps/Shared
- discovery promote: load instance and call update() (fixes 'Resource.update is not a function')
- discovery plugin cards: last-run time/status + Logs button
god_admin now exists at boot (seeded by docker-entrypoint), so the canonical
cross-resource super group nested into every resource's _admin group is god_admin,
not the legacy app_super_admin. docker-entrypoint no longer seeds or nests
app_super_admin (god_admin nests into the app_sso_* groups directly). isSuperAdmin
still recognizes a pre-existing app_super_admin as a migration alias until rebuild.
* feat: complete the group model (god_admin, site groups, aggregates), enforce naming, fix docs 500s + status dots (v1.26.0)
- seed god_admin + nest into app_super_admin; auto-provision site groups (S_super_admin, S_hosts_*/S_apps_* aggregates, S_everyone) on site create + self-heal on Directory load
- map service resources to the app kind (site_local_app_<slug>_*); nest per-resource groups into site aggregates (physical inheritance lattice)
- enforce the group naming convention server-side on POST /groups; surface god_admin + site groups on the site resource modal
- fix in-app /docs/<slug> 500s (Dockerfile never copied docs/); serve doc images at /docs/images
- fix Directory status dots (neutral grey when agent endpoint unreachable); align Profile/API cards full-width
- group resolver: keep the site slug verbatim (site_local not re-slugified)
- bump to 1.26.0
* fix: use verbatim resource slugs in group names (matches access-request tests + live convention)
The group naming inserts a kind segment (resourceGroupCns(site, kind, slug, level)),
but the access-request tests + the live directory convention are verbatim
({site}_{slug}_{level} -- the kind is carried in the resource slug, e.g. host_theta-env).
For bare test slugs this produced site_x_host_artest-host_x_access instead of the
expected site_x_artest-host_x_access, so the requester was never removed from the
auto-provisioned access group and every request 409'd. resourceGroupCns is now
(site, slug, level) with the verbatim slug; the kind is used only to pick the
aggregate the group nests into.
Closes the end-user half of the directory and adds nested LDAP groups.
The directory could describe the lab but could not tell anyone what they had
or how to reach it, and several of the paths meant to do so were silently
returning nothing:
- GET /api/discovery/me resolved groups from req.user.groups, which does not
exist (req.user carries memberOf), so it returned only isPublic resources
for every human caller -- "My Services" was blank for everyone. The same
read made isDirectoryAdmin() false for real admins.
- The portal's "Discover More Services" called the admin-gated endpoint and
swallowed the 403, so it never rendered for non-admins at all.
- Services reported no address, because /me had reimplemented getMyAccess
without its parent-walking resolution.
Adds the catalog at /, self-service access requests, and admin access
visibility (per-resource counts, and the reverse "what can this user reach").
Nested groups come in two halves. groupOfNames.member already accepts a group
DN, so nesting needs no schema -- what it needs is resolution, which no
released OpenLDAP performs. The all-in-one image therefore builds slapd from a
pinned master commit for the nestgroup overlay, and the app computes the
closure itself when pointed at a server without it. Both paths are covered.
member-values is deliberately left out of nestgroup-flags: it expands `member`
when reading a group, which destroys the distinction between "listed here" and
"reachable through a nested group" and is not recoverable afterwards.
Full suite green in both resolution modes: 215 passed, 2 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Add Resource audit fields (created/updated by/on) and site-slug group prefixing
Resource had no created_by/created_on/updated_by/updated_on fields at all,
unlike proxy's Host and jump-host's ApiToken which already track this --
needed for the upcoming resource-modal footer. @simpleworkjs/orm has no
auto-timestamp hook, so these are set explicitly in the directory-admin
route handlers on every create/update.
Also: when a host/service resource is created, its two auto-created LDAP
groups (<slug>_access/_admin) now get prefixed with the nearest ancestor
site's slug (via a new Resource.findAncestorSiteSlug walk), so groups from
different sites don't collide/look identical. Falls back to today's
unprefixed naming when a resource has no site ancestor.
Included the checked-in dev inventory.sqlite's ALTER TABLE for the new
columns, since @simpleworkjs/orm's sync() only creates missing tables, never
alters existing ones -- the raw model change alone would have broken every
Resource read/write against this file with "no such column: created_by".
* Migrate Resource modal onto app.modal's tabs/footer/URL, add Children tab
The Directory's resource modal was a separate, hand-rolled, always-in-DOM
Bootstrap modal, independent of the shared app.modal singleton -- migrating
it onto app.modal (now published with tabs/footer/url support in
@simpleworkjs/frontend 0.2.6) is the pilot for standardizing entity modals
across the stack.
- General/Details/Associated LDAP Groups/Children tabs, replacing the old
single long form (Details keeps every kind-conditional container
unchanged; toggleFormFields() didn't need to change at all).
- Footer shows created/updated by/on (via the new Resource audit fields)
and the Save button; Groups/Children tabs are hidden in add-mode since
they need an existing resource id.
- New Children tab lists a resource's existing children (reusing the
already-loaded edges/resourcesById data, no new endpoint) and an "Add
Child Resource" button that reuses openAddModal's existing preset-parent
support. Folded the pre-existing generic "Relationships (Graph Edges)"
section in underneath, under an "advanced" subheading, rather than
dropping it or giving it a 5th tab of its own.
- GET /directory/:slug (mirroring the existing /users/:uid precedent) plus
a client-side app.modal.deepLinkSlug() check makes a resource's modal
linkable and directly loadable.
- Converted the groups/edges lists from jq-repeat to plain manual DOM
rendering: jq-repeat's MutationObserver-based scope (re)registration for
an element that's destroyed and recreated on every modal open runs
asynchronously, so populating synchronously right after open() (as
refreshGroupsUI/refreshEdgesUI must) raced it -- on the second and later
opens, the old scope's destroy() ran after the new data was pushed onto
it, silently discarding it. Manual rendering (matching the new Children
tab) sidesteps the race entirely.
- The #res-name/#res-kind auto-slug handler is now bound via
app.modal.on() (delegated) instead of directly -- a direct bind would
have silently stopped firing after the first Add/Edit, since the modal
body is rebuilt from scratch on every open().
Verified live against the running dev stack: tabs/footer/groups/children
all render and populate correctly (including on a second open, confirming
the jq-repeat race fix), the address bar updates to /directory/{slug} and
reverts on close, browser Back closes the modal via popstate without a
page reload, and a resource created under a Site gets correctly
site-slug-prefixed LDAP groups.
Rewire onto @simpleworkjs/directory-schema, /ldap, and /app-stack. The
directory discovery API now returns the {results} envelope via explicit
/resources, /resources/:slug, /graph, /me handlers and routes every read
through projectResource/projectResources, which unconditionally strips
client_secret_hash (and any /secret|password|privatekey/i key) and reduces
metadata to a public allowlist for non-admins — closing the leak where the ORM
serialized metadata wholesale. The dead routes/api_discovery.js (mounted after
the 404 catcher) is removed; ?group= now returns 200 instead of 404. user_ldap
+ group_ldap take escapeFilter/escapeDN + makeClient/withClient from the shared
ldap package (posix/write-side stays app-local; cert validation unchanged).
build_info unified to {buildVersion,buildHash,buildYear}; ldapts ^8.1.8. New
tests/discovery.test.js locks in the envelope + no-secrets guarantees. Lockfile
regenerated from the registry (no file:/link:).
Co-Authored-By: Claude <noreply@anthropic.com>
wmantly