node-nmap (the vendored library, not our code) treats ANY stderr
output from the nmap binary as a fatal scan error -- including nmap's
own harmless RTT-calibration warning ("RTTVAR has grown to over N
seconds, decreasing to M"), which it prints *during* a scan that goes
on to complete normally. That meant a real, successful scan (valid XML
already sitting in the library's rawData) got discarded and reported
as a failed run with zero hosts discovered -- not just log noise as
initially assumed.
Recover in our own plugin code by detecting this specific known-benign
message and manually re-running node-nmap's own XML-parse-then-complete
path when there's actually output to parse. A genuine parse failure or
any other error message still rejects exactly as before -- this only
widens the recovery path.
Three independent copies of the same bug: routes/discovery.js's
POST /discovery/promote/:slug (the actual "Promote" button in the UI)
and services/discovery_reconciler.js's autoPromote path both called
ResourceGroup.create() directly with no existence check -- unlike
routes/api_directory_admin.js's own ensureResourceGroup, which already
carried a comment describing this exact "groups appear 3x" bug and
fixing it, just not everywhere it occurred. ResourceGroup has no DB
unique constraint on (resourceId, groupCn), so a resource promoted
more than once (retried UI click, or the same LXC discovered from
multiple Proxmox cluster nodes) silently accumulated duplicate
access/admin rows every time. Added ResourceGroup.ensure() (the
existing check-then-create pattern, now on the model) and switched all
three call sites to it. New regression test in tests/reconciler.test.js.
Also: GET /api/directory-admin/resources ran a full group-model
self-heal fan-out (ensureSiteGroups per site + provisionResourceGroups
per resource, each several sequential LDAP round-trips) unconditionally
on every single list -- confirmed via code read as the actual
bottleneck once a directory has more than a handful of resources, not
data volume. Moved healing to where resources actually change instead
(POST/PUT /resources, POST /discovery/promote/:slug -- PUT had none at
all before this), and added POST /resources/heal-groups as an explicit
on-demand equivalent for backfilling a directory seeded before this
change.
Implements TODO items "service-to-service auth model" and "no-inbound
relay automation" together -- the first was never really "no credential
type exists", it was that nothing wired one of the credential types that
ALREADY exist (theta-proxy, jump-host, and this app each already have
their own self-service API token system, models/api_token.js) into an
actual inter-service call. This is that wiring, not a new invented
credential type.
- utils/proxy_client.js: ensureRelayRoute({host, ip, targetPort}) calls
theta-proxy's real Host API (GET/POST/PUT /api/host) using a `prx_...`
token an operator mints on theta-proxy and stores in OpenBao
(secret/integrations/theta-proxy), same pattern as agent_keys.js.
Idempotent and best-effort -- never fails the caller if the token/URL
isn't configured, since this is an enhancement on top of a working
join, not a join requirement.
- POST /api/site/spokes accepts optional noInbound/meshIp/publicHost
fields; when a spoke reports itself no-inbound, the master
best-effort creates/updates the matching relay route automatically.
SiteSpoke gained the fields + a relayNote for visibility.
Verified against a REAL running theta-proxy container (not mocked):
booted it standalone, logged in as the local admin, minted a real
`prx_` token via its actual API, and drove ensureRelayRoute() against
it for real. Caught a real bug doing this: GET /api/host/:item wraps
the record in `{item, results}`, not flat -- the mocked unit tests
(which I wrote first) all had the flat shape baked in and passed
cleanly, so this only surfaced against the real API. Fixed in both the
implementation and the unit tests' mocked response shape.
Rolls up this pass's multi-site work: live catalog replication (spokes
stay synced after joining, not just a one-time snapshot), identical
agent-signing keys across sites, coordinated master promotion with real
old-master demotion, the UI to actually see and use any of it, and two
real bugs found only by live two-container testing (site-promote's dead
authorization check, and masterJoinKey/replicationPushToken leaking to
the browser via GET /api/site/config).
See CHANGELOG.md for the full list.
The shipped join flow (v2.2.0-v2.3.0) was a one-time snapshot: a spoke's
catalog never updated after joining. This adds the two pieces that were
explicitly designed but missing:
- Live replication: a spoke registers its own endpoint with the master
right after joining (POST /api/site/spokes, Bearer join-key), receiving
a pushToken. Every successful catalog write on the master now fires a
fire-and-forget resync ping (utils/site_replicate.js) at every known
spoke, concurrently -- one unreachable spoke never blocks or delays
another (wired into the existing write-gate middleware in
api_directory_admin.js). The spoke's POST /api/site/resync handler
reuses the already-tested export+import path rather than applying a
partial diff.
- Identical directories: POST /api/site/export now best-effort includes
the master's agent-signing key; a spoke adopts it via agent_keys.adopt()
on both join and every resync, so every site's sso-manager can validly
sign a command for any agent enrolled anywhere -- the accepted tradeoff
discussed for this deployment's scale (blast radius for simplicity).
New SiteSpoke model tracks registered spokes (endpoint + pushToken);
registered it in models/index.js (a real bug the e2e test below caught --
SiteSpoke.list() 500'd with "Cannot read properties of null (reading
'adapter')" until the model was added to initORM's model list).
Verified end-to-end against docker-compose.multisite-e2e.yml: mint join
key -> join with selfUrl -> write a NEW resource on master post-join ->
poll the spoke -> it shows up within a few seconds via the resync push,
no manual re-join needed. MULTISITE E2E PASS.
Unit tests: nodejs/tests/site_replicate.test.js (concurrent fan-out, one
failing spoke doesn't block another, empty-registry and list()-throws
edge cases).
Server endpoints for joining a spoke to a master directory (MULTI_SITE_SPEC.md).
This pass is server-only; setup.sh wiring and the UI are the next layer.
- Site join keys (SiteJoinKey model, stj_ prefix): mint/revoke/delete/list,
hashed at rest, shown once — the same model as agent join keys.
- POST /api/site/export (master, Bearer stj_ key, no admin session): returns the
local LDAP tree (slapcat LDIF) + resource catalog + siteSlug + baseDn.
- POST /api/site/join (spoke, admin): { masterUrl, joinKey } pulls the master
export, imports resources (upsert by slug) + LDAP (ldapadd -c), and persists
the spoke role. Refused if already a spoke.
- Persisted site role: utils/site_config.js keeps isMaster/masterUrl/siteSlug in
/config/site.json (env seeds defaults); site-status/site-promote now use it.
- Unit tests (site_join, site_config) with in-memory stubs, wired into npm test.
- docs/site-join.md + docs router entry.
- Repairs the corrupted multi-site emojis (crown/bolt) in directory.ejs.
- .gitguardian.yml ignores the generic-password false positive on reading the
LDAP bind credential from runtime config (never a hardcoded secret).
ldapbuild now pulls ghcr.io/theta42/openldap-nestgroup:<pinned commit>
instead of compiling OpenLDAP from source on every build.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
- GET /api/directory-admin/resources let every kind:'host' resource through
regardless of promotion status, so "Auto-promote to Directory" unchecked on
a discovery plugin never kept unpromoted devices out of the Directory tab.
- GET /api/directory-admin/site-status queried the nonexistent Resource.subType
column instead of metadata.subType, throwing SequelizeDatabaseError.
- Added a "Show ignored" toggle to Discovered Inventory (off by default).
- Untracked nodejs/config/inventory.sqlite -- the app's default runtime DB,
not a fixture, committed by mistake across 13 prior releases.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Removed "Why this over the alternatives"; fixed stale links to the old
per-repo GitHub Pages site to point at the unified theta-suite docs site;
made explicit this is deployed as part of Theta Suite, not standalone; added
the agent capability/install screenshots to the gallery.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(v2.0.0): bump version to 2.0.0, add Multi-Site Master badge, site-status API, and Master promotion UI
* fix(test): update test script to run unit tests without requiring live Redis socket
Mail sending fell back to a hardcoded noreply@theta42.com From address when
smtp.from wasn't set, which authenticated relays reject with "Sender is not
same as SMTP authenticate username" since no relay authorized this account
to send as that address. Falls back to smtp.user first now.
Also: catalog card titles now read name-then-icon instead of icon-then-name,
and a handful of docs corrections found in an accuracy pass (configuration.md
missing the OpenBao/live-config layer, plugins.md undercounting plugin types,
vault.md describing OpenBao dev-mode/root-token access that doesn't reflect
the real production setup, orphaned discovery.md/vault.md pages linked in,
README's required-groups list missing app_sso_directory_admin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0113gCdnfSCuZr6xvPDxTo3D
Test Email always failed with "Email.send is not a function":
models/email.js exports {Mail}, and the handler required the module and
called .send on it directly. Every other caller destructures it.
Test SMS failed with "Unexpected token '<'": it POSTed to
https://api.voip.ms/v1.0/sms/send with Basic auth, an endpoint that does
not exist. VoIP.ms's REST API is a GET against voip.ms/api/v1/rest.php
with api_username/api_password and method=sendSMS, so the fabricated URL
returned HTML and response.json() threw.
Worse, ALL SMS delivery was broken. models/sms.js called
PluginInstance.find({...}) but the ORM has no find -- the query method is
list({where}) -- so it threw on every send, before it could even fall
back to the direct VoIP.ms path. OTP-by-SMS and notifications were dead.
Both test endpoints now send through the same senders every real message
uses. A test that reimplements delivery proves nothing about whether real
delivery works, which is how two broken paths went unnoticed. Failures
report as 400 with the underlying reason rather than an opaque 500.
Adds a guard suite that fails the build on any call to a non-existent ORM
static, on requiring models/email without destructuring {Mail}, and on
any reference to the bogus api.voip.ms host.
Also: the Install Agent modal now leads with the join-key flow. v1.30.0
shipped join keys in the API and documented the modal as the place to get
one, but the modal still only did the pre-register flow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JOIN KEYS
v1.29.0 required an admin to pre-register every machine before its agent
would be spoken to. The security model was right; the workflow was not --
installing the agent should be enough to add a host.
POST /api/agent/join-keys mints one credential an operator hands out. A
host presenting it is enrolled automatically and immediately issued its
OWN per-agent token plus the public key it must pin, delivered in the
config frame. The join key is a bootstrap credential, never the host's
identity, so one key stays convenient without becoming a fleet-wide
skeleton key: every host remains individually revocable.
DIRECTORY
Collapsing the tree did nothing. applyTreeCollapse found the caret with
`.tree-caret i` and returned early when absent -- Font Awesome's SVG mode
rewrites <i> to <svg>, so that selector matched nothing and the early
return skipped setting hideBelowDepth. State now lives on the caret
button and is rotated by CSS.
The Discovery Plugins delete button called deleteDiscoveryPlugin(), which
was never defined. The pane also had no .actionMessage, and confirmations
render into one -- without it the promise never settles, so an awaited
confirmation hangs forever and the action silently never happens.
Plugin instances can now be edited.
DISCOVERY
A fresh install presented its own five containers as unmanaged
discoveries. The Docker plugin now recognises the stack's compose project
and attaches each container to the service it implements. Container slugs
came from the container id, which changes on recreate, so every deploy
minted a new resource and orphaned the old one.
DOCS
/docs/discovery 404'd (no slug entry) and `agents` pointed at plugins.md,
leaving docs/agents.md unreachable. Adds docs/discovery.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SECURITY
/api/agent/ws authenticated nothing. There was no agent registry, so any
client reaching the SSO could register as a node, publish discovery and
telemetry into the admin view, and receive commands -- including a signed
arbitrary_bash -- addressed to a token it guessed. Tokens were generated
in the BROWSER and never recorded server-side, so there was nothing to
validate against and no way to revoke one.
Agents are now rows in a new Agent table, authenticated by SHA-256 token
hash before the connection is registered or the welcome payload is sent.
Tokens are minted by POST /api/agent/enroll and shown once. Revoke and
rotate drop the live socket immediately. All agent actions are audited.
The Ed25519 command-signing key was generated in the AgentManager
constructor, so it changed on every restart and the public_key pinned in
an agent's agent.yml stopped matching. It now lives in OpenBao at
secret/agent/signing-key; if it cannot be loaded the SSO refuses to send
high-risk commands rather than signing with a key no agent has seen.
DIRECTORY
Agents bind to a host resource instead of being matched by hostname, and
a bound agent's discovery is written onto that resource -- previously the
one source running ON the host contributed nothing to the directory.
The resource tree is collapsible, with state persisted per browser.
DISCOVERY
The Proxmox plugin zipped MACs and IPs from two flat lists by index,
attributing addresses to the wrong NIC on multi-NIC guests. NICs are now
keyed by MAC. Adds an endpoint resource parenting each node, sourceId/
vmid/node identity, container-interface filtering, node IP/MAC, and
offline-node handling.
The reconciler could make a resource its own parent, named hosts after
their MAC address, had a dead isIp() regex (\\. matches a backslash),
merged across kinds, and re-read the whole inventory per resource.
Dockerfile.test-runner never copied nodejs/plugins, so every plugin test
suite failed in CI as "Cannot find module".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- group names match docs/GROUPS.md: {site}_{kind}_{name}_{level} (kind always present; services -> app kind); updated resolver + tests + access_request test
- site resource carries only god_admin + site-wide groups
- groups no longer appear 3x: idempotent ResourceGroup linking (self-heal was creating duplicates on every Directory load)
- /api/agent/* no longer 404s: REST router mounts unconditionally (was gated on the WS server)
- shared-secrets: slug regex allows underscores; GET list uses static pathFor (fixes 's.path is not a function')
- vault Apps tab: new GET /api/vault/apps + Minted apps list + purpose text; /docs/vault help link + docs cover Apps/Shared
- discovery promote: load instance and call update() (fixes 'Resource.update is not a function')
- discovery plugin cards: last-run time/status + Logs button
god_admin now exists at boot (seeded by docker-entrypoint), so the canonical
cross-resource super group nested into every resource's _admin group is god_admin,
not the legacy app_super_admin. docker-entrypoint no longer seeds or nests
app_super_admin (god_admin nests into the app_sso_* groups directly). isSuperAdmin
still recognizes a pre-existing app_super_admin as a migration alias until rebuild.
* feat: complete the group model (god_admin, site groups, aggregates), enforce naming, fix docs 500s + status dots (v1.26.0)
- seed god_admin + nest into app_super_admin; auto-provision site groups (S_super_admin, S_hosts_*/S_apps_* aggregates, S_everyone) on site create + self-heal on Directory load
- map service resources to the app kind (site_local_app_<slug>_*); nest per-resource groups into site aggregates (physical inheritance lattice)
- enforce the group naming convention server-side on POST /groups; surface god_admin + site groups on the site resource modal
- fix in-app /docs/<slug> 500s (Dockerfile never copied docs/); serve doc images at /docs/images
- fix Directory status dots (neutral grey when agent endpoint unreachable); align Profile/API cards full-width
- group resolver: keep the site slug verbatim (site_local not re-slugified)
- bump to 1.26.0
* fix: use verbatim resource slugs in group names (matches access-request tests + live convention)
The group naming inserts a kind segment (resourceGroupCns(site, kind, slug, level)),
but the access-request tests + the live directory convention are verbatim
({site}_{slug}_{level} -- the kind is carried in the resource slug, e.g. host_theta-env).
For bare test slugs this produced site_x_host_artest-host_x_access instead of the
expected site_x_artest-host_x_access, so the requester was never removed from the
auto-provisioned access group and every request 409'd. resourceGroupCns is now
(site, slug, level) with the verbatim slug; the kind is used only to pick the
aggregate the group nests into.
Keep the release version in sync with the v1.25.0 tag (the changelog was bumped
but package.json was left at 1.23.0, which would trigger a false update-check
banner).
Co-Authored-By: Claude <noreply@anthropic.com>
- /api/vault proxy now injects X-Vault-Token: the proxy declared its request
hook with http-proxy-middleware v3 syntax (on: { proxyReq }), which the
installed HPM v2 silently ignores — so every vault call reached OpenBao
unauthenticated (the recurring 403). Rewritten as v2 onProxyReq.
- Header injection ordered before fixRequestBody (the body write flushes
headers; setting X-Vault-Token after it failed on every POST/PUT).
- initORM add-only schema heal: sequelize.sync() never ALTERs, so newer columns
(PluginInstance.lastLog) are now added via describeTable + addColumn.
- Long-lived external-app tokens via sso-app role (768h periodic); VaultAppToken
stores each app token's accessor and renews it at boot + every 6h; re-minting
revokes the previous token via its accessor.
- Wire-level tests for the vault proxy + app-token accessor lifecycle.
- package.json + lockfile bumped to 1.23.0.
Co-Authored-By: Claude <noreply@anthropic.com>
- New admin Agents page (nav + /agents route + views/agents.ejs): live list of
connected theta-agent hosts with CPU/RAM/disk/ZFS/GPU telemetry and online
status, updated live over socket.io ('agent.telemetry'/'agent.discovery').
- Auth + admin-gate the /api/agent REST router (it was mounted without
middleware.auth — anyone could list nodes / send commands). The agent
WebSocket (/api/agent/ws) is unaffected (handled by the raw wss upgrade with
its own token auth).
- package.json + lockfile bumped to 1.22.0 to match the tag.
Co-Authored-By: Claude <noreply@anthropic.com>
- vault_broker: always reconcile policy content before serving a cached
token (compare-and-skip), so stale stored policies can't cause a recurring
403 'permission denied'; policy content is parsed live by OpenBao, so edits
apply to existing tokens immediately.
- Shared secrets: publish to secret/shared/<owner>/<slug>; grant read to users
and apps by editing the grantee's policy content (live-applied). New
SharedSecret/SharedSecretGrant ORM models, /api/shared-secrets router, and a
Shared tab in the vault UI.
- package.json + lockfile bumped to 1.21.0 to match the tag.
Co-Authored-By: Claude <noreply@anthropic.com>
The v1.20.2 release tag was created but nodejs/package.json (and the
lockfile) were left at 1.20.1, so the deployed app's buildVersion lagged
its own release tag and the update-check banner falsely reported a newer
version. Bump the version fields to match the tag.
GET /api/conf no longer returns smtp.pass / oauth.jwtSecret in cleartext
(masked to ********). POST treats a blank or ******** secret submission as
"keep the stored value," so editing the From address or token lifetimes no
longer requires re-entering or leaks the SMTP password / JWT secret. The /conf
form fields carry a leave-unchanged hint. Storage stays in OpenBao at
secret/sso-manager/conf (unchanged); no theta-suite policy change needed.
Co-Authored-By: Claude <noreply@anthropic.com>
Generalize the half-built discovery plugins into a real plugin system: plugin
TYPES (the plugins/<category>/<type>.js modules with manifests) and loadable,
configurable, multi-copy plugin INSTANCES (PluginInstance ORM model) managed
from a dedicated /plugins page and /api/plugins API, with per-instance secrets
in OpenBao at secret/plugins/<id>/conf.
- plugin_registry.js: getTypes/getModule/splitConfig/mask + required-field helpers
- PluginInstance model (Sequelize): id/pluginType/category/name/slug(unique)/
enabled/cron/config(json, non-secret)/lastRun*; registered in models/index.js
- plugin_secrets.js: read/write/remove/mergeForRun over @simpleworkjs/bao-conf
- scheduler.js: schedules from the DB registry; per-instance stable BullMQ
JobScheduler ids (plugin:<id>) for load/unload; legacy migration from
conf.discovery.plugins on first boot (idempotent, empty-table-guarded)
- api_plugins.js (replaces routes/plugins.js): types/list/get/create/update/
secrets/test/load/unload/run/delete/runs; admin-gated; secrets always masked
- /plugins page (plugins.ejs) + nav; Agents & Scheduler tab removed from
/directory; /docs/agents aliased to /docs/plugins
- proxmox/unifi/nmap gained manifests (configSchema/validate/run alias)
- tests/plugins.test.js: registry unit + plugin_secrets (mocked bao-conf) +
PluginInstance model round-trip/unique-slug
- docs (plugins.md, vault.md, _config.yml, API.md) + 1.16.1 -> 1.17.0
Requires theta-suite >= v1.30.1 for the sso-broker secret/plugins/* grant;
fails-soft with a clear error if absent.
Co-Authored-By: Claude <noreply@anthropic.com>
Both view routes did server-side auth via req.user, but this app's auth-token is
a header set by client JS (localStorage), not a cookie — so req.user is
undefined on a browser navigation. permission.byGroup(undefined,...) throws
status 401, and the middleware.auth gate on /vault threw Auth.errors.login()
(401) for the same reason.
Both routes now render the shell unconditionally (like /users, /directory) and
gate client-side. conf.ejs already called app.auth.forceLogin; vault.ejs now
derives isAdmin + personal namespace from /api/user/me after forceLogin
instead of server-rendering them. /api/conf and /api/vault still enforce
app_sso_admin + OpenBao scope server-side — only the view-route gating moved
client-side where the session lives. Also removed a dead duplicate /conf route.
Co-authored-by: Claude <noreply@anthropic.com>
wmantly