Generalize the half-built discovery plugins into a real plugin system: plugin
TYPES (the plugins/<category>/<type>.js modules with manifests) and loadable,
configurable, multi-copy plugin INSTANCES (PluginInstance ORM model) managed
from a dedicated /plugins page and /api/plugins API, with per-instance secrets
in OpenBao at secret/plugins/<id>/conf.
- plugin_registry.js: getTypes/getModule/splitConfig/mask + required-field helpers
- PluginInstance model (Sequelize): id/pluginType/category/name/slug(unique)/
enabled/cron/config(json, non-secret)/lastRun*; registered in models/index.js
- plugin_secrets.js: read/write/remove/mergeForRun over @simpleworkjs/bao-conf
- scheduler.js: schedules from the DB registry; per-instance stable BullMQ
JobScheduler ids (plugin:<id>) for load/unload; legacy migration from
conf.discovery.plugins on first boot (idempotent, empty-table-guarded)
- api_plugins.js (replaces routes/plugins.js): types/list/get/create/update/
secrets/test/load/unload/run/delete/runs; admin-gated; secrets always masked
- /plugins page (plugins.ejs) + nav; Agents & Scheduler tab removed from
/directory; /docs/agents aliased to /docs/plugins
- proxmox/unifi/nmap gained manifests (configSchema/validate/run alias)
- tests/plugins.test.js: registry unit + plugin_secrets (mocked bao-conf) +
PluginInstance model round-trip/unique-slug
- docs (plugins.md, vault.md, _config.yml, API.md) + 1.16.1 -> 1.17.0
Requires theta-suite >= v1.30.1 for the sso-broker secret/plugins/* grant;
fails-soft with a clear error if absent.
Co-Authored-By: Claude <noreply@anthropic.com>
Closes the end-user half of the directory and adds nested LDAP groups.
The directory could describe the lab but could not tell anyone what they had
or how to reach it, and several of the paths meant to do so were silently
returning nothing:
- GET /api/discovery/me resolved groups from req.user.groups, which does not
exist (req.user carries memberOf), so it returned only isPublic resources
for every human caller -- "My Services" was blank for everyone. The same
read made isDirectoryAdmin() false for real admins.
- The portal's "Discover More Services" called the admin-gated endpoint and
swallowed the 403, so it never rendered for non-admins at all.
- Services reported no address, because /me had reimplemented getMyAccess
without its parent-walking resolution.
Adds the catalog at /, self-service access requests, and admin access
visibility (per-resource counts, and the reverse "what can this user reach").
Nested groups come in two halves. groupOfNames.member already accepts a group
DN, so nesting needs no schema -- what it needs is resolution, which no
released OpenLDAP performs. The all-in-one image therefore builds slapd from a
pinned master commit for the nestgroup overlay, and the app computes the
closure itself when pointed at a server without it. Both paths are covered.
member-values is deliberately left out of nestgroup-flags: it expands `member`
when reading a group, which destroys the distinction between "listed here" and
"reachable through a nested group" and is not recoverable afterwards.
Full suite green in both resolution modes: 215 passed, 2 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Add Resource audit fields (created/updated by/on) and site-slug group prefixing
Resource had no created_by/created_on/updated_by/updated_on fields at all,
unlike proxy's Host and jump-host's ApiToken which already track this --
needed for the upcoming resource-modal footer. @simpleworkjs/orm has no
auto-timestamp hook, so these are set explicitly in the directory-admin
route handlers on every create/update.
Also: when a host/service resource is created, its two auto-created LDAP
groups (<slug>_access/_admin) now get prefixed with the nearest ancestor
site's slug (via a new Resource.findAncestorSiteSlug walk), so groups from
different sites don't collide/look identical. Falls back to today's
unprefixed naming when a resource has no site ancestor.
Included the checked-in dev inventory.sqlite's ALTER TABLE for the new
columns, since @simpleworkjs/orm's sync() only creates missing tables, never
alters existing ones -- the raw model change alone would have broken every
Resource read/write against this file with "no such column: created_by".
* Migrate Resource modal onto app.modal's tabs/footer/URL, add Children tab
The Directory's resource modal was a separate, hand-rolled, always-in-DOM
Bootstrap modal, independent of the shared app.modal singleton -- migrating
it onto app.modal (now published with tabs/footer/url support in
@simpleworkjs/frontend 0.2.6) is the pilot for standardizing entity modals
across the stack.
- General/Details/Associated LDAP Groups/Children tabs, replacing the old
single long form (Details keeps every kind-conditional container
unchanged; toggleFormFields() didn't need to change at all).
- Footer shows created/updated by/on (via the new Resource audit fields)
and the Save button; Groups/Children tabs are hidden in add-mode since
they need an existing resource id.
- New Children tab lists a resource's existing children (reusing the
already-loaded edges/resourcesById data, no new endpoint) and an "Add
Child Resource" button that reuses openAddModal's existing preset-parent
support. Folded the pre-existing generic "Relationships (Graph Edges)"
section in underneath, under an "advanced" subheading, rather than
dropping it or giving it a 5th tab of its own.
- GET /directory/:slug (mirroring the existing /users/:uid precedent) plus
a client-side app.modal.deepLinkSlug() check makes a resource's modal
linkable and directly loadable.
- Converted the groups/edges lists from jq-repeat to plain manual DOM
rendering: jq-repeat's MutationObserver-based scope (re)registration for
an element that's destroyed and recreated on every modal open runs
asynchronously, so populating synchronously right after open() (as
refreshGroupsUI/refreshEdgesUI must) raced it -- on the second and later
opens, the old scope's destroy() ran after the new data was pushed onto
it, silently discarding it. Manual rendering (matching the new Children
tab) sidesteps the race entirely.
- The #res-name/#res-kind auto-slug handler is now bound via
app.modal.on() (delegated) instead of directly -- a direct bind would
have silently stopped firing after the first Add/Edit, since the modal
body is rebuilt from scratch on every open().
Verified live against the running dev stack: tabs/footer/groups/children
all render and populate correctly (including on a second open, confirming
the jq-repeat race fix), the address bar updates to /directory/{slug} and
reverts on close, browser Back closes the modal via popstate without a
page reload, and a resource created under a Site gets correctly
site-slug-prefixed LDAP groups.
routes/group.js's add/removeMember never called User.clearCache(), unlike
the isServiceAccount handling in routes/user.js (which does this
deliberately, with a comment explaining exactly why). isServiceAccount is
derived at User.get() time from app_sso_service_account membership and
cached for 5 minutes -- so adding or removing a user from ANY group via
this route left group-derived state (isServiceAccount, and by extension
anything else that reads memberOf off a cached User) stale for up to 5
minutes.
In production this manifested as a real user's account appearing to
"vanish": users.ejs's People tab filters out anything with
isServiceAccount truthy, so once that user's membership in
app_sso_service_account changed, they'd disappear from the tab anyone
actually looks at for up to 5 minutes -- looking exactly like data loss,
though the account was never touched. Found by investigating a live "lost
users" report: the account had isServiceAccount: 'yes' and was in fact
still fully present, just hidden.
This does not explain how the account came to be a member of
app_sso_service_account in the first place (unresolved -- possibly a
manual/accidental group-membership change via the Groups UI, which has no
guardrail against adding a real person to what's meant to be a marker
group for non-person accounts). It does fix a real correctness gap: any
admin group-membership change now takes effect immediately instead of on
a timer.
Verified against a real LDAP+Redis harness: the new test fails on the
unfixed code (stale isServiceAccount immediately after the PUT) and
passes with the fix. Full suite: 189/191 passing (2 pre-existing skips).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Native confirm() blocks all further browser events on the page (found
live, mid browser-automation testing, on directory.ejs's "Rotate Client
Secret" -- it froze the tab). Every call site across the app was removed
in favor of app.messages.action/confirm/toast and app.modal.open; this
static check (scans views/ and public/js|lib/js for bare alert(/confirm(/
prompt() calls) keeps a regression from shipping unnoticed the way the
oauth_client.js DELETE bug just did.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
OAuthClient wraps @simpleworkjs/orm's Resource model, whose instance
delete method is .delete() -- not .remove(), which is what model-redis's
Table instances (e.g. this app's ApiToken, AuthToken) use. The DELETE
route called the wrong one, so every delete silently 500'd; the route's
try/catch turned it into a plain JSON error response rather than a thrown
exception, and the existing tests' cleanup-only delete calls (afterAll,
end of the rotate test) never checked the response status, so the bug
shipped unnoticed. The Directory Management UI was never affected --
routes/api_directory_admin.js's DELETE routes already used .delete()
correctly throughout.
Found and root-caused live against a real deployment's SSO API, then
reproduced and fixed against a local docker stack with a rebuilt image:
confirmed DELETE returned a genuine 500 before the fix and a real 200 +
404-on-subsequent-GET after.
Adds two dedicated tests (PUT and DELETE persistence, each verified by a
follow-up GET rather than trusting the mutating response alone), and
hardens the existing rotate test's incidental delete call with real
assertions. Verified the new DELETE test fails on the old code and
passes on the fix. Full suite (189 tests, real LDAP + Redis) passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rewire onto @simpleworkjs/directory-schema, /ldap, and /app-stack. The
directory discovery API now returns the {results} envelope via explicit
/resources, /resources/:slug, /graph, /me handlers and routes every read
through projectResource/projectResources, which unconditionally strips
client_secret_hash (and any /secret|password|privatekey/i key) and reduces
metadata to a public allowlist for non-admins — closing the leak where the ORM
serialized metadata wholesale. The dead routes/api_discovery.js (mounted after
the 404 catcher) is removed; ?group= now returns 200 instead of 404. user_ldap
+ group_ldap take escapeFilter/escapeDN + makeClient/withClient from the shared
ldap package (posix/write-side stays app-local; cert validation unchanged).
build_info unified to {buildVersion,buildHash,buildYear}; ldapts ^8.1.8. New
tests/discovery.test.js locks in the envelope + no-secrets guarantees. Lockfile
regenerated from the registry (no file:/link:).
Co-Authored-By: Claude <noreply@anthropic.com>
The ORM Model.toJSON() serializes only schema fields, so the mapped
client_id/scopes/redirect_uris/... that OAuthClient.get() attaches to
the wrapped Resource were stripped from GET /api/oauth/client[/:id]
responses. client_id came back undefined; the theta-env bootstrap then
POSTed /api/oauth/client/undefined/rotate and got a 500, aborting stack
bring-up whenever proxy-secrets.js lacked a usable secret.
- OAuthClient.get() now emits an explicit public toJSON (client_id, name,
slug, scopes, redirect_uris, allowed_groups, token_lifetime, is_valid),
deliberately omitting client_secret_hash so it can't leak over the API.
- OAuthClient.get() null-guards Resource.get() (which returns null, not
throws) and returns a clean 404 for an unknown/undefined id instead of
crashing on r.kind.
- Regression tests: list/get expose client_id + hide the secret hash, the
list-then-rotate bootstrap path, and unknown-id -> 4xx not 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docker-compose.test.yml spins up the all-in-one OpenLDAP image, a
standalone Redis, and a test-runner that seeds the test user and runs
jest against them. globalSetup honors REDIS_URL; tests/setup.js
initializes the ORM and flushes test Redis keys before the run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add conf.ldap.ldapsHost / conf.ldap.ldapsPort so the /integrations page
can advertise an internal-only LDAPS hostname separate from the public
OAuth issuer. This avoids forcing admins to port-forward 636 publicly.
- routes/index.js derives LDAPS URL from ldapsHost/ldapsPort with issuer fallback
- integrations.ejs adds a contextual help panel explaining TLS hostname
validation, the public-issuer default, and recommended internal-DNS /
Docker-internal alternatives
- conf/base.js, secrets.js.example, DEPLOYMENT.md, docs/configuration.md,
and docs/ldap.md document and expose the new options
- Add tests/integrations.test.js for default and custom ldapsHost behavior
- Bump version to 1.1.17
Co-authored-by: Claude <noreply@anthropic.com>
- New GitHub Actions workflow: builds the real Dockerfile.openldap
image, starts it, seeds the LDAP fixtures the test suite expects
(uid 'test' + 'wmantly', matching the existing "wmantly is always
present in the test LDAP" assumption in several test files), then
runs the full Jest suite against it on Node 18/20/22. This repo
previously had unit tests but no automated workflow running them.
- Found while building this: the bundled default ppolicy entry
(docker-entrypoint.sh + ops/ldap-setup.sh) sets pwdLockout: FALSE,
which is backwards -- it silently makes the admin "deactivate user"
action a no-op for auto-lockout-after-failed-attempts (a related
but distinct ppolicy feature from pwdAccountLockedTime). Fixed to
TRUE in both places; ldap-setup.sh also gets a drift-correction
path so an existing deployment can pick up the fix by re-running it.
- Separately, deactivating a user still doesn't block their LDAP bind
in the bundled image even with this fix -- filed as #68, since it's
a deeper OpenLDAP ppolicy overlay question unrelated to the CI/test
setup here. tests/user_admin.test.js now soft-skips that specific
assertion (with a console warning pointing at #68) instead of
failing, so this known environment gap doesn't block CI.
theta42/proxy fronts an arbitrary number of hosts behind SSO, each with its
own callback URL (https://<host>/__proxy_auth/callback) — proxy's own code
comment already assumed "a wildcard redirect URI covers all", but no
wildcard matching existed here, so every proxied host's callback had to be
registered on the shared OAuth client individually or /oauth/authorize
would reject it with InvalidRedirectURI.
Add `*` (one hostname label) / `**` (any number of labels) wildcard support
to redirect_uri matching, e.g. `https://**.example.com/__proxy_auth/callback`
now covers every host proxy fronts under example.com. Exact matches still
work exactly as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reported: creating any user via the API failed with
{"name":"InvalidSyntaxError","message":"gidNumber: value #0 invalid per syntax Code: 0x15"}
Root cause: addPosixGroup() computes the next gidNumber as
`Math.max(...groups.map(i => i.gidNumber)) + 1`. theta-env's
bootstrap.js creates the first admin via raw ldapadd with a hardcoded
uidNumber/gidNumber (10000) directly on the user entry, but never
creates a matching posixGroup entry -- so on a theta-env-bootstrapped
directory there are zero posixGroup entries, `Math.max()` on an empty
array is `-Infinity` in JS (not 0), and `-Infinity + 1` stringifies to
"-Infinity" -- an invalid LDAP integer, rejected by the directory. This
broke every single user creation, not just this one.
Separately: the reporter's intended scheme is for organically-created
users to start at uidNumber/gidNumber 1500, distinct from the
bootstrap admin's reserved 10000. Fixing the crash with a bare "floor
of 1500" alone wouldn't achieve that, since addPosixAccount's own
Math.max() would still find the admin's posixAccount entry (uidNumber
10000, found via a different, correctly-indexed search) and allocate
10001 for the next user.
Added a shared nextPosixId(entries, key) helper: takes the highest
existing value strictly below conf.ldap.uidGidReservedFloor (default
9000) plus one, or conf.ldap.uidGidMin (default 1500) if there are no
such entries. Ids at/above the reserved floor -- like the bootstrap
admin's 10000 -- are ignored entirely when computing the next
available number, so real users always start at 1500 and grow upward
regardless of the admin's reserved id.
Verified against a real theta-env deployment end to end:
- Reproduced the exact reported crash on a fresh bootstrap
- After the fix: first real user gets uidNumber/gidNumber "1500",
second gets "1501" -- admin's 10000 never enters the calculation
- New unit tests (nodejs/tests/posix_id.test.js, no LDAP required):
6/6 pass, covering the empty-array case, the reserved-floor
exclusion, and the NaN-from-missing-value case
- npm test: 18/18 passing tests still pass (unchanged); the other 155
failures are pre-existing/environmental (no LDAP server in this
sandbox) -- confirmed via git stash before starting this fix
wmantly