Merge pull request #129 from theta42/release-v1.31.1

release v1.31.1: sso v1.17.2 + /vault policy fix (setup.sh)

CHANGELOG.md

@@ -8,6 +8,259 @@ orchestration code; see each submodule's own `CHANGELOG.md`
 [sso-manager-node](https://github.com/theta42/sso-manager-node/blob/master/CHANGELOG.md))
 for what changed inside the apps it composes.
 
+## [v1.31.1] - 2026-08-01
+
+Pairs the sso v1.17.2 post-deploy fixes with the theta-suite half of the
+`/vault` secrets-list 403 fix (the `sso-admin` OpenBao policy grant that lives
+in `setup.sh`), and rolls the `sso-manager-node` submodule gitlink to v1.17.2.
+`proxy` (v1.13.1), `jump-host` (v1.14.1), and `ldap-client` (v1.23.0) are
+unchanged.
+
+### Changed (theta-suite)
+- **`setup.sh` — `sso-admin` policy**: added a `list` grant on the bare KV mount
+  root `secret/metadata` so an admin can list the top-level dirs in the `/vault`
+  UI. `secret/metadata/*` already covered nested paths, but not the mount root
+  itself — so the secrets list 403'd. (The matching per-user/per-app directory
+  grants ship in sso v1.17.2's `vault_broker.js`.)
+- **`setup.sh` — `ensure_policy`**: now always (re)writes the policy instead of
+  skipping when it exists. `bao policy write` is an idempotent overwrite, so a
+  re-run applies policy edits (like the new grant above) instead of stranding
+  the old HCL with "already exists — keeping."
+
+### Changed (submodule gitlinks)
+- **sso-manager-node**: `v1.17.1` → `v1.17.2` — the post-deploy fixes (auto-slug
+  plugins, schedule dropdown, `/profile` rendering, plugin-edit persistence,
+  nmap in the image, the sso-side `/vault` policy grants) plus the SMS (VoIP.ms)
+  and Terms-of-Service configuration on `/conf`. Full changelog below.
+
+### Deploy
+Operators upgrading from v1.31.0:
+1. `git pull` and `git submodule update --init --recursive`.
+2. Re-run `./setup.sh` — **required**: applies the new `sso-admin`
+   `secret/metadata` list grant and the `ensure_policy` always-write refresh
+   (idempotent). Per-user vault policies self-heal on the next `/vault` visit
+   (sso v1.17.2 re-writes them).
+3. `docker compose build && docker compose up -d` — the rebuild installs `nmap`
+   in the sso image (fixes the nmap plugin "not found" error).
+
+### Bundled submodule release notes
+
+#### sso-manager-node v1.17.2 — post-deploy fixes + SMS/TOS on /conf
+
+Post-deploy fixes from testing the v1.31.0 stack, plus the SMS (VoIP.ms) and
+Terms-of-Service configuration the `/conf` page was missing.
+
+##### Fixed
+- **Plugin slug is now auto-generated** from the instance name — the New Plugin
+  modal no longer asks for a Slug (it derives a stable, unique handle from the
+  name, appending `-2`, `-3`, … on collision). The generated slug still shows in
+  the table and the Edit (read-only) modal. `POST /api/plugins` `slug` is now
+  optional; an explicit slug is still accepted and validated.
+- **Plugin schedule is a dropdown**, not a raw cron box: Hourly / Daily /
+  Weekly, plus **Custom** which reveals the raw 5-field cron input. Stored value
+  is still a cron string, so the server is unchanged.
+- **`/vault` secrets list no longer 403s.** The per-user, per-app, and admin
+  OpenBao policies granted `list` only on `secret/metadata/.../*` (nested
+  paths), never on the directory path itself — so listing a directory's
+  *contents* (which checks `list` on the directory, e.g.
+  `secret/metadata/users/<uid>` or the mount root `secret/metadata`) was denied.
+  `vault_broker.js`'s `userPolicyHcl`/`appPolicyHcl` now also grant `list` on the
+  bare directory path, and `ensurePolicy` now always re-writes the policy
+  (idempotent) so already-created `user-<uid>` policies pick up the new grant on
+  the next vault-page visit. The matching `sso-admin` mount-root grant ships in
+  theta-suite v1.31.1 (`setup.sh`), where `ensure_policy` is likewise made
+  always-write so re-running `./setup.sh` applies policy edits.
+- **`/profile` no longer shows literal `{{…}}` tags.** Three template fragments
+  sat outside the `jq-repeat="user"` scope, so they rendered raw: the card
+  header `Profile: {{user.uid}}`, the `Members of {{user.uid}}'s Group` tab
+  label, and the Admin Actions block's `{{#isActive}}`/`{{#isInactive}}`
+  buttons. The header/label are now populated by JS (the `Members` label
+  already had a setter pointing at a missing id); the Admin Actions block is
+  moved inside the scope so `{{uid}}`/`{{#isActive}}`/`{{#isInactive}}` render
+  and the correct Activate/Deactivate button shows.
+- **Editing a plugin now persists.** The Edit modal had been prefilled with the
+  masked secret values and rendered them as fields, but `PUT /:id` only saves
+  non-secret config — so an edited secret was silently dropped. The Edit modal
+  now shows **non-secret fields only** (secrets have their own Edit-Secrets
+  modal), removing the confusion.
+- **nmap plugin: "NMAP not found at command location: nmap"** — the `nmap`
+  binary was not installed in the app image. `Dockerfile.openldap` now `apk
+  add`s `nmap` in the runtime stage, and `plugins/discovery/nmap.js` translates
+  the opaque node-nmap spawn-missing error into an actionable `lastError`.
+
+##### Added
+- **SMS (VoIP.ms) configuration on `/conf`.** The existing VoIP.ms SMS sender
+  (`models/sms.js`, used for 2FA OTP delivery) was configurable only via env /
+  config files. It now has an SMS card on `/conf` (API username, DID, API
+  password), saved to OpenBao at `secret/sso-manager/conf` under `voipms`, with
+  the API password masked (`********`) and leave-blank-to-keep — mirroring the
+  SMTP card exactly. `models/sms.js` reads `conf.voipms.*` at call time, so a
+  saved change takes effect live without a restart.
+- **Terms of Service editor moved to `/conf`** from the admin Overview
+  dashboard, where it never belonged. The same `app.tos.get`/`update` flow,
+  the "require all users to re-accept" checkbox, and the `app_sso_admin` gate
+  (matching `routes/tos.js`'s PUT gate) are preserved. The Overview page keeps
+  stats, notifications, and metrics.
+
+## [v1.31.0] - 2026-08-01
+
+Roll-up release: bumps the composed submodules to their latest tags so a fresh
+`git clone` + `./setup.sh` deploys the SSO Manager plugin system, the `/conf`
+SMTP/OAuth secret masking, and the ldap-client changelog. `proxy` (v1.13.1) and
+`jump-host` (v1.14.1) were already at latest and are unchanged.
+
+### Changed (submodule gitlinks)
+- **sso-manager-node**: `v1.16.1` → `v1.17.1` (the plugin system shipped in
+  v1.17.0, plus the v1.17.1 `/conf` secret-masking hardening).
+- **ldap-client**: `v1.1.1` → `v1.23.0` — a CHANGELOG-only release (the new
+  `CHANGELOG.md` documenting v1.1.0/v1.0.0; **no code change** — the "UI polish"
+  tag message is misleading, the v1.1.1…v1.23.0 diff is `CHANGELOG.md` only).
+
+### Deploy
+Operators upgrading from a prior release:
+1. `git pull` and `git submodule update --init --recursive` (or a fresh clone).
+2. Re-run `./setup.sh` — this is **required** if you haven't yet applied the
+   v1.30.1 `sso-broker` OpenBao policy grant for `secret/plugins/*` (idempotent;
+   it grants the existing `SSO_VAULT_TOKEN` access live, so plugin-secrets
+   storage works).
+3. `docker compose build && docker compose up -d`. Existing
+   `conf.discovery.plugins` setups auto-migrate into `PluginInstance` rows +
+   OpenBao secrets on first boot of sso v1.17.x.
+
+### Bundled submodule release notes
+
+#### sso-manager-node v1.17.0 — real plugin system (loadable instances + OpenBao secrets)
+
+## [1.17.0] - 2026-08-01
+
+A real **plugin system**: the half-built discovery plugins (statically
+configured in `sso-secrets.js`, only toggleable for cron/enabled) become
+**configurable, loadable/unloadable plugin instances** you manage from a
+dedicated **Plugins** page and the `/api/plugins` API, with multiple runtime
+copies of each type and per-instance secrets stored in OpenBao.
+
+### Added
+- **Plugin instances** — a new `PluginInstance` ORM model
+  (`nodejs/models/plugin_instance.js`, Sequelize) is the registry of
+  configured, scheduled plugin copies. Each has a `pluginType`, a unique
+  `slug` (the discovery source name), a cron schedule, an `enabled` flag
+  (load/unload), non-secret `config` (JSON), and last-run bookkeeping. Multiple
+  instances of the same type are supported.
+- **Plugin registry** (`nodejs/services/plugin_registry.js`) — generalizes the
+  one-shot discovery-plugin scan in `scheduler.js`. Plugin types are modules
+  under `nodejs/plugins/<category>/<type>.js` exporting a manifest
+  (`type`, `category`, `name`, `description`, `configSchema`, `validate`,
+  `run`/`discover`). Exposes `getTypes`, `getModule`, `splitConfig` (secret vs
+  non-secret), `mask`, and required-field helpers for the UI/API.
+- **Per-instance secrets in OpenBao** (`nodejs/utils/plugin_secrets.js`) —
+  `configSchema` fields flagged `secret:true` (e.g. a Proxmox `tokenSecret`,
+  UniFi `password`) are stored at `secret/plugins/<instance-id>/conf`, never in
+  the DB. The UI only ever sees masked (`********`) values. Plugins run
+  in-process (BullMQ workers), so they need no OpenBao token of their own — the
+  SSO reads/writes via the `sso-broker` token. **Requires theta-suite ≥ v1.30.1**
+  for the `sso-broker` policy grant on `secret/plugins/*`; the API fails-soft
+  with a clear error if absent.
+- **`/api/plugins` API** (`nodejs/routes/api_plugins.js`, replaces the old
+  `routes/plugins.js`) — `GET /types`, list/get/create/update/update-secrets/
+  test/load/unload/run/delete/runs. Admin-only
+  (`app_sso_admin` / `app_sso_directory_admin` / `app_super_admin`).
+- **Plugins page** (`/plugins`, `views/plugins.ejs`) + nav entry — instance
+  table with New/Edit/Edit-Secrets/Test/Run-now/Load/Unload/Delete, config forms
+  rendered from each type's `configSchema`.
+- **`validate`** ("Test" button) on the built-in Proxmox/UniFi/Nmap plugins.
+
+### Changed
+- `services/scheduler.js` now schedules from the `PluginInstance` table instead
+  of static `conf.discovery.plugins` + a Redis override hash. Each instance owns
+  a stable BullMQ JobScheduler id (`plugin:<instanceId>`) so load/unload
+  upsert/remove one schedule without disturbing the rest. Discovery plugins
+  reconcile results under the instance's `slug`.
+- The three discovery plugins (`plugins/discovery/{proxmox,unifi,nmap}.js`)
+  gained manifests (`configSchema`, `validate`, `run` alias). `nmap`'s
+  `targetRange` is non-secret; Proxmox `tokenSecret` and UniFi `password` are
+  secret.
+- The `/plugins` page route renders the page instead of redirecting to
+  `/directory`; the **Agents & Scheduler** tab was removed from `/directory`
+  (plugins are now managed on the Plugins page). The `/docs/agents` link is
+  aliased to `/docs/plugins`.
+- `docs/plugins.md`, `docs/vault.md`, `docs/_config.yml` (nav), and `API.md`
+  (Plugin Endpoints section) document the new system.
+
+### Legacy migration
+On first boot of v1.17.0, if the `PluginInstance` table is empty **and**
+`conf.discovery.plugins` has entries, one instance per configured type is seeded
+automatically (secret fields copied into OpenBao). After that the static
+config is ignored — manage plugins from the UI/API. Idempotent (guarded by the
+empty-table check).
+
+### Prerequisite
+**theta-suite ≥ v1.30.1** — re-run `./setup.sh` after upgrading so the
+`sso-broker` OpenBao policy is granted `secret/plugins/*`. Without it, storing
+plugin secrets fails with a clear error.
+
+
+#### sso-manager-node v1.17.1 — mask SMTP/OAuth secrets + leave-blank-to-keep on /conf
+
+## [1.17.1] - 2026-08-01
+
+Hardens the **runtime SMTP/OAuth secret handling** on the `/conf` admin page to
+match the plugin-secrets discipline: the SMTP password and OAuth JWT secret are
+no longer returned in cleartext by `GET /api/conf` or round-tripped through the
+form. They remain saved in OpenBao at `secret/sso-manager/conf` at runtime
+(unchanged) — only how they're surfaced to the admin changes.
+
+### Changed
+- **`GET /api/conf`** now masks `smtp.pass` and `oauth.jwtSecret` to `********`
+  (was: returned in cleartext). Non-secret fields (host, port, user, from,
+  secure, issuer, token lifetimes) are returned as before.
+- **`POST /api/conf`** now treats a blank or `********` secret-field submission
+  as "keep the current stored value" — so an admin editing the From address or
+  token lifetimes no longer has to re-enter (or leak) the SMTP password / JWT
+  secret. Only a genuinely new, non-blank value overwrites. The preserved values
+  are re-applied to live `conf` immediately, as before.
+- **`/conf` page** (`views/conf.ejs`): the Password and JWT Secret fields carry
+  a "leave unchanged to keep the current value stored in OpenBao" hint; the page
+  copy notes secret fields are masked. No JSON-textarea editing is involved —
+  SMTP is and remains configured through structured form fields.
+
+### Notes
+- SMTP (and OAuth) config was **already** saved to OpenBao at runtime before
+  this release (via `POST /api/conf` → `baoConf.set('sso-manager/conf')`, and
+  overlaid back at boot by `bao-conf.init`). This release closes the
+  cleartext-exposure gap; it does not move the storage path.
+- No theta-suite policy change required — `secret/sso-manager/conf` was already
+  granted to the `sso-broker` policy.
+
+
+#### ldap-client v1.23.0 — CHANGELOG-only (no code change)
+
+Adds a `CHANGELOG.md` documenting v1.1.0 (`app_super_admin` / `app_jump_admin`
+group support in SSSD access filters; the sso/jump-host TLS-validation
+divergence) and v1.0.0 (initial SSSD LDAP auth release). No source changes vs
+v1.1.1; the v1.23.0 tag commit only adds this file.
+
+## [v1.30.1] - 2026-08-01
+
+Prerequisite release for the SSO Manager plugin system (shipped in
+sso-manager-node v1.17.0). Grants the `sso-broker` OpenBao policy access to the
+new per-instance plugin secrets namespace so the SSO can store plugin secrets in
+OpenBao instead of `sso-secrets.js`.
+
+### Changed (theta-suite orchestration)
+- **`setup.sh`**: added `secret/data/plugins/*` (CRUD+list) and
+  `secret/metadata/plugins/*` (list/read/delete) to the `sso-broker` policy
+  HCL. `ensure_policy sso-broker` is idempotent, so re-running `./setup.sh`
+  immediately grants the existing `SSO_VAULT_TOKEN` access to `secret/plugins/*`
+  (policies are evaluated live; the token keeps its id). The SSO side fails-soft
+  with a clear error if this grant is absent.
+- **Docs**: `docs/secrets.md` (new "Plugin secrets" section + `sso-broker`
+  policy row) and `docs/architecture.md` (sso-manager access row) now list
+  `secret/plugins/*`.
+
+> The plugin system itself (configurable plugin instances, load/unload, UI/API,
+> multi-copy, secrets in OpenBao) is in sso-manager-node v1.17.0; theta-suite
+> will bump its submodule gitlink to that release next.
+
 ## [v1.30.0] - 2026-08-01
 
 The project is renamed **theta-env → theta-suite** — it has grown from a

docs/architecture.md

@@ -99,7 +99,7 @@ the paths it needs:
 
 | Service | env var | Policy | Access |
 |---------|---------|--------|--------|
-| sso-manager | `SSO_VAULT_TOKEN` | `sso-broker` | `secret/sso-manager/conf`, `secret/users/*`, `secret/apps/*`; also mints per-user + per-app tokens |
+| sso-manager | `SSO_VAULT_TOKEN` | `sso-broker` | `secret/sso-manager/conf`, `secret/users/*`, `secret/apps/*`, `secret/plugins/*`; also mints per-user + per-app tokens |
 | proxy | `PROXY_VAULT_TOKEN` | `proxy` | `secret/proxy/conf` (read) |
 | jump-host | `JUMP_VAULT_TOKEN` | `jump-host` | `secret/jump-host/conf` (read) |
 

docs/secrets.md

@@ -60,7 +60,7 @@ never passed to a service container.
 
 | Policy | Capabilities | Held by |
 |---|---|---|
-| `sso-broker` | read/write `secret/sso-manager/conf`, `secret/users/*`, `secret/apps/*`; `update` on `auth/token/create/sso-broker`; `update` on `sys/policies/acl/user-*`, `app-*`, `sso-admin` | SSO (`SSO_VAULT_TOKEN`) |
+| `sso-broker` | read/write `secret/sso-manager/conf`, `secret/users/*`, `secret/apps/*`, `secret/plugins/*`; `update` on `auth/token/create/sso-broker`; `update` on `sys/policies/acl/user-*`, `app-*`, `sso-admin` | SSO (`SSO_VAULT_TOKEN`) |
 | `sso-admin` | read/write/list all of `secret/*` | admin UI sessions (minted by the broker) |
 | `proxy` | read `secret/proxy/conf` | proxy (`PROXY_VAULT_TOKEN`) |
 | `jump-host` | read `secret/jump-host/conf` | jump host (`JUMP_VAULT_TOKEN`) |
@@ -150,6 +150,31 @@ const data = await baoConf.get('apps/my-service/conf'); // secret/data/apps/my-s
 await baoConf.set('apps/my-service/conf', { db_password: '...' });
 ```
 
+## Plugin secrets
+
+The SSO Manager's plugin system (configurable plugin instances you create,
+edit, load/unload, and run from the **Plugins** page) stores each instance's
+secrets in its own OpenBao namespace, `secret/plugins/<instance-id>/conf`,
+rather than in the static `sso-secrets.js` `discovery.plugins` block. The
+SSO reads and writes these server-side through the `sso-broker` token (the
+plugin runs in-process as a BullMQ worker, so it needs no token of its own),
+and the admin UI only ever sees masked (`********`) values.
+
+- A **plugin type** is a module under `nodejs/plugins/<category>/<type>.js`
+  exporting a manifest (`configSchema` declares which fields are `secret`).
+- A **plugin instance** is a configured, loadable/unloadable copy of a type,
+  tracked in the `PluginInstance` table; you can have multiple instances of the
+  same type (e.g. two Proxmox endpoints with their own tokens).
+- Non-secret config lives in the DB row; only the `secret:true` field values
+  live in `secret/plugins/<instance-id>/conf`.
+
+Deleting an instance removes both the DB row and its `secret/plugins/<id>/*`
+namespace. Legacy `discovery.plugins` entries in `sso-secrets.js` are migrated
+to instances automatically on the first boot of SSO Manager ≥ v1.17.0 (the
+secret fields are copied into OpenBao at that point). See the SSO Manager
+[plugins docs](https://theta42.github.io/sso-manager-node/plugins.html) for the
+UI/API reference.
+
 ## Operator rotation
 
 If a secret is exposed (or just on a routine schedule), rotate it at the
@@ -192,4 +217,5 @@ re-mint the per-app tokens.
   git-destructive operation you can opt into.
 - **Per-app secrets beyond boot config** (e.g. the proxy's DNS-provider creds,
   the jump host's per-user LDAP SSH keys) moving into OpenBao — only the
-  boot-critical `*-secrets.js` contents moved in this phase.
+  boot-critical `*-secrets.js` contents moved in this phase. (Plugin instance
+  secrets *are* in OpenBao, at `secret/plugins/<id>/conf` — see above.)

setup.sh

@@ -712,12 +712,12 @@ bao_run() { docker exec -e BAO_TOKEN="$VAULT_TOKEN" openbao bao "$@"; }
 # Write an ACL policy from stdin HCL only if it does not already exist.
 ensure_policy() {
 	local name="$1"
-	if bao_run policy read "$name" >/dev/null 2>&1; then
-		info "  policy ${name} already exists — keeping."
-	else
-		info "  writing policy ${name}..."
-		docker exec -i -e BAO_TOKEN="$VAULT_TOKEN" openbao bao policy write "$name" - >/dev/null
-	fi
+	# Always (re)write: `bao policy write` is an idempotent overwrite, so this
+	# applies policy edits on a re-run instead of stranding the old HCL
+	# forever ("already exists — keeping" silently dropped upgrades — e.g.
+	# the secret/metadata mount-root list grant added for the /vault fix).
+	info "  writing policy ${name}..."
+	docker exec -i -e BAO_TOKEN="$VAULT_TOKEN" openbao bao policy write "$name" - >/dev/null
 }
 
 # Read KEY= from ./.env (empty if absent) — reuse a previously minted token
@@ -769,7 +769,9 @@ seed_app_conf() {
 info "Configuring OpenBao policies..."
 # sso-broker — sso's authority to read/write its own conf, mint per-user and
 # per-app tokens (auth/token/create/sso-broker), and create the matching
-# user-<uid> / app-<name> / sso-admin policies.
+# user-<uid> / app-<name> / sso-admin policies. secret/plugins/* holds per-instance
+# plugin secrets managed by the SSO plugin system (configurable plugin copies,
+# loaded/unloaded at runtime — see sso-manager-node docs/plugins.md).
 ensure_policy sso-broker <<'HCL'
 path "secret/data/sso-manager/conf" { capabilities = ["create", "read", "update", "delete", "list"] }
 path "secret/metadata/sso-manager/conf" { capabilities = ["list", "read", "delete"] }
@@ -777,14 +779,20 @@ path "secret/data/users/*" { capabilities = ["create", "read", "update", "delete
 path "secret/metadata/users/*" { capabilities = ["list", "read", "delete"] }
 path "secret/data/apps/*" { capabilities = ["create", "read", "update", "delete", "list"] }
 path "secret/metadata/apps/*" { capabilities = ["list", "read", "delete"] }
+path "secret/data/plugins/*" { capabilities = ["create", "read", "update", "delete", "list"] }
+path "secret/metadata/plugins/*" { capabilities = ["list", "read", "delete"] }
 path "auth/token/create/sso-broker" { capabilities = ["update"] }
 path "sys/policies/acl/user-*" { capabilities = ["create", "read", "update", "delete", "list"] }
 path "sys/policies/acl/app-*" { capabilities = ["create", "read", "update", "delete", "list"] }
 path "sys/policies/acl/sso-admin" { capabilities = ["create", "read", "update", "delete", "list"] }
 HCL
 # sso-admin — admin users in the vault UI: read/write/list everything under secret/.
+# The bare `secret/metadata` grant lets an admin LIST the KV mount root (the
+# top-level dirs); `secret/metadata/*` covers nested paths but not the root
+# itself, so without it the /vault secrets list 403s.
 ensure_policy sso-admin <<'HCL'
 path "secret/data/*" { capabilities = ["create", "read", "update", "delete", "list"] }
+path "secret/metadata" { capabilities = ["list", "read", "delete"] }
 path "secret/metadata/*" { capabilities = ["list", "read", "delete"] }
 HCL
 # proxy / jump-host — read only their own boot conf.