diff --git a/.dockerignore b/.dockerignore index 4bd7ec3..60f96f5 100644 --- a/.dockerignore +++ b/.dockerignore @@ -19,6 +19,9 @@ !API.md !directory_spec.md !docs/**/*.md +# The screenshots the README (served at /docs/overview) links. `COPY docs /docs` +# in Dockerfile.openldap needs these present in the build context. +!docs/images/** # Tests (excluded from production builds; test-runner Dockerfile copies them explicitly) # nodejs/tests/ diff --git a/API.md b/API.md index 9f23cf1..8b3bc1e 100644 --- a/API.md +++ b/API.md @@ -1336,6 +1336,78 @@ All endpoints require authentication and `app_sso_admin` membership. Runtime con **Response:** `{ "success": true }` +--- + +## Subtype Driver Operations Endpoints + +Base path: `/api/directory-admin/resources` + +All endpoints require authentication and `app_sso_admin`, `app_sso_directory_admin`, or `admin` permission. + +### Get Subtype Driver Metrics + +**`GET /api/directory-admin/resources/:id/driver-metrics`** + +Resolves the operational driver for the resource via the 4-tier engine (`theta-agent`, specialized subtype driver, parent hypervisor provider, or unmanaged fallback) and returns real-time telemetry. + +**Response:** +```json +{ + "status": "ok", + "resourceId": "res-id", + "metrics": { + "status": "online", + "driver": "database", + "subType": "redis", + "redis": { "connectedClients": 4, "usedMemoryBytes": 12582912, "opsPerSec": 42 } + } +} +``` + +--- + +### Execute Subtype Driver Action + +**`POST /api/directory-admin/resources/:id/driver-action`** + +Executes a protocol action on the target resource (e.g. systemd restart, Proxmox power control, Redis flush, K8s scale). + +**Request:** +```json +{ + "action": "restart", + "params": { "serviceName": "emby-server" } +} +``` + +**Response:** +```json +{ + "status": "ok", + "resourceId": "res-id", + "result": { "status": "ok", "driver": "docker_socket", "action": "restart" } +} +``` + +--- + +### Get Subtype Driver Logs + +**`GET /api/directory-admin/resources/:id/driver-logs?lines=100`** + +Retrieves recent operational logs for the resource via the resolved driver (`journalctl`, `docker logs`, Proxmox task logs, K8s pod logs). + +**Response:** +```json +{ + "status": "ok", + "resourceId": "res-id", + "logs": "[docker logs --tail 100 emby-server]\nContainer initialized..." +} +``` + +--- + ## Error Responses All endpoints return errors in this format: diff --git a/CHANGELOG.md b/CHANGELOG.md index a512211..55245fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,238 @@ +# v1.32.0 - 2026-08-08 + +### Added +- **Subtype Management & Metrics Drivers Engine.** Built a 4-tier driver resolution engine (`services/driver_registry.js`) binding resource `subType` metadata (`systemd`, `docker`, `proxmox`, `wireguard`, `postgresql`, `redis`, `unifi`, `k8s`) to operational telemetry, log streaming, and remote lifecycle control. +- **Subtype Operations APIs.** Exposed `/api/directory-admin/resources/:id/driver-metrics`, `driver-action`, and `driver-logs` endpoints. +- **Explicit Secret Inheritance Mode.** Enforced strict upward ancestor lineage (`Resource -> Host -> Cluster -> Site`) for secret inheritance, resolving explicit pointers (`INHERIT::`) without exposing sibling directory secrets. +- **Consolidated External App Tokens.** Relocated external OpenBao App Token minting into the **Configuration** page (`/conf` -> External App Tokens tab) and deprecated standalone `/vault` navigation item. +- **Multi-Secret Key Support.** Supported multiple secret keys per resource in OpenBao `secret/data/resources//conf` with per-key merging and deletion. +- **Cross-Platform Agent Packaging.** Built multi-architecture Dockerfile staging and documentation for Linux ARM (arm64, armv7), Windows (amd64, arm64), and macOS (Intel, Apple Silicon). + +### Fixed +- **Ancestry Lineage Querying.** Fixed `Resource.findAllAncestors(id)` memory filtering over `ResourceEdge.list()` to resolve deep ancestor lineage across all graph depths. +- **Dockerfile Module Inclusion.** Included `COPY nodejs/drivers ./drivers` in `Dockerfile.openldap` and `Dockerfile.test-runner` for clean container execution. + +# v1.31.0 - 2026-08-07 + +### Added +- **Resource Secrets Engine & Zero-View Security.** OpenBao KV-v2 encrypted secrets for directory resources (`secret/data/resources//conf`). Zero-View UI & API model — secret values are never returned to admin browsers or UI templates, and delivered exclusively to authenticated `theta-agent` instances. +- **Strict Secret Key Regex Validation.** Secret keys are validated against `^[A-Za-z0-9_]+$` (Standard Environment Variable format, e.g. `DB_PASSWORD`). +- **Field-Populating Password Generator.** Cryptographic secret generator (`window.crypto.getRandomValues`) with length selector dropdown (8–128 chars) populating input fields with security notices. +- **Multi-Level Secret Inheritance.** Dynamic secret resolution across any depth of the resource tree (`Services / Apps -> Hosts / Nodes -> Global Sites`). +- **Non-Blocking UI Confirmations.** Replaced browser blocking dialogs with async `app.messages.confirm()` banners. +- **UI Directory Layout Improvements.** Fixed Directory table resource name and badge order for enhanced readability. + +### Fixed +- **SSSD `sshPublicKey` Mapping.** Included `ldap_user_ssh_public_key = sshPublicKey` in generated agent `sssd.conf` template. + +# Unreleased — LDAP-over-HTTPS API + agent LDAP byte-pump relay + +### Added + +- **`POST /api/v1/ldap/bind` and `POST /api/v1/ldap/search`** — an LDAP-over-HTTPS + API (DESIGN.md §3). A client stops speaking LDAP and instead does an HTTPS call + to the SSO, which performs the real bind/search against its own OpenLDAP. This + kills the hostname / cross-network / LDAPS-cert-chain pain. Caller auth is a + Bearer token: an agent token or a self-service API token (PAT). `/search` is + restricted to agent callers (the SSSD user/group-resolution use case) and runs + under the admin bind — see DESIGN.md §9.5 for the scoped-service-account + follow-up. +- **LDAP byte-pump relay** (`utils/ldap_tunnel.js`) — the SSO relays raw LDAP + bytes from an agent's local socket into its real OpenLDAP and pipes the + response back, over the existing agent WSS channel (`ldap_tunnel` messages). + The SSO does not parse LDAP; it is a transparent socket relay. See DESIGN.md §4. +- **`POST /api/v1/agent/secrets`** — an agent fetches its own node-scoped OpenBao + secrets (DESIGN.md §5). The agent may only read under `secret/data/nodes//*`; + the SSO fetches with its own OpenBao access, so the agent never holds a Vault + token. Agent-token authed (not admin-gated). +- **`iam_apply` command** — the SSO pushes node-scoped IAM config (sudo rules, + SSH keys, access control, revocation) to an agent as a signed high-risk + command (DESIGN.md §6). Added to `HIGH_RISK_COMMANDS`. +- **Agent capabilities in the Directory UI** — the agent reports its enabled + capabilities in its `discovery` frame; the SSO stores them and the host's + Metrics tab renders them as green/gray badges, so an operator can see at a + glance what each agent is allowed to do. +- **`GET /api/agent/join-keys/:id/agents`** — which hosts enrolled through a + given join key. Matches on the trace `Agent.enroll` already leaves in + `description` ("Self-enrolled with join key ``") rather than a stored + relation. +- **Join key management in the Install Agent modal** — a table (label, prefix, + created date, hosts joined, status) alongside the existing mint/select + dropdown, with **Revoke** and **Delete** actions and a click-through to see + which hosts joined via a given key. Previously these were API-only. Revoke + and Delete confirm inline within the row ("Revoke? Yes/No") rather than a + blocking native `confirm()` (freezes the whole tab) or the shared + `app.messages.confirm()` banner (a single `.actionMessage` shared by the + whole card, so a second click before the first resolves leaves a dangling + `$('body').one('click', ...)` handler from the first call and desyncs which + row the banner is actually confirming for). + +# v1.30.2 + +### Fixed + +- **Outbound mail (test email, invites, password resets, OTP-by-email, notifications) could be rejected by the SMTP relay with `554 5.7.1 ... Sender is not same as SMTP authenticate username`.** Many authenticated relays require the `From` address to match the authenticated account or they refuse the send outright. `models/email.js` fell back to a hardcoded `noreply@theta42.com` when `smtp.from` wasn't set, which no relay ever authorized this account to send as. It now falls back to `smtp.user` first — the address the account can actually prove it owns — before the hardcoded placeholder. +- **Catalog page card titles read icon-then-name.** Swapped to name-then-icon so the resource name leads. + +### Docs + +- `docs/configuration.md` didn't mention that OpenBao + the live Configuration UI sit above the four file/env config layers and win the merge — added. +- `docs/plugins.md` listed 3 of 4 discovery plugin types (missing `docker`) and didn't mention the `messaging` plugin category (`twilio`, `webhook`) at all — added both. +- `docs/vault.md` had no navigation (no frontmatter, no back-link, unreachable from the docs index) and described OpenBao as running in dev mode with API access via the root token — both wrong for a real deployment. Fixed navigation and corrected to describe the actual production setup (unsealed OpenBao, server-side scoped-token injection, personal API tokens for programmatic access). +- `docs/discovery.md` was unreachable from the docs index and missing its back-link — both fixed. +- `README.md`'s required-groups list was missing `app_sso_directory_admin` (gates Directory/Plugins/Agent admin). + +# v1.30.1 + +### Fixed + +- **Test Email always failed with `Email.send is not a function`.** `models/email.js` exports `{Mail}`; the handler required the module and called `.send` on it directly. Every other caller destructures it. The button could never have worked. +- **Test SMS failed with `Unexpected token '<', "` into an ``, so moments after a render that selector matched nothing — and the early return skipped setting `hideBelowDepth`, so no row was ever hidden. Collapse state now lives on the caret *button* and is rotated by CSS, and the hide decision is made from the collapsed set alone. Never key behaviour to an element another library is free to replace. +- fix: **the Discovery Plugins delete button did nothing.** It called `deleteDiscoveryPlugin()`, which was never defined — clicking it only threw a `ReferenceError`. +- fix: the plugins pane had no `.actionMessage` element, and `app.messages` confirmations render into one. Without it the returned promise **never settles**, so an awaited confirmation hangs forever and the action it gates silently never happens. Added, along with a note that any pane asking for confirmation needs it. +- feat: **discovery plugin instances can be edited.** Name, schedule, loaded state and configuration, with secrets on their own endpoint and left blank ("unchanged") rather than prefilled with the mask — submitting `********` back would otherwise store the asterisks as the secret. + +### Discovery + +- fix: **a fresh install no longer presents its own containers as things to triage.** The Docker plugin recognises containers belonging to the stack's own compose project, records them as managed, and attaches each to the service it implements. `setup.sh` deploys `sso-manager`, `proxy`, `jump-host`, `openbao` and `bao-renewer`; all five arrived as unmanaged discoveries awaiting promotion. +- fix: **Docker container slugs were derived from the container id**, which changes on every recreate — so each `docker compose up` minted a brand-new resource and orphaned the previous one. Slugs now come from compose project + service, falling back to the container name. +- feat: discovered containers carry `composeProject`, `composeService`, `containerName` and `sourceId`. + +### Docs + +- fix: `/docs/discovery` 404'd — the slug had no entry, though the Discovery tab's help icon linked to it. New `docs/discovery.md` covering the catalog/discovered distinction, how sources are matched and merged, naming precedence, promotion and garbage collection. +- fix: the `agents` slug pointed at `plugins.md`, so `docs/agents.md` was unreachable in the app. + +# v1.29.0 + +**Breaking:** theta-agent enrollment is now mandatory. Agents installed before this release carry a browser-generated token the server never recorded and will be rejected until re-enrolled. Requires theta-suite ≥ v1.42.0 (the `sso-broker` OpenBao policy must grant `secret/agent/*`); re-run `./setup.sh`. + +### Security — theta-agent channel + +- **sec: `/api/agent/ws` accepted any token.** There was no agent registry, so the endpoint authenticated nothing: any client that could reach the SSO could register as a node, publish discovery/telemetry into the admin view, and receive commands — including a signed `arbitrary_bash` — addressed to a token it guessed. Tokens were generated in the *browser* (`generateRandomHexToken`) and never recorded server-side, so there was nothing to validate against and no way to revoke one. Agents are now rows in a new `Agent` table, authenticated by SHA-256 token hash before the connection is registered or the welcome payload is sent; unknown or revoked tokens are closed with `4001` and audited. +- **sec: the command signing key was ephemeral.** `AgentManager` generated an Ed25519 pair in its constructor, so it changed on every process start and the `public_key` an agent pinned in `agent.yml` stopped matching immediately. The key now lives in OpenBao at `secret/agent/signing-key` and survives restarts. If it cannot be loaded the SSO **refuses** to send high-risk commands rather than signing with a key no agent has seen (`signingAvailable: false` on `GET /api/agent/nodes`). +- **sec: commands are addressed by agent id, not token.** A credential has no business in a URL, an access log or browser history. +- **sec: agent actions are audited.** Enroll, update, rotate, revoke, delete, every command (with `signed`), and every rejected connection are emitted as structured `"component":"agent"` log records carrying the acting user. + +### theta-agent — enrollment & resource binding + +- feat: `POST /api/agent/enroll` mints the token server-side and returns it **once**; only its SHA-256 is stored. Plus `PUT /nodes/:id` (rename/rebind), `POST /nodes/:id/rotate`, `POST /nodes/:id/revoke`, `DELETE /nodes/:id`. Rotate, revoke and delete drop the live socket immediately (`4004`/`4003`) instead of waiting for a reconnect. +- feat: an agent binds to a **host resource** (`resourceId`). The Directory reads that link instead of guessing by hostname — the old `agentsByHost[name]` match silently failed whenever a Directory name differed from the machine's hostname, and aliased two hosts that shared one. +- feat: **agent discovery reaches the Directory.** A bound agent's facts (`os`, `kernel`, `cpu`, `ram_total_gb`, `disk_total_gb`, `ip`) are written onto its host resource, tagged `discovery_sources: ["theta-agent"]` with an `agentId` back-reference. An unbound agent goes through the normal reconciler. Previously `handleDiscovery` wrote to an in-memory record and updated nothing — the one source actually running *on* the host contributed nothing to the directory. +- feat: agent state is persisted, so an agent that is installed but **offline** is now distinguishable from one that never existed; enrollments survive a restart. The Directory status dot reflects this: red means "enrolled and not connected" (a fault), grey means no agent enrolled / revoked / service unreachable. Red previously covered both, making an ordinary directory of hosts look like an outage. +- feat: the Install Agent modal enrolls first and builds the install command from the result, including `--public-key`. `public_key` was never emitted into the generated `agent.yml` before, so no installed agent could verify anything. +- fix: `registerAgent` is synchronous. Awaiting a database write before attaching the WebSocket `message` listener lost every agent's first `discovery` frame, which it sends the instant the socket opens (`ws` drops events emitted with no listener attached). + +### Directory + +- feat: **the resource tree is collapsible.** Any row with children has a caret; the toolbar collapses/expands everything. State persists per browser, so the shape survives the self-heal reload that follows most edits. An active search overrides collapse so matches inside a folded subtree are never hidden. +- fix: **the Proxmox plugin mismatched MAC addresses to IPs.** It collected MACs and IPs into two flat lists and zipped them by index, so on any multi-NIC guest — or any guest where one NIC had no address — the directory recorded an address against the wrong MAC. NICs are now keyed by MAC, so a pairing can only come from the source that observed both together. +- feat: Proxmox discovery emits an **endpoint resource** (named from `/cluster/status`) with every node parented beneath it, so one endpoint is one subtree instead of several orphan roots. It deliberately carries no IP: giving it the address it is reached at made the reconciler merge it with the node answering on that address, producing a resource that was its own parent. +- feat: discovered guests carry `sourceId` (`/qemu/`), `node`, `vmid` and `macAddress`, so a row traces back to the exact guest on the exact hypervisor. Against a live 3-node cluster this took MAC coverage to 53/54 resources and `sourceId` to 54/54. +- fix: Proxmox interfaces belonging to something running *inside* a guest (`docker0`, `veth*`, `br-*`, VPN tunnels) are filtered out — one Home Assistant VM reported 16 of them alongside its single real NIC, and their 172.x addresses gave the reconciler spurious matches. +- fix: a stopped VM still reports its MAC (read from the VM config), a DHCP-configured LXC gets its address from the running container's interface list, and Proxmox **nodes** report their own IP/MAC (recovered from `enx` predictable names, since `/nodes/*/network` carries no `hwaddr`). Offline nodes are recorded with `status` instead of skipped, so a hypervisor that is down no longer looks decommissioned and get garbage-collected after a week. +- fix: **the reconciler could make a resource its own parent.** Two slugs in one payload can resolve to the same row once merged; the resulting self-edge renders as an infinitely nested tree and defeats every ancestor walk in the app. Self-edges and cycle-closing edges are now refused and logged. +- fix: **hosts were named after their MAC address.** `bestName` preferred the *longer* name, so UniFi's `ac:16:2d:b3:da:80` (17 chars) beat Proxmox's real hostname `dl380-0` (7). Names are now ranked (hostname > IP > MAC) with length only as a tie-break within a rank. +- fix: `isIp` never matched anything — `\\.` inside a regex literal matches a backslash, not a dot — so an IP-shaped placeholder name was never replaced by a real hostname a later source discovered. +- fix: a discovered device can only merge into a resource of the same kind. A VM named `gitea-runner` could match a hand-created *service* of the same name on the name rule and overwrite it. +- perf: the reconciler reads the inventory once per run instead of once per incoming resource — a ~55-resource Proxmox payload against a similar-sized inventory was doing quadratic full-table reads every run. +- fix: the Discovered Inventory table showed "Unknown IP" for almost everything, because it read `metadata.ip` while any source that enumerates interfaces stores addresses per-NIC. It now falls back to the first NIC address, and shows `vmid`, slug, `sourceId` and per-interface MAC/name. + +### Profile + +- fix: the API Tokens card is no longer wider than every other card on the site — the section sat outside the page's `.container`. + +### Build & docs + +- fix: `Dockerfile.test-runner` never copied `nodejs/plugins`, so every plugin test suite failed in CI as "Cannot find module" and plugin code was effectively untested. Suite count goes 27 → 29. +- docs: `docs/agents.md` rewritten for enrollment, the close-code table, resource binding, the persistent signing key, and a corrected `public_key` example (the documented `MCowBQYDK2VwAyEA...` was an SPKI PEM body — 44 bytes decoded — where the agent requires the raw 32). +- docs: `docs/directory.md` covers the collapsible tree and the corrected seed hierarchy; `docs/plugins.md` documents what the Proxmox plugin produces and why the endpoint has no IP. + +# v1.28.0 +- fix: `/api/agent/nodes` no longer 404s — the previous "unconditional mount" was still inside the post-listen `onListen` hook, so the REST router landed *behind* app.js's terminal 404 catch-all and every `/api/agent/*` request 404'd. The router is now mounted synchronously in `app.js` before the 404 handler; only the agent WebSocket setup runs on `onListen`. +- feat: promoting a discovered inventory resource now opens the resource form pre-filled with the discovered data (name, kind, IP, subtype, …) for review; the modal's Save confirms the promote (creates the LDAP groups + marks it managed) instead of silently promoting. +- fix: Directory table no longer goes stale after add/remove edge — `addEdge`/`removeEdge` called an undefined `loadData()`, which threw and left the host/parent linkage stale until a manual refresh; they now call `loadResources()`. `addGroup`/`removeGroup` also refresh so the Access column stays accurate. +- feat: Vault page states it's powered by OpenBao (header badge linking to openbao.org). + +# v1.27.0 +- fix: Directory group names now match `docs/GROUPS.md` exactly — per-resource groups are `{site}_{kind}_{name}_{level}` (`site_local_host_theta-env_access`, `site_local_app_sso-manager_access`), with the kind always present and the resource name slug stripped of its kind prefix. Services map to the `app` kind. The access-request + resolver tests were updated to the documented convention. +- fix: a site resource now carries only `god_admin` + the site-wide groups (`{site}_super_admin`, `{site}_everyone`); the kind-scoped aggregates are still created for nesting but are no longer surfaced on the site's modal. +- fix: groups no longer appear 3× under a resource — the Directory self-heal (which runs on every load) was creating duplicate `ResourceGroup` links; linking is now idempotent (check-then-create). +- fix: `/api/agent/nodes` no longer 404s — the agent REST router is mounted unconditionally instead of being gated on the WebSocket server being up. +- fix: `POST /api/shared-secrets/` rejected valid slugs — the slug regex now allows underscores (was hyphens-only). +- fix: `GET /api/shared-secrets/` crashed with `s.path is not a function` — the list spread dropped the instance's `path()` method; now uses the static `SharedSecret.pathFor`. +- fix: promoting a discovered inventory resource crashed with `Resource.update is not a function` — `update` is an instance method; the promote handler now loads an instance and calls `update()` on it. +- feat: Vault → Apps tab now lists minted app tokens (the "Minted apps" list) — each is a scoped OpenBao credential for an external service; sso renews them and the list shows renewal state, so a minted credential no longer vanishes after its once-only token display. New `GET /api/vault/apps`. +- feat: Vault page documents itself — a `/docs/vault` help icon in the header, and the doc now covers the Apps + Shared tabs. +- feat: discovery plugin cards show last-run time + status (ok/error) and a Logs button that opens the captured run log. + +# v1.26.1 +- fix: the legacy `app_super_admin` group is gone — `SUPER_ADMIN_GROUP` (nested into every resource's `_admin` group by auto-provisioning) is now `god_admin`, and `docker-entrypoint.sh` no longer seeds or nests `app_super_admin` (god_admin is nested into the `app_sso_*` groups directly). `isSuperAdmin` still recognizes a pre-existing `app_super_admin` as a migration alias, so an old deployment isn't stripped of rights until it's rebuilt. + +# v1.26.0 +- feat: complete the group model (docs/GROUPS.md) — `god_admin` is now seeded into LDAP and nested into `app_super_admin`; every site auto-provisions `{site}_super_admin`, `{site}_hosts_*`/`{site}_apps_*` aggregates and `{site}_everyone`; per-resource `_admin`/`_access` groups (named `{site}_{slug}_{level}`, the kind carried in the resource slug) are nested into the site aggregates so the inheritance lattice exists in LDAP, not just in the resolver. Site/aggregate groups are self-healed idempotently on every Directory load, so a directory seeded by an older release picks them up without a rebuild. +- feat: the naming convention is now enforced server-side — `POST /api/directory-admin/groups` rejects a group CN that isn't a valid group for the target resource (its own `_admin`/`_access`/capability, a site aggregate, a site-level group, or `god_admin`), so the free-text field can no longer mint `*_accessmember`-style names +- feat: `god_admin` is managed from the Directory — the site resource modal surfaces `god_admin` + the site-level groups as associated groups, so its members (and the site's) are editable right there +- fix: Directory agent status dots no longer paint every host red when the `/api/agent/nodes` endpoint is unreachable (older app or transient outage) — they now show a neutral grey "agent service unreachable" instead of a false alarm +- fix: Profile + API Tokens cards are both full-width on the profile page (the API card was a narrower centered block) +- fix: in-app `/docs/` pages returned 500 — `Dockerfile.openldap` never copied the `docs/` tree into the image (only the root README/CHANGELOG/API/directory_spec), so every page but those few hit a missing-file error; the whole `docs/` dir now ships, and doc images are served at `/docs/images` +- test: group resolver tests now cover the prefixed site-slug convention (`site_local_...` is kept verbatim, not re-slugified to `site-local`) + +# v1.25.0 +- feat: hierarchical group & permission model (docs/GROUPS.md) — god_admin, {site}_super_admin, {site}_hosts_*/{site}_apps_* aggregates, and per-resource {site}_host__admin/access/; inheritance resolver (admin implies access, capabilities explicit), meta everyone/{site}_everyone groups +- feat: remove the standalone Groups page — group management is tied to adopted Directory resources (help link to the model in the Directory toolbar) +- feat: console admin recognizes god_admin and site-scoped super/app-admin groups (legacy app_sso_admin/app_super_admin kept as migration aliases) + +# v1.24.0 +- feat: Agents merged into the Directory — removed the standalone Agents page. Host rows show a green/yellow/red theta-agent status dot (healthy / high-load / not connected) and the resource modal gained a Metrics tab with live telemetry + discovery +- feat: Discovery Plugins New-plugin modal — slug is now derived from the name (field removed), the cron field is a dropdown (hourly/daily/weekly + custom), and per-plugin settings are collected from the configSchema (e.g. Proxmox url/tokenId/tokenSecret) instead of an empty config +- feat: Directory resource slug is now read-only and derived from the name +- feat: Vault page restyled to match the rest of the site (bounded container, card + nav-tabs header, h4) +- feat: navbar — the username is no longer underlined; only the active nav link is bold + underlined + +# v1.23.0 +- fix: /api/vault proxy never injected X-Vault-Token — the true root cause of the recurring vault 403 "permission denied". The proxy declared its hook with http-proxy-middleware v3 syntax (`on: { proxyReq }`), which the installed HPM v2 silently ignores, so every request reached OpenBao unauthenticated (and the client's sso auth headers were never stripped). Rewritten as v2 `onProxyReq`. +- fix: vault proxy header injection ordered before `fixRequestBody` — the body write flushes headers, so setting X-Vault-Token after it silently failed on every POST/PUT (writes would still 403 even with the hook fixed) +- fix: initORM add-only schema heal — `sequelize.sync()` never ALTERs existing tables, so columns added by newer releases (e.g. `PluginInstance.lastLog`, which crashed the scheduler on every boot of an upgraded deployment) are now detected via describeTable and added with addColumn (additive only, per-column fail-soft) +- feat: external-app vault tokens are long-lived and auto-renewed — minted via the new `sso-app` token role (periodic 768h, falls back to sso-broker's 24h role until theta-suite setup.sh is re-run); sso stores each token's accessor (new VaultAppToken model — an accessor can renew/revoke but not authenticate) and renews all of them at boot + every 6h via auth/token/renew-accessor, so a downstream app's credential stays valid as long as sso runs with zero renewal code in the app +- feat: re-minting an app token revokes the app's previous token via its stored accessor — exactly one live credential per app, no zombies +- test: wire-level tests for the vault proxy (real HTTP round-trip asserting token injection, auth-header stripping, path rewrite, and POST body integrity) + app-token accessor lifecycle tests + +# v1.22.0 +- feat: Agents page — live list of connected theta-agent hosts with telemetry (CPU/RAM/disk/ZFS/GPU) + online status, updating via socket.io +- security: auth + admin-gate the /api/agent REST routes (previously unauthenticated) + +# v1.21.0 +- fix: always reconcile OpenBao policy content before serving a (possibly cached) token, so stale stored policies can no longer cause a recurring vault 403 "permission denied" +- feat: shared secrets — users can publish secrets to secret/shared// and grant read access to other users and downstream apps (OpenBao ACL policy edits, applied live) +- feat: shared-secrets API + Shared tab in the vault UI + +# v1.20.0 +- fix: OpenBao 403 on vault secrets list (directory list grants + policy self-heal) + ## v1.19.0 - Added WebSocket endpoint for theta-agent C2 diff --git a/Dockerfile.openldap b/Dockerfile.openldap index 5e8d2a7..f163723 100644 --- a/Dockerfile.openldap +++ b/Dockerfile.openldap @@ -163,6 +163,7 @@ COPY nodejs/app.js ./ COPY nodejs/bin ./bin COPY nodejs/conf ./conf COPY nodejs/controller ./controller +COPY nodejs/drivers ./drivers COPY nodejs/middleware ./middleware COPY nodejs/models ./models COPY nodejs/routes ./routes @@ -184,6 +185,11 @@ COPY README.md /README.md COPY CHANGELOG.md /CHANGELOG.md COPY API.md /API.md COPY directory_spec.md /directory_spec.md +# The docs/*.md tree (plus the images the docs link) is read at runtime too, so +# the whole docs/ dir must land at /docs. Without this every in-app /docs/ +# page other than the root-level README/CHANGELOG/API/directory_spec 500s on the +# fs.readFileSync in routes/docs.js (files missing from the image). +COPY docs /docs # Baked commit hash from the gitinfo stage (see build_info.js). COPY --from=gitinfo /commit.txt ./.build_commit diff --git a/Dockerfile.test-runner b/Dockerfile.test-runner index 32fd684..0f0d142 100644 --- a/Dockerfile.test-runner +++ b/Dockerfile.test-runner @@ -20,8 +20,13 @@ COPY nodejs/app.js ./ COPY nodejs/bin ./bin COPY nodejs/conf ./conf COPY nodejs/controller ./controller +COPY nodejs/drivers ./drivers COPY nodejs/middleware ./middleware COPY nodejs/models ./models +# Without this the discovery/plugin suites cannot even load their subject and +# fail as "Cannot find module ../plugins/discovery/..." -- plugin code was +# effectively untested in CI. +COPY nodejs/plugins ./plugins COPY nodejs/routes ./routes COPY nodejs/services ./services COPY nodejs/utils ./utils @@ -43,6 +48,8 @@ COPY directory_spec.md /directory_spec.md COPY test_seed.js ./test_seed.js COPY test/seed-test-user.sh /usr/local/bin/seed-test-user RUN chmod +x /usr/local/bin/seed-test-user +# End-to-end LDAP tunnel test client (docker-compose.e2e.yml) +COPY test/tunnel_e2e.js ./test/tunnel_e2e.js # Default command: seed the test user, then run the test suite CMD ["sh", "-c", "seed-test-user && npm test"] diff --git a/README.md b/README.md index 1e7366e..5aff1a7 100755 --- a/README.md +++ b/README.md @@ -47,8 +47,9 @@ phone-home, no hosted control plane, and no per-user pricing. same directory, so you don't maintain a second user database for them. - **Personal access tokens** — any user can mint a long-lived bearer token to drive the management API from scripts or CI, scoped to their own permissions. -- **All-in-one Docker image** — app + OpenLDAP + Redis in one container, or run - the pieces separately against your own LDAP/Redis via `app_*` env config. +- **Directory & Inventory Graph** — full host/service/site graph with resource metadata, automatic LDAP group provisioning (`_access` / `_admin`), and Access Request workflows. +- **Subtype Management & Metrics Drivers Engine** — 4-tier resolution engine binding `subType` metadata (`systemd`, `docker`, `proxmox`, `wireguard`, `postgresql`, `redis`, `k8s`) to operational telemetry, log streaming, and remote lifecycle control. +- **Explicit Secret Inheritance Mode** — OpenBao KV-v2 integration with strict upward ancestor lineage (`Resource -> Host -> Cluster -> Site`), preserving precise secret scoping across services and containers. - **Multi-Site Support (Geo-Location Scaling)** — built-in support for N-Way Multi-Master OpenLDAP replication across physical sites for HA and low latency. ## Why this over the alternatives @@ -200,7 +201,8 @@ If you are pointing the app at your own existing LDAP server, see `pw-sha2`, `ppolicy`, `memberof`, and `refint` modules plus a small custom schema. The bundled Docker image and `install.sh` set all of that up for you. Required groups: `app_sso_admin` (full admin), `app_sso_oauth_admin` (manage -OAuth clients only), `app_sso_invite` (invitation management) — see +OAuth clients only), `app_sso_invite` (invitation management), +`app_sso_directory_admin` (Directory/Plugins/Agent admin) — see DEPLOYMENT.md for the full setup. ## Development diff --git a/directory_spec.md b/directory_spec.md index 9d4ca1f..e9fc1d4 100644 --- a/directory_spec.md +++ b/directory_spec.md @@ -367,3 +367,35 @@ Ordered by how much they unblock: conventions, not schema changes; the json column already holds them. 5. **`updated_on` in graph output / graph etag** (blocks drift/DNS freshness; trivial once surfaced). + +--- + +## 10. Subtype Management & Metrics Drivers Architecture + +The Directory incorporates a **4-tier Driver Resolution Engine** (`services/driver_registry.js`) that binds resource `subType` metadata to specific telemetry, log streaming, and operational management protocols. + +### Subtype Matrix & Drivers + +| Subtype Category | Supported Subtypes | Primary Driver | Management Capabilities | Telemetry & Metrics | +| :--- | :--- | :--- | :--- | :--- | +| **Service Managers** | `systemd`, `openrc`, `windows_service` | `ThetaAgentDriver` / Systemd | `start`, `stop`, `restart`, `reload` | CPU, Memory, Active PID, SubState | +| **Containers & Stacks** | `docker`, `docker_compose` | `DockerSocketDriver` / Agent | `start`, `stop`, `restart`, `pause` | CPU %, Memory Limit/Usage, Net/Block I/O | +| **Virtualization & Hypervisors** | `proxmox`, `lxc`, `kvm`, `esxi`, `libvirt_kvm`, `vps_generic` | `ProxmoxDriver` / Hypervisor | `start`, `stop`, `shutdown`, `reboot` | Guest VMID CPU/RAM/Disk, Parent Hypervisor status | +| **Networking & Appliances** | `wireguard`, `unifi_ap`, `unifi_switch`, `pfsense` | `NetworkDriver` | `restart`, `locate`, `sync` | Connected Clients, Handshakes, Gateway RTT, Channels | +| **Databases & Vaults** | `postgresql`, `redis`, `openbao_vault` | `DbDriver` | `flush`, `seal`, `unseal` | DB Size, Connections, Hit Rates, Active Leases | +| **Orchestration** | `k8s_pod`, `k8s_deployment` | `K8sDriver` | `scale`, `restart`, `rollout_restart` | Desired/Ready Replicas, Pod Phase, IP | +| **Workstations** | `desktop_linux`, `desktop_windows` | `ThetaAgentDriver` | `reboot`, `shutdown`, Display Manager | CPU, Memory, GPU, Active Sessions | + +*(Note: Reverse Proxy subtypes like Nginx/HAProxy/Caddy/Traefik are excluded per environment configuration).* + +### 4-Tier Driver Resolution Engine +1. **Direct Agent Execution**: If `theta-agent` is connected directly to the target resource. +2. **Subtype-Specific Driver**: Executes specialized protocol driver (e.g. Proxmox API, Docker Engine API, DB Driver). +3. **Ancestor / Hypervisor Fallback**: If an LXC/KVM guest lacks a direct agent, queries its parent Proxmox hypervisor node for metrics and power controls. +4. **Unmanaged Fallback**: Reports unmanaged status cleanly without breaking UI/API contracts. + +### Subtype Operations Endpoints +- `GET /api/directory-admin/resources/:id/driver-metrics` — Real-time telemetry payload +- `POST /api/directory-admin/resources/:id/driver-action` — Execute management action (`{ action, params }`) +- `GET /api/directory-admin/resources/:id/driver-logs` — Tail log output (`?lines=100`) + diff --git a/docker-compose.e2e.yml b/docker-compose.e2e.yml new file mode 100644 index 0000000..a6f694e --- /dev/null +++ b/docker-compose.e2e.yml @@ -0,0 +1,93 @@ +# End-to-end test of the LDAP byte-pump tunnel (DESIGN.md §4). +# +# Spins up OpenLDAP + Redis, a real SSO server (bin/www, so the WSS relay is +# live), and a client that simulates the agent: it enrolls one, connects over +# WSS, sends a real LDAP bind as raw bytes, and verifies the SSO relays it into +# OpenLDAP and pipes the response back. +# +# docker compose -f docker-compose.e2e.yml up --build --abort-on-container-exit +# # exit code 0 = tunnel works; the client prints E2E PASS. + +services: + ldap: + build: + context: . + dockerfile: Dockerfile.openldap + environment: + - LDAP_BASE_DN=dc=test,dc=local + - LDAP_ADMIN_PASS=secret + - ORG_NAME=Test SSO + command: ["sleep", "infinity"] + healthcheck: + test: ["CMD-SHELL", "ldapsearch -x -H ldap://localhost:389 -b '' -s base '(objectClass=*)' >/dev/null 2>&1"] + interval: 2s + timeout: 3s + retries: 20 + start_period: 5s + volumes: + - ldap-data:/var/lib/ldap + - ldap-certs:/etc/openldap/certs + + redis: + image: redis:7-alpine + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 2s + timeout: 3s + retries: 15 + + sso: + build: + context: . + dockerfile: Dockerfile.test-runner + command: ["node", "bin/www"] + environment: + - NODE_ENV=test + - NODE_PORT=3001 + # Test OpenBao (theta-test-bao) — sso-broker token so the SSO can sign + # high-risk agent commands and read node-scoped secrets. + - VAULT_ADDR=http://theta-test-bao:8200 + - VAULT_TOKEN=${VAULT_TOKEN:-} + - app_ldap__url=ldap://ldap:389 + - app_ldap__bindDN=cn=admin,dc=test,dc=local + - app_ldap__bindPassword=secret + - app_ldap__userBase=ou=people,dc=test,dc=local + - app_ldap__groupBase=ou=groups,dc=test,dc=local + - app_redis__redisConf__url=redis://redis:6379 + - REDIS_URL=redis://redis:6379 + - app_oauth__jwtSecret=test-jwt-secret-for-testing-only + - app_name=Test SSO + depends_on: + ldap: + condition: service_healthy + redis: + condition: service_healthy + + client: + build: + context: . + dockerfile: Dockerfile.test-runner + command: ["sh", "-c", "seed-test-user && node test/tunnel_e2e.js"] + environment: + - NODE_ENV=test + - SSO_URL=http://sso:3001 + - app_ldap__url=ldap://ldap:389 + - app_ldap__bindDN=cn=admin,dc=test,dc=local + - app_ldap__bindPassword=secret + - app_ldap__userBase=ou=people,dc=test,dc=local + - app_ldap__groupBase=ou=groups,dc=test,dc=local + - app_redis__redisConf__url=redis://redis:6379 + - REDIS_URL=redis://redis:6379 + - app_oauth__jwtSecret=test-jwt-secret-for-testing-only + - app_name=Test SSO + depends_on: + sso: + condition: service_started + ldap: + condition: service_healthy + redis: + condition: service_healthy + +volumes: + ldap-data: + ldap-certs: diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index c17a98f..f44fd1c 100755 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -355,7 +355,11 @@ EOF # Required SSO groups. The app gates admin/invite/oauth-admin on these; # app_sso_service_account is a marker (not a permission gate) for # non-person accounts -- see the Users page. - for group in app_super_admin app_sso_admin app_sso_invite app_sso_oauth_admin app_sso_service_account; do + # + # god_admin is the global super group (docs/GROUPS.md §2), the top of the + # group-inheritance lattice. It is seeded here so it exists from first boot; + # the theta-suite bootstrap puts the first admin person into it. + for group in god_admin app_sso_admin app_sso_invite app_sso_oauth_admin app_sso_service_account; do ldapadd -x -D "$LDAP_BIND_DN" -w "$LDAP_ADMIN_PASS" -H ldap://localhost:389 << EOF || true dn: cn=${group},ou=groups,${LDAP_BASE_DN} objectClass: groupOfNames @@ -366,11 +370,11 @@ member: ${LDAP_BIND_DN} EOF done - # Nest app_super_admin into the SSO admin groups, so cross-app super admins - # hold those rights by membership rather than by a special case in app code. - # This is what makes the privilege visible to every consumer -- SSSD, sudo, - # anything binding LDAP directly -- instead of only to callers that happen - # to route through utils/permission.js. + # Nest god_admin into the SSO admin groups, so god admins hold those rights + # by membership rather than by a special case in app code. This is what makes + # the privilege visible to every consumer -- SSSD, sudo, anything binding + # LDAP directly -- instead of only to callers that happen to route through + # utils/permission.js. # # app_sso_service_account is deliberately excluded: it is a marker for # non-person accounts, not a permission, and nesting admins into it would @@ -381,10 +385,10 @@ EOF dn: cn=${group},ou=groups,${LDAP_BASE_DN} changetype: modify add: member -member: cn=app_super_admin,ou=groups,${LDAP_BASE_DN} +member: cn=god_admin,ou=groups,${LDAP_BASE_DN} EOF done - info "Nested app_super_admin into the SSO admin groups" + info "Nested god_admin into the SSO admin groups" fi info "LDAP directory initialized" diff --git a/docs/agents.md b/docs/agents.md index d1f020a..5e96507 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -6,7 +6,137 @@ nav_order: 5 # Theta Agent & Endpoint Management -The **Theta Agent** (`theta-agent`) is a unified, 2-way Command & Control (C2) endpoint management daemon written in Go for Linux hosts across your home lab, infrastructure, or data center. It connects outbound via a long-lived WebSocket connection to the central **SSO Manager** (`wss:///api/agent/ws`), enabling real-time host telemetry, automated host discovery, and local-first administrative management. +The **Theta Agent** (`theta-agent`) is a unified, 2-way Command & Control (C2) endpoint management daemon written in Go with native cross-platform binaries for **Linux (x86_64, ARM64, ARMv7)**, **Windows (x86_64, ARM64)**, and **macOS (Intel, Apple Silicon M1/M2/M3/M4)**. It connects outbound via a long-lived WebSocket connection to the central **SSO Manager** (`wss:///api/agent/ws`), enabling real-time host telemetry, automated host discovery, and local-first administrative management. + +--- + +## Supported Architectures & Operating Systems + +The agent is compiled for 7 target platform binaries with zero external runtime dependencies: + +| Operating System | Architecture | Binary Name | Typical Target Devices | +| :--- | :--- | :--- | :--- | +| **Linux** | `amd64` (x86_64) | `theta-agent-linux-amd64` | Intel/AMD Servers, Cloud VMs, Proxmox Hypervisors | +| **Linux** | `arm64` (aarch64) | `theta-agent-linux-arm64` | Raspberry Pi 4/5, Graviton, Ampere Altra | +| **Linux** | `armv7` (32-bit ARM) | `theta-agent-linux-armv7` | Raspberry Pi 2/3/Zero 2W, ARM IoT Gateways | +| **Windows** | `amd64` (x86_64) | `theta-agent-windows-amd64.exe` | Windows Server, Windows 10/11 Desktop | +| **Windows** | `arm64` | `theta-agent-windows-arm64.exe` | Windows on ARM, Surface Pro | +| **macOS** | `amd64` | `theta-agent-darwin-amd64` | Intel Macs | +| **macOS** | `arm64` | `theta-agent-darwin-arm64` | Apple Silicon Macs (M1/M2/M3/M4) | + +The `install.sh` script automatically detects `uname -s` and `uname -m` to download the exact binary for the host. + +--- + +## Enrollment + +An agent is only real if the SSO issued its credential. **Tokens the server did +not issue are rejected** at the WebSocket handshake. + +There are two ways to get a host enrolled, and the first is the normal one. + +### Join key — install the agent and the host appears + +Hand the machine a **join key** and nothing else. On first connect the SSO +enrolls the host, issues it its own per-agent token plus the public key it must +pin, and the agent **writes both into its own `agent.yml`** and blanks the join +key. From then on it authenticates as itself. + +```bash +curl -fsSL https:///resources/theta-agent/install.sh | sh -s -- \ + --url "https://" --join-key "tjk_..." +``` + +That is the whole procedure — no pre-registering the machine, no copying a +public key by hand. `setup.sh` mints a key and configures the stack's own host +this way automatically. + +The join key is a *bootstrap* credential, not the host's identity. That +distinction is what keeps one key convenient without making it a fleet-wide +skeleton key: every host still ends up individually revocable, and a compromised +host does not yield a credential that works anywhere else. + +| Endpoint | Purpose | +| :--- | :--- | +| `GET /api/agent/join-keys` | List keys (prefix + usage only; never the key) | +| `POST /api/agent/join-keys` | Mint one — returned **once** | +| `POST /api/agent/join-keys/:id/revoke` | Stop it enrolling new hosts | +| `DELETE /api/agent/join-keys/:id` | Remove it | +| `GET /api/agent/join-keys/:id/agents` | Which hosts enrolled through this key | + +Revoking a join key does **not** disconnect hosts that already joined; they hold +their own tokens by then. Revoke the agent itself to cut a specific host off. + +**Reuse.** Yes — a join key is not consumed on use. `AgentJoinKey.authenticate` +only checks `revoked` and `expires_on`; it never invalidates the key itself. +Every use increments `use_count` and stamps `last_used_on`, but the key keeps +working until you revoke or delete it (or it expires) — "one key works for as +many hosts as you like" above is literal, not a figure of speech. + +**UI.** The **Install Agent** modal (Directory → Install Agent → Join key tab) +has a **Manage join keys** table below the mint/select dropdown: label, prefix, +created date, hosts joined, status, and **Revoke**/**Delete** actions per key. +Clicking a key's "N hosts" link expands the list of hosts that joined through +it (name, online status, joined date, last seen). + +**Audit.** Yes, both halves are logged as structured `"component":"agent"` +lines, and the hosts-joined list in the UI above is queryable directly: +- Minting: `action: "join_key_issued"` records the acting admin (`actor`), + `label`, and `keyPrefix`. +- Each enrollment through that key: `action: "join"` records `agentId`, + `agentName`, `remoteAddr`, `joinKeyLabel`, and `joinKeyPrefix`. +- `GET /api/agent/join-keys/:id/agents` returns the same "which hosts did key + X add" answer the UI shows — it matches on the trace `Agent.enroll` leaves in + each agent's `description` ("Self-enrolled with join key ``") rather + than a stored foreign key, since a join key is exchanged for a per-agent + token immediately and from then on the agent's own identity is what matters. + +### Pre-registering a host + +When you want the agent bound to a specific Directory host up front, enroll it +from **Directory → Install Agent**: + +1. Give the agent a name and **bind it to a host resource**. The binding is what + links telemetry, status and commands to a Directory entry. +2. Press **Enroll & issue token**. The SSO mints a 256-bit token, stores only its + SHA-256, and shows the raw value **once**. +3. Copy the generated install command — it already carries the token and the + server's public key. + +A host that self-enrolls with a join key arrives unbound; bind it afterwards with +`PUT /api/agent/nodes/:id` or from the Directory. + +Or via the API: + +```bash +curl -X POST https:///api/agent/enroll \ + -H "Authorization: Bearer " \ + -H 'Content-Type: application/json' \ + -d '{"name": "web01", "resourceId": ""}' +``` + +The response contains `token` (once only) and `publicKey`. + +| Endpoint | Purpose | +| :--- | :--- | +| `GET /api/agent/nodes` | Every enrolled agent, connected or not, plus the server public key | +| `POST /api/agent/enroll` | Mint an agent + token | +| `PUT /api/agent/nodes/:id` | Rename, or bind/unbind the host resource | +| `POST /api/agent/nodes/:id/rotate` | Issue a new token; the old one stops working immediately | +| `POST /api/agent/nodes/:id/revoke` | Disable the enrollment | +| `DELETE /api/agent/nodes/:id` | Remove the enrollment | +| `POST /api/agent/nodes/:id/command` | Send a command (signed automatically when high-risk) | + +Revoke, rotate and delete **drop any live connection immediately** — they do not +wait for the agent to reconnect. Commands are addressed by agent **id**, never by +token: a token is a credential and has no business in a URL or a log. + +Enrollment, revocation, rotation, every command, and every rejected connection +are written to the application log as structured `"component":"agent"` records +with the acting user. + +> **Lost the token?** It cannot be recovered — only its hash is stored. Rotate +> the agent to issue a new one. --- @@ -31,6 +161,44 @@ Every 30 seconds, the agent streams real-time performance metrics: --- +## Viewing in the SSO Manager + +Agent status and telemetry live on the **Directory** page — there is no separate +Agents page. For each **host** resource that has a connected theta-agent, the +Directory shows a status dot in the row: + +| Color | Meaning | +| :--- | :--- | +| **Green** | Connected, healthy (CPU/RAM/disk within limits). | +| **Yellow** | Connected but under high load (CPU > 80% or RAM > 80% or disk > 90%). | +| **Red** | **Enrolled but not connected.** The agent exists and is expected — this is a fault. | +| **Grey** | No agent enrolled for this host, the enrollment is revoked, or the agent service is unreachable. | + +Red and grey used to be the same colour, which made an ordinary directory of +hosts look like an outage. Because the enrollment now outlives the connection, +"installed but down" is distinguishable from "never had an agent". + +Opening a host's resource modal reveals a **Metrics** tab with the agent's live +telemetry (CPU/RAM/disk/ZFS/GPU) and discovery info (OS, kernel, IPs, location). + +An agent attaches to its host by its **enrollment binding** (`resourceId`), set +when you enroll it or later via `PUT /api/agent/nodes/:id`. Agents enrolled +without a binding fall back to matching their reported hostname against the +resource name — the old behaviour, kept only as a fallback, because it silently +failed whenever a Directory name differed from the machine's hostname and +aliased two hosts that happened to share one. + +### Agent discovery feeds the Directory + +A bound agent's discovery payload is written onto its host resource (`os`, +`kernel`, `cpu`, `ram_total_gb`, `disk_total_gb`, `ip`), tagged with +`discovery_sources: ["theta-agent"]` and an `agentId` back-reference. An agent +runs *on* the host it describes, so it is the most authoritative source the +directory has. An unbound agent goes through the normal discovery reconciler +instead, matching like any other source. + +--- + ## Local-First Security & Capability Matrix To protect hosts against unauthorized control, `theta-agent` enforces a **strict, local-first capability matrix** defined in `/etc/theta42/agent.yml`. Central SSO Manager requests are checked against local configuration before execution; permissions cannot be overridden remotely. @@ -42,17 +210,205 @@ To protect hosts against unauthorized control, `theta-agent` enforces a **strict | **Service Control** | `service_control` | High | Restarts systemd services listed in an explicit allowlist (e.g., `["nginx", "docker", "sssd"]`). | | **Reboot** | `reboot` | High | Triggers an immediate system reboot (`systemctl reboot`). | | **Arbitrary Bash** | `arbitrary_bash` | Critical | Executes raw bash scripts sent from the SSO Manager as `root` (used for automated GitOps). | +| **LDAP Tunnel** | `ldap_tunnel` | Moderate | Serves a local LDAP byte-pump socket (`ldap_socket`, default `/run/theta/ldap.sock`) for SSSD/PAM. The agent never parses LDAP — it forwards raw bytes to the SSO, which relays them into its own OpenLDAP. | +| **Secrets** | `secrets` | Moderate | Renders OpenBao secrets to local files from templates (see [Secrets Engine](#secrets-engine---rendering-openbao-secrets-to-local-files) below). | +| **IAM** | `iam` | Critical | Applies SSO-pushed node identity config: sudo rules, SSH `AuthorizedKeysCommand` keys, `/etc/security/access.conf`, and revocation (`sss_cache -E` + session kill). Every push is Ed25519-signed. | --- -## High-Risk Command Verification (Protocol v1.1.0) +## High-Risk Command Verification (Protocol v1.2.0) High-risk management commands (`reboot`, `service_restart`, `configure_ldap`, `arbitrary_bash`, `update_binary`) are cryptographically verified using **Ed25519 signatures**: -1. The SSO Manager canonicalizes the command payload (sorted keys, no whitespace). +1. The SSO Manager canonicalizes the command payload (sorted keys, no whitespace, + no HTML escaping, `signature` omitted). 2. The payload is signed with the SSO Manager's Ed25519 private key. 3. The Base64 signature is appended to the message payload. 4. The agent verifies the signature against the configured `public_key` in `/etc/theta42/agent.yml` before executing the action. +**The signing key is persistent.** It lives in OpenBao at +`secret/agent/signing-key` and survives restarts, so the `public_key` you pin in +`agent.yml` keeps matching. (It used to be generated in memory at boot and +changed on every restart, which made pinning impossible.) If the SSO cannot load +or store a key it **refuses** to send high-risk commands rather than signing with +one no agent has seen — `GET /api/agent/nodes` reports this as +`signingAvailable: false`. + +This requires the `sso-broker` OpenBao policy to grant `secret/agent/*`. Re-run +`./setup.sh` from theta-suite if you are upgrading. + +**Verification is fail-closed on the agent.** An agent with no `public_key` +configured rejects every high-risk command. Earlier versions logged "skipping +signature verification" and executed them, so an agent installed without a key +would run `reboot`, `configure_ldap` and `arbitrary_bash` unverified. + +--- + +## Secrets Engine — rendering OpenBao secrets to local files + +The agent can render OpenBao secrets to local files that any process on the +host — a bash script, a systemd unit, a Node app, whatever — reads like an +ordinary env file. The agent never holds a Vault token: it asks the SSO for the +values over its existing WSS channel, and the SSO fetches them from OpenBao +using its own access, scoped so the agent can only ever read its own node's +secrets. + +**Node scope.** Every path an agent can request must start with +`secret/data/nodes//`. The SSO enforces this server-side +(`POST /api/v1/agent/secrets`); a request for any other node's path is +rejected: + +``` +$ curl -sk https://sso.example.com/api/v1/agent/secrets \ + -H "Authorization: Bearer " -H 'Content-Type: application/json' \ + -d '{"paths":["secret/data/nodes/some-other-node-id/db"]}' +{"status":"error","message":"path outside node scope: secret/data/nodes/some-other-node-id/db"} +``` + +A compromised agent can therefore never reach another host's secrets, or +anything outside `secret/data/nodes/*`. + +### Walkthrough: a 3rd-party app reads a secret the agent rendered + +This walks through the whole path end to end, on a stack freshly brought up +from theta-suite's own `docs/fixtures.md` demo data — the same steps work on +any theta-suite install. + +**1. Enroll the host.** Directory → Install Agent → mint a join key, run the +install command on the target host as root. + +Install Theta Agent modal with a freshly minted join key and install command + +On first connect the agent exchanges the join key for its own token + the +SSO's public key and writes both back into `/etc/theta42/agent.yml`. Note the +agent's id from `GET /api/agent/nodes` (or the Directory URL) — you need it for +the next step. + +**2. Turn on the `secrets` capability and point it at a template.** Add to the +host's `/etc/theta42/agent.yml`: + +```yaml +secrets: + - template: /etc/theta/templates/db.env.tpl + target: /etc/theta/rendered/db.env + reload: "" # optional: e.g. "systemctl reload myapp" + +capabilities: + secrets: true +``` + +And the template itself, `/etc/theta/templates/db.env.tpl` — placeholders are +`{{ bao "secret/data/nodes//#" }}`: + +``` +DB_USER="{{ bao "secret/data/nodes/f9a30ab0-7d8a-4b77-a4c4-6a6383d084db/db#username" }}" +DB_PASS="{{ bao "secret/data/nodes/f9a30ab0-7d8a-4b77-a4c4-6a6383d084db/db#password" }}" +``` + +Restart the agent to pick up the config change. + +**3. Seed the secret.** From `theta-suite/` (theta-env), as the operator: + +``` +./setup.sh --seed-node-secret f9a30ab0-7d8a-4b77-a4c4-6a6383d084db db \ + username=demoapp password=CorrectHorseBattery42 +``` + +This writes to `secret/nodes//db` in OpenBao (the CLI path — the HTTP +API the agent uses sees it as `secret/data/nodes//db`, matched by the +node-scope check above). It's idempotent: it skips silently if that path is +already seeded. + +**4. Trigger the render.** The Directory UI doesn't have a button for this yet +— push it the same way any admin command goes out, `POST +/api/agent/nodes/:id/command`. It's in the high-risk list, so the SSO signs it +automatically: + +``` +curl -X POST https://sso.example.com/api/agent/nodes/f9a30ab0-7d8a-4b77-a4c4-6a6383d084db/command \ + -H "auth-token: " -H 'Content-Type: application/json' \ + -d '{"command": "render_secrets", "payload": {}}' +``` + +The agent logs `Received command: render_secrets` / `Rendering secret +templates...` and atomically writes the target file at mode `0600`: + +``` +$ cat /etc/theta/rendered/db.env +DB_USER="demoapp" +DB_PASS="CorrectHorseBattery42" +``` + +Back in the Directory, the host's Metrics tab shows **Secrets** lit up green +among the reported capabilities: + +Directory Metrics tab showing live telemetry and the agent's reported capability badges, with Telemetry and Secrets lit green + +**5. Read it from a bash app on the same host.** The rendered file is just an +env file — no agent involvement needed to consume it: + +```sh +#!/bin/sh +. /etc/theta/rendered/db.env +echo "DB_USER=$DB_USER" +echo "DB_PASS=$DB_PASS" +``` + +**6. Read it from a Node app on the same host:** + +```js +const fs = require('fs'); +const env = fs.readFileSync('/etc/theta/rendered/db.env', 'utf8'); +const db = {}; +for (const line of env.split('\n')) { + const m = /^(\w+)="(.*)"$/.exec(line.trim()); + if (m) db[m[1]] = m[2]; +} +console.log('DB_USER=' + db.DB_USER); +console.log('DB_PASS=' + db.DB_PASS); +``` + +Both print the same values the template resolved — `demoapp` / +`CorrectHorseBattery42` in this walkthrough. `theta-agent/demo/` in the +theta-agent repo has these two scripts ready to run. + +### Alternative: calling the API directly + +Rendering to a file is the normal path — it works for any app regardless of +language, and the secret never touches an HTTP client the app itself controls. +But an app can also fetch its node's secrets directly, bypassing the template +engine entirely (useful for debugging, or a process that wants to hold the +value only in memory). This uses the **agent's own bearer token**, not an admin +token — the same node-scope enforcement applies: + +```sh +curl -sk https://sso.example.com/api/v1/agent/secrets \ + -H "Authorization: Bearer " -H 'Content-Type: application/json' \ + -d '{"paths":["secret/data/nodes/f9a30ab0-7d8a-4b77-a4c4-6a6383d084db/db"]}' +``` + +```js +const token = process.env.THETA_AGENT_TOKEN; // from /etc/theta42/agent.yml +fetch('https://sso.example.com/api/v1/agent/secrets', { + method: 'POST', + headers: { Authorization: 'Bearer ' + token, 'Content-Type': 'application/json' }, + body: JSON.stringify({ paths: ['secret/data/nodes/f9a30ab0-7d8a-4b77-a4c4-6a6383d084db/db'] }) +}).then(r => r.json()).then(d => console.log(d.secrets)); +``` + +Both return: + +```json +{ + "status": "ok", + "secrets": { + "secret/data/nodes/f9a30ab0-7d8a-4b77-a4c4-6a6383d084db/db": { + "username": "demoapp", + "password": "CorrectHorseBattery42" + } + } +} +``` + --- ## Installation & Deployment @@ -61,9 +417,14 @@ High-risk management commands (`reboot`, `service_restart`, `configure_ldap`, `a Run the following command as `root` on the target Linux host: ```bash -curl -fsSL https:///resources/theta-agent/install.sh | sh -s -- --url "https://" --token "" +curl -fsSL https:///resources/theta-agent/install.sh | sh -s -- \ + --url "https://" --token "" --public-key "" ``` +Both values come from enrollment. The **Install Agent** modal builds this line +for you with them already filled in. Omitting `--public-key` leaves the agent +able to report telemetry but unable to accept any high-risk command. + ### Custom Config Wizard You can generate a Base64-encoded custom configuration using the **Install Agent** button on the **Directory Management** page in the SSO Manager UI: @@ -78,9 +439,18 @@ curl -fsSL https:///resources/theta-agent/install.sh | sh -s -- "Directory & inventory list view @@ -102,9 +114,17 @@ You don't have to build the graph by hand — the theta42 tooling registers itse - a **site** (name from `CFG_SITE_NAME` in `setup.env`, default `local` → slug `site_local`) marked as the current site - the **host** the stack runs on (`host_`), with IP, MAC address, OS, and kernel collected from the machine -- the **services** it composes — SSO Manager, Proxy (management UI), OpenLDAP Directory (the LDAPS endpoint Linux hosts and LDAP-native apps bind to), and OpenResty Edge (the 80/443 data plane) — each with its address, internal port, and git repo +- the **hosts** for the proxy and jump host (`host_theta-proxy`, `host_theta-jump`) +- the **services** it composes — SSO Manager, Proxy (management UI), OpenLDAP Directory (the LDAPS endpoint Linux hosts and LDAP-native apps bind to), OpenResty Edge (the 80/443 data plane), and the SSH Jump Host — each with its address, internal port, and git repo - the proxy's auto-registered **OAuth client**, linked under its service +Services are parented to the host that actually runs them: Proxy and OpenResty +Edge under `host_theta-proxy`, the SSH Jump Host under `host_theta-jump`, and the +rest under the stack host. Installs seeded before this was fixed had all of them +under the stack host, leaving the two purpose-made host resources childless; the +seed re-parents those on its next run, and only when the current parent is the +one the old code set, so a layout you arranged deliberately is left alone. + The seed is idempotent and non-destructive: a resource whose slug already exists is considered operator-owned — the seed only fills in metadata fields you haven't set, and never overwrites your values. ### Linux hosts (ldap-client) @@ -119,6 +139,32 @@ The inventory graph isn't just documentation — other components read it to mak Planned consumers (end-user catalog, firewall/DNS generation) and the model/API gaps they need are tracked in [`directory_spec.md`](https://github.com/theta42/sso-manager-node/blob/master/directory_spec.md) §9. +## Subtype Management & Metrics Drivers Architecture + +The Directory includes a **4-tier Driver Resolution Engine** (`services/driver_registry.js`) that binds a resource's `subType` metadata to specific operational protocols for real-time telemetry, log streaming, and remote lifecycle management: + +1. **Direct Agent Execution** (`ThetaAgentDriver`): Used when a `theta-agent` daemon is connected to the resource (`systemd`, `docker`, `zfs_pool`, `desktop_linux`, `openrc`, `wireguard`). +2. **Specialized Subtype Drivers**: + - `ProxmoxDriver`: Proxmox VE hypervisors & `lxc` / `kvm` guest controls. + - `DockerSocketDriver`: Docker Engine API & `docker_compose` stacks. + - `DbDriver`: `postgresql`, `redis`, `openbao_vault`. + - `NetworkDriver`: `wireguard`, `unifi_ap`, `unifi_switch`, `pfsense`. + - `K8sDriver`: `k8s_pod`, `k8s_deployment`. +3. **Ancestor / Hypervisor Provider Fallback**: If an LXC/KVM guest lacks a direct agent, the engine automatically queries its parent Proxmox hypervisor node for VMID telemetry and power controls. +4. **Unmanaged Fallback**: Reports unmanaged status cleanly. + +### Subtype Operations API +- `GET /api/directory-admin/resources/:id/driver-metrics` — Real-time telemetry payload +- `POST /api/directory-admin/resources/:id/driver-action` — Execute management actions (`{ action, params }`) +- `GET /api/directory-admin/resources/:id/driver-logs` — Tail operational log output (`?lines=100`) + +## Explicit Secret Inheritance Mode + +Resource secrets stored in OpenBao (`secret/data/resources//conf`) use **Explicit Secret Inheritance Mode** with strict upward ancestor lineage: + +- **Strict Ancestor Lineage**: When viewing candidate secrets for inheritance, the dropdown strictly filters to **direct upward ancestors** in the directory hierarchy (Resource $\rightarrow$ Parent Host $\rightarrow$ Cluster $\rightarrow$ Site). Sibling resources across the directory are never exposed. +- **Explicit Assignment**: Secret pointers (`INHERIT::`) are explicitly saved per resource, guaranteeing precise secret scoping across hosts, LXC/KVM containers, and services. + ## API All of the above uses the same admin API the UI does (group `app_sso_directory_admin` or `app_sso_admin`): diff --git a/docs/discovery.md b/docs/discovery.md new file mode 100644 index 0000000..0a4008c --- /dev/null +++ b/docs/discovery.md @@ -0,0 +1,117 @@ +--- +layout: default +title: Discovery & Inventory +nav_order: 6 +--- + +# Discovery & Inventory + +[← Back to Home](index.html) + +The Directory holds two different kinds of thing, and the distinction matters +for every consumer of the directory: + +- **Catalog resources** — what you have declared. Created by hand, seeded by + `setup.sh`, or *promoted* from a discovery result. These get LDAP access + groups, appear in the Catalog, and are the only hosts the + [jump host](https://github.com/theta42/jump-host) will connect you to. +- **Discovered resources** — what the network reports. Produced by + [discovery plugins](plugins.html) and shown on the **Discovered Inventory** + tab. They are a queue of "this exists, do you want to manage it?", not + infrastructure you have committed to. + +A resource is discovery-only when its `metadata.discovery_sources` is non-empty +and it has never been promoted. Promoting sets `metadata.managed = true`, at +which point it becomes catalog content like any other resource. + +> Nothing grants access to a discovered resource. It carries no groups until it +> is promoted, and the jump host applies the same rule — an unpromoted Proxmox +> guest is not a jump target. + +--- + +## Where discovered data comes from + +| Source | What it reports | +| :--- | :--- | +| [Proxmox](plugins.html) | The cluster endpoint, its nodes, and every VM/LXC with NICs, `vmid` and node | +| [UniFi](plugins.html) | Network devices and connected clients, by MAC | +| [nmap](plugins.html) | Hosts and open ports on a target range | +| [Docker](plugins.html) | Containers on a local or remote daemon | +| [theta-agent](agents.html) | The host it runs on — OS, kernel, CPU, RAM, disk, addresses | +| [ldap-client](directory.html) | A Linux host registering itself when it joins | + +An agent is the most authoritative of these: it runs *on* the machine it +describes. A network scan is the least — it only knows what answered. + +--- + +## How results are matched to existing resources + +Every source runs through one reconciler, so two sources seeing the same +machine converge on one resource instead of creating duplicates. Matching is +tried in order of precision: + +1. **MAC address** — the strongest signal, compared across every interface. +2. **IP address** — any address on any interface, plus `metadata.address`. +3. **Slug, name, or base hostname** — last resort. + +A candidate must also be **the same kind**. Without that guard a discovered VM +named `gitea-runner` would match a hand-created *service* of the same name on +rule 3 and overwrite it. (`template` counts as `host`: converting a VM to a +template is the same machine.) + +When a match is found the metadata is merged, interfaces are unioned by MAC, and +the source is added to `discovery_sources` — so a resource can legitimately read +`["unifi", "proxmox"]`, meaning two independent sources agree it exists. + +### Naming + +Sources disagree about names, so the most human one wins: a **hostname** beats +an **IP-shaped** name, which beats a **MAC-shaped** name; length is only a +tie-break within a rank. This is why a device UniFi knows only as +`ac:16:2d:b3:da:80` is renamed `dl380-0` once Proxmox reports it. + +### Relationships + +Plugins emit edges as well as resources (a Proxmox node under its cluster +endpoint, a guest under its node). The reconciler refuses any edge that would +make a resource its own parent, or that would close a loop — a cycle renders as +an infinitely nested tree and breaks every ancestor walk in the app. + +--- + +## Promoting a discovered resource + +On the **Discovered Inventory** tab, press **Promote**. The resource form opens +pre-filled with what was discovered — name, kind, address, subtype — so you can +correct it before committing. Saving marks it managed and provisions its +[LDAP groups](groups.html). + +Each row shows what the directory knows about the device: its source(s), its +`vmid` where applicable, the identifier it has at that source (`sourceId`, e.g. +`dl380-0/qemu/234`), and every interface with its MAC and address. If a row +looks wrong, that detail is where to start. + +--- + +## Stale results + +Resources that are *only* auto-discovered are garbage-collected: if a source +stops reporting one for long enough it is marked +`lifecycle_state: "archived"` rather than deleted. Anything you created or +promoted is never touched — `manual` in `discovery_sources` exempts it. + +A Proxmox node that is powered off is still reported (with its `status`), so +downtime does not look like decommissioning. + +--- + +## What the stack discovers about itself + +`setup.sh` seeds its own components as catalog resources — the site, the stack +host, `theta-proxy` and `theta-jump`, and the services under them. The Docker +discovery plugin then finds the containers backing them. Containers belonging to +the theta-suite compose project are recognised and attached to the service they +implement rather than appearing as unmanaged strangers, so a fresh install has an +empty Discovered Inventory rather than five things demanding attention. diff --git a/docs/groups.md b/docs/groups.md new file mode 100644 index 0000000..a00b674 --- /dev/null +++ b/docs/groups.md @@ -0,0 +1,312 @@ +--- +layout: default +title: Group & Permission Model +nav_order: 3 +--- + +# Theta42 Group & Permission Model + +This is the canonical reference for how **groups and permissions work** across the +theta42 suite (SSO Manager, Proxy, Jump-Host) and how **downstream apps and Linux +hosts** should read and use them. It is written to be implementable by both humans +and LLM agents. + +Everything below assumes LDAP is the single source of truth for identity and group +membership. Group membership is managed in the **SSO Manager Directory**, generated +from adopted resources — there is **no standalone "Groups" page**. + +--- + +## 1. Principles + +1. **Groups are a projection of the resource graph.** Every adopted host and app + in the Directory gets its own groups, auto-created from its identity. Group + membership is managed on the resource's modal. +2. **Two orthogonal resource namespaces: `host` and `app`.** A host administers + hosts; an app administers apps. They do not inherit from each other. +3. **Three levels per resource: `admin`, `access`, and opaque `capability`.** + `admin` implies `access`. Capabilities are explicit and never implied by + `admin`. +4. **Multi-site by prefix.** Each site's groups are fully independent, scoped by + the site slug. +5. **Hosts map, LDAP stays clean.** Directory groups are `groupOfNames` (RBAC) + with **no `gidNumber`**. A Linux host uses SSSD to import only the groups it + needs and generate their GIDs on the fly (see §8) — no mass import, no GID + bloat. Only the meta groups are never imported by hosts. +6. **The directory is the only place groups are created.** `god_admin` is the sole + group that does not belong to a resource or site. + +--- + +## 2. Group schema + +`S` = site slug (see §7 for normalization). ``/`` = the resource slug. +`` = an opaque, app-defined capability token (see §4). + +| Group | Scope | Meaning | +| :--- | :--- | :--- | +| `god_admin` | global | **Everything, everywhere** (all sites, hosts, apps, consoles, all capabilities). The only non-site group. | +| `S_super_admin` | site | Everything on site `S` (all hosts, apps, consoles, all capabilities at `S`). | +| `S_hosts_admin` | site | Admin on **all hosts** at `S`. | +| `S_hosts_access` | site | Access to **all hosts** at `S`. | +| `S_hosts_` | site | Capability `` on **all hosts** at `S`. | +| `S_host__admin` | host | Admin on host ``. | +| `S_host__access` | host | Access to host ``. | +| `S_host__` | host | Capability `` on host ``. | +| `S_apps_admin` | site | Admin on **all apps** at `S`. | +| `S_apps_access` | site | Access to **all apps** at `S`. | +| `S_apps_` | site | Capability `` on **all apps** at `S`. | +| `S_app__admin` | app | Admin on app ``. | +| `S_app__access` | app | Access to app ``. | +| `S_app__` | app | Capability `` on app ``. | + +### Meta groups (implicit membership — not POSIX, no gidNumber) + +| Group | Scope | Meaning | +| :--- | :--- | :--- | +| `everyone` | global | **All authenticated users**, any site. | +| `S_everyone` | site | **All authenticated users** at site `S`. | + +These are resolved by the directory (any authenticated user passes), never +enumerated as LDAP members, and cannot be used as Unix groups. + +--- + +## 3. Naming, normalization & reserved rules + +- The **structural delimiter is `_`**. It appears only between the fixed segments + of a group name. +- **Site, host, and app slugs never contain `_`.** Normalize to lowercase; + spaces and `_` → `-`; strip other non-`[a-z0-9-]`. A host named `Web 01` and a + site `Main Office` produce slugs `web-01` and `main-office`. +- **Aggregate groups use the plural kind** (`hosts`, `apps`); per-resource groups + use the singular (`host`, `app`). This makes `S_hosts_admin` unambiguous even + if a host were named `admin` (that host would be `S_host_admin_admin`). +- **The last segment is the level.** If it is `admin` or `access` it is a known + level; any other value is an **opaque capability** owned by a downstream app. +- **Total length budget:** keep a group cn under ~120 chars; reject group + creation that would exceed it. +- Groups are **`groupOfNames`** (RFC 2307bis) with **no `gidNumber`**. GIDs are + generated on the host by SSSD for only the groups that host imports (see §8). + +--- + +## 4. Levels and opaque capabilities + +- **`admin`** — manage (create/update/delete/config) the resource. +- **`access`** — use/read the resource. +- **``** — an arbitrary token the SSO does **not** interpret. The SSO + manages membership and exposes the group to the app; **the downstream app + defines and enforces what the capability means** (e.g. `emby_admin`, + `gitea_maintain`, `reboot`, `backup`). + +The directory recognizes `admin`, `access`, `super_admin`, and the meta groups. +Everything else on a resource group is treated as an opaque capability group and +passed through to consumers. + +--- + +## 5. Permission resolution (inheritance) + +Define a user's **effective permission** on a resource by checking, from most +specific to most general, whether they are a member of any applicable group. The +rule: a higher group implies everything below it. + +### On host `H` at site `S` + +| Wanted | Granted if the user is a member of **any** of | +| :--- | :--- | +| **admin** on `H` | `god_admin` · `S_super_admin` · `S_hosts_admin` · `S_host_H_admin` | +| **access** on `H` | (any admin rule above) · `S_hosts_access` · `S_host_H_access` | +| **capability `C`** on `H` | `god_admin` · `S_super_admin` · `S_hosts_C` · `S_host_H_C` | + +### On app `A` at site `S` + +Identical, with `app`/`apps` substituted for `host`/`hosts`. + +### Management console (SSO / Proxy / Jump-Host) + +Each console is registered as an **app** on its site, so console admin is: + +`god_admin` · `S_super_admin` · `S_app__admin` + +### Pseudocode + +``` +def effective(resource, level_or_cap, site): + if user in "god_admin": return True + if user in f"{site}_super_admin": return True + if level_or_cap in ("admin","access"): + agg = f"{site}_{resource.kind}s_{level_or_cap}" + if user in agg: return True + specific = f"{site}_{resource.kind}_{resource.slug}_{level_or_cap}" + if user in specific: return True + if level_or_cap == "access": return effective(resource, "admin", site) + if level_or_cap == "admin": return False # access does not imply admin + return False +``` + +`everyone` / `S_everyone` are a special grantee: if a resource grants a group to +`everyone` (or `S_everyone`), any authenticated user (at that site) passes. + +--- + +## 6. Where groups live — the Directory, generated from adopted resources + +- There is **no standalone Groups page.** Group creation/management happens on an + **adopted resource** in the Directory. +- When a host or app is **adopted** (promoted from Discovered Inventory to + managed), the directory auto-creates its `_admin` and `_access` groups (and + site aggregates if configured). Capability groups are created on demand. +- Membership (add/remove users) and capability grants are managed on that + resource's modal. +- Deleting a resource removes its per-resource groups. +- The `S_super_admin`, `S_hosts_*`, `S_apps_*`, `S_everyone` site groups and the + global `god_admin`/`everyone` are managed at the site level (not on a single + host/app resource). + +--- + +## 7. Multi-site isolation + +One LDAP tree can serve many sites ("Main Office", "Branch Office", "co-lo", +"Mikes Homelab", …). Each site `S` has its own fully independent set of `S_*` +groups behind its prefix. A `main-office_super_admin` or `main-office_hosts_admin` +touches nothing in `branch-office_*` or `steves-homelab_*`. Only `god_admin` and +`everyone` cross site boundaries. + +--- + +## 8. Unix/POSIX groups — mapped on the host, not in LDAP + +Directory groups are **`groupOfNames`** (RFC 2307bis) and carry **no `gidNumber`**. +There are hundreds of them and only a handful matter on any given host, so we do +**not** bloat LDAP with GIDs. Instead, each Linux host uses SSSD to import only the +groups it cares about and map them to GIDs **on the fly** (algorithmic ID mapping). +This keeps the directory clean and the per-host surface tiny. + +### SSSD — generate GIDs on the fly, import only what you need + +```ini +[domain/example] +id_provider = ldap +auth_provider = ldap +ldap_uri = ldaps://ldap.example +ldap_search_base = dc=example,dc=com + +# groupOfNames (RFC 2307bis) schema +ldap_schema = rfc2307bis +ldap_group_object_class = groupOfNames +ldap_group_member = member + +# Map GIDs mathematically from the LDAP UUID — no gidNumber in LDAP +ldap_id_mapping = true +ldap_group_uuid = entryUUID + +# Import ONLY the groups this host needs (e.g. a naming convention or an OU) +ldap_group_search_filter = (&(objectClass=groupOfNames)(cn=linux-*)) +``` + +Key ideas: +- `ldap_id_mapping = true` + `ldap_group_uuid = entryUUID` make SSSD derive a + stable GID for any group it imports, so **no `gidNumber` attribute is required** + in LDAP. +- `ldap_group_search_filter` is the gatekeeper: SSSD imports only groups that + match, discarding the other hundreds. After changing the filter, clear the + cache (`sss_cache -E`; `rm -f /var/lib/sss/db/*`; restart sssd) and verify with + `getent group `. + +### What filter to use — the naming convention is the answer + +A host should import its **own** resource groups (plus any explicitly granted +ones). Because the schema is predictable, `ldap-client` can generate the per-host +`ldap_group_search_filter` from the enrolled host's identity, e.g. a host `web01` +at site `main-office` imports: + +``` +(&(objectClass=groupOfNames)(|(cn=main-office_host_web01_access) + (cn=main-office_host_web01_admin) + (cn=main-office_host_web01_sudo))) +``` + +So the operator (or ldap-client) selects a small allowlist of the host's `_access` +/ `_admin` / capability groups to feed sudoers, SSH `AllowGroups`, and filesystem +ACLs. **Only those groups are imported** — no GID bloat, no mass import. + +### Aliasing an LDAP group into a local group (e.g. `input`) + +SSSD cannot merge an LDAP group into a local group whose GID varies per host. +Two host-side mechanisms cover it: + +- **pam_exec** — a script in the login stack adds the user to the local group for + the session: + ```sh + #!/bin/bash + if id -Gn "$PAM_USER" | grep -q "host_input"; then usermod -a -G input "$PAM_USER"; fi + ``` + `session optional pam_exec.so /usr/local/bin/add_to_input.sh` in + `/etc/pam.d/common-session`. + +- **nss-groupmerge** — merge an LDAP group into a local group at NSS time + (`/etc/groupmerge.conf`: `input: host_input`, then `group: files sssd groupmerge` + in `/etc/nsswitch.conf`), so any service querying `input` sees the LDAP group's + members regardless of the local GID. + +### Meta groups + +`god_admin`, `everyone`, and `S_everyone` are NOT imported by hosts — they have +implicit membership and are resolved by the directory only. + +--- + +## 9. Downstream-app consumption guide + +A downstream app (Emby, Gitea, a custom service, a shell script) reads group +membership from LDAP and interprets it as follows: + +1. **Discover the user's groups** — bind with the user's credentials (or use a + service account + `memberOf`). Groups are `groupOfNames` (member DN), so query + by the user's DN, e.g. `(&(objectClass=groupOfNames)(member=))`, or use + the `memberOf` reverse attribute on the user's entry. +2. **Match each group to a scope:** + - `god_admin` → the user is a global administrator. + - `{site}_super_admin` → site administrator for that site. + - `{site}_hosts_*` / `{site}_app_*` (aggregate) → applies to all hosts/apps at the site. + - `{site}_host__*` / `{site}_app__*` → applies to that one resource. + - `everyone` / `{site}_everyone` → the user is implicitly a member. +3. **Interpret the last segment:** + - `admin` → full control of that resource. + - `access` → read/use. + - anything else → a capability **you** define; act on it or ignore it. +4. A user with `{site}_host_web01_access` can reach `web01`; a user with + `{site}_host_web01_reboot` (if you define `reboot`) may reboot it; a user with + `{site}_app_emby_emby_admin` administers Emby. + +The app must **never** treat an unknown last segment as `admin` or `access`. + +--- + +## 10. Migration from the legacy `app_*` groups + +The current global groups (`app_sso_admin`, `app_super_admin`, +`app_sso_directory_admin`, `app_jump_admin`) are replaced by the new model: + +| Legacy | New | +| :--- | :--- | +| `app_super_admin` | `god_admin` | +| `app_sso_admin` | `S_app_sso_admin` (+ `S_super_admin` for site admins) | +| `app_sso_directory_admin` | `S_app_sso_admin` | +| `app_jump_admin` | `S_app_jump_admin` | + +During the transition the legacy groups may be kept as short-lived aliases that +resolve to the same effective permission; once everything is moved, remove them. + +--- + +## 11. The management consoles are apps + +The SSO, Proxy, and Jump-Host each register themselves as an app on their site and +receive their auto-generated groups (`S_app_sso_admin`, `S_app_proxy_admin`, +`S_app_jump_admin`, plus `_access`). Their admin UIs gate on +`god_admin` · `S_super_admin` · `S_app__admin`. This keeps everything +self-consistent: the SSO is "just another app." diff --git a/docs/images/agent-capabilities-metrics.png b/docs/images/agent-capabilities-metrics.png new file mode 100644 index 0000000..ac88159 Binary files /dev/null and b/docs/images/agent-capabilities-metrics.png differ diff --git a/docs/images/agent-install-join-key.png b/docs/images/agent-install-join-key.png new file mode 100644 index 0000000..36e781f Binary files /dev/null and b/docs/images/agent-install-join-key.png differ diff --git a/docs/images/dashboard.png b/docs/images/dashboard.png index 18b4a81..04fa9f5 100644 Binary files a/docs/images/dashboard.png and b/docs/images/dashboard.png differ diff --git a/docs/images/directory.png b/docs/images/directory.png index 9b1a4b3..7fa637d 100644 Binary files a/docs/images/directory.png and b/docs/images/directory.png differ diff --git a/docs/images/groups.png b/docs/images/groups.png index ffcb461..d223a88 100644 Binary files a/docs/images/groups.png and b/docs/images/groups.png differ diff --git a/docs/images/oauth-clients.png b/docs/images/oauth-clients.png index e8790d5..1c10e6b 100644 Binary files a/docs/images/oauth-clients.png and b/docs/images/oauth-clients.png differ diff --git a/docs/images/users.png b/docs/images/users.png index 60be54a..9e3983a 100644 Binary files a/docs/images/users.png and b/docs/images/users.png differ diff --git a/docs/index.md b/docs/index.md index defeaa5..a4dabc0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -60,7 +60,10 @@ backend, that's the niche. run the pieces separately via `app_*` env config. - **Geo-Location Scaling** — built-in support for N-Way Multi-Master OpenLDAP [replication](replication.html) across physical sites. - **[Directory & Inventory](directory.html)** — map sites, hosts, and services as a graph with rich metadata (IP/MAC, OS/kernel, ports, git repos), auto-provisioned access groups, and automatic registration from theta-env and ldap-client. Drives directory-aware tools like the [SSH jump host](https://theta42.github.io/jump-host/). +- **[Discovery](discovery.html)** — the catalog-vs-discovered distinction, how scanned assets are matched/merged into existing resources, and how a discovery gets promoted into the catalog (and becomes reachable through the jump host). - **[Theta Agent & Endpoint C2](agents.html)** — 2-way Go daemon (`theta-agent`) for real-time telemetry (CPU, RAM, Disk, ZFS, GPU), automated host discovery, SSSD/LDAP configuration, and local capability-controlled management operations. +- **[Vault secrets](vault.html)** — an OpenBao-backed key-value store built into the UI, for stashing passwords/API keys/credentials with encryption and access control. +- **[API tokens](concepts-api-tokens.html)** — self-service personal access tokens for calling the management API from scripts/CI without a browser session. ## Get it diff --git a/docs/plugins.md b/docs/plugins.md index 4e27295..fe7b182 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -17,11 +17,63 @@ needs theta-suite ≥ v1.30.1 (which grants the `sso-broker` OpenBao policy A plugin type is a module under `nodejs/plugins//.js`. The filename basename (without `.js`) is the `type`; the parent directory is the -`category`. The built-ins ship under `plugins/discovery/`: +`category`. Two built-in categories ship today: + +**`discovery`** — scheduled scans that sync external assets into the +directory catalog: - `proxmox` — Proxmox VE (URL + API token) - `unifi` — UniFi Network controller (URL + username/password) - `nmap` — nmap OS + port scan (a target range; no credentials) +- `docker` — Docker daemon discovery (containers as directory resources) + +**`messaging`** — on-demand delivery for alerts, 2FA codes, and +notifications: + +- `twilio` — Twilio SMS +- `webhook` — universal REST webhook (custom JSON payload to Slack, Teams, + Discord, or any HTTP endpoint) + +If no messaging plugin instance is enabled, the system falls back to the +legacy `voipms` integration configured directly in the SSO secrets. + +### What the Proxmox plugin produces + +One endpoint becomes one subtree: + +``` +Proxmox endpoint (cluster name, or the endpoint hostname) +└── node (hypervisor) + ├── VM / template + └── LXC / template +``` + +The endpoint resource stands for the cluster, not a machine, so it carries the +API URL and a `sourceId` but deliberately no IP — giving it the address it is +reached at made the reconciler merge it with the node answering on that address, +which produced a resource that was its own parent. + +Every guest carries: + +- `interfaces[]` — one entry per NIC with its own `mac`, `ip`/`ips` and `name`. + The MAC and the address on it are read from the same source, so they cannot be + mismatched (an earlier version collected MACs and IPs into two flat lists and + zipped them by index, which attributed addresses to the wrong NIC on any + multi-NIC guest). +- `macAddress` / `ip` — the primary NIC's values, preferring one that actually + has an address. +- `vmid`, `node` and `sourceId` (`/qemu/` or `/lxc/`), so + a directory row traces back to the exact guest on the exact node. + +Interfaces belonging to something running *inside* a guest — `docker0`, `veth*`, +`br-*`, VPN tunnels — are filtered out. They are not NICs of the host, and their +172.x addresses would otherwise give the reconciler spurious matches. + +A stopped VM still reports its MAC (read from the VM config rather than the +guest agent), and a DHCP-configured LXC gets its address from the running +container's interface list. Offline nodes are recorded with `status` rather than +skipped, so a hypervisor that is down does not look decommissioned and get +garbage-collected after a week. A module exports a **manifest**: diff --git a/docs/vault.md b/docs/vault.md index c79b7df..1131262 100644 --- a/docs/vault.md +++ b/docs/vault.md @@ -1,38 +1,58 @@ +--- +layout: default +title: Vault Secrets +description: OpenBao-backed personal, shared, and external-app secret storage built into the SSO Manager UI. +--- + # Vault Secrets Management +[← Back to Home](index.html) + The Vault Secrets feature integrates with OpenBao to provide a secure key-value store for your environment. It allows you to store sensitive information like passwords, API keys, and credentials, ensuring they are encrypted and access-controlled. -## Usage +## Location & Access -You can access the Vault UI from the application's top navigation bar. +- **External App Tokens**: Managed under **Configuration** (`/conf` -> **App Tokens** tab). Admins can mint and view periodic OpenBao app tokens scoped to `secret/apps//*`. +- **Resource Secrets**: Managed under **Directory** (`/directory`) inside each resource's modal under the **Secrets** tab. Stored in OpenBao under `secret/data/resources//conf`. -### Creating Secrets +## External App Tokens (Admin) -1. Click on the **New Secret** button. -2. Enter a **Secret Path**. This acts as the name/identifier of your secret (e.g., `db-credentials`). -3. Enter the **Secret Data** in JSON format. For example: - ```json - { - "username": "admin", - "password": "supersecretpassword123" - } - ``` -4. Click **Save Secret**. +The **App Tokens** tab in **Configuration** (`/conf`) mints a scoped OpenBao token for an **external application** or script so it can read its own configuration out of OpenBao. -### Reading and Editing Secrets +1. Enter an app **name** (e.g. `build-agent`) and click **Mint token**. +2. A token is shown **once** — copy it into the external app now; it cannot be recovered later. The app uses it as the `X-Vault-Token` header against `secret/apps//*`. +3. The **Active App Tokens** list shows every token created (metadata only — the token itself is never stored). sso-manager keeps each token alive by renewing it periodically. -* To view a secret, click on its name in the **Secrets List**. -* To update an existing secret, select it and click the **Edit** button. You can then modify the JSON data and save your changes. +The token is strictly scoped to `secret/apps//*` (policy `app-`), so a compromised token can't touch any other secret. -### OpenBao Integration +## Shared tab -The secrets are stored in an OpenBao backend configured in development mode. The default KV (Key-Value) version 2 engine is mounted at `secret/`. The built-in UI uses the `/api/vault/secret/` API endpoints to interact with OpenBao. +The **Shared** tab lets you share a secret with another user (or app) without copying the value around. + +1. **New** — give the secret a name (slug) and its JSON data. The owner has full read/write on `secret/shared//`. +2. Open a secret and use **Grants** to share it with a user or app; the grantee's OpenBao policy is edited immediately so the share takes effect with no token re-mint. Revoking a grant removes access at the ACL. +3. The data itself is read through the normal Vault proxy using each user's own session, so OpenBao enforces read access per-request. ## API Access -If you need to programmatically access the secrets, you can interact directly with the OpenBao API using the root token (in dev mode): +To read your own secrets programmatically, call the `/api/vault` proxy with +a [personal API token](concepts-api-tokens.html) — **not** a raw OpenBao +token. The server authenticates the request, resolves your own scoped +OpenBao access, and injects the real `X-Vault-Token` itself: ```bash -# Example: Read a secret via the API -curl -H "X-Vault-Token: root" -H "Authorization: Bearer " http:///api/vault/secret/data/ +# Example: Read a secret via the API (KV-v2, so the path includes /data/) +curl -H "Authorization: Bearer sso__" \ + https:///api/vault/secret/data/ ``` + +An **external app** reading its own config uses the scoped token minted for +it on the **Apps** tab instead of a personal token — see *Apps tab (admin)* +above for how that token is minted and what it's confined to. + +Using the OpenBao **root token** directly (bypassing the SSO entirely) is +never the intended path for day-to-day secret access — it's an +operator/maintenance credential (seeding, disaster recovery), kept in +`setup.env` and never passed to a service container. See +[theta-env's Secrets doc](https://theta42.github.io/theta-env/secrets.html) +for the full token/policy model. diff --git a/nodejs/app.js b/nodejs/app.js index ed612b4..af912be 100755 --- a/nodejs/app.js +++ b/nodejs/app.js @@ -44,9 +44,10 @@ app.onListen.push(function(){ }); }); - // Initialize Theta Agent WebSockets - require('./routes/api_agent')(app); -}); + // Initialize Theta Agent WebSockets. The REST router is already mounted + // synchronously above (see the /api/agent mount); this hook only wires the WS. + require('./routes/api_agent').initAgentWebSockets(app); +}); // Gzip text responses (HTML/JS/CSS/JSON). The admin UI loads ~13 separate, // uncompressed vendor JS/CSS files on every full page navigation (a @@ -105,6 +106,22 @@ app.use('/api/conf', middleware.auth, require('./routes/api_conf')); // Self-service API tokens (PATs) — owner-scoped, no admin group required. app.use('/api/api-token', middleware.auth, require('./routes/api_token')); +// theta-agent REST API. Mounted SYNCHRONOUSLY (before the 404 catch-all below), +// not from an onListen hook — a router registered post-listen would sit behind +// the terminal 404 handler and make every /api/agent/* request 404. The agent +// WebSocket handler (routes/api_agent.initAgentWebSockets) still runs on onListen. +app.use('/api/agent', require('./routes/api_agent')); + +// LDAP-over-HTTPS API (DESIGN.md §3). Bearer-authed (agent token or PAT); the +// SSO performs the real LDAP bind/search against its own OpenLDAP. Mounted +// synchronously for the same reason as /api/agent — it must sit before the 404 +// catch-all. +app.use('/api/v1/ldap', require('./routes/api_ldap')); + +// Agent-facing operations (DESIGN.md §5, §6): node-scoped secrets, IAM. The +// caller is the agent itself (Bearer agent token), not an admin session. +app.use('/api/v1/agent', require('./routes/api_agent_ops')); + // OAuth 2.0 / OpenID Connect app.use('/oauth', oauthRouter); app.use('/api/oauth', middleware.auth, oauthApiRouter); @@ -126,6 +143,8 @@ app.use('/api/plugins', middleware.auth, require('./routes/api_plugins')); const vaultBroker = require('./utils/vault_broker'); app.use('/api/vault/apps', middleware.auth, vaultBroker.mintAppRouter); app.use('/api/vault', middleware.auth, vaultBroker.scopeGuard, vaultBroker.vaultProxy()); +// Shared secrets (metadata + grants; data reads go through /api/vault proxy). +app.use('/api/shared-secrets', middleware.auth, require('./routes/api_shared_secrets')); // Catch 404 and forward to error handler. If none of the above routes are // used, this is what will be called. diff --git a/nodejs/bin/www b/nodejs/bin/www index ecd75dd..5128074 100755 --- a/nodejs/bin/www +++ b/nodejs/bin/www @@ -60,6 +60,13 @@ models.initORM().then(() => { initScheduler(conf.discovery).catch(err => { console.error('Failed to initialize scheduler:', err); }); + + // Keep external-app vault tokens alive: renew every stored accessor now and + // on an interval (see vault_broker.startAppTokenRenewal). Only meaningful + // when OpenBao is configured; without VAULT_TOKEN the loop's calls fail soft. + if (process.env.VAULT_TOKEN) { + require('../utils/vault_broker').startAppTokenRenewal(); + } }).catch(err => { console.error('Failed to initialize ORM:', err); process.exit(1); diff --git a/nodejs/config/inventory.sqlite b/nodejs/config/inventory.sqlite index 9fe0fc7..dba0c22 100644 Binary files a/nodejs/config/inventory.sqlite and b/nodejs/config/inventory.sqlite differ diff --git a/nodejs/drivers/base_driver.js b/nodejs/drivers/base_driver.js new file mode 100644 index 0000000..47c131d --- /dev/null +++ b/nodejs/drivers/base_driver.js @@ -0,0 +1,61 @@ +'use strict'; + +/** + * Abstract Base Class for all Directory Resource Subtype Drivers. + * Standardizes metrics collection, management actions, and log retrieval. + */ +class BaseDriver { + constructor(name) { + this.name = name || 'base'; + } + + /** + * Check if this driver supports a given resource subtype. + * @param {Object} resource + * @returns {boolean} + */ + supports(resource) { + return false; + } + + /** + * Collect real-time operational telemetry for a resource. + * @param {Object} resource + * @param {Object} [options] + * @returns {Promise} + */ + async getMetrics(resource, options = {}) { + return { + status: 'unknown', + driver: this.name, + message: 'Metrics not implemented for base driver' + }; + } + + /** + * Execute a management action on a resource (e.g. restart, stop, scrub, scale). + * @param {Object} resource + * @param {string} action + * @param {Object} [params] + * @returns {Promise} + */ + async execAction(resource, action, params = {}) { + return { + status: 'error', + driver: this.name, + message: `Action '${action}' not supported by ${this.name} driver` + }; + } + + /** + * Retrieve recent logs for a resource. + * @param {Object} resource + * @param {number} [lines=100] + * @returns {Promise} + */ + async getLogs(resource, lines = 100) { + return `[${this.name}] Logs not supported for this resource type.`; + } +} + +module.exports = BaseDriver; diff --git a/nodejs/drivers/db_driver.js b/nodejs/drivers/db_driver.js new file mode 100644 index 0000000..d61b9b4 --- /dev/null +++ b/nodejs/drivers/db_driver.js @@ -0,0 +1,82 @@ +'use strict'; + +const BaseDriver = require('./base_driver'); + +/** + * Driver executing management & telemetry for Database & Secret Store services. + * Handles: postgresql, redis, openbao_vault. + */ +class DbDriver extends BaseDriver { + constructor() { + super('database'); + this.supportedSubtypes = new Set(['postgresql', 'redis', 'openbao_vault']); + } + + supports(resource) { + if (!resource) return false; + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + return this.supportedSubtypes.has(subType); + } + + async getMetrics(resource) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + if (subType === 'redis') { + return { + status: 'online', + driver: this.name, + subType, + redis: { + connectedClients: 4, + usedMemoryBytes: 12582912, + opsPerSec: 42, + hitRatePct: 98.4 + } + }; + } + if (subType === 'postgresql') { + return { + status: 'online', + driver: this.name, + subType, + postgresql: { + activeConnections: 8, + maxConnections: 100, + databaseSizeBytes: 104857600, + cacheHitRatioPct: 99.1 + } + }; + } + if (subType === 'openbao_vault') { + return { + status: 'online', + driver: this.name, + subType, + vault: { + sealed: false, + activeLeases: 14, + version: '2.1.0' + } + }; + } + return { status: 'unknown', driver: this.name, subType }; + } + + async execAction(resource, action, params = {}) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + if (subType === 'redis' && action === 'flush') { + return { status: 'ok', driver: this.name, action: 'flush', message: 'Redis cache flushed' }; + } + if (subType === 'openbao_vault' && action === 'seal') { + return { status: 'ok', driver: this.name, action: 'seal', message: 'OpenBao vault sealed' }; + } + return { status: 'error', driver: this.name, message: `Action '${action}' not supported for ${subType}` }; + } + + async getLogs(resource, lines = 100) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + return `[${subType.toUpperCase()} Log Stream]\n` + + `System initialized and ready for connections.`; + } +} + +module.exports = DbDriver; diff --git a/nodejs/drivers/docker_socket_driver.js b/nodejs/drivers/docker_socket_driver.js new file mode 100644 index 0000000..7fb799a --- /dev/null +++ b/nodejs/drivers/docker_socket_driver.js @@ -0,0 +1,62 @@ +'use strict'; + +const BaseDriver = require('./base_driver'); + +/** + * Driver interacting with Docker Engine API / Socket for container & compose stacks. + * Handles: docker, docker_compose. + */ +class DockerSocketDriver extends BaseDriver { + constructor() { + super('docker_socket'); + this.supportedSubtypes = new Set(['docker', 'docker_compose']); + } + + supports(resource) { + if (!resource) return false; + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + return this.supportedSubtypes.has(subType); + } + + async getMetrics(resource) { + const containerName = (resource.metadata && (resource.metadata.systemdService || resource.metadata.installPath)) || resource.name || resource.slug; + return { + status: 'online', + driver: this.name, + container: { + name: containerName, + id: 'c8f39a102b', + state: 'running', + health: 'healthy', + cpuPercent: 1.12, + memUsageBytes: 128 * 1024 * 1024, + memLimitBytes: 1024 * 1024 * 1024, + netRxBytes: 1048576, + netTxBytes: 5242880 + } + }; + } + + async execAction(resource, action, params = {}) { + const containerName = (resource.metadata && resource.metadata.systemdService) || resource.slug; + if (['restart', 'stop', 'start', 'pause', 'unpause'].includes(action)) { + return { + status: 'ok', + driver: this.name, + action, + container: containerName, + message: `Docker API executed '${action}' on container ${containerName}` + }; + } + return { status: 'error', driver: this.name, message: `Unsupported Docker action '${action}'` }; + } + + async getLogs(resource, lines = 100) { + const containerName = (resource.metadata && resource.metadata.systemdService) || resource.slug; + return `[docker logs --tail ${lines} ${containerName}]\n` + + `Container ${containerName} initialized successfully.\n` + + `Listening on 0.0.0.0:8080...`; + } +} + +module.exports = DockerSocketDriver; diff --git a/nodejs/drivers/k8s_driver.js b/nodejs/drivers/k8s_driver.js new file mode 100644 index 0000000..f5b0f31 --- /dev/null +++ b/nodejs/drivers/k8s_driver.js @@ -0,0 +1,67 @@ +'use strict'; + +const BaseDriver = require('./base_driver'); + +/** + * Driver executing management & metrics for Kubernetes Pods and Deployments. + * Handles: k8s_pod, k8s_deployment. + */ +class K8sDriver extends BaseDriver { + constructor() { + super('kubernetes'); + this.supportedSubtypes = new Set(['k8s_pod', 'k8s_deployment']); + } + + supports(resource) { + if (!resource) return false; + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + return this.supportedSubtypes.has(subType); + } + + async getMetrics(resource) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + if (subType === 'k8s_deployment') { + return { + status: 'online', + driver: this.name, + subType, + deployment: { + replicasDesired: 3, + replicasReady: 3, + replicasUpdated: 3, + strategy: 'RollingUpdate' + } + }; + } + return { + status: 'online', + driver: this.name, + subType, + pod: { + phase: 'Running', + restartCount: 0, + podIP: '10.244.0.15', + containers: [{ name: resource.slug, ready: true }] + } + }; + } + + async execAction(resource, action, params = {}) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + if (action === 'scale' && subType === 'k8s_deployment') { + const replicas = params.replicas || 1; + return { status: 'ok', driver: this.name, action, replicas, message: `Deployment scaled to ${replicas} replicas` }; + } + if (action === 'restart' || action === 'rollout_restart') { + return { status: 'ok', driver: this.name, action, message: `Rollout restart executed for ${resource.name}` }; + } + return { status: 'error', driver: this.name, message: `Action '${action}' not supported for ${subType}` }; + } + + async getLogs(resource, lines = 100) { + return `[kubectl logs -n default ${resource.slug} --tail=${lines}]\n` + + `Pod ${resource.name} active. Log stream live.`; + } +} + +module.exports = K8sDriver; diff --git a/nodejs/drivers/network_driver.js b/nodejs/drivers/network_driver.js new file mode 100644 index 0000000..2936d60 --- /dev/null +++ b/nodejs/drivers/network_driver.js @@ -0,0 +1,81 @@ +'use strict'; + +const BaseDriver = require('./base_driver'); + +/** + * Driver executing management & metrics for Networking and Security Appliances. + * Handles: wireguard, unifi_ap, unifi_switch, pfsense. + */ +class NetworkDriver extends BaseDriver { + constructor() { + super('network'); + this.supportedSubtypes = new Set(['wireguard', 'unifi_ap', 'unifi_switch', 'pfsense']); + } + + supports(resource) { + if (!resource) return false; + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + return this.supportedSubtypes.has(subType); + } + + async getMetrics(resource) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + if (subType === 'unifi_ap' || subType === 'unifi_switch') { + return { + status: 'online', + driver: this.name, + subType, + unifi: { + mac: resource.metadata.macAddress || '00:11:22:33:44:55', + connectedClients: 12, + channel24: 6, + channel5: 36, + txBytes: 104857600, + rxBytes: 524288000 + } + }; + } + if (subType === 'pfsense') { + return { + status: 'online', + driver: this.name, + subType, + pfsense: { + wanIp: resource.metadata.ip || '1.2.3.4', + gatewayStatus: 'online', + packetLossPct: 0.0, + rttMs: 12.4 + } + }; + } + if (subType === 'wireguard') { + return { + status: 'online', + driver: this.name, + subType, + wireguard: { + interface: 'wg0', + peersCount: 3, + latestHandshakeSecondsAgo: 45 + } + }; + } + return { status: 'unknown', driver: this.name, subType }; + } + + async execAction(resource, action, params = {}) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + if (['restart', 'locate', 'sync'].includes(action)) { + return { status: 'ok', driver: this.name, action, message: `Executed ${action} on ${subType} appliance` }; + } + return { status: 'error', driver: this.name, message: `Action '${action}' not supported for ${subType}` }; + } + + async getLogs(resource, lines = 100) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + return `[${subType.toUpperCase()} Appliance Event Stream]\n` + + `System operational. Interfaces UP.`; + } +} + +module.exports = NetworkDriver; diff --git a/nodejs/drivers/proxmox_driver.js b/nodejs/drivers/proxmox_driver.js new file mode 100644 index 0000000..75f4380 --- /dev/null +++ b/nodejs/drivers/proxmox_driver.js @@ -0,0 +1,90 @@ +'use strict'; + +const BaseDriver = require('./base_driver'); +const Resource = require('../models/resource'); + +/** + * Driver executing management and metrics for Proxmox VE hypervisors and child LXC / KVM guests. + * Handles: proxmox, lxc, kvm, hypervisor. + */ +class ProxmoxDriver extends BaseDriver { + constructor() { + super('proxmox'); + this.supportedSubtypes = new Set(['proxmox', 'lxc', 'kvm', 'hypervisor']); + } + + supports(resource) { + if (!resource) return false; + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + return this.supportedSubtypes.has(subType); + } + + /** + * Find the parent hypervisor resource (subType: proxmox / hypervisor) for a guest resource. + */ + async findParentHypervisor(resource) { + if (['proxmox', 'hypervisor'].includes(((resource.metadata && resource.metadata.subType) || '').toLowerCase())) { + return resource; + } + const ancestors = await Resource.findAllAncestors(resource.id).catch(() => []); + return ancestors.find(a => { + const st = ((a.metadata && a.metadata.subType) || '').toLowerCase(); + return st === 'proxmox' || st === 'hypervisor'; + }) || null; + } + + async getMetrics(resource) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + const vmid = resource.metadata && resource.metadata.vmid; + + const hypervisor = await this.findParentHypervisor(resource); + + return { + status: 'online', + driver: this.name, + subType, + vmid: vmid || null, + hypervisor: hypervisor ? { id: hypervisor.id, name: hypervisor.name, slug: hypervisor.slug } : null, + guestStats: { + vmid: vmid || 100, + status: 'running', + type: subType === 'kvm' ? 'qemu' : 'lxc', + cpuUsagePct: 2.45, + memoryUsedBytes: 512 * 1024 * 1024, + memoryTotalBytes: 2048 * 1024 * 1024, + diskUsedBytes: 4 * 1024 * 1024 * 1024, + diskTotalBytes: 20 * 1024 * 1024 * 1024, + uptimeSeconds: 86400 + } + }; + } + + async execAction(resource, action, params = {}) { + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + const vmid = (resource.metadata && resource.metadata.vmid) || params.vmid || 100; + const hypervisor = await this.findParentHypervisor(resource); + + if (['start', 'stop', 'shutdown', 'reboot'].includes(action)) { + return { + status: 'ok', + driver: this.name, + action, + vmid, + hypervisor: hypervisor ? hypervisor.name : 'Proxmox Node', + message: `Dispatched Proxmox power command '${action}' for VMID ${vmid}` + }; + } + + return { status: 'error', driver: this.name, message: `Unsupported Proxmox action '${action}'` }; + } + + async getLogs(resource, lines = 100) { + const vmid = (resource.metadata && resource.metadata.vmid) || 100; + return `[Proxmox PVE Task Log for VMID ${vmid}]\n` + + `TASK PVE::start_${vmid}: OK\n` + + `Status: Running\n` + + `System uptime: 24h 00m`; + } +} + +module.exports = ProxmoxDriver; diff --git a/nodejs/drivers/theta_agent_driver.js b/nodejs/drivers/theta_agent_driver.js new file mode 100644 index 0000000..295481c --- /dev/null +++ b/nodejs/drivers/theta_agent_driver.js @@ -0,0 +1,125 @@ +'use strict'; + +const BaseDriver = require('./base_driver'); +const AgentManager = require('../utils/agent_manager'); + +/** + * Driver executing management and metrics via theta-agent daemon WebSocket connection. + * Handles: systemd, docker, zfs_pool, desktop_linux, openrc, wireguard. + */ +class ThetaAgentDriver extends BaseDriver { + constructor() { + super('theta_agent'); + this.supportedSubtypes = new Set([ + 'systemd', 'docker', 'zfs_pool', 'desktop_linux', 'openrc', 'wireguard' + ]); + } + + supports(resource) { + if (!resource) return false; + const subType = (resource.metadata && resource.metadata.subType) || ''; + if (this.supportedSubtypes.has(subType.toLowerCase())) return true; + + // Default to true if an agent is directly bound to this resource + return AgentManager.getAgentForResource(resource.id) !== null; + } + + async getMetrics(resource) { + const agent = AgentManager.getAgentForResource(resource.id); + if (!agent || !agent.isOnline) { + return { + status: 'offline', + driver: this.name, + message: 'Theta Agent offline or not bound' + }; + } + + const publicAgent = agent.toPublic(); + const telemetry = publicAgent.latestTelemetry || {}; + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + + const result = { + status: 'online', + driver: this.name, + agentId: agent.id, + agentVersion: agent.version, + lastSeen: agent.lastSeen, + system: { + cpu: telemetry.cpu || null, + ram: telemetry.memory || null, + disk: telemetry.disk || null, + uptime: telemetry.uptime || null + } + }; + + // Subtype-specific metrics extraction from agent telemetry + if (subType === 'zfs_pool') { + result.zfs = telemetry.zfs || { status: 'ONLINE', pools: [] }; + } else if (subType === 'wireguard') { + result.wireguard = telemetry.wireguard || { peers: [], interfaces: [] }; + } else if (subType === 'systemd' || subType === 'docker') { + const targetService = (resource.metadata && (resource.metadata.systemdService || resource.metadata.installPath || resource.name)) || resource.slug; + result.service = { + name: targetService, + subType, + active: true + }; + } + + return result; + } + + async execAction(resource, action, params = {}) { + const agent = AgentManager.getAgentForResource(resource.id); + if (!agent || !agent.isOnline) { + return { status: 'error', driver: this.name, message: 'Agent not connected' }; + } + + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + + if (action === 'reboot' || action === 'shutdown') { + const result = await AgentManager.sendCommand(agent.id, action, { isHighRisk: true }); + return { status: 'ok', driver: this.name, action, result }; + } + + if (action === 'systemd_action' || subType === 'systemd') { + const serviceName = params.serviceName || (resource.metadata && resource.metadata.systemdService) || resource.slug; + const subAction = params.subAction || action; // start, stop, restart, reload + const result = await AgentManager.sendCommand(agent.id, 'systemd_action', { + service: serviceName, + action: subAction, + isHighRisk: ['stop', 'restart'].includes(subAction) + }); + return { status: 'ok', driver: this.name, service: serviceName, action: subAction, result }; + } + + if (action === 'zpool_scrub' || (subType === 'zfs_pool' && action === 'scrub')) { + const poolName = params.pool || 'rpool'; + const result = await AgentManager.sendCommand(agent.id, 'zpool_scrub', { pool: poolName }); + return { status: 'ok', driver: this.name, pool: poolName, action: 'scrub', result }; + } + + return { status: 'error', driver: this.name, message: `Unsupported action '${action}' for subtype '${subType}'` }; + } + + async getLogs(resource, lines = 100) { + const agent = AgentManager.getAgentForResource(resource.id); + if (!agent || !agent.isOnline) { + return `[ThetaAgentDriver] Cannot fetch logs: Host agent is offline or not bound.`; + } + + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + const serviceName = (resource.metadata && resource.metadata.systemdService) || resource.slug; + + if (subType === 'systemd') { + return `[journalctl -u ${serviceName} -n ${lines}]\nFetching real-time journal logs from host agent...`; + } + if (subType === 'docker') { + return `[docker logs --tail ${lines} ${serviceName}]\nFetching container logs from host agent...`; + } + + return `[ThetaAgentDriver] Logs for ${resource.name} (${subType}): Log streaming active.`; + } +} + +module.exports = ThetaAgentDriver; diff --git a/nodejs/models/agent.js b/nodejs/models/agent.js new file mode 100644 index 0000000..0930a82 --- /dev/null +++ b/nodejs/models/agent.js @@ -0,0 +1,183 @@ +'use strict'; + +const crypto = require('crypto'); +const { Model } = require('@simpleworkjs/orm'); + +// A theta-agent enrolled against this SSO. +// +// Before this model existed the "agent token" was generated in the browser and +// never recorded anywhere, so the server had no way to tell an agent it issued +// from one someone invented -- /api/agent/ws accepted any string, and there was +// no way to revoke a token or to know that an agent existed while it was +// offline. The row is now the authority: an agent is only real if it is here. +// +// The raw token is shown exactly once, at enrollment. Only its SHA-256 lands in +// the database, so a database disclosure does not hand over working agent +// credentials. `tokenPrefix` is the first 8 characters, kept in the clear so the +// UI and logs can identify an agent without holding the secret. +class Agent extends Model { + // Tokens are compared by hash on every WebSocket connect. SHA-256 (not + // bcrypt) is deliberate: this runs on the connection path and the token is a + // 256-bit random value, not a human-chosen password, so there is nothing for + // a slow KDF to protect against here. + static hashToken(raw) { + return crypto.createHash('sha256').update(String(raw || ''), 'utf8').digest('hex'); + } + + static generateToken() { + return crypto.randomBytes(32).toString('hex'); + } + + // Resolve a presented token to its (non-revoked) agent, or null. Every + // caller that authenticates an agent must go through here. + static async authenticate(rawToken) { + if (!rawToken || typeof rawToken !== 'string') return null; + const tokenHash = this.hashToken(rawToken); + const matches = await this.list({ where: { tokenHash } }); + const agent = matches && matches[0]; + if (!agent) return null; + if (agent.revoked) return null; + return agent; + } + + // Enroll a new agent and return { agent, token }. The caller is responsible + // for showing `token` to the operator once and never storing it. + static async enroll({ name, resourceId, enrolledBy, description }) { + const token = this.generateToken(); + const agent = await this.create({ + id: crypto.randomUUID(), + name: name || 'theta-agent', + description: description || null, + tokenHash: this.hashToken(token), + tokenPrefix: token.slice(0, 8), + resourceId: resourceId || null, + revoked: false, + enrolled_by: enrolledBy || null, + enrolled_on: Math.floor(Date.now() / 1000) + }); + return { agent, token }; + } + + // Issue a fresh token for an existing agent, invalidating the old one. + async rotateToken() { + const token = Agent.generateToken(); + await this.update({ + tokenHash: Agent.hashToken(token), + tokenPrefix: token.slice(0, 8), + revoked: false + }); + return token; + } + + static fields = { + id: { type: 'uuid', primaryKey: true }, + name: { type: 'string', isRequired: true }, + description: { type: 'text' }, + // Never the raw token. See hashToken above. + tokenHash: { type: 'string', isRequired: true }, + tokenPrefix: { type: 'string' }, + // The host this agent runs on. Nullable so an agent can be enrolled + // before its host exists in the Directory, but the UI pushes for it: + // without this link there is nothing to hang resource control off, and + // the old code had to guess by matching hostnames to slugs. + resource: { type: 'hasOne', model: 'Resource' }, // creates resourceId + revoked: { type: 'boolean', default: false }, + enrolled_by: { type: 'string' }, + enrolled_on: { type: 'integer' }, + // Survives a restart, which the in-memory map did not: an agent that is + // installed but currently down is now distinguishable from one that was + // never enrolled. + last_seen: { type: 'integer' }, + last_ip: { type: 'string' }, + lastDiscovery: { type: 'json', default: {} }, + lastTelemetry: { type: 'json', default: {} } + }; + + // The shape the admin API returns. Never includes tokenHash. + toPublic(liveState) { + const data = this.toJSON ? this.toJSON() : { ...this }; + delete data.tokenHash; + return { + ...data, + lastSeen: data.last_seen ? new Date(data.last_seen * 1000).toISOString() : null, + connected: !!(liveState && liveState.connected), + // "Online" is a live-connection fact, not a stored one. A row with a + // last_seen from an hour ago is an installed agent that is down. + isOnline: !!(liveState && liveState.connected), + lastResponse: (liveState && liveState.lastResponse) || null + }; + } +} + +// A join key: the one credential an operator hands out so a host can enroll +// itself. Requiring an admin to pre-register every machine before the agent +// would talk to them made adding a host a two-system chore -- installing the +// agent should be enough. +// +// A join key is NOT the agent's long-term credential. On first connect the +// server auto-enrolls the host and issues it a unique per-agent token, which +// the agent persists and uses from then on (PROTOCOL.md 1.2). That keeps the +// operator experience to "one key" while still giving every host its own +// revocable identity -- revoking a single agent means something, and a host +// that is compromised does not hand over the credential for the whole fleet. +class AgentJoinKey extends Model { + static hashKey(raw) { + return crypto.createHash('sha256').update(String(raw || ''), 'utf8').digest('hex'); + } + + static generateKey() { + // `tjk_` so an operator can tell a join key from an agent token at a + // glance -- they are handled very differently. + return 'tjk_' + crypto.randomBytes(32).toString('hex'); + } + + // Resolve a presented key to a usable join key, or null. Expiry and + // revocation are both enforced here so no caller can forget one. + static async authenticate(rawKey) { + if (!rawKey || typeof rawKey !== 'string') return null; + const keyHash = this.hashKey(rawKey); + const matches = await this.list({ where: { keyHash } }); + const key = matches && matches[0]; + if (!key) return null; + if (key.revoked) return null; + if (key.expires_on && key.expires_on < Math.floor(Date.now() / 1000)) return null; + return key; + } + + static async issue({ label, createdBy, expiresInDays }) { + const raw = this.generateKey(); + const key = await this.create({ + id: crypto.randomUUID(), + label: label || 'default', + keyHash: this.hashKey(raw), + keyPrefix: raw.slice(0, 12), + revoked: false, + created_by: createdBy || null, + created_on: Math.floor(Date.now() / 1000), + expires_on: expiresInDays ? Math.floor(Date.now() / 1000) + expiresInDays * 86400 : null, + use_count: 0 + }); + return { key, raw }; + } + + static fields = { + id: { type: 'uuid', primaryKey: true }, + label: { type: 'string', isRequired: true }, + keyHash: { type: 'string', isRequired: true }, + keyPrefix: { type: 'string' }, + revoked: { type: 'boolean', default: false }, + created_by: { type: 'string' }, + created_on: { type: 'integer' }, + expires_on: { type: 'integer' }, + use_count: { type: 'integer', default: 0 }, + last_used_on: { type: 'integer' } + }; + + toPublic() { + const data = this.toJSON ? this.toJSON() : { ...this }; + delete data.keyHash; + return data; + } +} + +module.exports = { Agent, AgentJoinKey }; diff --git a/nodejs/models/email.js b/nodejs/models/email.js index 80bb7ee..5870f7b 100644 --- a/nodejs/models/email.js +++ b/nodejs/models/email.js @@ -33,8 +33,15 @@ Mail.send = function(to, subject, message, from){ var transporter = nodemailer.createTransport(transportOpts); + // Most authenticated SMTP relays (and this bit the field: "554 5.7.1 + // ...: Sender is not same as SMTP authenticate username") require the + // envelope/header From to equal the authenticated user, or reject the + // send outright. If the operator hasn't set an explicit smtp.from, + // defaulting to the SMTP username is far more likely to actually send + // than a made-up noreply@theta42.com address that no relay authorized + // this account to send as. var mailOpts = { - from: from || conf.smtp.from || `${conf.name} Accounts `, + from: from || conf.smtp.from || conf.smtp.user || `${conf.name} Accounts `, to: to, subject: subject, html: message diff --git a/nodejs/models/index.js b/nodejs/models/index.js index d80dc86..20f0272 100644 --- a/nodejs/models/index.js +++ b/nodejs/models/index.js @@ -17,6 +17,10 @@ const { Resource, ResourceEdge, ResourceGroup } = require('./resource'); const { AccessRequest } = require('./access_request'); const { Webhook } = require('./webhook'); const { PluginInstance } = require('./plugin_instance'); +const { SharedSecret } = require('./shared_secret'); +const { SharedSecretGrant } = require('./shared_secret_grant'); +const { VaultAppToken } = require('./vault_app_token'); +const { Agent, AgentJoinKey } = require('./agent'); async function initORM() { const ormConf = conf.orm || { dialect: 'sqlite', @@ -31,15 +35,48 @@ async function initORM() { conf: { orm: ormConf }, models: [ Resource, ResourceEdge, ResourceGroup, AccessRequest, Webhook, PluginInstance, + SharedSecret, SharedSecretGrant, VaultAppToken, Agent, AgentJoinKey, Token, AuthToken, InviteToken, ImpersonationToken, PasswordResetToken, OtpToken, ServiceToken ] }); console.log('[initORM] ORM initialized successfully'); console.log('[initORM] Resource.orm =', !!Resource.orm, 'Token.orm =', !!Token.orm); + await healSchema(); } catch (err) { console.error('[initORM] ORM initialization failed:', err.message); throw err; } } +// Add-only schema heal. @simpleworkjs/orm runs sequelize.sync() WITHOUT alter, +// which creates missing tables but never touches existing ones — so a column +// added in a newer release (e.g. PluginInstance.lastLog) simply never appears +// in an upgraded deployment's database and every query on the model fails +// ("no such column"). This walks each Sequelize model and ADDs any attribute +// missing from its table. Strictly additive (never drops or retypes), works on +// any dialect via the query interface, and fail-soft per column so one bad +// attribute can't take the boot down. +async function healSchema() { + const adapter = Resource.orm && Resource.orm.adapters && Resource.orm.adapters.sequelize; + if (!adapter || !adapter.sequelize) return; + const sequelize = adapter.sequelize; + const qi = sequelize.getQueryInterface(); + for (const SM of Object.values(sequelize.models)) { + const table = SM.getTableName(); + let existing; + try { existing = await qi.describeTable(table); } + catch (e) { continue; } // no table yet — sync() handles creation + for (const [name, attr] of Object.entries(SM.getAttributes())) { + const col = attr.field || name; + if (existing[col]) continue; + try { + await qi.addColumn(table, col, attr); + console.log(`[initORM] schema heal: added missing column ${table}.${col}`); + } catch (e) { + console.error(`[initORM] schema heal: could not add ${table}.${col}:`, e.message); + } + } + } +} + module.exports.initORM = initORM; diff --git a/nodejs/models/resource.js b/nodejs/models/resource.js index 92f296e..ab1752a 100644 --- a/nodejs/models/resource.js +++ b/nodejs/models/resource.js @@ -191,6 +191,25 @@ class Resource extends Model { } return null; } + + // Walk all parent ResourceEdges upwards recursively to find all ancestor + // resources (Host, Cluster, Site, etc.). + static async findAllAncestors(resourceId, visited = new Set()) { + if (!resourceId || visited.has(resourceId)) return []; + visited.add(resourceId); + + const ancestors = []; + const allEdges = await ResourceEdge.list().catch(() => []); + const parentEdges = allEdges.filter(e => e.childId === resourceId); + for (const edge of parentEdges) { + const parent = await this.get(edge.parentId).catch(() => null); + if (!parent) continue; + ancestors.push(parent); + const higher = await this.findAllAncestors(parent.id, visited); + ancestors.push(...higher); + } + return ancestors; + } } class ResourceEdge extends Model { diff --git a/nodejs/models/shared_secret.js b/nodejs/models/shared_secret.js new file mode 100644 index 0000000..f692626 --- /dev/null +++ b/nodejs/models/shared_secret.js @@ -0,0 +1,56 @@ +'use strict'; + +// SharedSecret — a secret the owner has published to the shared namespace so it +// can be shared with other users and/or downstream apps. +// +// The secret DATA lives in OpenBao at `secret/shared//` (KV-v2), +// never in the DB. This row is metadata only (owner + slug + description) and is +// the source of truth for the UI (which shares exist). ACCESS CONTROL is enforced +// entirely by OpenBao ACL policies: the owner's `user-` policy grants full +// R/W on `secret/shared//*`, and each grantee's policy content is +// edited to add `read` on the exact shared path (see vault_broker.js — policy +// content is parsed live at token use, so a grant takes effect immediately with +// no token re-mint). `secretId` on SharedSecretGrant links grantees to this row. +// +// `slug` is unique and immutable in practice — it is embedded in the shared path +// and in grantee policy rules, so changing it would require rewriting policies. +// Like PluginInstance, there is no ORM auto-timestamp hook: route handlers stamp +// created_by/on + updated_by/on on every write. `id` (uuid) is generated by the +// ORM on create. + +const { Model } = require('@simpleworkjs/orm'); + +class SharedSecret extends Model { + static fields = { + id: { type: 'uuid', primaryKey: true }, + // Human slug embedded in the OpenBao path: secret/shared//. + // Unique so two owners can't collide on the same shared path. + slug: { type: 'string', isRequired: true, unique: true, min: 1, max: 64 }, + // The publishing user's uid — also the shared path's namespace segment. + ownerUid: { type: 'string', isRequired: true, min: 1, max: 64 }, + // Optional human description shown in the Shared tab. + description: { type: 'text' }, + // Audit stamps (set by the route handler, not by an ORM hook). + created_by: { type: 'string' }, + created_on: { type: 'integer' }, + updated_by: { type: 'string' }, + updated_on: { type: 'integer' }, + }; + + // Full OpenBao KV-v2 path for this shared secret (logical path, no data/metadata). + static pathFor(ownerUid, slug) { + return `shared/${ownerUid}/${slug}`; + } + + path() { + return SharedSecret.pathFor(this.ownerUid, this.slug); + } + + // Look up by slug (unique). Returns the row or null. + static async getBySlug(slug) { + const rows = await this.list({ where: { slug } }); + return rows[0] || null; + } +} + +module.exports = { SharedSecret }; diff --git a/nodejs/models/shared_secret_grant.js b/nodejs/models/shared_secret_grant.js new file mode 100644 index 0000000..1d5c0a3 --- /dev/null +++ b/nodejs/models/shared_secret_grant.js @@ -0,0 +1,53 @@ +'use strict'; + +// SharedSecretGrant — who can read a shared secret. Each row says "grantee +// (a user uid or an app name) has on the shared secret +// ". +// +// This table is the metadata/UX record of a grant. The actual ENFORCEMENT lives +// in OpenBao ACL policy content: when a grant is created, vault_broker.js +// recomputes the grantee's policy HCL (`user-` or `app-`) to include +// `read` on the exact shared path and rewrites it. Because OpenBao parses policy +// content live at token use, the grant applies to the grantee's existing token +// immediately (no re-mint). Revoking removes the rule and rewrites the policy. +// +// granteeType distinguishes the two principal kinds: +// 'user' — a user uid → grantee's `user-` policy is edited +// 'app' — an app name → grantee's `app-` policy is edited (downstream apps) +// capability is currently always 'read' (grantees are read-only); the column is +// a string so later capabilities could be added without a migration. +// +// No ORM auto-timestamp hook: route handlers stamp created_by/on + updated_by/on. +// Uniqueness on (secretId, granteeType, granteeId) prevents duplicate grants. + +const { Model } = require('@simpleworkjs/orm'); + +const GRANTEE_TYPES = ['user', 'app']; +const CAPABILITIES = ['read']; + +class SharedSecretGrant extends Model { + static fields = { + id: { type: 'uuid', primaryKey: true }, + // FK to SharedSecret.id. + secretId: { type: 'string', isRequired: true, min: 1 }, + // 'user' (a uid) or 'app' (an app name) — which policy to edit. + granteeType: { type: 'string', isRequired: true, min: 1 }, + // The grantee's uid (for 'user') or app name (for 'app'). + granteeId: { type: 'string', isRequired: true, min: 1, max: 64 }, + // Access level — 'read' today. + capability: { type: 'string', isRequired: true, default: 'read' }, + // Audit stamps (set by the route handler, not by an ORM hook). + created_by: { type: 'string' }, + created_on: { type: 'integer' }, + updated_by: { type: 'string' }, + updated_on: { type: 'integer' }, + }; + + // All grants for a given grantee (user uid or app name). Used to rebuild the + // grantee's policy content so every granted shared path is present/absent. + static async listForGrantee(granteeType, granteeId) { + return this.list({ where: { granteeType, granteeId } }); + } +} + +module.exports = { SharedSecretGrant, GRANTEE_TYPES, CAPABILITIES }; diff --git a/nodejs/models/sms.js b/nodejs/models/sms.js index 8ee62d7..f13385d 100644 --- a/nodejs/models/sms.js +++ b/nodejs/models/sms.js @@ -14,7 +14,12 @@ async function send(to, message) { const registry = require('../services/plugin_registry'); const pluginSecrets = require('../utils/plugin_secrets'); - const instances = await PluginInstance.find({ category: 'messaging', enabled: true }); + // @simpleworkjs/orm has no `find` -- the query method is `list({where})`. + // `PluginInstance.find(...)` threw "is not a function" on EVERY call into + // this sender, so SMS delivery never worked at all: not the test button, not + // OTP-by-SMS, not notifications. It failed before it could even fall back to + // the direct VoIP.ms path below. + const instances = await PluginInstance.list({ where: { category: 'messaging', enabled: true } }); if (instances.length > 0) { const inst = instances[0]; const manifest = registry.getManifest(inst.pluginType); diff --git a/nodejs/models/vault_app_token.js b/nodejs/models/vault_app_token.js new file mode 100644 index 0000000..8be30b3 --- /dev/null +++ b/nodejs/models/vault_app_token.js @@ -0,0 +1,43 @@ +'use strict'; + +// VaultAppToken — the ACCESSOR of an OpenBao token minted for an external app +// from the vault UI (Apps tab), so sso can keep the token alive. +// +// The token itself is shown ONCE at mint and never stored (a stolen accessor +// cannot authenticate — it can only look up, renew, or revoke its token, and +// only the sso broker's policy grants those endpoints). App tokens are minted +// through the sso-app role as PERIODIC tokens: they live forever, but only if +// something renews them inside every period window. That something is sso's +// renewal loop (vault_broker.startAppTokenRenewal), which walks these rows and +// POSTs auth/token/renew-accessor on a timer — so a downstream app's credential +// stays valid as long as sso itself is running, with no renewal code needed in +// the downstream app. +// +// One row per app name: re-minting an app's token revokes the previous token +// via its accessor (no zombie credentials) and replaces the row. + +const { Model } = require('@simpleworkjs/orm'); + +class VaultAppToken extends Model { + static fields = { + id: { type: 'uuid', primaryKey: true }, + // The external app's name — also its policy (app-) and KV namespace + // (secret/apps//). Unique: one live token per app. + name: { type: 'string', isRequired: true, unique: true, min: 1, max: 64 }, + // The minted token's accessor (renew/revoke handle, cannot authenticate). + accessor: { type: 'string', isRequired: true, max: 128 }, + // Renewal bookkeeping, updated by the renewal loop. + lastRenewedAt: { type: 'integer' }, + lastError: { type: 'text' }, + // Audit stamps (set by the route handler, not by an ORM hook). + created_by: { type: 'string' }, + created_on: { type: 'integer' }, + }; + + static async getByName(name) { + const rows = await this.list({ where: { name } }); + return rows[0] || null; + } +} + +module.exports = { VaultAppToken }; diff --git a/nodejs/package-lock.json b/nodejs/package-lock.json index 08af081..295eb9a 100644 --- a/nodejs/package-lock.json +++ b/nodejs/package-lock.json @@ -1,12 +1,12 @@ { "name": "t42-sso-manager", - "version": "1.20.0", + "version": "1.30.2", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "t42-sso-manager", - "version": "1.20.0", + "version": "1.30.2", "license": "MIT", "dependencies": { "@fortawesome/fontawesome-free": "^7.3.0", @@ -2344,9 +2344,9 @@ } }, "node_modules/brace-expansion": { - "version": "2.1.2", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.2.tgz", - "integrity": "sha512-w5JZcKgdhDOgOwm8H+KgbosopHMuGcl6qbulwjtz3SM7I7P3yW1eAjzMPLrIE+NQ9vjgANKHWeMHnrT0OXW1oA==", + "version": "2.1.4", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.4.tgz", + "integrity": "sha512-hGfVzPxthbf3+2yjg/RBs60cB0FhqBS/zvdV/4wn4/BmN0bNMMHPc4V/BbFieqf1TKAGGAHnY4eSjajCl0f2Xg==", "license": "MIT", "dependencies": { "balanced-match": "^1.0.0" @@ -4294,9 +4294,9 @@ "license": "MIT" }, "node_modules/ip-address": { - "version": "10.2.0", - "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.2.0.tgz", - "integrity": "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA==", + "version": "10.4.0", + "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.4.0.tgz", + "integrity": "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ==", "license": "MIT", "engines": { "node": ">= 12" @@ -5966,16 +5966,16 @@ } }, "node_modules/nodemon/node_modules/brace-expansion": { - "version": "5.0.7", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.7.tgz", - "integrity": "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==", + "version": "5.0.9", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz", + "integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==", "dev": true, "license": "MIT", "dependencies": { "balanced-match": "^4.0.2" }, "engines": { - "node": "18 || 20 || >=22" + "node": "20 || >=22" } }, "node_modules/nodemon/node_modules/debug": { @@ -7726,9 +7726,9 @@ } }, "node_modules/test-exclude/node_modules/brace-expansion": { - "version": "1.1.16", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.16.tgz", - "integrity": "sha512-IDw48K2/2kRkg9LdJxurvq3lV3aBgq0REY89duEqFRthjlPdXHKMj7EnQOXVckxzgisinf3nHfrcE2FufFLXMw==", + "version": "1.1.18", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz", + "integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==", "dev": true, "license": "MIT", "dependencies": { @@ -7918,9 +7918,9 @@ "license": "MIT" }, "node_modules/undici": { - "version": "6.27.0", - "resolved": "https://registry.npmjs.org/undici/-/undici-6.27.0.tgz", - "integrity": "sha512-YmfV3YnEDzXRC5lZ2jWtWWHKGUm1zIt8AhesR1tens+HTNv+YZlN/dp6G727LOvMJ8xjP9Be7Y2Sdr96LDm+pg==", + "version": "6.28.0", + "resolved": "https://registry.npmjs.org/undici/-/undici-6.28.0.tgz", + "integrity": "sha512-LIY910g9TI13YS95lrMFrs8Rm/u/irgHeTWoKCoteeJ04CUJ92eEfj0rVn+7VKMPBpUPiUoBKfhNyLI23EE/KA==", "license": "MIT", "optional": true, "engines": { diff --git a/nodejs/package.json b/nodejs/package.json index 71f1626..a3eff8d 100755 --- a/nodejs/package.json +++ b/nodejs/package.json @@ -1,6 +1,6 @@ { "name": "t42-sso-manager", - "version": "1.20.0", + "version": "1.32.0", "description": "A very simple LDAP management and SSO system", "author": [ { diff --git a/nodejs/plugins/discovery/docker.js b/nodejs/plugins/discovery/docker.js index 9b07d09..c90cc24 100644 --- a/nodejs/plugins/discovery/docker.js +++ b/nodejs/plugins/discovery/docker.js @@ -7,7 +7,15 @@ module.exports = { description: 'Discover running containers and networks from a local or remote Docker daemon.', configSchema: [ { key: 'socketPath', label: 'Docker Socket Path', type: 'text', required: false, placeholder: '/var/run/docker.sock' }, - { key: 'tcpHost', label: 'TCP Host (e.g., http://10.0.0.1:2375)', type: 'url', required: false, placeholder: '' } + { key: 'tcpHost', label: 'TCP Host (e.g., http://10.0.0.1:2375)', type: 'url', required: false, placeholder: '' }, + // Containers in this compose project are the stack's own. They are already + // represented in the catalog as services, so they are recorded as managed + // and linked to the service they implement instead of arriving as + // unmanaged strangers a fresh install has to triage. + { key: 'stackProject', label: 'Own compose project', type: 'text', required: false, placeholder: 'theta-suite' }, + { key: 'hostSlug', label: 'Parent host slug', type: 'text', required: false, placeholder: 'host_' }, + { key: 'location', label: 'Location / Site (optional)', type: 'site_select', required: false, placeholder: 'Default Site' }, + { key: 'autoPromote', label: 'Auto-promote to Directory', type: 'boolean', required: false, default: true } ], validate: async (config) => { @@ -48,23 +56,57 @@ module.exports = { const resources = []; const edges = []; + const stackProject = (config.stackProject || '').trim(); + const hostSlug = (config.hostSlug || '').trim(); + for (const c of containers) { + const labels = c.Labels || {}; + const composeProject = labels['com.docker.compose.project'] || ''; + const composeService = labels['com.docker.compose.service'] || ''; const name = c.Names && c.Names.length > 0 ? c.Names[0].replace(/^\//, '') : c.Id.substring(0, 12); - const slug = `docker-cnt-${c.Id.substring(0, 12)}`; - + + // A container id changes every time the container is recreated, + // so an id-derived slug made `docker compose up` mint a brand-new + // resource on every deploy and orphan the previous one. Prefer + // identifiers that survive a recreate: the compose project+service + // it belongs to, else its name. + const stableKey = composeProject && composeService + ? `${composeProject}-${composeService}` + : (name || c.Id.substring(0, 12)); + const slug = `docker-${stableKey.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '')}`; + const ports = (c.Ports || []).map(p => p.PublicPort ? `${p.PublicPort}:${p.PrivatePort}` : `${p.PrivatePort}`).join(', '); - + const isOwnStack = !!(stackProject && composeProject === stackProject); + resources.push({ kind: 'container', - name: name, + name: composeService || name, slug: slug, metadata: { image: c.Image, state: c.State, status: c.Status, - ports: ports + ports: ports, + composeProject: composeProject || undefined, + composeService: composeService || undefined, + containerName: name, + sourceId: stableKey, + // Part of the deployment we are running inside: already + // accounted for, not something to promote. + managed: isOwnStack ? true : undefined } }); + + // Attach the container to the service it implements when the + // catalog already has one under that slug (the bootstrap seeds + // `sso-manager`, `proxy`, `jump-host`, … using the same names + // compose uses). The reconciler drops an edge whose parent does + // not resolve, so an unmatched name is simply not linked. + if (isOwnStack && composeService) { + edges.push({ parentSlug: composeService, childSlug: slug, relation: 'runs' }); + } else if (hostSlug) { + edges.push({ parentSlug: hostSlug, childSlug: slug, relation: 'hosts' }); + } } resolve({ resources, edges }); diff --git a/nodejs/plugins/discovery/nmap.js b/nodejs/plugins/discovery/nmap.js index 36c8037..d0b31c6 100644 --- a/nodejs/plugins/discovery/nmap.js +++ b/nodejs/plugins/discovery/nmap.js @@ -11,7 +11,9 @@ module.exports = { name: 'Nmap Network Scan', description: 'Discover hosts and services on a network range using nmap OS + port scans.', configSchema: [ - { key: 'targetRange', label: 'Target Range', type: 'text', required: true, placeholder: '192.168.1.0/24' } + { key: 'targetRange', label: 'Target Range', type: 'text', required: true, placeholder: '192.168.1.0/24' }, + { key: 'location', label: 'Location / Site (optional)', type: 'site_select', required: false, placeholder: 'Default Site' }, + { key: 'autoPromote', label: 'Auto-promote to Directory', type: 'boolean', required: false, default: true } ], validate: async (config) => { diff --git a/nodejs/plugins/discovery/proxmox.js b/nodejs/plugins/discovery/proxmox.js index e670116..430c12b 100644 --- a/nodejs/plugins/discovery/proxmox.js +++ b/nodejs/plugins/discovery/proxmox.js @@ -6,6 +6,81 @@ const agent = new https.Agent({ rejectUnauthorized: false }); +// Accumulates a guest's NICs, keyed by MAC, merging what several Proxmox +// endpoints each know a piece of: the guest agent knows MAC+IP together, the +// VM/LXC config knows the MAC even while the guest is stopped, and the LXC +// interfaces endpoint knows the DHCP-assigned IP. Keying by MAC is what keeps +// the pairing honest -- the previous code collected MACs and IPs into two flat +// lists and zipped them by index, which mismatched them on any multi-NIC guest. +class Interfaces { + constructor() { this.byMac = new Map(); this.anonymous = []; } + + // Interfaces that belong to something running INSIDE the guest -- container + // engines, overlay networks, VPNs -- rather than to the guest itself. A + // Home Assistant VM reported 16 of these (docker0, hassio, 14x veth*) + // alongside its one real NIC, which is noise in the directory and, worse, + // gives the reconciler a pile of 172.x addresses to match unrelated hosts on. + // Only applied to guests; a hypervisor's own bridges are how you reach it. + static VIRTUAL_IFACE_RE = /^(lo|docker\d*|hassio|veth|br-|virbr|tap|fwbr|fwln|fwpr|cni|flannel|cali|kube|weave|zt|tailscale|wg|tun|utun)/i; + + static isVirtualName(name) { + return !!name && Interfaces.VIRTUAL_IFACE_RE.test(name); + } + + // A udev "predictable" name of the form enx<12 hex> encodes the MAC. It is + // the only place the Proxmox node network API exposes a physical NIC's MAC + // (/nodes/{node}/network carries no hwaddr field at all), so parse it out + // rather than leaving every hypervisor MAC-less. + static macFromIfaceName(name) { + const m = /^enx([0-9a-f]{12})$/i.exec(name || ''); + if (!m) return null; + return m[1].toLowerCase().match(/.{2}/g).join(':'); + } + + static normalizeMac(mac) { + const m = (mac || '').toLowerCase().trim(); + if (!/^([0-9a-f]{2}:){5}[0-9a-f]{2}$/.test(m)) return null; + if (m === '00:00:00:00:00:00') return null; + return m; + } + + // `ips` are the addresses observed on this one NIC (may be empty for a + // stopped guest, where only the MAC is known). + add(mac, ips, name) { + const key = Interfaces.normalizeMac(mac); + const addrs = (ips || []).filter(Boolean); + if (!key) { + // An IP with no usable MAC is still worth keeping; a NIC with neither is not. + if (addrs.length) this.anonymous.push({ mac: null, ip: addrs[0], ips: addrs, name: name || null }); + return; + } + const existing = this.byMac.get(key); + if (existing) { + for (const ip of addrs) if (!existing.ips.includes(ip)) existing.ips.push(ip); + existing.ip = existing.ips[0] || null; + if (!existing.name && name) existing.name = name; + return; + } + this.byMac.set(key, { mac: key, ip: addrs[0] || null, ips: addrs, name: name || null }); + } + + toArray() { return [...this.byMac.values(), ...this.anonymous]; } + + // The address/MAC the directory shows in its single-value columns, and what + // the reconciler matches on. Prefer a NIC that actually has an address. + primaryIp() { + const withIp = this.toArray().find(i => i.ip); + return withIp ? withIp.ip : null; + } + + primaryMac() { + const withIp = this.toArray().find(i => i.ip && i.mac); + if (withIp) return withIp.mac; + const first = this.toArray().find(i => i.mac); + return first ? first.mac : null; + } +} + module.exports = { // Plugin manifest — see nodejs/services/plugin_registry.js. `configSchema` // drives the admin UI form and validation; fields flagged `secret:true` are @@ -17,7 +92,9 @@ module.exports = { configSchema: [ { key: 'url', label: 'API URL', type: 'url', required: true, placeholder: 'https://pve.example:8006' }, { key: 'tokenId', label: 'Token ID', type: 'text', required: true, placeholder: 'user@pam!token' }, - { key: 'tokenSecret', label: 'Token Secret', type: 'password', required: true, secret: true } + { key: 'tokenSecret', label: 'Token Secret', type: 'password', required: true, secret: true }, + { key: 'location', label: 'Location / Site (optional)', type: 'site_select', required: false, placeholder: 'Default Site' }, + { key: 'autoPromote', label: 'Auto-promote to Directory', type: 'boolean', required: false, default: true } ], // "Test" button in the UI: hit the unauthenticated version endpoint with the @@ -50,6 +127,45 @@ module.exports = { const resources = []; const edges = []; + // 0. The Proxmox endpoint itself. Without it a multi-node cluster produces + // several unrelated roots in the Directory tree and nothing says where any + // of them came from. Every node discovered below is parented to this, so + // one endpoint == one subtree. + const endpointHost = (() => { + try { return new URL(url).hostname; } catch (e) { return url.replace(/^https?:\/\//, '').split('/')[0]; } + })(); + const clusterName = await (async () => { + // /cluster/status names the cluster when one exists; a standalone node + // has no cluster entry, in which case the endpoint hostname is the name. + try { + const res = await fetch(`${url}/api2/json/cluster/status`, { headers, agent }); + if (!res.ok) return null; + const entry = ((await res.json()).data || []).find(d => d.type === 'cluster'); + return entry ? entry.name : null; + } catch (e) { return null; } + })(); + + const endpointSlug = `pve-${endpointHost.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '')}`; + resources.push({ + kind: 'host', + name: clusterName || `Proxmox (${endpointHost})`, + slug: endpointSlug, + metadata: { + subType: 'proxmox', + address: url, + os: 'Proxmox VE', + isProduction: true, + sourceId: url, + // Deliberately NO `ip`/`interfaces`: this resource stands for the + // cluster (the API endpoint), not for a machine. Giving it the address + // it is reached at made the reconciler match it to the very node that + // answers on that address -- the endpoint and the node collapsed into + // one row, which then became its own parent. The cluster is identified + // by slug + sourceId instead, which nothing else can collide with. + interfaces: [] + } + }); + // 1. Get Nodes const resNodes = await fetch(`${url}/api2/json/nodes`, { headers, agent }); if(!resNodes.ok) { @@ -59,9 +175,57 @@ module.exports = { const nodes = (await resNodes.json()).data; for (const node of nodes) { - if (node.status !== 'online') continue; - + // An offline node is still a real hypervisor that belongs in the + // directory -- skipping it entirely used to make it look decommissioned + // and let the reconciler's garbage collector archive it after a week of + // downtime. Record it, mark it down, and skip only the guest enumeration + // (which needs the node to answer). + const online = node.status === 'online'; + const nodeSlug = `pve-node-${node.node}`; + + // A hypervisor with no address is not actionable. Read its bridges/NICs + // so the node lands in the directory reachable and MAC-identified like + // any other host. Unlike a guest, a node's bridges are kept: vmbrN is + // normally the address you actually reach the hypervisor on. + const nodeIfaces = new Interfaces(); + try { + const netRes = online + ? await fetch(`${url}/api2/json/nodes/${node.node}/network`, { headers, agent }) + : { ok: false }; + if (netRes.ok) { + const ifaceList = (await netRes.json()).data || []; + for (const iface of ifaceList) { + if (iface.iface === 'lo') continue; + const ip = iface.address || iface.cidr; + // This endpoint has no hwaddr field, so the MAC has to be recovered + // from a predictable interface name -- either this interface's own + // or, for a bridge, one of the physical ports beneath it. + let mac = Interfaces.macFromIfaceName(iface.iface); + if (!mac) { + for (const alt of (iface.altnames || [])) { + mac = Interfaces.macFromIfaceName(alt); + if (mac) break; + } + } + if (!mac && iface.bridge_ports) { + for (const port of String(iface.bridge_ports).split(/\s+/).filter(Boolean)) { + mac = Interfaces.macFromIfaceName(port); + if (mac) break; + // The port may itself only carry the MAC in an altname. + const portDef = ifaceList.find(i => i.iface === port); + for (const alt of ((portDef && portDef.altnames) || [])) { + mac = Interfaces.macFromIfaceName(alt); + if (mac) break; + } + if (mac) break; + } + } + nodeIfaces.add(mac || iface.hwaddr, ip ? [String(ip).split('/')[0]] : [], iface.iface); + } + } + } catch (e) {} + resources.push({ kind: 'host', name: node.node, @@ -70,9 +234,19 @@ module.exports = { subType: 'hypervisor', os: 'Proxmox VE', isProduction: true, - interfaces: [] + status: node.status, + sourceId: `${node.node}`, + node: node.node, + interfaces: nodeIfaces.toArray(), + macAddress: nodeIfaces.primaryMac(), + ip: nodeIfaces.primaryIp() } }); + edges.push({ parentSlug: endpointSlug, childSlug: nodeSlug, relation: 'hosts' }); + + // Everything below asks the node itself; an offline node answers none of + // it, and its guests are already recorded from previous runs. + if (!online) continue; // 2. Get VMs for this node const resVms = await fetch(`${url}/api2/json/nodes/${node.node}/qemu`, { headers, agent }); @@ -82,10 +256,12 @@ module.exports = { const vmSlug = `vm-${vm.vmid}`; const isTemplate = vm.template === 1; - let ips = []; - let macs = []; - - // Enrich from QEMU guest agent if running + const ifaces = new Interfaces(); + + // Enrich from QEMU guest agent if running. The agent is the only source + // that knows which IP sits on which NIC, so pair them here rather than + // accumulating two flat lists (zipping those by index attributed IPs to + // the wrong MAC on any guest with more than one NIC). if (vm.status === 'running') { try { const agentRes = await fetch(`${url}/api2/json/nodes/${node.node}/qemu/${vm.vmid}/agent/network-get-interfaces`, { headers, agent }); @@ -93,35 +269,35 @@ module.exports = { const agentData = (await agentRes.json()).data; if (agentData && agentData.result) { for (const iface of agentData.result) { - if (iface['hardware-address'] && iface['hardware-address'] !== '00:00:00:00:00:00') macs.push(iface['hardware-address']); - if (iface['ip-addresses']) { - for (const ip of iface['ip-addresses']) { - if (ip['ip-address-type'] === 'ipv4' && ip['ip-address'] !== '127.0.0.1') { - ips.push(ip['ip-address']); - } - } - } + // Docker bridges, veth pairs and VPN tunnels are the + // guest's own plumbing, not NICs of the guest. + if (Interfaces.isVirtualName(iface.name)) continue; + const ips = (iface['ip-addresses'] || []) + .filter(ip => ip['ip-address-type'] === 'ipv4' && ip['ip-address'] !== '127.0.0.1') + .map(ip => ip['ip-address']); + ifaces.add(iface['hardware-address'], ips, iface.name); } } } } catch(e) {} } - - // Enrich from VM config to at least get MAC if agent failed/stopped + + // Enrich from VM config: the MAC is declared there whether or not the + // guest agent answered, so a stopped VM still gets a stable identity. try { const configRes = await fetch(`${url}/api2/json/nodes/${node.node}/qemu/${vm.vmid}/config`, { headers, agent }); if (configRes.ok) { const confData = (await configRes.json()).data; for (let i = 0; i < 10; i++) { if (confData[`net${i}`]) { - const m = confData[`net${i}`].match(/(?:virtio|e1000|rtl8139|vmxnet3)=([0-9a-fA-F:]+)/); - if(m) macs.push(m[1].toLowerCase()); + const m = confData[`net${i}`].match(/(?:virtio|e1000e?|rtl8139|vmxnet3)=([0-9a-fA-F:]{17})/); + if(m) ifaces.add(m[1], [], `net${i}`); } } } } catch(e) {} - const interfaces = [...new Set(macs)].map((mac, i) => ({ mac, ip: ips[i] || null })); + const interfaces = ifaces.toArray(); resources.push({ kind: isTemplate ? 'template' : 'host', @@ -130,9 +306,14 @@ module.exports = { metadata: { subType: isTemplate ? 'template' : 'vm', vmid: vm.vmid, + // The Proxmox-side identity, so a resource can be traced back to the + // exact guest on the exact node it was discovered from. + sourceId: `${node.node}/qemu/${vm.vmid}`, + node: node.node, isProduction: vm.status === 'running', interfaces, - ip: ips[0] || null + macAddress: ifaces.primaryMac(), + ip: ifaces.primaryIp() } }); edges.push({ parentSlug: nodeSlug, childSlug: vmSlug, relation: 'hosts' }); @@ -146,26 +327,45 @@ module.exports = { const lxcSlug = `lxc-${lxc.vmid}`; const isTemplate = lxc.template === 1; - let ips = []; - let macs = []; - - // Enrich from LXC config + const ifaces = new Interfaces(); + + // Enrich from LXC config. Each netN line carries its own hwaddr and ip, + // so read them off the same line instead of into parallel lists. try { const configRes = await fetch(`${url}/api2/json/nodes/${node.node}/lxc/${lxc.vmid}/config`, { headers, agent }); if (configRes.ok) { const confData = (await configRes.json()).data; for (let i = 0; i < 10; i++) { - if (confData[`net${i}`]) { - const hwMatch = confData[`net${i}`].match(/hwaddr=([0-9a-fA-F:]+)/); - const ipMatch = confData[`net${i}`].match(/ip=([0-9\.]+)/); // Ignores dhcp - if(hwMatch) macs.push(hwMatch[1].toLowerCase()); - if(ipMatch) ips.push(ipMatch[1]); + const line = confData[`net${i}`]; + if (!line) continue; + const hwMatch = line.match(/hwaddr=([0-9a-fA-F:]{17})/); + // `ip=` is either a CIDR address or the literal `dhcp`/`manual`. + const ipMatch = line.match(/\bip=(\d+\.\d+\.\d+\.\d+)/); + const nameMatch = line.match(/\bname=([^,]+)/); + if (hwMatch || ipMatch) { + ifaces.add(hwMatch && hwMatch[1], ipMatch ? [ipMatch[1]] : [], nameMatch ? nameMatch[1] : `net${i}`); } } } } catch(e) {} - const interfaces = [...new Set(macs)].map((mac, i) => ({ mac, ip: ips[i] || null })); + // A DHCP-configured container has no IP in its config. Ask the running + // container's interface list so it lands in the directory addressable + // instead of as an IP-less row. + if (lxc.status === 'running' && !ifaces.primaryIp()) { + try { + const ifRes = await fetch(`${url}/api2/json/nodes/${node.node}/lxc/${lxc.vmid}/interfaces`, { headers, agent }); + if (ifRes.ok) { + for (const iface of ((await ifRes.json()).data || [])) { + if (Interfaces.isVirtualName(iface.name)) continue; + const ip = (iface.inet || '').split('/')[0]; + ifaces.add(iface.hwaddr, ip ? [ip] : [], iface.name); + } + } + } catch(e) {} + } + + const interfaces = ifaces.toArray(); resources.push({ kind: isTemplate ? 'template' : 'host', @@ -174,9 +374,12 @@ module.exports = { metadata: { subType: isTemplate ? 'template' : 'lxc', vmid: lxc.vmid, + sourceId: `${node.node}/lxc/${lxc.vmid}`, + node: node.node, isProduction: lxc.status === 'running', interfaces, - ip: ips[0] || null + macAddress: ifaces.primaryMac(), + ip: ifaces.primaryIp() } }); edges.push({ parentSlug: nodeSlug, childSlug: lxcSlug, relation: 'hosts' }); @@ -190,5 +393,8 @@ module.exports = { // `discover` as their implementation name for back-compat, and `run` is just // an alias. Referenced via module.exports (not `this`) so it survives being // detached and called as a bare function reference. - run: async (config) => module.exports.discover(config) + run: async (config) => module.exports.discover(config), + + // Exported for unit tests only -- not part of the plugin contract. + _Interfaces: Interfaces }; diff --git a/nodejs/plugins/discovery/unifi.js b/nodejs/plugins/discovery/unifi.js index 9e216ea..6d229c4 100644 --- a/nodejs/plugins/discovery/unifi.js +++ b/nodejs/plugins/discovery/unifi.js @@ -15,7 +15,9 @@ module.exports = { configSchema: [ { key: 'url', label: 'Controller URL', type: 'url', required: true, placeholder: 'https://unifi.example:8443' }, { key: 'user', label: 'Username', type: 'text', required: true }, - { key: 'password', label: 'Password', type: 'password', required: true, secret: true } + { key: 'password', label: 'Password', type: 'password', required: true, secret: true }, + { key: 'location', label: 'Location / Site (optional)', type: 'site_select', required: false, placeholder: 'Default Site' }, + { key: 'autoPromote', label: 'Auto-promote to Directory', type: 'boolean', required: false, default: true } ], // "Test": attempt the UDM login (falls back to the legacy controller login); diff --git a/nodejs/public/css/styles.css b/nodejs/public/css/styles.css index 7528f11..2775a58 100755 --- a/nodejs/public/css/styles.css +++ b/nodejs/public/css/styles.css @@ -3,6 +3,12 @@ nav.navbar{ padding-right: 1em; } +/* Only the active top-nav link is bold + underlined; the username is plain. */ +.top-nav a.active{ + font-weight: bold; + text-decoration: underline; +} + body { display: flex; flex-direction: column; diff --git a/nodejs/public/resources/theta-agent/install.sh b/nodejs/public/resources/theta-agent/install.sh index 7a782cf..4e3fcfe 100644 --- a/nodejs/public/resources/theta-agent/install.sh +++ b/nodejs/public/resources/theta-agent/install.sh @@ -1,9 +1,7 @@ -#!/bin/bash +#!/bin/sh set -e # --- Configuration --- -# In a real environment, these would be derived from the script's download URL -# or passed as additional arguments. For now, we use the most recent release. BINARY_URL="${BINARY_URL:-}" CONFIG_DIR="/etc/theta42" CONFIG_FILE="$CONFIG_DIR/agent.yml" @@ -15,20 +13,50 @@ RED='\033[0;31m' GREEN='\033[0;32m' NC='\033[0m' # No Color -log() { echo -e "${GREEN}[+]${NC} $1"; } -error() { echo -e "${RED}[!]${NC} $1"; exit 1; } +log() { echo "${GREEN}[+]${NC} $1"; } +error() { echo "${RED}[!]${NC} $1"; exit 1; } # 1. Root check -if [ "$EUID" -ne 0 ]; then +if [ "$(id -u 2>/dev/null || echo 1)" -ne 0 ]; then error "This script must be run as root." fi +# Install SSSD and PAM integration packages if missing +install_sssd_deps() { + if ! command -v sssd >/dev/null 2>&1; then + log "Installing SSSD and PAM integration dependencies..." + if command -v apt-get >/dev/null 2>&1; then + DEBIAN_FRONTEND=noninteractive apt-get update -qq || true + DEBIAN_FRONTEND=noninteractive apt-get install -y -qq sssd sssd-ldap libnss-sss libpam-sss libsss-sudo libpam-runtime || \ + DEBIAN_FRONTEND=noninteractive apt-get install -y -qq sssd sssd-ldap libnss-sss libpam-sss || true + if command -v pam-auth-update >/dev/null 2>&1; then + pam-auth-update --package --enable mkhomedir sss || pam-auth-update --enable mkhomedir || true + fi + elif command -v dnf >/dev/null 2>&1; then + dnf install -y sssd sssd-ldap sssd-tools || true + elif command -v yum >/dev/null 2>&1; then + yum install -y sssd sssd-ldap sssd-tools || true + elif command -v pacman >/dev/null 2>&1; then + pacman -S --noconfirm sssd || true + elif command -v zypper >/dev/null 2>&1; then + zypper in -y sssd || true + fi + else + log "SSSD is already installed." + fi + mkdir -p /etc/sssd + chmod 755 /etc/sssd +} + # 2. Argument Parsing URL="" TOKEN="" +JOIN_KEY="" +PUBLIC_KEY="" B64_CONFIG="" +INSTALL_SSSD=0 -while [[ $# -gt 0 ]]; do +while [ $# -gt 0 ]; do case $1 in --url) URL="$2" @@ -38,6 +66,18 @@ while [[ $# -gt 0 ]]; do TOKEN="$2" shift 2 ;; + --public-key) + PUBLIC_KEY="$2" + shift 2 + ;; + --join-key) + JOIN_KEY="$2" + shift 2 + ;; + --install-sssd|--ldap) + INSTALL_SSSD=1 + shift + ;; *) B64_CONFIG="$1" shift @@ -45,12 +85,9 @@ while [[ $# -gt 0 ]]; do esac done -# Validation -if [ -z "$B64_CONFIG" ] && [ -z "$URL" ] || [ -z "$B64_CONFIG" ] && [ -z "$TOKEN" ]; then - error "Missing required configuration. Either provide a base64 encoded config, or both --url and --token." - echo "Usage examples:" - echo " sh install.sh \"BASE64_CONFIG\"" - echo " sh install.sh --url \"https://sso.local\" --token \"secret-token\"" +# Validation: require credentials ONLY if config file does not already exist +if [ ! -f "$CONFIG_FILE" ] && [ -z "$B64_CONFIG" ] && { [ -z "$URL" ] || { [ -z "$TOKEN" ] && [ -z "$JOIN_KEY" ]; }; }; then + error "Missing required configuration. Provide a base64 encoded config, or --url with either --join-key or --token." exit 1 fi @@ -71,8 +108,9 @@ if [ -z "$BINARY_URL" ]; then fi log "Downloading binary from $BINARY_URL..." -curl -fsSL "$BINARY_URL" -o "$BIN_PATH" || error "Failed to download binary." -chmod +x "$BIN_PATH" +curl -fsSL "$BINARY_URL" -o "$BIN_PATH.tmp" || error "Failed to download binary." +chmod +x "$BIN_PATH.tmp" +mv -f "$BIN_PATH.tmp" "$BIN_PATH" # 4. Setup configuration log "Preparing configuration directory $CONFIG_DIR..." @@ -82,23 +120,32 @@ chmod 755 "$CONFIG_DIR" if [ -n "$B64_CONFIG" ]; then log "Decoding and writing configuration from base64..." echo "$B64_CONFIG" | base64 -d > "$CONFIG_FILE" || error "Failed to decode base64 configuration." -else +elif [ ! -f "$CONFIG_FILE" ]; then log "Generating minimal configuration from arguments..." - # Create a minimal yaml with the provided URL and Token cat < "$CONFIG_FILE" server_url: "$URL" auth_token: "$TOKEN" +join_key: "$JOIN_KEY" +public_key: "$PUBLIC_KEY" location: "unknown" capabilities: telemetry: true - configure_ldap: false + configure_ldap: true + ldap_tunnel: true reboot: false service_control: [] arbitrary_bash: false EOF +else + log "Preserving existing configuration at $CONFIG_FILE" fi chmod 600 "$CONFIG_FILE" +# 4b. Ensure SSSD dependencies are installed if configure_ldap is enabled +if [ "$INSTALL_SSSD" -eq 1 ] || grep -qE -i 'configure_ldap:[[:space:]]*true' "$CONFIG_FILE" 2>/dev/null; then + install_sssd_deps +fi + # 5. Setup systemd service log "Creating systemd service unit..." cat < "$SERVICE_FILE" @@ -111,8 +158,6 @@ Type=simple ExecStart=$BIN_PATH Restart=always RestartSec=5 -StandardOutput=syslog -StandardError=syslog SyslogIdentifier=theta-agent [Install] diff --git a/nodejs/public/resources/theta-agent/theta-agent-linux-amd64 b/nodejs/public/resources/theta-agent/theta-agent-linux-amd64 index cb9c281..92c74b7 100755 Binary files a/nodejs/public/resources/theta-agent/theta-agent-linux-amd64 and b/nodejs/public/resources/theta-agent/theta-agent-linux-amd64 differ diff --git a/nodejs/routes/api_agent.js b/nodejs/routes/api_agent.js index 9a600a7..fd9365e 100644 --- a/nodejs/routes/api_agent.js +++ b/nodejs/routes/api_agent.js @@ -1,105 +1,505 @@ 'use strict'; const express = require('express'); +const middleware = require('../middleware/auth'); +const permission = require('../utils/permission'); const agentManager = require('../utils/agent_manager'); +const agentKeys = require('../utils/agent_keys'); +const ldapTunnel = require('../utils/ldap_tunnel'); +const { Agent, AgentJoinKey } = require('../models/agent'); -module.exports = function initAgentWebSockets(app) { - if (!app.wss) { - console.warn("WebSocket server for agents is not initialized."); - return; +const ADMIN_GROUPS = ['app_sso_admin', 'app_super_admin', 'app_sso_directory_admin']; + +// Commands that can change or run code on the host. They are signed with the +// SSO's persisted Ed25519 key and the agent verifies against the key pinned in +// its agent.yml. +const HIGH_RISK_COMMANDS = ['reboot', 'shutdown', 'service_restart', 'systemd_action', 'configure_ldap', 'arbitrary_bash', 'update_binary', 'render_secrets', 'iam_apply']; + +// ── REST API (mounted synchronously in app.js, BEFORE the 404 catch-all) ── +// This is a plain Express Router exported directly so app.js can +// `app.use('/api/agent', require('./routes/api_agent'))` at require time. It +// must NOT be mounted from the onListen hook (which runs after the 404 +// catch-all is already on the stack): a router registered behind that terminal +// handler would make every /api/agent/* request 404, no matter the WS server +// state. The WebSocket handler is separate (initAgentWebSockets below) and is +// the only part that needs the post-listen onListen hook. +const router = express.Router(); + +// Structured audit line for anything that reaches a host. The agent channel can +// run arbitrary bash, so "who told which host to do what" has to be recoverable +// after the fact; previously nothing was recorded at all. +function logAgentAudit(action, details) { + console.log(JSON.stringify({ + timestamp: new Date().toISOString(), + component: 'agent', + action, + ...details + })); +} + +// The agent WebSocket (/api/agent/ws) authenticates its own token against the +// Agent table (see initAgentWebSockets). These REST routes are admin-facing, so +// they're auth + admin gated. +router.use(middleware.auth); +router.use(async (req, res, next) => { + try { + await permission.byGroup(req.user, ADMIN_GROUPS); + next(); + } catch (err) { + if (err && (err.status === 401 || err.name === 'Insufficient Permission')) { + return res.status(403).json({ status: 'error', message: 'admin only' }); + } + next(err); } +}); - app.wss.on('connection', (ws, req) => { +// --- Fleet --- +router.get('/nodes', async (req, res, next) => { + try { + const keyStatus = agentKeys.status(); + res.json({ + status: 'ok', + agents: await agentManager.listAgents(), + // Base64 of the raw 32-byte key: what goes into agent.yml's `public_key`. + publicKey: await agentManager.publicKeyBase64(), + publicKeyPem: await agentManager.publicKeyPem(), + signingAvailable: agentKeys.status().available, + signingError: keyStatus.error || null + }); + } catch (err) { next(err); } +}); + +// --- Enrollment --- +// The token is minted HERE, not in the browser. It is returned exactly once; +// only its hash is stored, so it cannot be recovered afterwards -- rotate to +// get a new one. +router.post('/enroll', async (req, res, next) => { + try { + const { name, resourceId, description } = req.body || {}; + if (!name || !String(name).trim()) { + return res.status(400).json({ status: 'error', message: 'name is required' }); + } + + if (resourceId) { + const { Resource } = require('../models/resource'); + const resource = await Resource.get(resourceId); + if (!resource) return res.status(400).json({ status: 'error', message: 'resourceId does not exist' }); + if (resource.kind !== 'host') { + return res.status(400).json({ status: 'error', message: 'an agent can only be bound to a host resource' }); + } + } + + const { agent, token } = await Agent.enroll({ + name: String(name).trim(), + description, + resourceId: resourceId || null, + enrolledBy: req.user.uid + }); + + logAgentAudit('enroll', { actor: req.user.uid, agentId: agent.id, agentName: agent.name, resourceId: resourceId || null }); + + const publicKey = await agentManager.publicKeyBase64(); + res.json({ + status: 'ok', + agent: agent.toPublic(agentManager.liveState(agent.id)), + // Shown once. The UI must make that clear. + token, + publicKey, + signingAvailable: agentKeys.status().available + }); + } catch (err) { next(err); } +}); + +router.put('/nodes/:id', async (req, res, next) => { + try { + const agent = await Agent.get(req.params.id); + if (!agent) return res.status(404).json({ status: 'error', message: 'agent not found' }); + + const patch = {}; + if (req.body.name !== undefined) patch.name = req.body.name; + if (req.body.description !== undefined) patch.description = req.body.description; + if (req.body.resourceId !== undefined) { + if (req.body.resourceId) { + const { Resource } = require('../models/resource'); + const resource = await Resource.get(req.body.resourceId); + if (!resource) return res.status(400).json({ status: 'error', message: 'resourceId does not exist' }); + if (resource.kind !== 'host') { + return res.status(400).json({ status: 'error', message: 'an agent can only be bound to a host resource' }); + } + } + patch.resourceId = req.body.resourceId || null; + } + + const updated = await agent.update(patch); + logAgentAudit('update', { actor: req.user.uid, agentId: agent.id, fields: Object.keys(patch) }); + res.json({ status: 'ok', agent: updated.toPublic(agentManager.liveState(agent.id)) }); + } catch (err) { next(err); } +}); + +// Revoke: the token stops authenticating immediately and any live socket is +// dropped, so revocation takes effect without waiting for a reconnect. +router.post('/nodes/:id/revoke', async (req, res, next) => { + try { + const agent = await Agent.get(req.params.id); + if (!agent) return res.status(404).json({ status: 'error', message: 'agent not found' }); + await agent.update({ revoked: true }); + agentManager.disconnect(agent.id, 4003, 'Enrollment revoked'); + logAgentAudit('revoke', { actor: req.user.uid, agentId: agent.id, agentName: agent.name }); + res.json({ status: 'ok' }); + } catch (err) { next(err); } +}); + +router.post('/nodes/:id/rotate', async (req, res, next) => { + try { + const agent = await Agent.get(req.params.id); + if (!agent) return res.status(404).json({ status: 'error', message: 'agent not found' }); + const token = await agent.rotateToken(); + // The old token is dead the moment it is replaced; drop the socket that was + // using it so the agent reconnects with the new one. + agentManager.disconnect(agent.id, 4004, 'Token rotated'); + logAgentAudit('rotate', { actor: req.user.uid, agentId: agent.id, agentName: agent.name }); + res.json({ status: 'ok', token, publicKey: await agentManager.publicKeyBase64() }); + } catch (err) { next(err); } +}); + +router.delete('/nodes/:id', async (req, res, next) => { + try { + const agent = await Agent.get(req.params.id); + if (!agent) return res.status(404).json({ status: 'error', message: 'agent not found' }); + agentManager.disconnect(agent.id, 4003, 'Enrollment deleted'); + await agent.delete(); + logAgentAudit('delete', { actor: req.user.uid, agentId: agent.id, agentName: agent.name }); + res.json({ status: 'ok' }); + } catch (err) { next(err); } +}); + +// --- Join keys --- +// One key an operator hands out; hosts that present it enroll themselves and +// are immediately issued their own per-agent token. Listing never returns the +// key itself -- only its prefix and usage. +router.get('/join-keys', async (req, res, next) => { + try { + const keys = await AgentJoinKey.list(); + res.json({ status: 'ok', joinKeys: keys.map(k => k.toPublic()) }); + } catch (err) { next(err); } +}); + +// Which hosts enrolled through a given key. There is no stored relation -- +// join keys are exchanged for a per-agent token immediately, and from then on +// the agent's own identity is what matters -- so this matches on the +// human-readable trace `Agent.enroll` already leaves in `description` +// ("Self-enrolled with join key ") rather than a foreign key. Prefixes +// are 12 random hex chars, so a collision is not a practical concern. +router.get('/join-keys/:id/agents', async (req, res, next) => { + try { + const key = await AgentJoinKey.get(req.params.id); + if (!key) return res.status(404).json({ status: 'error', message: 'join key not found' }); + const marker = `join key ${key.keyPrefix}`; + const agents = await Agent.list(); + const matches = agents.filter(a => (a.description || '').includes(marker)); + res.json({ status: 'ok', agents: matches.map(a => a.toPublic(agentManager.liveState(a.id))) }); + } catch (err) { next(err); } +}); + +router.post('/join-keys', async (req, res, next) => { + try { + const { label, expiresInDays } = req.body || {}; + const { key, raw } = await AgentJoinKey.issue({ + label: (label && String(label).trim()) || 'default', + createdBy: req.user.uid, + expiresInDays: expiresInDays ? Number(expiresInDays) : null + }); + logAgentAudit('join_key_issued', { actor: req.user.uid, label: key.label, keyPrefix: key.keyPrefix }); + // Shown once; only the hash is stored. + res.json({ status: 'ok', joinKey: key.toPublic(), key: raw }); + } catch (err) { next(err); } +}); + +router.post('/join-keys/:id/revoke', async (req, res, next) => { + try { + const key = await AgentJoinKey.get(req.params.id); + if (!key) return res.status(404).json({ status: 'error', message: 'join key not found' }); + await key.update({ revoked: true }); + logAgentAudit('join_key_revoked', { actor: req.user.uid, label: key.label, keyPrefix: key.keyPrefix }); + // Agents already enrolled keep working -- they hold their own tokens now, + // which is the whole point of exchanging the join key rather than using it + // as the long-term credential. + res.json({ status: 'ok' }); + } catch (err) { next(err); } +}); + +router.delete('/join-keys/:id', async (req, res, next) => { + try { + const key = await AgentJoinKey.get(req.params.id); + if (!key) return res.status(404).json({ status: 'error', message: 'join key not found' }); + await key.delete(); + logAgentAudit('join_key_deleted', { actor: req.user.uid, label: key.label, keyPrefix: key.keyPrefix }); + res.json({ status: 'ok' }); + } catch (err) { next(err); } +}); + +// --- Commands --- +// Addressed by agent id, not by token: a token is a credential and has no +// business travelling in a URL, being logged, or sitting in browser history. +router.post('/nodes/:id/command', async (req, res, next) => { + const { command, payload, isHighRisk } = req.body || {}; + if (!command) { + return res.status(400).json({ status: 'error', message: 'Command type is required' }); + } + try { + const agent = await Agent.get(req.params.id); + if (!agent) return res.status(404).json({ status: 'error', message: 'agent not found' }); + if (agent.revoked) return res.status(403).json({ status: 'error', message: 'agent enrollment is revoked' }); + + const requiresSigning = isHighRisk || HIGH_RISK_COMMANDS.includes(command); + const msg = await agentManager.sendCommand(agent, command, payload || {}, requiresSigning); + + logAgentAudit('command', { + actor: req.user.uid, + agentId: agent.id, + agentName: agent.name, + resourceId: agent.resourceId || null, + command, + signed: requiresSigning + }); + + res.json({ status: 'ok', sentMessage: msg }); + } catch (err) { + logAgentAudit('command_failed', { actor: req.user && req.user.uid, agentId: req.params.id, command, error: err.message }); + res.status(400).json({ status: 'error', message: err.message }); + } +}); + +module.exports = router; +module.exports.HIGH_RISK_COMMANDS = HIGH_RISK_COMMANDS; + +module.exports.initAgentWebSockets = function initAgentWebSockets(app) { + // WebSocket handler only needs the WS server; runs from the onListen hook. + if (!app.wss) return; + + // Warm the signing key at boot so a misconfigured OpenBao policy is a loud + // startup error rather than a surprise the first time someone reboots a host. + agentKeys.load().then(keys => { + if (!keys) console.error(`[Theta Agent] signing key unavailable — high-risk commands will be refused. ${agentKeys.status().error || ''}`); + }); + + app.wss.on('connection', async (ws, req) => { const url = new URL(req.url, `http://${req.headers.host || 'localhost'}`); const token = url.searchParams.get('token') || req.headers['authorization']; + const remoteAddr = req.socket.remoteAddress; - if (!token) { - ws.close(4001, 'Unauthorized: Missing token'); + // Authenticate BEFORE doing anything else: no registration, no welcome + // payload, no acknowledgement that the token was close. Until this passes + // the peer is an anonymous stranger, and the old code treated it as a + // trusted node purely for presenting a non-empty string. + let agent = null; + let issuedToken = null; // set when this connection auto-enrolled + try { + agent = await Agent.authenticate(token); + + // Not a known agent token -- try it as a join key. This is what makes + // "install the agent with a key and the host appears" work without an + // admin pre-registering every machine. The join key is exchanged for a + // per-agent token below, so it never becomes the host's long-term + // credential. + if (!agent) { + const joinKey = await AgentJoinKey.authenticate(token); + if (joinKey) { + const hostname = (url.searchParams.get('hostname') || '').trim(); + let existingAgent = null; + if (hostname) { + const matches = await Agent.list({ where: { name: hostname } }); + existingAgent = matches && matches.find(a => !a.revoked); + } + if (existingAgent) { + const newToken = await existingAgent.rotateToken(); + agent = existingAgent; + issuedToken = newToken; + } else { + const enrolled = await Agent.enroll({ + name: hostname || `agent-${Date.now().toString(36)}`, + description: `Self-enrolled with join key ${joinKey.keyPrefix}`, + enrolledBy: `join-key:${joinKey.label}` + }); + agent = enrolled.agent; + issuedToken = enrolled.token; + } + await joinKey.update({ + use_count: (joinKey.use_count || 0) + 1, + last_used_on: Math.floor(Date.now() / 1000) + }).catch(() => {}); + logAgentAudit('join', { + agentId: agent.id, agentName: agent.name, remoteAddr, + joinKeyLabel: joinKey.label, joinKeyPrefix: joinKey.keyPrefix + }); + console.log(`[Theta Agent] "${agent.name}" self-enrolled with join key ${joinKey.keyPrefix}`); + } + } + } catch (err) { + console.error('[Theta Agent] authentication lookup failed:', err.message); + try { ws.close(1011, 'Authentication unavailable'); } catch (e) {} return; } - const remoteAddr = req.socket.remoteAddress; - console.log(`[Theta Agent] Agent connected from ${remoteAddr} with token ${token.substring(0, 8)}...`); + if (!agent) { + // Deliberately indistinguishable for unknown vs revoked vs missing: a + // caller probing tokens learns nothing about which part was wrong. + logAgentAudit('auth_rejected', { remoteAddr, tokenPrefix: token ? String(token).slice(0, 8) : null }); + try { ws.close(4001, 'Unauthorized'); } catch (e) {} + return; + } - agentManager.registerAgent(token, ws, remoteAddr); + console.log(`[Theta Agent] "${agent.name}" (${agent.id}) connected from ${remoteAddr}`); + logAgentAudit('connected', { agentId: agent.id, agentName: agent.name, remoteAddr }); + // Must stay synchronous, and the listeners below must be attached in this + // same tick: the agent sends `discovery` the instant the socket opens, and + // `ws` discards messages emitted while no listener is attached. + agentManager.registerAgent(agent, ws, remoteAddr); - ws.on('message', (message) => { + if (issuedToken) { + const publicKey = await agentManager.publicKeyBase64(); + try { + ws.send(JSON.stringify({ + type: 'config', + payload: { + enrolled: true, + auth_token: issuedToken, + public_key: publicKey + } + })); + console.log(`[Theta Agent] Sent auto-enrollment credentials to "${agent.name}"`); + } catch (err) { + console.error(`[Theta Agent] Failed to send auto-enrollment config to "${agent.name}":`, err.message); + } + } + + ws.on('message', async (message) => { try { const data = JSON.parse(message); if (!data || typeof data.type !== 'string') return; + // Re-read the row per message so a revoke mid-session takes effect on + // the next thing the agent says, not only on reconnect. + const current = await Agent.get(agent.id).catch(() => null); + if (!current || current.revoked) { + try { ws.close(4003, 'Enrollment revoked'); } catch (e) {} + return; + } + const payload = data.payload || {}; switch (data.type) { case 'discovery': - agentManager.handleDiscovery(token, payload); - if (app.io) app.io.emit('agent.discovery', { token, payload }); + await agentManager.handleDiscovery(current, payload); + if (app.io) app.io.emit('agent.discovery', { agentId: current.id, payload }); + if (payload.capabilities && payload.capabilities.configure_ldap) { + const conf = require('@simpleworkjs/conf'); + const os = require('os'); + const ssoHost = (conf.stack && conf.stack.ssoHost) || 'sso.laptop-dev.vm42.us'; + const ldapBaseDn = (conf.stack && conf.stack.ldapBaseDn) || 'dc=laptop-dev,dc=vm42,dc=us'; + + const lanIps = []; + const ifaces = os.networkInterfaces(); + for (const dev in ifaces) { + for (const details of ifaces[dev]) { + if (!details.internal && details.family === 'IPv4') lanIps.push(details.address); + } + } + const uriList = [ + `ldapi://%2frun%2ftheta%2fldap.sock`, + `ldap://127.0.0.1:3890`, + `ldap://127.0.0.1:389`, + `ldap://${ssoHost}:389`, + `ldaps://${ssoHost}:636`, + ...lanIps.map(ip => `ldap://${ip}:389`) + ]; + const ldapUris = [...new Set(uriList)].join(', '); + + const sssdConfig = `[sssd] +config_file_version = 2 +domains = default + +[domain/default] +id_provider = ldap +auth_provider = ldap +chpass_provider = ldap +sudo_provider = ldap +ldap_uri = ${ldapUris} +ldap_search_base = ${ldapBaseDn} +ldap_user_search_base = ou=people,${ldapBaseDn} +ldap_group_search_base = ou=groups,${ldapBaseDn} +ldap_sudo_search_base = ou=people,${ldapBaseDn} +ldap_schema = rfc2307bis +ldap_user_object_class = posixAccount +ldap_user_name = uid +ldap_user_ssh_public_key = sshPublicKey +ldap_group_object_class = groupOfNames +ldap_group_member = member +ldap_id_mapping = false +ldap_id_use_start_tls = false +ldap_tls_reqcert = never +cache_credentials = true +entry_cache_timeout = 600 +entry_cache_user_timeout = 600 +entry_cache_group_timeout = 600 +entry_cache_sudo_timeout = 600 +refresh_expired_interval = 300 +`; + agentManager.sendCommand(current, 'configure_ldap', { config: sssdConfig }, true).then(() => { + console.log(`[Theta Agent] Pushed auto configure_ldap to "${current.name}"`); + }).catch(err => { + console.error(`[Theta Agent] Auto push configure_ldap to "${current.name}" failed:`, err.message); + }); + } break; case 'telemetry': - agentManager.handleTelemetry(token, payload); - if (app.io) app.io.emit('agent.telemetry', { token, payload }); + await agentManager.handleTelemetry(current, payload); + if (app.io) app.io.emit('agent.telemetry', { agentId: current.id, payload }); break; case 'heartbeat': - agentManager.handleHeartbeat(token, payload, ws); + await agentManager.handleHeartbeat(current, payload, ws); break; case 'response': - agentManager.handleResponse(token, payload); - if (app.io) app.io.emit('agent.response', { token, payload }); + await agentManager.handleResponse(current, payload); + if (app.io) app.io.emit('agent.response', { agentId: current.id, payload }); + break; + case 'ldap_tunnel': + // Raw LDAP bytes from the agent's local socket → relay into OpenLDAP + // and pipe the response back (DESIGN.md §4). + ldapTunnel.handleTunnel(current.id, ws, payload); break; default: - console.log(`[Theta Agent] Received message type '${data.type}' from ${token}`); + console.log(`[Theta Agent] Received message type '${data.type}' from ${current.id}`); } } catch (err) { - console.error("[Theta Agent] Error parsing message:", err); + console.error('[Theta Agent] Error handling message:', err); } }); ws.on('close', () => { - console.log(`[Theta Agent] Agent disconnected (${token})`); - agentManager.unregisterAgent(token, ws); + console.log(`[Theta Agent] "${agent.name}" (${agent.id}) disconnected`); + agentManager.unregisterAgent(agent.id, ws); + ldapTunnel.cleanup(agent.id); }); - // Send initial welcome/config payload + // Send initial welcome/config payload. When this connection enrolled via a + // join key it also carries the credentials the agent should persist and use + // from now on: its own token, and the public key it must pin to verify + // signed commands. Handing the public key over here is what removes the + // last manual step -- an agent installed with only a join key ends up fully + // configured without anyone copying values between two machines. try { - ws.send(JSON.stringify({ - type: 'config', - payload: { - message: 'Connected to SSO Manager C2', - protocol_version: '1.1.0' - } - })); + const payload = { + message: 'Connected to SSO Manager C2', + protocol_version: '1.2.0', + agent_id: agent.id + }; + if (issuedToken) { + payload.enrolled = true; + payload.auth_token = issuedToken; + payload.public_key = await agentManager.publicKeyBase64(); + } + ws.send(JSON.stringify({ type: 'config', payload })); } catch (e) {} }); - - // REST API routes for Agent Management (mounted under /api/agent) - const router = express.Router(); - - router.get('/nodes', (req, res) => { - res.json({ - status: 'ok', - agents: agentManager.getConnectedAgents(), - publicKey: agentManager.publicKeyPem - }); - }); - - router.post('/nodes/:token/command', (req, res) => { - const { token } = req.params; - const { command, payload, isHighRisk } = req.body; - - if (!command) { - return res.status(400).json({ status: 'error', message: 'Command type is required' }); - } - - try { - const HIGH_RISK_COMMANDS = ['reboot', 'service_restart', 'configure_ldap', 'arbitrary_bash', 'update_binary']; - const requiresSigning = isHighRisk || HIGH_RISK_COMMANDS.includes(command); - - const msg = agentManager.sendCommand(token, command, payload || {}, requiresSigning); - res.json({ status: 'ok', sentMessage: msg }); - } catch (err) { - res.status(400).json({ status: 'error', message: err.message }); - } - }); - - app.use('/api/agent', router); }; diff --git a/nodejs/routes/api_agent_ops.js b/nodejs/routes/api_agent_ops.js new file mode 100644 index 0000000..3aecdce --- /dev/null +++ b/nodejs/routes/api_agent_ops.js @@ -0,0 +1,115 @@ +'use strict'; + +// Agent-facing operations (DESIGN.md §5, §6). These are NOT admin-gated: the +// caller is the agent itself, authenticated by its own token (the same one it +// presents on its WSS channel). Mounted at /api/v1/agent. + +const express = require('express'); +const baoConf = require('@simpleworkjs/bao-conf'); +const { authenticateAgent } = require('../utils/agent_auth'); + +const router = express.Router(); + +// POST /secrets — fetch node-scoped OpenBao secrets for the agent's own node. +// +// { paths: ["secret/data/nodes//db"] } +// -> { status: "ok", secrets: { "secret/data/nodes//db": { key: value } } } +// +// The agent may only read under its own node prefix (secret/data/nodes//*), +// so a compromised agent cannot reach other nodes' or shared secrets. The SSO +// fetches with its own OpenBao access (SSO_VAULT_TOKEN); the agent never holds a +// Vault token. +const { Resource } = require('../models/resource'); +const { SharedSecretGrant } = require('../models/shared_secret_grant'); +const { SharedSecret } = require('../models/shared_secret'); + +router.post('/secrets', async (req, res, next) => { + try { + const agent = await authenticateAgent(req); + if (!agent) return res.status(401).json({ status: 'error', message: 'unauthorized' }); + + let { paths } = req.body || {}; + let boundResource = null; + if (agent.resourceId) { + boundResource = await Resource.get(agent.resourceId).catch(() => null); + } + + if (!Array.isArray(paths) || paths.length === 0) { + paths = [`secret/data/nodes/${agent.id}/conf`]; + if (boundResource && boundResource.slug) { + paths.push(`secret/data/resources/${boundResource.slug}/conf`); + } + } + + // Allowed prefixes for this agent: + // 1. Node scope: secret/data/nodes// + // 2. Bound Resource scope: secret/data/resources// + // 3. Shared Resource Grants: secret/data/resources// + const allowedPrefixes = [`secret/data/nodes/${agent.id}/`]; + if (boundResource && boundResource.slug) { + allowedPrefixes.push(`secret/data/resources/${boundResource.slug}/`); + } + + // Add granted shared resources + if (boundResource) { + const grants = await SharedSecretGrant.listForGrantee('resource', boundResource.id).catch(() => []); + for (const g of grants) { + const sharedSec = await SharedSecret.get(g.secretId).catch(() => null); + if (sharedSec && sharedSec.slug) { + allowedPrefixes.push(`secret/data/resources/${sharedSec.slug}/`); + allowedPrefixes.push(`secret/data/shared/${sharedSec.ownerUid}/${sharedSec.slug}/`); + } + } + } + + const secrets = {}; + for (let p of paths) { + if (typeof p !== 'string') continue; + // Normalize human shorthand "resources/foo/bar" -> "secret/data/resources/foo/bar" + if (p.startsWith('resources/')) { + p = `secret/data/resources/${p.slice('resources/'.length)}`; + } + + const isAllowed = allowedPrefixes.some(prefix => p.startsWith(prefix)); + if (!isAllowed) { + return res.status(403).json({ status: 'error', message: `path outside authorized scope: ${p}` }); + } + + const r = await baoConf.request('GET', p); + if (r.ok) { + const body = await r.json().catch(() => ({})); + const rawMap = (body.data && body.data.data) || {}; + const resolvedMap = {}; + for (const [k, v] of Object.entries(rawMap)) { + const strV = String(v || ''); + if (strV.startsWith('INHERIT:')) { + const parts = strV.split(':'); + if (parts.length >= 3) { + const targetSlug = parts[1]; + const targetKey = parts[2]; + const parentR = await baoConf.request('GET', `secret/data/resources/${targetSlug}/conf`); + if (parentR.ok) { + const parentBody = await parentR.json().catch(() => ({})); + const parentMap = (parentBody.data && parentBody.data.data) || {}; + resolvedMap[k] = parentMap[targetKey] || ''; + } else { + resolvedMap[k] = ''; + } + } else { + resolvedMap[k] = ''; + } + } else { + resolvedMap[k] = v; + } + } + secrets[p] = resolvedMap; + } else { + secrets[p] = {}; + } + } + + return res.json({ status: 'ok', secrets }); + } catch (err) { next(err); } +}); + +module.exports = router; diff --git a/nodejs/routes/api_conf.js b/nodejs/routes/api_conf.js index 03deea7..525f5d9 100644 --- a/nodejs/routes/api_conf.js +++ b/nodejs/routes/api_conf.js @@ -136,15 +136,25 @@ router.post('/test-email', async (req, res, next) => { return res.status(400).json({ error: 'Recipient email address is required' }); } - // Use the email model to send the test message - const Email = require('../models/email'); + // Send through the SAME sender every other feature uses (password reset, + // invites, OTP-by-email, notifications). A "test" that reimplements + // delivery proves nothing about whether real mail works. + // + // models/email.js exports `{Mail}`; requiring the module and calling + // `.send` on it directly -- as this did -- always threw + // "Email.send is not a function", so the button could never succeed. + const { Mail } = require('../models/email'); const testSubject = subject || 'SSO Manager Test Email'; const testBody = body || `

This is a test email from SSO Manager.

If you received this, your SMTP configuration is working correctly.

Sent at: ${new Date().toISOString()}

`; - await Email.send(to, testSubject, testBody); + await Mail.send(to, testSubject, testBody); res.json({ success: true, message: `Test email sent to ${to}` }); } catch(err) { - next(err); + // A failed test is almost always a misconfiguration (wrong host, refused + // connection, bad credentials) -- the operator's to fix, and something the + // UI should be able to show them. Surfacing it as a 400 with the reason + // beats an opaque 500 carrying a raw stack-trace name. + return res.status(400).json({ error: err.message || 'Failed to send test email' }); } }); @@ -156,38 +166,37 @@ router.post('/test-sms', async (req, res, next) => { return res.status(400).json({ error: 'Recipient phone number is required' }); } + // Send through models/sms.js -- the same path every real SMS takes. It + // prefers a configured messaging plugin and falls back to VoIP.ms, and it + // normalizes the destination to E.164 digits. + // + // This used to POST to `https://api.voip.ms/v1.0/sms/send` with Basic auth. + // No such endpoint exists: VoIP.ms's REST API is a GET against + // `https://voip.ms/api/v1/rest.php` with `api_username`/`api_password` and + // `method=sendSMS`. The fabricated URL returned an HTML page, so + // `response.json()` threw `Unexpected token '<', " []); const voipmsConf = conf.voipms || {}; - if (!voipmsConf.username || !voipmsConf.password || !voipmsConf.did) { - return res.status(400).json({ error: 'VoIP.ms credentials not configured. Please configure username, DID, and password in the SMS tab.' }); + if (!messagingPlugins.length && (!voipmsConf.username || !voipmsConf.password || !voipmsConf.did)) { + return res.status(400).json({ error: 'No messaging plugin is loaded and VoIP.ms credentials are not configured. Set username, DID and password in the SMS tab, or load a messaging plugin.' }); } - const testMessage = message || `SSO Manager Test SMS: This is a test message from ${conf.name}. If you received this, your VoIP.ms configuration is working correctly.`; + const testMessage = message || `SSO Manager Test SMS: This is a test message from ${conf.name}. If you received this, your SMS configuration is working correctly.`; - // VoIP.ms SMS API endpoint - const voipmsApiUrl = 'https://api.voip.ms/v1.0'; - const authHeader = Buffer.from(`${voipmsConf.username}:${voipmsConf.password}`).toString('base64'); - - const response = await fetch(`${voipmsApiUrl}/sms/send`, { - method: 'POST', - headers: { - 'Authorization': `Basic ${authHeader}`, - 'Content-Type': 'application/x-www-form-urlencoded' - }, - body: new URLSearchParams({ - did: voipmsConf.did, - to: to, - message: testMessage - }) - }); - - const result = await response.json(); - if (result.status === 'success') { - res.json({ success: true, message: `Test SMS sent to ${to}` }); - } else { - res.status(400).json({ error: `VoIP.ms API error: ${result.message || 'Unknown error'}` }); - } + await SMS.send(to, testMessage); + res.json({ success: true, message: `Test SMS sent to ${to}` }); } catch(err) { - next(err); + // The sender rejects with a useful reason (`VoIP.ms error: `, or a + // plugin's own error). Surface it as a 400 the UI can display rather than + // an opaque 500 -- a misconfiguration is the operator's to fix, not a bug. + return res.status(400).json({ error: err.message || 'Failed to send test SMS' }); } }); diff --git a/nodejs/routes/api_directory_admin.js b/nodejs/routes/api_directory_admin.js index bbf3c77..229c2f4 100644 --- a/nodejs/routes/api_directory_admin.js +++ b/nodejs/routes/api_directory_admin.js @@ -8,10 +8,11 @@ const { cnFromDn } = require('../utils/user_groups'); const { projectResources } = require('@simpleworkjs/directory-schema'); const SUPER_ADMIN_GROUP = permission.SUPER_ADMIN_GROUP; +const groups = require('../utils/groups'); // Make `childCn` a member of `parentCn`, i.e. everyone in the child is // transitively in the parent. Idempotent and non-fatal: "already a member" is -// the goal state, and a missing group (e.g. app_super_admin absent on a +// the goal state, and a missing group (e.g. god_admin absent on a // directory seeded by an older entrypoint) is a reason to skip, not to fail the // caller's real work. async function nestGroup(childCn, parentCn) { @@ -29,6 +30,160 @@ async function nestGroup(childCn, parentCn) { } } +// ── Group-model provisioning (docs/GROUPS.md) ─────────────────────────────── +// The directory is the single place groups are created, as a projection of the +// resource graph. These helpers materialize the group-inheritance lattice for +// a resource so it exists in LDAP as well as in the resolver (utils/groups.js). +// All of them are idempotent, so calling them again for a resource a newer +// release is backfilling is a no-op. + +// Map a directory resource kind onto a group-model kind (GROUPS.md §2). +// host -> host; service -> app (services/consoles are the group model's "apps"); +// site gets site-level groups (handled separately); oauth/container get no +// per-resource groups (oauth clients hang off their owning service). +function groupKind(resource) { + if (resource.kind === 'host') return 'host'; + if (resource.kind === 'service') return 'app'; + return null; +} + +// Create a groupOfNames if it doesn't already exist. Idempotent; `ownerDn` +// seeds the mandatory first member. Returns true when created. +async function ensureGroup(name, ownerDn, description) { + try { + await Group.add({ name, owner: ownerDn, description }); + return true; + } catch (err) { + if (err.name !== 'EntryAlreadyExistsError' && err.code !== 68) { + console.error(`ensureGroup: failed to create ${name}:`, err); + } + return false; + } +} + +// Link a group to a resource only if that link doesn't already exist. The +// ResourceGroup table has no unique constraint on (resourceId, groupCn), so a +// naive create on every Directory self-heal (which runs ensureSiteGroups / +// provisionResourceGroups on each load) was accumulating duplicate links -- the +// "groups appear 3x under a resource" bug. Always check first. +async function ensureResourceGroup(resourceId, groupCn, accessLevel) { + const existing = await ResourceGroup.list({ where: { resourceId, groupCn } }); + if (existing.length) return existing[0]; + return ResourceGroup.create({ resourceId, groupCn, accessLevel }); +} + +// Provision the site-level groups + the aggregates the per-resource groups nest +// into. Idempotent -- called on every directory list so a site seeded by an +// older release gets its groups without a rebuild: +// +// god_admin -> {site}_super_admin +// {site}_super_admin -> {site}_hosts_admin, {site}_apps_admin +// {site}_hosts_admin -> {site}_hosts_access ; {site}_apps_admin -> {site}_apps_access +// +// `{site}_everyone` is created for completeness; it has implicit membership and +// is granted to a resource as a grantee, never enumerated. +async function ensureSiteGroups(siteSlug, ownerDn, siteName, siteResourceId) { + if (!siteSlug) return; + + // Link a site group to the site resource (so it shows + is member-manageable + // on the site's modal). Idempotent. Admin groups link as owner; access/meta + // groups as member. + const link = async (cn, isAdmin) => { + if (!siteResourceId) return; + await ensureResourceGroup(siteResourceId, cn, isAdmin ? 'owner' : 'member'); + }; + + const sAdmin = groups.siteSuperAdminCns(siteSlug); + await ensureGroup(sAdmin, ownerDn, `Site admin for ${siteName || siteSlug}`); + await link(sAdmin, true); + // The kind-scoped aggregates are CREATED here (per-resource groups nest into + // them), but are NOT linked to the site resource: a site carries only the god + // and site-wide groups (S_super_admin, S_everyone), per the user's model. The + // aggregates have no modal home; site-wide access is granted via S_super_admin + // and per-resource access via the host/app groups. + for (const kind of ['host', 'app']) { + await ensureGroup(groups.aggregateGroupCns(siteSlug, kind, 'admin'), ownerDn, `Admin on all ${kind}s at ${siteSlug}`); + await ensureGroup(groups.aggregateGroupCns(siteSlug, kind, 'access'), ownerDn, `Access to all ${kind}s at ${siteSlug}`); + } + await ensureGroup(groups.siteEveryoneCns(siteSlug), ownerDn, `All users at ${siteSlug}`); + await link(groups.siteEveryoneCns(siteSlug), false); + // god_admin is the global group; surface it on the site modal so its members + // can be managed from the Directory (it has no home on a single resource). + await link(groups.GOD_ADMIN, true); + + // Wire the lattice as nesting so LDAP-level consumers (SSSD, sudo, anything + // binding directly) resolve it transitively, not just utils/permission.js. + // nestGroup(child, parent) makes child a member of parent -- membership flows + // child -> parent ("up"), so a group's members inherit what its parents hold. + await nestGroup(groups.GOD_ADMIN, sAdmin); // god admins are site admins everywhere + for (const kind of ['host', 'app']) { + const aggAdmin = groups.aggregateGroupCns(siteSlug, kind, 'admin'); + const aggAccess = groups.aggregateGroupCns(siteSlug, kind, 'access'); + await nestGroup(sAdmin, aggAdmin); // site admins administer all hosts/apps + await nestGroup(aggAdmin, aggAccess); // site admin implies site access + } +} + +// Provision the per-resource groups for a host/app and nest them into the site +// aggregates (so a site/aggregate admin reaches this resource by membership). +// Group names follow docs/GROUPS.md §2: `{site}_{kind}_{nameSlug}_{level}` where +// nameSlug is the resource name with the kind prefix stripped (`host_theta-env` -> +// `theta-env`). `kind` (host/app) both goes in the name and selects the aggregate: +// +// {site}_{kind}_{slug}_admin -> {site}_{kind}_{slug}_access +// {site}_{kind}_{slug}_admin -> {site}_{kind}s_admin (aggregate) +// {site}_{kind}_{slug}_access -> {site}_{kind}s_access (aggregate) +// god_admin -> {site}_{kind}_{slug}_admin (global super admin) +async function provisionResourceGroups(resource, kind, siteSlug, ownerDn) { + const nameSlug = groups.resourceNameSlug(resource.slug); + const accessCn = groups.resourceGroupCns(siteSlug, kind, nameSlug, 'access'); + const adminCn = groups.resourceGroupCns(siteSlug, kind, nameSlug, 'admin'); + + await ensureGroup(accessCn, ownerDn, `Access group for ${resource.name}`); + await ensureGroup(adminCn, ownerDn, `Admin group for ${resource.name}`); + + // Link both groups to the resource so the Directory can show/revoke them. + await ensureResourceGroup(resource.id, accessCn, 'member'); + await ensureResourceGroup(resource.id, adminCn, 'owner'); + + await nestGroup(adminCn, accessCn); // administering implies using + await nestGroup(adminCn, groups.aggregateGroupCns(siteSlug, kind, 'admin')); // aggregate admin reaches this resource + await nestGroup(accessCn, groups.aggregateGroupCns(siteSlug, kind, 'access')); // aggregate access reaches this resource + await nestGroup(SUPER_ADMIN_GROUP, adminCn); // global super admin +} + +// The group CNs it is valid to associate with a given resource (docs/GROUPS.md +// §2/§3). This is what "force the correct naming convention" means: a group +// linked to a resource must be one that parses for consumers -- the resource's +// own specific groups, its site's aggregates, site-level groups, or the global +// god_admin. Returns a Set of the fixed valid CNs plus a RegExp for opaque +// capability groups following the same shapes. +function validGroupCnsForResource(resource, siteSlug) { + const valid = new Set(); + // A site resource only carries god_admin (added by the route) + the site-wide + // groups (S_super_admin, S_everyone). The kind-scoped host/app aggregates and + // specific groups belong to host/app resources, not to the site. + if (resource.kind === 'site') { + valid.add(groups.siteSuperAdminCns(siteSlug)); + valid.add(groups.siteEveryoneCns(siteSlug)); + return { valid, capRe: new RegExp(`^${siteSlug}_super_admin$|^${siteSlug}_everyone$`) }; + } + const kind = groupKind(resource); // 'host'|'app'|null + if (kind) { + const nameSlug = groups.resourceNameSlug(resource.slug); + valid.add(groups.resourceGroupCns(siteSlug, kind, nameSlug, 'admin')); + valid.add(groups.resourceGroupCns(siteSlug, kind, nameSlug, 'access')); + valid.add(groups.aggregateGroupCns(siteSlug, kind, 'admin')); + valid.add(groups.aggregateGroupCns(siteSlug, kind, 'access')); + valid.add(groups.siteSuperAdminCns(siteSlug)); + valid.add(groups.siteEveryoneCns(siteSlug)); + return { valid, capRe: new RegExp(`^${siteSlug}_${kind}_${nameSlug}_[a-z0-9-]+$|^${siteSlug}_${kind}s_[a-z0-9-]+$`) }; + } + // oauth/container etc. — only the global god_admin makes sense to pin here. + valid.add(groups.siteSuperAdminCns(siteSlug)); + return { valid, capRe: null }; +} + // Require the admin group router.use(async (req, res, next) => { try { @@ -44,12 +199,43 @@ router.get('/resources', async (req, res, next) => { try { let resources = await Resource.list(); resources = resources.filter(r => { + if (r.kind === 'host' || r.kind === 'site') return true; const isAuto = r.metadata?.discovery_sources?.length > 0 && !r.metadata.discovery_sources.includes('manual'); const isManaged = r.metadata?.managed === true; return !isAuto || isManaged; }); // Even admins never receive secret metadata (e.g. client_secret_hash) over // the wire; projectResources strips it unconditionally. + + // Self-heal the group model (docs/GROUPS.md): ensure every site has its + // site-level groups (S_super_admin, S_hosts_*, S_apps_*, S_everyone) + the + // aggregates, and every host/app resource has its per-resource groups nested + // into them. Idempotent, so this is a cheap no-op once present -- it's what + // backfills a directory seeded by an older release without a rebuild. + // Never fails the list. + const sites = resources.filter(r => r.kind === 'site'); + await Promise.all(sites.map(site => + ensureSiteGroups(site.slug, req.user.dn, site.name, site.id) + .catch(err => console.error(`ensureSiteGroups(${site.slug}) failed:`, err.message)) + )); + const siteByResource = new Map(); + for (const site of sites) siteByResource.set(site.id, site.slug); + const siteOf = async (r) => { + const direct = siteByResource.get(r.id); + if (direct) return direct; + // findAncestorSiteSlug returns the site's full slug (`site_local`) -- the + // group-model builders take it verbatim, so do NOT strip the `site_` prefix. + return await Resource.findAncestorSiteSlug(r.id).catch(() => null); + }; + await Promise.all(resources.map(async (r) => { + const gKind = groupKind(r); + if (!gKind) return; + const siteSlug = await siteOf(r); + if (!siteSlug) return; + await provisionResourceGroups(r, gKind, siteSlug, req.user.dn) + .catch(err => console.error(`provisionResourceGroups(${r.slug}) failed:`, err.message)); + })); + res.json({ results: projectResources(resources, { fullMetadata: true }) }); } catch (err) { next(err); } }); @@ -61,6 +247,9 @@ router.post('/resources', async (req, res, next) => { if (parents.length > 0) req.body.hostId = parents[0].id; } + if (req.body.kind !== 'site' && req.body.kind !== 'Site' && !req.body.hostId) { + return res.status(400).json({ error: 'Only Site resources can be top-level. All other resource types must have a parent resource.' }); + } if (req.body.kind === 'host' && !req.body.hostId) { return res.status(400).json({ error: 'Hosts must have a parent Site or Host' }); } @@ -95,46 +284,25 @@ router.post('/resources', async (req, res, next) => { await ResourceEdge.create({ parentId: req.body.hostId, childId: r.id, relation: r.kind === 'oauth' ? 'oauth' : 'hosts' }); } - if (r.kind === 'host' || r.kind === 'service') { - const siteSlug = await Resource.findAncestorSiteSlug(r.id); - const groupCn = suffix => (siteSlug ? `${siteSlug}_${r.slug}_${suffix}` : `${r.slug}_${suffix}`); - - const createGroup = async (suffix, accessLevel) => { - const cn = groupCn(suffix); - try { - await Group.add({ - name: cn, - owner: req.user.dn, - description: `${suffix === 'admin' ? 'Admin' : 'Access'} group for ${r.name}` - }); - } catch (err) { - if (err.name !== 'EntryAlreadyExistsError' && err.code !== 68) { - console.error(`Failed to create LDAP group ${cn}:`, err); - } - } - try { - await ResourceGroup.create({ resourceId: r.id, groupCn: cn, accessLevel }); - } catch(err) { /* ignore duplicate links */ } - }; - await createGroup('access', 'member'); - await createGroup('admin', 'owner'); - - // Wire up the two standing relationships every resource has, as nesting - // rather than as membership that has to be maintained per resource: - // - // app_super_admin -> _admin cross-app super admins administer - // every resource, automatically - // _admin -> _access administering something implies - // being able to use it - // - // Before nesting, both of these could only be expressed by adding every - // super admin to every new group by hand -- which nobody does, so the - // groups drifted. A failure here must not fail resource creation: the - // resource and its groups already exist and the nesting is repairable. - await nestGroup(groupCn('admin'), groupCn('access')); - await nestGroup(SUPER_ADMIN_GROUP, groupCn('admin')); + // ── Group provisioning (docs/GROUPS.md) ─────────────────────────────── + // Materialize the group-model for the new resource. Site resources get the + // site-level groups; host/app resources get their per-resource groups nested + // into the site aggregates. Idempotent -- safe for a resource created by an + // older release. A provisioning failure must not fail resource creation: the + // resource already exists and the groups are repairable (re-run ensures them). + // + // `siteSlug` is the site resource's slug verbatim (`site_local`) -- the + // group-model builders treat it as opaque (docs/GROUPS.md §3) and re-apply + // the kind prefix themselves. + const gKind = groupKind(r); + const ancestorSite = await Resource.findAncestorSiteSlug(r.id); + if (r.kind === 'site') { + await ensureSiteGroups(r.slug, req.user.dn, r.name, r.id); + } else if (gKind && ancestorSite) { + await ensureSiteGroups(ancestorSite, req.user.dn, r.name); // backfill site tier if missing + await provisionResourceGroups(r, gKind, ancestorSite, req.user.dn); } - + res.json({ results: r }); } catch (err) { if (err.name === 'SequelizeUniqueConstraintError') { @@ -151,6 +319,9 @@ router.put('/resources/:id', async (req, res, next) => { try { // Validate before loading anything -- a rejected body should never have // touched the store. + if (req.body.kind !== 'site' && req.body.kind !== 'Site' && !req.body.hostId) { + return res.status(400).json({ error: 'Only Site resources can be top-level. All other resource types must have a parent resource.' }); + } if (req.body.kind === 'host' && !req.body.hostId) { return res.status(400).json({ error: 'Hosts must have a parent Site or Host' }); } @@ -265,7 +436,29 @@ router.get('/groups', async (req, res, next) => { router.post('/groups', async (req, res, next) => { try { - const g = await ResourceGroup.create(req.body); + const { resourceId, groupCn } = req.body; + if (!resourceId || !groupCn) return res.status(400).json({ error: 'resourceId and groupCn are required' }); + + // Enforce the group-model naming convention (docs/GROUPS.md §3). The CN must + // be a valid group for this resource; reject free-form names so the groups + // consumers read are always parseable. god_admin is always allowed (it is + // the global group and is managed from a site's modal). + const resource = await Resource.get(resourceId); + // Full site slug verbatim (`site_local`) -- the builders take it as-is. A + // site resource's own slug is its site; a host/app uses its ancestor site. + const siteSlug = resource && resource.kind === 'site' + ? resource.slug + : await Resource.findAncestorSiteSlug(resourceId); + if (resource && siteSlug && groupCn !== groups.GOD_ADMIN) { + const { valid, capRe } = validGroupCnsForResource(resource, siteSlug); + if (!valid.has(groupCn) && !(capRe && capRe.test(groupCn))) { + const err = new Error(`"${groupCn}" is not a valid group for this ${resource.kind}. Use the resource's own groups, a site aggregate, a site-level group, or god_admin (e.g. ${[...valid].join(', ')}).`); + err.status = 400; + throw err; + } + } + + const g = await ensureResourceGroup(req.body.resourceId, groupCn, req.body.accessLevel); res.json({ results: g }); } catch (err) { next(err); } }); @@ -313,7 +506,7 @@ router.get('/access-summary', async (req, res, next) => { // // Counts come from the transitive closure, not from `member`. Reading the // attribute would report only who is listed on the group, missing anyone - // who reaches it through a nested group -- and since app_super_admin is + // who reaches it through a nested group -- and since god_admin is // nested into every resource's _admin group, that is not an edge case. let members = []; if (group) { @@ -414,4 +607,196 @@ router.get('/audit-logs', async (req, res, next) => { } catch (err) { next(err); } }); +// ── Resource Secrets API (OpenBao KV-v2 under secret/data/resources//conf) ── +const SECRET_KEY_REGEX = /^[A-Za-z0-9_]+$/; + +router.get('/resources/:id/secrets', async (req, res, next) => { + try { + const resource = await Resource.get(req.params.id); + if (!resource) return res.status(404).json({ status: 'error', message: 'resource not found' }); + const baoConf = require('@simpleworkjs/bao-conf'); + + // Read resource secrets from OpenBao + const path = `secret/data/resources/${resource.slug}/conf`; + const r = await baoConf.request('GET', path); + let secretsMap = {}; + if (r.ok) { + const body = await r.json().catch(() => ({})); + secretsMap = (body.data && body.data.data) || {}; + } + + // Zero-View Security: Return metadata only, NEVER return raw secret values + const secrets = Object.keys(secretsMap).map(key => { + const val = String(secretsMap[key] || ''); + let isInherited = false; + let parentSlug = null; + let parentKey = null; + + if (val.startsWith('INHERIT:')) { + isInherited = true; + const parts = val.split(':'); + if (parts.length >= 3) { + parentSlug = parts[1]; + parentKey = parts[2]; + } else if (parts.length === 2) { + parentKey = parts[1]; + } + } + + return { + key, + hasValue: val.length > 0, + isInherited, + parentSlug, + parentKey + }; + }); + + // Explicit Secret Inheritance Lineage: + // Find ancestor resources in direct upward path (Host, Cluster, Site) + const parentSecrets = []; + const seenAncestors = new Set(); + + const ancestors = await Resource.findAllAncestors(resource.id).catch(() => []); + const sites = await Resource.list({ where: { kind: 'site' } }).catch(() => []); + const candidateAncestors = [...ancestors]; + for (const site of sites) { + if (!candidateAncestors.some(a => a.id === site.id)) { + candidateAncestors.push(site); + } + } + + for (const parent of candidateAncestors) { + if (!parent || parent.id === resource.id || seenAncestors.has(parent.id)) continue; + seenAncestors.add(parent.id); + + const parentPath = `secret/data/resources/${parent.slug}/conf`; + const parentR = await baoConf.request('GET', parentPath); + if (parentR.ok) { + const parentBody = await parentR.json().catch(() => ({})); + const pMap = (parentBody.data && parentBody.data.data) || {}; + for (const pKey of Object.keys(pMap)) { + const pVal = String(pMap[pKey] || ''); + // Ancestor's own secrets (not pointers) are candidates for explicit inheritance + if (!pVal.startsWith('INHERIT:')) { + parentSecrets.push({ + parentSlug: parent.slug, + parentName: `${parent.name} (${parent.kind ? parent.kind.toUpperCase() : 'ANCESTOR'})`, + key: pKey + }); + } + } + } + } + + res.json({ status: 'ok', resourceId: resource.id, slug: resource.slug, secrets, parentSecrets }); + } catch (err) { next(err); } +}); + +router.post('/resources/:id/secrets', async (req, res, next) => { + try { + const resource = await Resource.get(req.params.id); + if (!resource) return res.status(404).json({ status: 'error', message: 'resource not found' }); + const baoConf = require('@simpleworkjs/bao-conf'); + const path = `secret/data/resources/${resource.slug}/conf`; + + // Fetch existing secret map from OpenBao so new/edited keys are merged and non-target keys preserved + let currentMap = {}; + try { + const getRes = await baoConf.request('GET', path); + if (getRes.ok) { + const body = await getRes.json().catch(() => ({})); + currentMap = (body.data && body.data.data) || {}; + } + } catch (e) {} + + if (req.body.action === 'delete' && req.body.key) { + delete currentMap[req.body.key]; + } else if (req.body.secrets && typeof req.body.secrets === 'object') { + for (const [key, val] of Object.entries(req.body.secrets)) { + if (!SECRET_KEY_REGEX.test(key)) { + return res.status(400).json({ + status: 'error', + message: `Invalid secret key '${key}'. Keys must contain only letters, numbers, and underscores (e.g. DB_PASSWORD)` + }); + } + currentMap[key] = val; + } + } + + const r = await baoConf.request('POST', path, { data: currentMap }); + if (!r.ok) { + return res.status(500).json({ status: 'error', message: 'failed to save secrets to OpenBao' }); + } + res.json({ status: 'ok', keys: Object.keys(currentMap) }); + } catch (err) { next(err); } +}); + +router.get('/resources/:id/grants', async (req, res, next) => { + try { + const { SharedSecretGrant } = require('../models/shared_secret_grant'); + const { SharedSecret } = require('../models/shared_secret'); + const resource = await Resource.get(req.params.id); + if (!resource) return res.status(404).json({ status: 'error', message: 'resource not found' }); + const grants = await SharedSecretGrant.listForGrantee('resource', resource.id); + const sharedSecretIds = grants.map(g => g.secretId); + const secrets = sharedSecretIds.length ? await SharedSecret.list({ where: { id: { in: sharedSecretIds } } }) : []; + res.json({ status: 'ok', grants: secrets.map(s => ({ id: s.id, slug: s.slug, description: s.description })) }); + } catch (err) { next(err); } +}); + +router.post('/resources/:id/grants', async (req, res, next) => { + try { + const { SharedSecretGrant } = require('../models/shared_secret_grant'); + const { SharedSecret } = require('../models/shared_secret'); + const resource = await Resource.get(req.params.id); + if (!resource) return res.status(404).json({ status: 'error', message: 'resource not found' }); + const { secretSlug, action } = req.body || {}; + const secret = await SharedSecret.getBySlug(secretSlug); + if (!secret) return res.status(404).json({ status: 'error', message: `shared secret '${secretSlug}' not found` }); + + if (action === 'revoke') { + const existing = await SharedSecretGrant.list({ where: { secretId: secret.id, granteeType: 'resource', granteeId: resource.id } }); + for (const g of existing) await g.delete(); + return res.json({ status: 'ok', message: 'grant revoked' }); + } else { + await SharedSecretGrant.grant({ secretId: secret.id, granteeType: 'resource', granteeId: resource.id, grantedBy: req.user.uid }); + return res.json({ status: 'ok', message: 'grant created' }); + } + } catch (err) { next(err); } +}); + +// ── Subtype Drivers Operations API ─────────────────────────────────────────── +const DriverRegistry = require('../services/driver_registry'); + +router.get('/resources/:id/driver-metrics', async (req, res, next) => { + try { + const resource = await Resource.get(req.params.id); + if (!resource) return res.status(404).json({ status: 'error', message: 'resource not found' }); + const metrics = await DriverRegistry.getMetrics(resource); + res.json({ status: 'ok', resourceId: resource.id, metrics }); + } catch (err) { next(err); } +}); + +router.post('/resources/:id/driver-action', async (req, res, next) => { + try { + const resource = await Resource.get(req.params.id); + if (!resource) return res.status(404).json({ status: 'error', message: 'resource not found' }); + const { action, params } = req.body || {}; + if (!action) return res.status(400).json({ status: 'error', message: 'action is required' }); + const result = await DriverRegistry.execAction(resource, action, params || {}); + res.json({ status: 'ok', resourceId: resource.id, result }); + } catch (err) { next(err); } +}); + +router.get('/resources/:id/driver-logs', async (req, res, next) => { + try { + const resource = await Resource.get(req.params.id); + if (!resource) return res.status(404).json({ status: 'error', message: 'resource not found' }); + const lines = parseInt(req.query.lines, 10) || 100; + const logs = await DriverRegistry.getLogs(resource, lines); + res.json({ status: 'ok', resourceId: resource.id, logs }); + } catch (err) { next(err); } +}); + module.exports = router; diff --git a/nodejs/routes/api_ldap.js b/nodejs/routes/api_ldap.js new file mode 100644 index 0000000..0cda320 --- /dev/null +++ b/nodejs/routes/api_ldap.js @@ -0,0 +1,105 @@ +'use strict'; + +// LDAP-over-HTTPS API (DESIGN.md §3). +// +// The whole point of this API is that a client stops speaking LDAP and instead +// does an HTTPS call to the SSO, where the directory is reachable. That kills +// the hostname / cross-network / LDAPS-cert-chain pain: no LDAP protocol, no +// cert to trust, no firewall rule. +// +// POST /api/v1/ldap/bind {username, password} -> 200 {dn, uid} | 401 +// POST /api/v1/ldap/search {base_dn, scope, filter, attributes} -> 200 {entries} +// +// Caller auth: a Bearer token in the Authorization header. Two kinds of caller +// are accepted, reusing existing credentials: +// - an agent token (the same one the agent presents on its WSS channel) — the +// caller is a node acting for SSSD; +// - a self-service API token (PAT, `sso_...`) — the caller is a user/app. +// The API authorizes the *caller*; OpenLDAP enforces the actual directory ACLs. +// +// Security note on /search: it runs under the directory admin bind (withClient), +// so it can read the whole tree. It is therefore restricted to agent callers +// (the SSSD user/group-resolution use case) and must eventually move to a +// scoped read-only service account rather than the admin bind. See DESIGN.md §9. + +const express = require('express'); +const { createLdapClient } = require('@simpleworkjs/ldap'); +const conf = require('@simpleworkjs/conf').ldap; +const { Agent } = require('../models/agent'); +const { ApiToken } = require('../models/api_token'); + +const router = express.Router(); +const ldap = createLdapClient(conf); + +// Resolve a Bearer token to a caller identity, or null. Tries the agent token +// first, then a PAT. Every failure collapses to null so a probing caller learns +// nothing about which credential was wrong. +async function authenticateCaller(req) { + const auth = req.headers['authorization'] || ''; + const m = /^Bearer\s+(.+)$/i.exec(auth); + if (!m) return null; + const token = String(m[1]).trim(); + if (!token) return null; + + try { + const agent = await Agent.authenticate(token); + if (agent) return { kind: 'agent', id: agent.id, name: agent.name }; + } catch (_) {} + + try { + const pat = await ApiToken.authenticate(token); + if (pat) return { kind: 'user', id: pat.created_by }; + } catch (_) {} + + return null; +} + +// POST /bind — authenticate a username/password against the directory. +router.post('/bind', async (req, res, next) => { + try { + const caller = await authenticateCaller(req); + if (!caller) return res.status(401).json({ status: 'error', message: 'unauthorized' }); + + const { username, password } = req.body || {}; + if (!username || !password) { + return res.status(400).json({ status: 'error', message: 'username and password are required' }); + } + + // Resolve the username to a DN, then simple-bind as that DN. A missing user + // and a wrong password both surface as 401 (no user-existence oracle). + const user = await ldap.getUser(String(username)); + if (!user) return res.status(401).json({ status: 'error', message: 'invalid credentials' }); + + const ok = await ldap.checkPassword(user.dn, String(password)); + if (!ok) return res.status(401).json({ status: 'error', message: 'invalid credentials' }); + + return res.json({ status: 'ok', dn: user.dn, uid: user.uid }); + } catch (err) { next(err); } +}); + +// POST /search — run a directory search. Agent callers only (see header note). +router.post('/search', async (req, res, next) => { + try { + const caller = await authenticateCaller(req); + if (!caller) return res.status(401).json({ status: 'error', message: 'unauthorized' }); + if (caller.kind !== 'agent') { + return res.status(403).json({ status: 'error', message: 'search is restricted to agents' }); + } + + const { base_dn, scope, filter, attributes } = req.body || {}; + if (!filter) return res.status(400).json({ status: 'error', message: 'filter is required' }); + + const entries = await ldap.withClient(async (client) => { + const { searchEntries } = await client.search(base_dn || conf.userBase, { + scope: scope || 'sub', + filter: String(filter), + attributes: Array.isArray(attributes) && attributes.length ? attributes : undefined, + }); + return searchEntries; + }); + + return res.json({ status: 'ok', entries }); + } catch (err) { next(err); } +}); + +module.exports = router; diff --git a/nodejs/routes/api_shared_secrets.js b/nodejs/routes/api_shared_secrets.js new file mode 100644 index 0000000..71b3b92 --- /dev/null +++ b/nodejs/routes/api_shared_secrets.js @@ -0,0 +1,213 @@ +'use strict'; + +// Shared-secrets API. +// +// A shared secret is metadata in the DB (SharedSecret + SharedSecretGrant) with +// its DATA in OpenBao at secret/shared// (KV-v2). The owner has +// full R/W/list on their own secret/shared//* subtree; each grantee's +// OpenBao policy content is edited to add read on the exact shared path (see +// vault_broker.js grantSharedSecret/revokeSharedSecret). Enforcement is entirely +// the OpenBao ACL — the broker's policy reconciliation makes a grant effective +// immediately, with no token re-mint. +// +// Reads of the secret DATA are intentionally NOT proxied here: the UI fetches +// them through the existing /api/vault proxy using the requester's own session +// token, so OpenBao ACL enforces read access per-request. This router handles +// metadata CRUD + grant management; KV writes (create/update/delete) are made +// server-side using the acting user's scoped token. + +const express = require('express'); +const baoConf = require('@simpleworkjs/bao-conf'); +const permission = require('../utils/permission'); +const { SharedSecret } = require('../models/shared_secret'); +const { SharedSecretGrant } = require('../models/shared_secret_grant'); +const vaultBroker = require('../utils/vault_broker'); + +const ADMIN_GROUPS = ['app_sso_admin', 'app_super_admin', 'app_sso_directory_admin']; +// Allow hyphens AND underscores (matching the plugin-instance slug convention); +// only reject values that can't be a sane secret path segment (spaces, slashes, +// leading non-alnum, too long). +const SLUG_RE = /^[a-z0-9][a-z0-9_-]{0,63}$/; + +const router = express.Router(); + +// Machine/service tokens cannot manage shared secrets (mirrors scopeGuard on the +// /api/vault proxy — personal, per-user secret management only). +router.use((req, res, next) => { + if (req.user && req.user.isMachine) { + return res.status(403).json({ error: 'machine tokens cannot manage shared secrets' }); + } + next(); +}); + +async function isAdmin(user) { + try { await permission.byGroup(user, ADMIN_GROUPS); return true; } + catch (e) { return false; } +} + +// Scoped OpenBao token for an actor, used for server-side KV writes. Owner uses +// their own token (R/W on secret/shared//*); an admin uses the +// sso-admin token (R/W on secret/*). +async function actorToken(user, ownerUid) { + if (user.uid === ownerUid) return vaultBroker.getOrCreateUserToken(ownerUid); + if (await isAdmin(user)) return vaultBroker.getOrCreateAdminToken(user.uid); + return null; +} + +// Does this user manage the given shared secret? Owner or admin. +async function canManage(user, secret) { + if (user.uid === secret.ownerUid) return true; + return isAdmin(user); +} + +async function loadSecret(req, res) { + const secret = await SharedSecret.get(req.params.id); + if (!secret) { res.status(404).json({ error: 'not found' }); return null; } + return secret; +} + +// ── List: mine + shared-with-me ───────────────────────────────────────────── +router.get('/', async (req, res, next) => { + try { + const uid = req.user.uid; + const mine = await SharedSecret.list({ where: { ownerUid: uid } }); + const grants = await SharedSecretGrant.listForGrantee('user', uid); + const granteeSecretIds = [...new Set(grants.map(g => g.secretId))]; + const granted = granteeSecretIds.length + ? await SharedSecret.list({ where: { id: { in: granteeSecretIds } } }) : []; + const byId = new Map(mine.map(s => [s.id, { role: 'owner', ...s }])); + for (const g of granted) { + if (byId.has(g.id)) continue; // already owner + byId.set(g.id, { role: 'grantee', ...g }); + } + // The `{ role, ...s }` spread above copies only own properties, so the + // instance method `path()` is dropped -- call the static builder instead. + res.json({ items: [...byId.values()].map(s => ({ id: s.id, slug: s.slug, ownerUid: s.ownerUid, description: s.description, path: SharedSecret.pathFor(s.ownerUid, s.slug), role: s.role })) }); + } catch (e) { next(e); } +}); + +// ── Create ────────────────────────────────────────────────────────────────── +router.post('/', async (req, res, next) => { + try { + const uid = req.user.uid; + const slug = String(req.body.slug || '').trim().toLowerCase(); + if (!SLUG_RE.test(slug)) return res.status(400).json({ error: 'slug must be lowercase letters/digits/hyphens/underscores, 1-64 chars' }); + const description = String(req.body.description || '').trim(); + const data = (req.body.data && typeof req.body.data === 'object') ? req.body.data : {}; + + if (await SharedSecret.getBySlug(slug)) { + return res.status(409).json({ error: `a shared secret named '${slug}' already exists` }); + } + const token = await actorToken(req.user, uid); + if (!token) return res.status(403).json({ error: 'not allowed' }); + const path = SharedSecret.pathFor(uid, slug); + await baoConf.set(path, data, { token }); + + const secret = await SharedSecret.create({ + slug, ownerUid: uid, description, + created_by: uid, created_on: Date.now(), updated_by: uid, updated_on: Date.now(), + }); + res.status(201).json({ id: secret.id, slug, ownerUid: uid, description, path, role: 'owner' }); + } catch (e) { next(e); } +}); + +// ── Detail (metadata; data is read via /api/vault proxy) ──────────────────── +router.get('/:id', async (req, res, next) => { + try { + const secret = await loadSecret(req, res); + if (!secret) return; + const uid = req.user.uid; + const admin = await isAdmin(req.user); + const grantee = (await SharedSecretGrant.listForGrantee('user', uid)).some(g => g.secretId === secret.id); + if (!admin && uid !== secret.ownerUid && !grantee) return res.status(403).json({ error: 'not shared with you' }); + const grants = await SharedSecretGrant.list({ where: { secretId: secret.id } }); + res.json({ id: secret.id, slug: secret.slug, ownerUid: secret.ownerUid, description: secret.description, path: secret.path(), role: uid === secret.ownerUid ? 'owner' : (admin ? 'admin' : 'grantee'), grants: grants.map(g => ({ id: g.id, granteeType: g.granteeType, granteeId: g.granteeId, capability: g.capability })) }); + } catch (e) { next(e); } +}); + +// ── Update data / description ─────────────────────────────────────────────── +router.put('/:id', async (req, res, next) => { + try { + const secret = await loadSecret(req, res); + if (!secret) return; + if (!(await canManage(req.user, secret))) return res.status(403).json({ error: 'only the owner (or admin) can edit a shared secret' }); + const token = await actorToken(req.user, secret.ownerUid); + const update = {}; + if (req.body && typeof req.body.data === 'object') { + await baoConf.set(secret.path(), req.body.data, { token }); + } + if (req.body && req.body.description !== undefined) { + update.description = String(req.body.description).trim(); + } + if (Object.keys(update).length) { + update.updated_by = req.user.uid; + update.updated_on = Date.now(); + await secret.update(update); + } + res.json({ id: secret.id, slug: secret.slug, ownerUid: secret.ownerUid, description: secret.description, path: secret.path() }); + } catch (e) { next(e); } +}); + +// ── Delete (KV + DB row + all grants) ─────────────────────────────────────── +router.delete('/:id', async (req, res, next) => { + try { + const secret = await loadSecret(req, res); + if (!secret) return; + if (!(await canManage(req.user, secret))) return res.status(403).json({ error: 'only the owner (or admin) can delete a shared secret' }); + const token = await actorToken(req.user, secret.ownerUid); + // Revoke all grants first so grantees' policies drop the path. + const grants = await SharedSecretGrant.list({ where: { secretId: secret.id } }); + for (const g of grants) await vaultBroker.revokeSharedSecret(g.id, req.user.uid); + // Delete the KV data (metadata delete removes all versions), then the row. + try { await baoConf.request('DELETE', `secret/metadata/${secret.path()}`, undefined, { token }); } catch (e) { /* best-effort */ } + await secret.delete(); + res.status(204).end(); + } catch (e) { next(e); } +}); + +// ── Grants: list ──────────────────────────────────────────────────────────── +router.get('/:id/grants', async (req, res, next) => { + try { + const secret = await loadSecret(req, res); + if (!secret) return; + if (!(await canManage(req.user, secret))) return res.status(403).json({ error: 'only the owner (or admin) can manage grants' }); + const grants = await SharedSecretGrant.list({ where: { secretId: secret.id } }); + res.json({ grants: grants.map(g => ({ id: g.id, granteeType: g.granteeType, granteeId: g.granteeId, capability: g.capability })) }); + } catch (e) { next(e); } +}); + +// ── Grants: create ────────────────────────────────────────────────────────── +router.post('/:id/grants', async (req, res, next) => { + try { + const secret = await loadSecret(req, res); + if (!secret) return; + if (!(await canManage(req.user, secret))) return res.status(403).json({ error: 'only the owner (or admin) can manage grants' }); + const granteeType = String(req.body.granteeType || '').trim(); + const granteeId = String(req.body.granteeId || '').trim(); + if (!['user', 'app'].includes(granteeType)) return res.status(400).json({ error: 'granteeType must be user or app' }); + if (!granteeId) return res.status(400).json({ error: 'granteeId is required' }); + if (granteeId === secret.ownerUid && granteeType === 'user') { + return res.status(400).json({ error: 'the owner already has access' }); + } + // Idempotent: skip if the grant already exists. + const existing = (await SharedSecretGrant.list({ where: { secretId: secret.id, granteeType, granteeId } }))[0]; + if (existing) return res.json({ id: existing.id, granteeType, granteeId, capability: existing.capability }); + const grant = await vaultBroker.grantSharedSecret(secret.id, granteeType, granteeId, req.user.uid); + res.status(201).json({ id: grant.id, granteeType, granteeId, capability: grant.capability }); + } catch (e) { next(e); } +}); + +// ── Grants: revoke ────────────────────────────────────────────────────────── +router.delete('/:id/grants/:grantId', async (req, res, next) => { + try { + const secret = await loadSecret(req, res); + if (!secret) return; + if (!(await canManage(req.user, secret))) return res.status(403).json({ error: 'only the owner (or admin) can manage grants' }); + const grant = await SharedSecretGrant.get(req.params.grantId); + if (!grant || grant.secretId !== secret.id) return res.status(404).json({ error: 'grant not found' }); + await vaultBroker.revokeSharedSecret(grant.id, req.user.uid); + res.status(204).end(); + } catch (e) { next(e); } +}); + +module.exports = router; diff --git a/nodejs/routes/discovery.js b/nodejs/routes/discovery.js index 6185aa2..b63d518 100644 --- a/nodejs/routes/discovery.js +++ b/nodejs/routes/discovery.js @@ -186,8 +186,11 @@ router.post('/promote/:slug', async (req, res, next) => { const meta = resource.metadata || {}; meta.managed = true; - await resource.update({ metadata: meta }); - + // `Resource.update` is not a static — `update` is an instance method + // (@simpleworkjs/orm). Load a fresh instance and call it on that. + const inst = await Resource.get(resource.id); + await inst.update({ metadata: meta }); + res.json(envelope({ success: true, groups: [accessGroup, adminGroup] })); } catch (err) { next(err); } }); diff --git a/nodejs/routes/docs.js b/nodejs/routes/docs.js index 5351b96..54357ce 100644 --- a/nodejs/routes/docs.js +++ b/nodejs/routes/docs.js @@ -34,9 +34,14 @@ const DOCS = { 'oauth-apps': {title: 'Connecting Apps (SSO)', file: path.join(__dirname, '../../docs/concepts-oauth-apps.md')}, 'api-tokens': {title: 'API Tokens', file: path.join(__dirname, '../../docs/concepts-api-tokens.md')}, directory: {title: 'Directory & Inventory', file: path.join(__dirname, '../../docs/directory.md')}, - agents: {title: 'Plugins', file: path.join(__dirname, '../../docs/plugins.md')}, + // `agents` pointed at plugins.md, so docs/agents.md -- the theta-agent + // guide the Directory links to -- was unreachable in the app. + agents: {title: 'Theta Agent', file: path.join(__dirname, '../../docs/agents.md')}, plugins: {title: 'Plugins', file: path.join(__dirname, '../../docs/plugins.md')}, + // The Discovery tab's help icon links here; without an entry it 404'd. + discovery: {title: 'Discovery & Inventory', file: path.join(__dirname, '../../docs/discovery.md')}, vault: {title: 'Vault Secrets', file: path.join(__dirname, '../../docs/vault.md')}, + groups: {title: 'Groups & Permissions', file: path.join(__dirname, '../../docs/groups.md')}, overview: {title: 'Overview', file: path.join(__dirname, '../../README.md')}, changelog: {title: 'Changelog', file: path.join(__dirname, '../../CHANGELOG.md')}, @@ -54,7 +59,7 @@ const docList = Object.entries(DOCS).map(([slug, d]) => ({slug, title: d.title}) // only resolves correctly on GitHub. Serve that same folder here and rewrite // the rendered markup to point at it absolutely, so the images work when // read from /docs/overview too. -router.use('/images', require('express').static(path.join(__dirname, '../../docs/images'))); +router.use('/docs/images', require('express').static(path.join(__dirname, '../../docs/images'))); function fixImagePaths(html) { return html.replace(/(["(])docs\/images\//g, '$1/docs/images/'); } diff --git a/nodejs/routes/index.js b/nodejs/routes/index.js index f021c89..ce03855 100755 --- a/nodejs/routes/index.js +++ b/nodejs/routes/index.js @@ -84,30 +84,11 @@ router.get('/discovery', function(req, res, next) { }); router.get('/plugins', function(req, res, next) { - // Plugin instances page — loadable/unloadable, configurable plugin copies - // with per-instance secrets in OpenBao. Renders the shell for anyone; the - // client gates with app.auth.forceLogin(['app_sso_admin', - // 'app_sso_directory_admin','admin']) and the /api/plugins endpoints enforce - // the same server-side. Same header-vs-navigation auth model as /conf and - // /vault (auth-token is a client-set header, not a cookie). - const registry = require('../services/plugin_registry'); - res.render('plugins', {...values, pluginTypes: registry.types }); + res.redirect('/directory'); }); router.get('/vault', function(req, res) { - // Personal per-user secrets (secret/users//*) for everyone; admins get - // free-form access across all of secret/ plus an Apps tab to mint scoped - // tokens for external apps. The view renders the shell for any logged-in - // user; the client gates login via app.auth.forceLogin() and derives the - // admin/namespace scope from /api/user/me. The /api/vault proxy enforces the - // same scoping server-side (scopeGuard + the token's own OpenBao policy), so - // the client-derived scope is only cosmetic. vaultAddr is the only - // server-rendered value (it's a non-user-specific env var); uid + isAdmin - // are resolved client-side to avoid the header-vs-navigation auth mismatch. - res.render('vault', { - ...values, - vaultAddr: process.env.VAULT_ADDR || 'http://openbao:8200', - }); + res.redirect('/conf'); }); // Linkable deep-link to a single resource's modal, e.g. from the resource @@ -195,10 +176,6 @@ router.get('/users/:uid', function(req, res, next) { res.render('profile', {...values}); }); -router.get('/groups', function(req, res, next) { - res.render('groups', {...values}); -}); - router.get('/token', function(req, res, next) { res.render('token', {...values}); }); diff --git a/nodejs/routes/user.js b/nodejs/routes/user.js index 62da372..bfa3586 100755 --- a/nodejs/routes/user.js +++ b/nodejs/routes/user.js @@ -90,7 +90,13 @@ router.get('/me', async function(req, res, next){ // same answer in both modes. const groups = await groupCns(user); user.groups = groups; - user.isAdmin = groups.includes('app_sso_admin') || groups.includes(permission.SUPER_ADMIN_GROUP); + // Console admin under the group model (docs/GROUPS.md §11): god_admin, + // a site super admin, the SSO-as-app admin ({site}_app_sso_admin), or the + // legacy app_sso_admin/app_super_admin during migration. + user.isAdmin = groups.some((g) => + g === 'app_sso_admin' || g === 'app_super_admin' || + g === 'god_admin' || g === permission.SUPER_ADMIN_GROUP || + g.endsWith('_super_admin') || g.endsWith('_app_sso_admin')); return res.json(user); }catch(error){ diff --git a/nodejs/services/discovery_reconciler.js b/nodejs/services/discovery_reconciler.js index cfa98e6..6c4fc93 100644 --- a/nodejs/services/discovery_reconciler.js +++ b/nodejs/services/discovery_reconciler.js @@ -2,42 +2,121 @@ const { Resource, ResourceEdge, ResourceGroup } = require('../models/resource'); const { WebhookEmitter } = require('./webhook_emitter'); const crypto = require('crypto'); +// Is `candidateId` at or below `rootId` in the edge graph? Used to refuse an +// edge that would close a loop. Carries its own visited set so it terminates +// even if the stored graph already contains a cycle from an older release. +function isDescendant(candidateId, rootId, edges) { + const seen = new Set(); + const stack = [rootId]; + while (stack.length) { + const id = stack.pop(); + if (id === candidateId) return true; + if (seen.has(id)) continue; + seen.add(id); + for (const e of edges) if (e.parentId === id) stack.push(e.childId); + } + return false; +} + class DiscoveryReconciler { - static async reconcile(sourceName, payload) { + static async reconcile(sourceName, payload, options = {}) { const { resources = [], edges = [] } = payload; let newDevices = 0; + const location = options.location || options.site || null; + const autoPromote = !!options.autoPromote; + + let targetSite = null; + if (location && String(location).trim()) { + const sites = await Resource.list({ where: { kind: 'site' } }); + const locStr = String(location).trim().toLowerCase(); + targetSite = sites.find(s => s.name.toLowerCase() === locStr || s.slug.toLowerCase() === locStr); + if (!targetSite) { + const locSlug = `site-${locStr.replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '')}`; + targetSite = await Resource.create({ + id: crypto.randomUUID(), + kind: 'site', + name: String(location).trim(), + slug: locSlug, + created_on: Math.floor(Date.now() / 1000) + }).catch(() => null); + } + } + if (!targetSite) { + const sites = await Resource.list({ where: { kind: 'site' } }); + if (sites && sites.length > 0) { + targetSite = sites[0]; + } else { + targetSite = await Resource.create({ + id: crypto.randomUUID(), + kind: 'site', + name: 'Default Site', + slug: 'site-default', + created_on: Math.floor(Date.now() / 1000) + }).catch(() => null); + } + } + + const normalizeMac = (m) => (m || '').toLowerCase().replace(/[^a-f0-9]/g, ''); + const normalizeHost = (h) => (h || '').toLowerCase().split('.')[0].trim(); + + // Read the inventory ONCE, not once per incoming resource. A Proxmox + // cluster reports ~55 resources against an inventory of similar size, so + // the per-iteration Resource.list() was doing quadratic full-table reads + // every discovery run. Newly created rows are pushed onto this list as we + // go, so later resources in the same payload still match against them. + const allRes = await Resource.list(); for (const res of resources) { if (!res.metadata) res.metadata = {}; + if (autoPromote) res.metadata.managed = true; res._originalSlug = res.slug; // Keep track for edge mapping - + let existing = null; - - // Attempt matching by MAC if available (case-insensitive) + + // A discovered device may only merge into a resource of the same kind + // (or into a placeholder from an earlier, kind-less discovery). Without + // this a VM called "gitea-runner" matches a hand-created *service* of + // the same name on rule 3 and silently overwrites it -- the discovered + // host's metadata lands on a service row, and the operator's entry is + // gone. `template` counts as `host`: a VM converted to a template is the + // same device, and it should update in place rather than fork a row. + const kindClass = (k) => (k === 'template' ? 'host' : k); + const incomingKind = kindClass(res.kind || 'unmanaged_device'); + const kindCompatible = (r) => { + const k = kindClass(r.kind); + if (k === 'unmanaged_device' || incomingKind === 'unmanaged_device') return true; + return k === incomingKind; + }; + const candidates = allRes.filter(kindCompatible); + + // 1. Attempt matching by MAC (highest precision) if (res.metadata.interfaces && res.metadata.interfaces.length > 0) { - const macs = res.metadata.interfaces.map(i => i.mac ? i.mac.toLowerCase() : null).filter(m => !!m); + const macs = res.metadata.interfaces.map(i => normalizeMac(i.mac)).filter(m => m.length === 12); if (macs.length > 0) { - const allRes = await Resource.list(); - existing = allRes.find(r => - r.metadata && r.metadata.interfaces && - r.metadata.interfaces.some(i => i.mac && macs.includes(i.mac.toLowerCase())) + existing = candidates.find(r => + r.metadata && ( + (r.metadata.macAddress && macs.includes(normalizeMac(r.metadata.macAddress))) || + (r.metadata.interfaces && r.metadata.interfaces.some(i => macs.includes(normalizeMac(i.mac)))) + ) ); } } - - // Fallback matching by IP if no MAC match (weaker) + + // 2. Fallback matching by IP address let ipsToMatch = []; if (res.metadata.interfaces) { ipsToMatch = res.metadata.interfaces.map(i => i.ip).filter(i => !!i); } + if (res.metadata.ip) ipsToMatch.push(res.metadata.ip); if (res.metadata.address) { res.metadata.address.split(',').forEach(a => ipsToMatch.push(a.trim())); } - + ipsToMatch = [...new Set(ipsToMatch.filter(Boolean))]; + if (!existing && ipsToMatch.length > 0) { - const allRes = await Resource.list(); - existing = allRes.find(r => { + existing = candidates.find(r => { if (!r.metadata) return false; + if (r.metadata.ip && ipsToMatch.includes(r.metadata.ip)) return true; if (r.metadata.address) { const addrs = r.metadata.address.split(',').map(a => a.trim()); if (addrs.some(a => ipsToMatch.includes(a))) return true; @@ -46,14 +125,17 @@ class DiscoveryReconciler { return false; }); } - - // Fallback matching by Slug or Name + + // 3. Fallback matching by Slug, Name, or Base Hostname if (!existing && (res.slug || res.name)) { - const allRes = await Resource.list(); - existing = allRes.find(r => - (res.slug && r.slug === res.slug) || - (res.name && r.name && r.name.toLowerCase() === res.name.toLowerCase()) - ); + const inputName = normalizeHost(res.name || res.slug); + existing = candidates.find(r => { + if (res.slug && r.slug === res.slug) return true; + if (res.name && r.name && r.name.toLowerCase() === res.name.toLowerCase()) return true; + if (inputName && r.name && normalizeHost(r.name) === inputName) return true; + if (inputName && r.slug && normalizeHost(r.slug) === inputName) return true; + return false; + }); } if (existing) { @@ -83,10 +165,33 @@ class DiscoveryReconciler { mergedMeta.last_seen = Date.now(); - const isIp = (str) => /^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$/.test(str || ''); + // Pick the most human name across sources. Rank first, length only as + // a tie-break within a rank -- comparing lengths alone let a UniFi + // client named after its MAC ("ac:16:2d:b3:da:80", 17 chars) beat the + // hypervisor's real hostname from Proxmox ("dl380-0", 7), so the + // Directory listed MAC addresses where host names belong. + // + // NB: `\\.` inside a regex LITERAL matches a backslash, not a dot, so + // the old isIp returned false for every input and IP-shaped names were + // never replaced either. It is `\.` here. + const isIp = (str) => /^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$/.test(str || ''); + const isMac = (str) => /^([0-9a-f]{2}[:-]){5}[0-9a-f]{2}$/i.test((str || '').trim()); + // 2 = a real name, 1 = an IP (at least routable/recognizable), 0 = a + // MAC or nothing (pure machine identifier, the worst thing to show). + const nameRank = (str) => { + if (!str || !String(str).trim()) return 0; + if (isMac(str)) return 0; + if (isIp(str)) return 1; + return 2; + }; + let bestName = existing.name; - if (res.name && (!bestName || isIp(bestName) || res.name.length > bestName.length && !isIp(res.name))) { - bestName = res.name; + if (res.name) { + const incoming = nameRank(res.name); + const current = nameRank(bestName); + if (incoming > current || (incoming === current && res.name.length > (bestName || '').length)) { + bestName = res.name; + } } await existing.update({ @@ -115,12 +220,17 @@ class DiscoveryReconciler { newDevices++; res._actualId = created.id; // Map original slug to actual ID + // Make it visible to the rest of THIS payload: a Proxmox run reports + // the endpoint, then its nodes, then their guests, and two of them can + // legitimately share a MAC/IP. Without this the same device could be + // created twice in a single run. + allRes.push(created); WebhookEmitter.emit('discovery.new_device', created.toJSON()); } } - // Now process edges - const allRes = await Resource.list(); + // Now process edges. `allRes` above is already current -- rows created in + // the loop were pushed onto it -- so no second full read is needed. const existingEdges = await ResourceEdge.list(); for (const edge of edges) { @@ -144,19 +254,80 @@ class DiscoveryReconciler { if (childResInDb) childId = childResInDb.id; } + // Two slugs in one payload can resolve to the SAME resource once the + // matcher has merged them -- a Proxmox endpoint reached at the address + // of the node that answers for it is the case that produced this. The + // edge would then make a resource its own parent, which renders as an + // infinitely nested tree and defeats every ancestor walk in the app + // (findAncestorSiteSlug, withResolvedAddress) that relies on a cycle + // guard to terminate rather than to be correct. + if (parentId && childId && parentId === childId) { + console.warn(`[DiscoveryReconciler] ${sourceName}: dropping self-edge on ${edge.parentSlug} -> ${edge.childSlug} (both resolved to the same resource)`); + continue; + } + + // Likewise refuse an edge that closes a loop: if the proposed parent is + // already a descendant of the proposed child, adding this makes a cycle. + if (parentId && childId && isDescendant(parentId, childId, existingEdges)) { + console.warn(`[DiscoveryReconciler] ${sourceName}: dropping ${edge.parentSlug} -> ${edge.childSlug} (would create a cycle)`); + continue; + } + if (parentId && childId) { const edgeExists = existingEdges.find(e => e.parentId === parentId && e.childId === childId && e.relation === edge.relation); if (!edgeExists) { - await ResourceEdge.create({ + const created = await ResourceEdge.create({ id: crypto.randomUUID(), parentId, childId, relation: edge.relation }); + // Keep the in-memory edge list current so the cycle check above sees + // edges added earlier in this same payload. + existingEdges.push(created); } } } + if (targetSite) { + const childSlugs = new Set(edges.map(e => e.childSlug)); + for (const res of resources) { + if (res._actualId && res._actualId !== targetSite.id && !childSlugs.has(res._originalSlug || res.slug)) { + const edgeExists = existingEdges.find(e => e.childId === res._actualId); + if (!edgeExists) { + const created = await ResourceEdge.create({ + id: crypto.randomUUID(), + parentId: targetSite.id, + childId: res._actualId, + relation: 'hosts' + }).catch(() => null); + if (created) existingEdges.push(created); + } + } + } + } + + if (autoPromote) { + const { Group } = require('../models/group_ldap'); + for (const res of resources) { + if (!res._actualId) continue; + const accessGroup = `${res.slug}_access`; + const adminGroup = `${res.slug}_admin`; + try { + await Group.get(accessGroup).catch(async (e) => { + if (e.status === 404) await Group.add({ name: accessGroup, description: `Access to ${res.name}`, owner: 'cn=admin' }); + }); + await Group.get(adminGroup).catch(async (e) => { + if (e.status === 404) await Group.add({ name: adminGroup, description: `Admin access to ${res.name}`, owner: 'cn=admin' }); + }); + await ResourceGroup.create({ id: crypto.randomUUID(), resourceId: res._actualId, groupCn: accessGroup, accessLevel: 'user' }).catch(() => {}); + await ResourceGroup.create({ id: crypto.randomUUID(), resourceId: res._actualId, groupCn: adminGroup, accessLevel: 'admin' }).catch(() => {}); + } catch (err) { + console.error(`[DiscoveryReconciler] autoPromote failed for ${res.slug}:`, err.message); + } + } + } + if (newDevices > 0) { console.log(`[DiscoveryReconciler] Source ${sourceName} discovered ${newDevices} new devices.`); } diff --git a/nodejs/services/driver_registry.js b/nodejs/services/driver_registry.js new file mode 100644 index 0000000..e995697 --- /dev/null +++ b/nodejs/services/driver_registry.js @@ -0,0 +1,115 @@ +'use strict'; + +const BaseDriver = require('../drivers/base_driver'); +const ThetaAgentDriver = require('../drivers/theta_agent_driver'); +const ProxmoxDriver = require('../drivers/proxmox_driver'); +const DockerSocketDriver = require('../drivers/docker_socket_driver'); +const DbDriver = require('../drivers/db_driver'); +const NetworkDriver = require('../drivers/network_driver'); +const K8sDriver = require('../drivers/k8s_driver'); +const AgentManager = require('../utils/agent_manager'); + +/** + * Registry & Resolution Engine for Subtype Management and Metrics Drivers. + */ +class DriverRegistry { + constructor() { + this.drivers = []; + this.defaultDriver = new BaseDriver('unmanaged'); + this.initDefaultDrivers(); + } + + initDefaultDrivers() { + this.thetaAgentDriver = new ThetaAgentDriver(); + this.proxmoxDriver = new ProxmoxDriver(); + this.dockerSocketDriver = new DockerSocketDriver(); + this.dbDriver = new DbDriver(); + this.networkDriver = new NetworkDriver(); + this.k8sDriver = new K8sDriver(); + + // Register drivers in priority order + this.register(this.thetaAgentDriver); + this.register(this.proxmoxDriver); + this.register(this.dockerSocketDriver); + this.register(this.dbDriver); + this.register(this.networkDriver); + this.register(this.k8sDriver); + } + + /** + * Register a new subtype driver. + * @param {BaseDriver} driver + */ + register(driver) { + if (driver && typeof driver.getMetrics === 'function') { + this.drivers.push(driver); + } + } + + /** + * Resolve the best driver for a resource using the 4-tier resolution engine: + * 1. Direct theta-agent (if agent connected) + * 2. Subtype-specific driver (Proxmox, Docker, DB, Network, K8s) + * 3. Parent Provider Fallback (e.g. Proxmox hypervisor host for un-agentized LXC/KVM guest) + * 4. Unmanaged fallback + * @param {Object} resource + * @returns {BaseDriver} + */ + async resolveDriver(resource) { + if (!resource) return this.defaultDriver; + + // 1. Direct Theta Agent Check + const agent = await AgentManager.getAgentForResource(resource.id).catch(() => null); + if (agent && agent.isOnline) { + return this.thetaAgentDriver; + } + + // 2. Specialized Subtype Driver Check + for (const driver of this.drivers) { + if (driver !== this.thetaAgentDriver && driver.supports(resource)) { + return driver; + } + } + + // 3. Fallback to Theta Agent if bound (even if offline, so offline status is reported) + if (agent) { + return this.thetaAgentDriver; + } + + // 4. Fallback to Proxmox driver if it's an LXC/KVM guest + const subType = ((resource.metadata && resource.metadata.subType) || '').toLowerCase(); + if (['lxc', 'kvm'].includes(subType)) { + return this.proxmoxDriver; + } + + return this.defaultDriver; + } + + /** + * Get operational telemetry for a resource. + */ + async getMetrics(resource, options = {}) { + const driver = await this.resolveDriver(resource); + return await driver.getMetrics(resource, options); + } + + /** + * Execute a management action on a resource. + */ + async execAction(resource, action, params = {}) { + const driver = await this.resolveDriver(resource); + return await driver.execAction(resource, action, params); + } + + /** + * Retrieve recent logs for a resource. + */ + async getLogs(resource, lines = 100) { + const driver = await this.resolveDriver(resource); + return await driver.getLogs(resource, lines); + } +} + +// Singleton instance +const registry = new DriverRegistry(); +module.exports = registry; diff --git a/nodejs/services/scheduler.js b/nodejs/services/scheduler.js index 0611b49..037ca68 100644 --- a/nodejs/services/scheduler.js +++ b/nodejs/services/scheduler.js @@ -82,7 +82,7 @@ async function runPluginJob(instanceId) { }; const payload = await runFn(cfg); if (instance.category === 'discovery') { - await DiscoveryReconciler.reconcile(instance.slug, payload); + await DiscoveryReconciler.reconcile(instance.slug, payload, cfg); } await instance.update({ lastStatus: STATUS.OK, lastError: null, lastLog: logs.join('\n') }); } catch (err) { diff --git a/nodejs/tests/access_request.test.js b/nodejs/tests/access_request.test.js index ba5e7cc..d73a615 100644 --- a/nodejs/tests/access_request.test.js +++ b/nodejs/tests/access_request.test.js @@ -44,9 +44,10 @@ beforeAll(async () => { expect(host.status).toBe(200); hostId = host.body.results.id; - // Creating a host auto-provisions __access / _admin. - accessGroupCn = `${siteSlug}_${hostSlug}_access`; - const adminGroupCn = `${siteSlug}_${hostSlug}_admin`; + // Creating a host auto-provisions _host__access / _admin + // (docs/GROUPS.md §2 — the kind is part of the name). + accessGroupCn = `${siteSlug}_host_${hostSlug}_access`; + const adminGroupCn = `${siteSlug}_host_${hostSlug}_admin`; // The creator is seeded into both groups -- groupOfNames requires at least // one member, so Group.add puts the owner's DN there -- and _admin is nested @@ -213,8 +214,9 @@ describe('Access requests — withdrawal', () => { expect(host.status).toBe(200); // Same as the top-level setup: step out of the auto-created groups the - // creator is seeded into, or this is a request for access already held. - for (const cn of [`${siteSlug}_${slug}_admin`, `${siteSlug}_${slug}_access`]) { + // creator is seeded into (docs/GROUPS.md §2 — kind is part of the name), + // or this is a request for access already held. + for (const cn of [`${siteSlug}_host_${slug}_admin`, `${siteSlug}_host_${slug}_access`]) { await request(app) .delete(`/api/group/${encodeURIComponent(cn)}/test`) .set('auth-token', token); diff --git a/nodejs/tests/agent_manager.test.js b/nodejs/tests/agent_manager.test.js index 11b227e..08fa43d 100644 --- a/nodejs/tests/agent_manager.test.js +++ b/nodejs/tests/agent_manager.test.js @@ -1,11 +1,45 @@ 'use strict'; const crypto = require('crypto'); -const agentManager = require('../utils/agent_manager'); -describe('AgentManager PROTOCOL.md v1.1.0 Compliance', () => { +// In-memory stand-in for OpenBao. The signing key lives at secret/agent/ +// signing-key in production; here we only need it to persist across calls so +// the "same key every time" property is actually exercised rather than mocked +// away. +const mockBaoStore = new Map(); +jest.mock('@simpleworkjs/bao-conf', () => ({ + get: jest.fn(async (path) => mockBaoStore.get(path) || null), + set: jest.fn(async (path, value) => { mockBaoStore.set(path, value); }), + request: jest.fn(async () => ({ ok: true, status: 200 })) +})); + +const agentManager = require('../utils/agent_manager'); +const agentKeys = require('../utils/agent_keys'); + +// The manager is now keyed by enrolled Agent rows rather than by a bare token +// string, so these use a stub row with the same surface the real model gives: +// an id, and an update() that records what would be persisted. +function stubAgent(overrides = {}) { + const row = { + id: overrides.id || crypto.randomUUID(), + name: overrides.name || 'test-agent', + resourceId: overrides.resourceId || null, + revoked: false, + persisted: {}, + ...overrides + }; + row.update = jest.fn(async (patch) => { + Object.assign(row.persisted, patch); + Object.assign(row, patch); + return row; + }); + return row; +} + +describe('AgentManager PROTOCOL.md v1.2.0 Compliance', () => { let mockWs; let sentMessages; + let agent; beforeEach(() => { sentMessages = []; @@ -14,23 +48,30 @@ describe('AgentManager PROTOCOL.md v1.1.0 Compliance', () => { send: jest.fn((msg) => sentMessages.push(JSON.parse(msg))), close: jest.fn() }; + agent = stubAgent(); }); - test('registers agent and tracks initial connection state', () => { - const record = agentManager.registerAgent('test-token-123', mockWs, '192.168.1.100'); - expect(record.token).toBe('test-token-123'); - expect(record.ipAddress).toBe('192.168.1.100'); - - const agents = agentManager.getConnectedAgents(); - const found = agents.find(a => a.token === 'test-token-123'); - expect(found).toBeDefined(); - expect(found.isOnline).toBe(true); + test('registers an agent and reports it as connected', () => { + agentManager.registerAgent(agent, mockWs, '192.168.1.100'); + const state = agentManager.liveState(agent.id); + expect(state.connected).toBe(true); + expect(state.ipAddress).toBe('192.168.1.100'); + expect(agentManager.isConnected(agent.id)).toBe(true); }); - test('processes discovery payload per PROTOCOL.md v1.1.0 Section 3.1', () => { - agentManager.registerAgent('test-token-123', mockWs, '192.168.1.100'); + // registerAgent must not be async: the WS `message` listener is attached in + // the same tick, and `ws` drops events emitted before a listener exists. An + // awaited DB write here swallowed every agent's first discovery frame, which + // is the one it sends immediately on connect. + test('registerAgent is synchronous so no message can be missed', () => { + const result = agentManager.registerAgent(agent, mockWs, '10.0.0.1'); + expect(result).toBeUndefined(); + expect(agentManager.isConnected(agent.id)).toBe(true); + }); - const discoveryPayload = { + test('persists discovery to the agent row (Section 3.1)', async () => { + agentManager.registerAgent(agent, mockWs, '192.168.1.100'); + await agentManager.handleDiscovery(agent, { hostname: 'node-01.local', ip_addresses: ['192.168.1.100', '10.0.0.5'], os: 'Ubuntu 24.04 LTS', @@ -39,41 +80,34 @@ describe('AgentManager PROTOCOL.md v1.1.0 Compliance', () => { ram_total_gb: 32.0, disk_total_gb: 500.0, location: 'dc-chicago-rack-4' - }; + }); - agentManager.handleDiscovery('test-token-123', discoveryPayload); - - const agents = agentManager.getConnectedAgents(); - const agent = agents.find(a => a.token === 'test-token-123'); - expect(agent.hostname).toBe('node-01.local'); - expect(agent.discovery.os).toBe('Ubuntu 24.04 LTS'); - expect(agent.discovery.ip_addresses).toEqual(['192.168.1.100', '10.0.0.5']); + const saved = agent.persisted.lastDiscovery; + expect(saved.hostname).toBe('node-01.local'); + expect(saved.os).toBe('Ubuntu 24.04 LTS'); + expect(saved.ip_addresses).toEqual(['192.168.1.100', '10.0.0.5']); + // Durable, not just in memory: an agent that goes offline keeps its facts. + expect(agent.persisted.last_seen).toEqual(expect.any(Number)); }); - test('processes telemetry payload per PROTOCOL.md v1.1.0 Section 3.2', () => { - agentManager.registerAgent('test-token-123', mockWs, '192.168.1.100'); - - const telemetryPayload = { + test('persists telemetry to the agent row (Section 3.2)', async () => { + agentManager.registerAgent(agent, mockWs, '192.168.1.100'); + await agentManager.handleTelemetry(agent, { cpu_usage_percent: 14.5, ram_usage_percent: 42.1, disk_usage_percent: 68.0, zfs_health: 'ONLINE', gpu_usage_percent: -1.0, timestamp: new Date().toISOString() - }; + }); - agentManager.handleTelemetry('test-token-123', telemetryPayload); - - const agents = agentManager.getConnectedAgents(); - const agent = agents.find(a => a.token === 'test-token-123'); - expect(agent.telemetry.cpu_usage_percent).toBe(14.5); - expect(agent.telemetry.zfs_health).toBe('ONLINE'); + expect(agent.persisted.lastTelemetry.cpu_usage_percent).toBe(14.5); + expect(agent.persisted.lastTelemetry.zfs_health).toBe('ONLINE'); }); - test('responds to heartbeat with heartbeat_ack per Section 3.3', () => { - agentManager.registerAgent('test-token-123', mockWs, '192.168.1.100'); - - agentManager.handleHeartbeat('test-token-123', { timestamp: new Date().toISOString() }, mockWs); + test('responds to heartbeat with heartbeat_ack (Section 3.3)', async () => { + agentManager.registerAgent(agent, mockWs, '192.168.1.100'); + await agentManager.handleHeartbeat(agent, { timestamp: new Date().toISOString() }, mockWs); expect(mockWs.send).toHaveBeenCalled(); const lastMsg = sentMessages[sentMessages.length - 1]; @@ -81,20 +115,92 @@ describe('AgentManager PROTOCOL.md v1.1.0 Compliance', () => { expect(lastMsg.payload.timestamp).toBeDefined(); }); - test('canonicalizes payload and signs high-risk commands using Ed25519 per Section 5', () => { - agentManager.registerAgent('test-token-123', mockWs, '192.168.1.100'); + test('canonicalizes and signs high-risk commands with Ed25519 (Section 5)', async () => { + agentManager.registerAgent(agent, mockWs, '192.168.1.100'); const rawPayload = { script: 'uptime', location: 'datacenter' }; - const msg = agentManager.sendCommand('test-token-123', 'arbitrary_bash', rawPayload, true); + const msg = await agentManager.sendCommand(agent, 'arbitrary_bash', rawPayload, true); expect(msg.type).toBe('arbitrary_bash'); - expect(msg.payload.signature).toBeDefined(); expect(typeof msg.payload.signature).toBe('string'); - // Verify signature with public key - const signatureBuffer = Buffer.from(msg.payload.signature, 'base64'); - const canonicalStr = agentManager.canonicalize(rawPayload); - const isValid = crypto.verify(null, Buffer.from(canonicalStr, 'utf8'), agentManager.publicKeyPem, signatureBuffer); + const keys = await agentKeys.load(); + const isValid = crypto.verify( + null, + Buffer.from(agentManager.canonicalize(rawPayload), 'utf8'), + crypto.createPublicKey(keys.publicKeyPem), + Buffer.from(msg.payload.signature, 'base64') + ); expect(isValid).toBe(true); }); + + // The canonical form has to match the Go agent's byte for byte. Go's + // encoding/json escapes <, > and & by default and JSON.stringify does not, so + // the agent uses SetEscapeHTML(false); this pins the server's half of that + // contract. See theta-agent TestCanonicalizeMatchesServerForm. + test('canonical form is sorted, unescaped, and omits the signature', () => { + const canonical = agentManager.canonicalize({ + script: 'echo a > b && c', + comment: 'x&y', + signature: 'should-not-appear' + }); + expect(canonical).toBe('{"comment":"x&y","script":"echo a > b && c"}'); + }); + + test('refuses to send to an agent that is not connected', async () => { + await expect(agentManager.sendCommand(agent, 'reload_config', {}, false)) + .rejects.toThrow(/not connected/); + }); + + // Revocation that only applies on the next reconnect is not revocation. + test('disconnect drops the live socket immediately', () => { + agentManager.registerAgent(agent, mockWs, '192.168.1.100'); + expect(agentManager.isConnected(agent.id)).toBe(true); + + const dropped = agentManager.disconnect(agent.id, 4003, 'Enrollment revoked'); + expect(dropped).toBe(true); + expect(mockWs.close).toHaveBeenCalledWith(4003, 'Enrollment revoked'); + expect(agentManager.isConnected(agent.id)).toBe(false); + }); + + test('a second connection for the same agent supersedes the first', () => { + agentManager.registerAgent(agent, mockWs, '192.168.1.100'); + const secondWs = { readyState: 1, send: jest.fn(), close: jest.fn() }; + agentManager.registerAgent(agent, secondWs, '192.168.1.101'); + + expect(mockWs.close).toHaveBeenCalledWith(4002, 'Superseded by new connection'); + expect(agentManager.liveState(agent.id).ipAddress).toBe('192.168.1.101'); + }); + + test('an unknown agent id is simply not connected', () => { + expect(agentManager.isConnected('no-such-agent')).toBe(false); + expect(agentManager.liveState('no-such-agent')).toEqual({ connected: false, lastResponse: null }); + }); +}); + +describe('agent signing key', () => { + // The old manager generated a key pair in its constructor, so it changed on + // every restart and the public_key pinned in agent.yml stopped matching. + test('the same key is returned across repeated loads', async () => { + const first = await agentKeys.load(); + const second = await agentKeys.load(); + expect(first.publicKeyBase64).toBe(second.publicKeyBase64); + }); + + test('the exported public key is the raw 32 bytes agents pin', async () => { + const keys = await agentKeys.load(); + expect(Buffer.from(keys.publicKeyBase64, 'base64')).toHaveLength(32); + }); + + test('rawPublicKeyBase64 strips the SPKI wrapper', () => { + const { publicKey } = crypto.generateKeyPairSync('ed25519', { + privateKeyEncoding: { type: 'pkcs8', format: 'pem' }, + publicKeyEncoding: { type: 'spki', format: 'pem' } + }); + const raw = Buffer.from(agentKeys.rawPublicKeyBase64(publicKey), 'base64'); + expect(raw).toHaveLength(32); + // and it is the tail of the DER encoding + const der = crypto.createPublicKey(publicKey).export({ type: 'spki', format: 'der' }); + expect(raw.equals(der.subarray(der.length - 32))).toBe(true); + }); }); diff --git a/nodejs/tests/api_agent_ops.test.js b/nodejs/tests/api_agent_ops.test.js new file mode 100644 index 0000000..c17de60 --- /dev/null +++ b/nodejs/tests/api_agent_ops.test.js @@ -0,0 +1,74 @@ +'use strict'; + +// Agent-facing ops (DESIGN.md §5): node-scoped secrets. OpenBao is not present +// in the test env, so @simpleworkjs/bao-conf is mocked. + +jest.mock('@simpleworkjs/bao-conf', () => ({ + request: jest.fn(async (method, path) => { + if (path.startsWith('secret/data/nodes/')) { + return { + ok: true, + status: 200, + json: async () => ({ data: { data: { username: 'alice', password: 's3cret' } } }), + }; + } + return { ok: false, status: 404, json: async () => ({}) }; + }), +})); + +const { request, app } = require('./setup'); +const { Agent } = require('../models/agent'); + +async function enrollAgent() { + const { agent, token } = await Agent.enroll({ + name: `ops-test-${Date.now().toString(36)}`, + description: 'api_agent_ops test', + enrolledBy: 'test' + }); + return { agent, token }; +} + +describe('Agent ops — POST /api/v1/agent/secrets', () => { + test('an agent can fetch its own node-scoped secrets', async () => { + const { agent, token } = await enrollAgent(); + const path = `secret/data/nodes/${agent.id}/db`; + const res = await request(app) + .post('/api/v1/agent/secrets') + .set('Authorization', `Bearer ${token}`) + .send({ paths: [path] }); + + expect(res.status).toBe(200); + expect(res.body.status).toBe('ok'); + expect(res.body.secrets[path]).toEqual({ username: 'alice', password: 's3cret' }); + }); + + test('a path outside the node scope is rejected', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/agent/secrets') + .set('Authorization', `Bearer ${token}`) + .send({ paths: ['secret/data/nodes/other-node/db'] }); + + expect(res.status).toBe(403); + }); + + test('no bearer token returns 401', async () => { + const res = await request(app) + .post('/api/v1/agent/secrets') + .send({ paths: ['secret/data/nodes/x/db'] }); + + expect(res.status).toBe(401); + }); + + test('missing paths defaults to agent node & resource secrets', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/agent/secrets') + .set('Authorization', `Bearer ${token}`) + .send({}); + + expect(res.status).toBe(200); + expect(res.body.status).toBe('ok'); + expect(res.body.secrets).toBeDefined(); + }); +}); diff --git a/nodejs/tests/api_ldap.test.js b/nodejs/tests/api_ldap.test.js new file mode 100644 index 0000000..a0eb562 --- /dev/null +++ b/nodejs/tests/api_ldap.test.js @@ -0,0 +1,126 @@ +'use strict'; + +// LDAP-over-HTTPS API (DESIGN.md §3). Exercises caller auth (agent token vs +// PAT), the bind flow against the real test OpenLDAP, and the agent-only search +// restriction. + +const { TEST_CREDS, request, app } = require('./setup'); +const { Agent } = require('../models/agent'); +const { ApiToken } = require('../models/api_token'); + +async function enrollAgent() { + const { agent, token } = await Agent.enroll({ + name: `ldap-test-${Date.now().toString(36)}`, + description: 'api_ldap test agent', + enrolledBy: 'test' + }); + return { agent, token }; +} + +async function makePat() { + const token = await ApiToken.add({ + name: 'ldap-test-pat', + description: 'api_ldap test', + created_by: 'test' + }); + return token._raw_token; +} + +describe('LDAP-over-HTTPS — POST /api/v1/ldap/bind', () => { + test('valid credentials return the bound DN', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/ldap/bind') + .set('Authorization', `Bearer ${token}`) + .send({ username: TEST_CREDS.uid, password: TEST_CREDS.password }); + + expect(res.status).toBe(200); + expect(res.body.status).toBe('ok'); + expect(res.body.uid).toBe(TEST_CREDS.uid); + expect(res.body.dn).toContain(TEST_CREDS.uid); + }); + + test('wrong password returns 401', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/ldap/bind') + .set('Authorization', `Bearer ${token}`) + .send({ username: TEST_CREDS.uid, password: 'wrong-password' }); + + expect(res.status).toBe(401); + }); + + test('unknown user returns 401 (no existence oracle)', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/ldap/bind') + .set('Authorization', `Bearer ${token}`) + .send({ username: 'no_such_user_xyz', password: 'whatever' }); + + expect(res.status).toBe(401); + }); + + test('a PAT caller can bind', async () => { + const pat = await makePat(); + const res = await request(app) + .post('/api/v1/ldap/bind') + .set('Authorization', `Bearer ${pat}`) + .send({ username: TEST_CREDS.uid, password: TEST_CREDS.password }); + + expect(res.status).toBe(200); + }); + + test('no bearer token returns 401', async () => { + const res = await request(app) + .post('/api/v1/ldap/bind') + .send({ username: TEST_CREDS.uid, password: TEST_CREDS.password }); + + expect(res.status).toBe(401); + }); + + test('missing username/password returns 400', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/ldap/bind') + .set('Authorization', `Bearer ${token}`) + .send({ username: TEST_CREDS.uid }); + + expect(res.status).toBe(400); + }); +}); + +describe('LDAP-over-HTTPS — POST /api/v1/ldap/search', () => { + test('an agent can search the user tree', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/ldap/search') + .set('Authorization', `Bearer ${token}`) + .send({ filter: `(uid=${TEST_CREDS.uid})`, attributes: ['uid', 'cn'] }); + + expect(res.status).toBe(200); + expect(res.body.status).toBe('ok'); + expect(Array.isArray(res.body.entries)).toBe(true); + expect(res.body.entries.length).toBeGreaterThan(0); + expect(res.body.entries[0].uid).toBe(TEST_CREDS.uid); + }); + + test('a PAT caller is denied search (agent-only)', async () => { + const pat = await makePat(); + const res = await request(app) + .post('/api/v1/ldap/search') + .set('Authorization', `Bearer ${pat}`) + .send({ filter: `(uid=${TEST_CREDS.uid})` }); + + expect(res.status).toBe(403); + }); + + test('missing filter returns 400', async () => { + const { token } = await enrollAgent(); + const res = await request(app) + .post('/api/v1/ldap/search') + .set('Authorization', `Bearer ${token}`) + .send({}); + + expect(res.status).toBe(400); + }); +}); diff --git a/nodejs/tests/discovery_naming.test.js b/nodejs/tests/discovery_naming.test.js new file mode 100644 index 0000000..c4299c7 --- /dev/null +++ b/nodejs/tests/discovery_naming.test.js @@ -0,0 +1,104 @@ +'use strict'; + +// Pure-logic coverage for the two reconciler rules that real Proxmox + UniFi +// data broke. Both were found by running discovery against a live cluster: +// the directory came back listing MAC addresses as host names, and one +// resource ended up as its own parent. + +// Mirrors the ranking in services/discovery_reconciler.js. Kept here (rather +// than exported) because it is a few lines of predicate that the reconciler +// applies inline while merging; if it grows, export it and drop this copy. +const isIp = (str) => /^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$/.test(str || ''); +const isMac = (str) => /^([0-9a-f]{2}[:-]){5}[0-9a-f]{2}$/i.test((str || '').trim()); +const nameRank = (str) => { + if (!str || !String(str).trim()) return 0; + if (isMac(str)) return 0; + if (isIp(str)) return 1; + return 2; +}; +function bestNameOf(existingName, incomingName) { + let best = existingName; + if (incomingName) { + const a = nameRank(incomingName); + const b = nameRank(best); + if (a > b || (a === b && incomingName.length > (best || '').length)) best = incomingName; + } + return best; +} + +describe('discovery name ranking', () => { + test('a real hostname beats a MAC even when shorter', () => { + // The exact regression: UniFi named the host by MAC, Proxmox knew the + // hostname, and length-only comparison kept the MAC. + expect(bestNameOf('ac:16:2d:b3:da:80', 'dl380-0')).toBe('dl380-0'); + }); + + test('a MAC never displaces a real hostname', () => { + expect(bestNameOf('dl380-0', 'ac:16:2d:b3:da:80')).toBe('dl380-0'); + }); + + test('a real hostname beats an IP-shaped name', () => { + expect(bestNameOf('192.168.1.27', 'hass.io')).toBe('hass.io'); + }); + + test('an IP beats a MAC', () => { + expect(bestNameOf('bc:24:11:3f:cd:c8', '192.168.1.27')).toBe('192.168.1.27'); + }); + + test('an IP does not displace a hostname', () => { + expect(bestNameOf('gitea-runner', '192.168.1.176')).toBe('gitea-runner'); + }); + + test('within the same rank the longer/more specific name wins', () => { + expect(bestNameOf('pve', 'pve-dl380-1')).toBe('pve-dl380-1'); + }); + + test('dash-separated MACs are recognized too', () => { + expect(bestNameOf('ac-16-2d-b3-da-80', 'dl380-0')).toBe('dl380-0'); + }); + + test('an empty existing name is always replaced', () => { + expect(bestNameOf('', 'anything')).toBe('anything'); + expect(bestNameOf(null, 'ac:16:2d:b3:da:80')).toBe('ac:16:2d:b3:da:80'); + }); +}); + +// Mirrors isDescendant() in the reconciler. +function isDescendant(candidateId, rootId, edges) { + const seen = new Set(); + const stack = [rootId]; + while (stack.length) { + const id = stack.pop(); + if (id === candidateId) return true; + if (seen.has(id)) continue; + seen.add(id); + for (const e of edges) if (e.parentId === id) stack.push(e.childId); + } + return false; +} + +describe('discovery edge cycle guard', () => { + const edges = [ + { parentId: 'cluster', childId: 'node1' }, + { parentId: 'node1', childId: 'vm1' }, + ]; + + test('detects a direct parent/child inversion', () => { + // Proposing node1 -> cluster when cluster -> node1 already exists. + expect(isDescendant('node1', 'cluster', edges)).toBe(true); + }); + + test('detects a deeper loop', () => { + expect(isDescendant('vm1', 'cluster', edges)).toBe(true); + }); + + test('allows an unrelated new parent', () => { + expect(isDescendant('node2', 'cluster', edges)).toBe(false); + }); + + test('terminates on a graph that already contains a cycle', () => { + // A self-edge written by an earlier release must not hang the walk. + const cyclic = [{ parentId: 'a', childId: 'a' }, { parentId: 'a', childId: 'b' }]; + expect(isDescendant('zzz', 'a', cyclic)).toBe(false); + }); +}); diff --git a/nodejs/tests/driver_registry.test.js b/nodejs/tests/driver_registry.test.js new file mode 100644 index 0000000..75d5082 --- /dev/null +++ b/nodejs/tests/driver_registry.test.js @@ -0,0 +1,86 @@ +'use strict'; + +jest.mock('@simpleworkjs/bao-conf', () => ({ + get: jest.fn(), + set: jest.fn(), + request: jest.fn(async () => ({ ok: true, status: 200, json: async () => ({}) })), +}), { virtual: true }); + +const DriverRegistry = require('../services/driver_registry'); + +describe('Subtype Driver Registry Engine', () => { + + test('resolves ProxmoxDriver for proxmox/hypervisor host subtype', async () => { + const resource = { + id: 'res-proxmox-1', + name: 'pve0', + kind: 'host', + metadata: { subType: 'proxmox' } + }; + const driver = await DriverRegistry.resolveDriver(resource); + expect(driver.name).toBe('proxmox'); + }); + + test('resolves DockerSocketDriver for docker/docker_compose subtype', async () => { + const resource = { + id: 'res-docker-1', + name: 'theta-suite-docker', + kind: 'service', + metadata: { subType: 'docker' } + }; + const driver = await DriverRegistry.resolveDriver(resource); + expect(driver.name).toBe('docker_socket'); + }); + + test('resolves DbDriver for redis, postgresql, openbao_vault subtypes', async () => { + const redisRes = { id: 'r1', metadata: { subType: 'redis' } }; + const pgRes = { id: 'r2', metadata: { subType: 'postgresql' } }; + const vaultRes = { id: 'r3', metadata: { subType: 'openbao_vault' } }; + + expect((await DriverRegistry.resolveDriver(redisRes)).name).toBe('database'); + expect((await DriverRegistry.resolveDriver(pgRes)).name).toBe('database'); + expect((await DriverRegistry.resolveDriver(vaultRes)).name).toBe('database'); + }); + + test('resolves NetworkDriver for wireguard, unifi_ap, pfsense', async () => { + const wgRes = { id: 'nw1', metadata: { subType: 'wireguard' } }; + const unifiRes = { id: 'nw2', metadata: { subType: 'unifi_ap' } }; + const pfRes = { id: 'nw3', metadata: { subType: 'pfsense' } }; + + expect((await DriverRegistry.resolveDriver(wgRes)).name).toBe('network'); + expect((await DriverRegistry.resolveDriver(unifiRes)).name).toBe('network'); + expect((await DriverRegistry.resolveDriver(pfRes)).name).toBe('network'); + }); + + test('resolves K8sDriver for k8s_pod and k8s_deployment', async () => { + const podRes = { id: 'k1', metadata: { subType: 'k8s_pod' } }; + const depRes = { id: 'k2', metadata: { subType: 'k8s_deployment' } }; + + expect((await DriverRegistry.resolveDriver(podRes)).name).toBe('kubernetes'); + expect((await DriverRegistry.resolveDriver(depRes)).name).toBe('kubernetes'); + }); + + test('returns unmanaged driver for unknown subtypes without agent', async () => { + const unknownRes = { id: 'u1', metadata: { subType: 'unknown_custom' } }; + const driver = await DriverRegistry.resolveDriver(unknownRes); + expect(driver.name).toBe('unmanaged'); + }); + + test('fetches metrics via resolved driver', async () => { + const redisRes = { id: 'r1', metadata: { subType: 'redis' } }; + const metrics = await DriverRegistry.getMetrics(redisRes); + expect(metrics.status).toBe('online'); + expect(metrics.driver).toBe('database'); + expect(metrics.redis).toBeDefined(); + expect(metrics.redis.connectedClients).toBeGreaterThan(0); + }); + + test('executes actions via resolved driver', async () => { + const dockerRes = { id: 'd1', slug: 'my-container', metadata: { subType: 'docker' } }; + const result = await DriverRegistry.execAction(dockerRes, 'restart'); + expect(result.status).toBe('ok'); + expect(result.driver).toBe('docker_socket'); + expect(result.action).toBe('restart'); + }); + +}); diff --git a/nodejs/tests/groups.test.js b/nodejs/tests/groups.test.js new file mode 100644 index 0000000..beeec52 --- /dev/null +++ b/nodejs/tests/groups.test.js @@ -0,0 +1,130 @@ +'use strict'; + +const { + slugify, + resourceGroupCns, + aggregateGroupCns, + siteSuperAdminCns, + siteEveryoneCns, + isKnownLevel, + levelGrants, + hasPermission, + GOD_ADMIN, +} = require('../utils/groups'); + +// Resource fixtures mirror the directory: hosts carry a `host_` prefix, services +// are stored bare. The builders take the *name* slug (kind stripped) + a kind, so +// a host `host_web-01` gives `main-office_host_web-01_*` and a service `emby` +// gives `main-office_app_emby_*` -- matching docs/GROUPS.md §2. +const HOST = { site: 'main-office', kind: 'host', slug: 'host_web-01' }; +const APP = { site: 'main-office', kind: 'app', slug: 'emby' }; +const SERVICE = { site: 'main-office', kind: 'service', slug: 'emby' }; +const OTHER_SITE_HOST = { site: 'branch-office', kind: 'host', slug: 'host_db' }; + +describe('slugify', () => { + test('lowercases, spaces and underscores become hyphens, no leading/trailing dash', () => { + expect(slugify('Web 01')).toBe('web-01'); + expect(slugify('Main Office')).toBe('main-office'); + expect(slugify('my_host')).toBe('my-host'); + expect(slugify(' Mixed CASE--name ')).toBe('mixed-case-name'); + expect(slugify('')).toBe(''); + }); +}); + +describe('group cn builders', () => { + test('per-resource names the kind + name slug (docs §2)', () => { + expect(resourceGroupCns('main-office', 'host', 'web-01', 'admin')).toBe('main-office_host_web-01_admin'); + expect(resourceGroupCns('main-office', 'app', 'emby', 'access')).toBe('main-office_app_emby_access'); + }); + test('a prefixed site slug is kept verbatim; the resource name slug is kind-stripped', () => { + expect(resourceGroupCns('site_local', 'host', 'theta-env', 'access')).toBe('site_local_host_theta-env_access'); + expect(resourceGroupCns('site_local', 'app', 'sso-manager', 'access')).toBe('site_local_app_sso-manager_access'); + }); + test('aggregate uses the plural kind', () => { + expect(aggregateGroupCns('main-office', 'host', 'admin')).toBe('main-office_hosts_admin'); + expect(aggregateGroupCns('main-office', 'app', 'access')).toBe('main-office_apps_access'); + }); + test('site super admin + everyone', () => { + expect(siteSuperAdminCns('main-office')).toBe('main-office_super_admin'); + expect(siteEveryoneCns('main-office')).toBe('main-office_everyone'); + }); + test('a directory site slug with a kind prefix is kept verbatim', () => { + expect(siteSuperAdminCns('site_local')).toBe('site_local_super_admin'); + expect(siteEveryoneCns('site_local')).toBe('site_local_everyone'); + expect(aggregateGroupCns('site_local', 'host', 'admin')).toBe('site_local_hosts_admin'); + }); + test('invalid kind throws', () => { + expect(() => resourceGroupCns('s', 'service', 'x', 'admin')).toThrow(); + expect(() => aggregateGroupCns('s', 'service', 'admin')).toThrow(); + }); +}); + +describe('levels', () => { + test('admin/access known; capabilities opaque', () => { + expect(isKnownLevel('admin')).toBe(true); + expect(isKnownLevel('access')).toBe(true); + expect(isKnownLevel('reboot')).toBe(false); + expect(isKnownLevel('emby_admin')).toBe(false); + }); + test('admin implies access; access does not imply admin', () => { + expect(levelGrants('admin', 'access')).toBe(true); + expect(levelGrants('access', 'admin')).toBe(false); + }); +}); + +describe('hasPermission — inheritance', () => { + test('god_admin grants everything everywhere', () => { + expect(hasPermission([GOD_ADMIN], HOST, 'admin')).toBe(true); + expect(hasPermission([GOD_ADMIN], HOST, 'access')).toBe(true); + expect(hasPermission([GOD_ADMIN], HOST, 'reboot')).toBe(true); + expect(hasPermission([GOD_ADMIN], OTHER_SITE_HOST, 'admin')).toBe(true); + }); + + test('site super admin grants everything on its site, not other sites', () => { + expect(hasPermission(['main-office_super_admin'], HOST, 'admin')).toBe(true); + expect(hasPermission(['main-office_super_admin'], HOST, 'reboot')).toBe(true); + expect(hasPermission(['main-office_super_admin'], OTHER_SITE_HOST, 'admin')).toBe(false); + }); + + test('aggregate (all hosts) grants on any host at the site', () => { + expect(hasPermission(['main-office_hosts_admin'], HOST, 'admin')).toBe(true); + expect(hasPermission(['main-office_hosts_access'], HOST, 'access')).toBe(true); + expect(hasPermission(['main-office_hosts_admin'], HOST, 'access')).toBe(true); + }); + + test('specific host group grants only that host', () => { + const cn = resourceGroupCns('main-office', 'host', 'web-01', 'admin'); + expect(hasPermission([cn], HOST, 'admin')).toBe(true); + expect(hasPermission([cn], OTHER_SITE_HOST, 'admin')).toBe(false); + }); + + test('admin implies access; access does not imply admin', () => { + expect(hasPermission([resourceGroupCns('main-office', 'host', 'web-01', 'admin')], HOST, 'access')).toBe(true); + expect(hasPermission([resourceGroupCns('main-office', 'host', 'web-01', 'access')], HOST, 'admin')).toBe(false); + }); + + test('capabilities are exact — admin does not grant a capability', () => { + expect(hasPermission([resourceGroupCns('main-office', 'host', 'web-01', 'reboot')], HOST, 'reboot')).toBe(true); + expect(hasPermission([resourceGroupCns('main-office', 'host', 'web-01', 'admin')], HOST, 'reboot')).toBe(false); + expect(hasPermission(['main-office_hosts_reboot'], HOST, 'reboot')).toBe(true); + }); + + test('hosts and apps are orthogonal namespaces', () => { + const hostAdmin = resourceGroupCns('main-office', 'host', 'web-01', 'admin'); + expect(hasPermission([hostAdmin], APP, 'access')).toBe(false); + const appAdmin = resourceGroupCns('main-office', 'app', 'emby', 'admin'); + expect(hasPermission([appAdmin], APP, 'access')).toBe(true); + }); + + test('a service maps to the app kind (docs §11)', () => { + // The directory `service` kind is the group model's `app`. + expect(hasPermission([resourceGroupCns('main-office', 'app', 'emby', 'admin')], SERVICE, 'admin')).toBe(true); + expect(hasPermission([resourceGroupCns('main-office', 'host', 'web-01', 'admin')], SERVICE, 'admin')).toBe(false); + }); + + test('cross-site isolation', () => { + const mainHostAdmin = resourceGroupCns('main-office', 'host', 'web-01', 'admin'); + expect(hasPermission([mainHostAdmin], OTHER_SITE_HOST, 'access')).toBe(false); + expect(hasPermission(['branch-office_hosts_admin'], OTHER_SITE_HOST, 'admin')).toBe(true); + }); +}); diff --git a/nodejs/tests/orm_method_guard.test.js b/nodejs/tests/orm_method_guard.test.js new file mode 100644 index 0000000..da76c8d --- /dev/null +++ b/nodejs/tests/orm_method_guard.test.js @@ -0,0 +1,118 @@ +'use strict'; + +const fs = require('fs'); +const path = require('path'); + +// @simpleworkjs/orm models expose `list`/`get`/`count`/`create` -- there is no +// `find`, `findOne`, `findAll` or `where`. Calling one is not a syntax error and +// nothing catches it until the line actually runs, so it can sit in a rarely +// exercised path indefinitely. +// +// It did: `models/sms.js` called `PluginInstance.find({...})`, which threw +// "is not a function" on EVERY SMS send -- the test button, OTP-by-SMS and +// notifications alike -- before it could even reach the VoIP.ms fallback. SMS +// delivery had simply never worked. +const ORM_MODELS = [ + 'Resource', 'ResourceEdge', 'ResourceGroup', 'AccessRequest', 'Webhook', + 'PluginInstance', 'SharedSecret', 'SharedSecretGrant', 'VaultAppToken', + 'Agent', 'AgentJoinKey', +]; +const MISSING_STATICS = ['find', 'findOne', 'findAll', 'findAndCountAll', 'where']; + +const ROOT = path.join(__dirname, '..'); +const SCAN_DIRS = ['models', 'routes', 'services', 'utils', 'plugins', 'controller', 'middleware']; + +function walk(dir, out = []) { + let entries; + try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch (e) { return out; } + for (const entry of entries) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) { + if (entry.name === 'node_modules') continue; + walk(full, out); + } else if (entry.name.endsWith('.js')) { + out.push(full); + } + } + return out; +} + +// Strip comments so a line *describing* the bug (like the one in models/sms.js) +// isn't reported as the bug. +function stripComments(src) { + return src + .replace(/\/\*[\s\S]*?\*\//g, '') + .replace(/(^|[^:])\/\/.*$/gm, '$1'); +} + +test('no source file calls an ORM static that does not exist', () => { + const pattern = new RegExp( + `\\b(${ORM_MODELS.join('|')})\\s*\\.\\s*(${MISSING_STATICS.join('|')})\\s*\\(`, + 'g' + ); + + const offenders = []; + for (const dir of SCAN_DIRS) { + for (const file of walk(path.join(ROOT, dir))) { + const src = stripComments(fs.readFileSync(file, 'utf8')); + src.split('\n').forEach((line, i) => { + const m = line.match(pattern); + if (m) offenders.push(`${path.relative(ROOT, file)}:${i + 1} — ${m.join(', ')}`); + }); + } + } + + expect(offenders).toEqual([]); +}); + +// models/email.js exports `{Mail}`, not a bare sender. Requiring the module and +// calling `.send` on it -- as routes/api_conf.js's test-email did -- always +// threw "Email.send is not a function", so the Test Email button could never +// have worked. +test('the email module exports Mail.send and callers destructure it', () => { + const mod = require('../models/email'); + expect(typeof mod.Mail).toBe('object'); + expect(typeof mod.Mail.send).toBe('function'); + // The bare module has no send() -- this is exactly the mistake to catch. + expect(mod.send).toBeUndefined(); + + const offenders = []; + for (const dir of SCAN_DIRS) { + for (const file of walk(path.join(ROOT, dir))) { + const src = stripComments(fs.readFileSync(file, 'utf8')); + // `X = require('...email')` followed by `X.send(` where X was not + // destructured. + const assigned = [...src.matchAll(/(?:const|let|var)\s+(\w+)\s*=\s*require\([^)]*models\/email[^)]*\)/g)] + .map(m => m[1]); + for (const name of assigned) { + if (new RegExp(`\\b${name}\\s*\\.\\s*send\\s*\\(`).test(src)) { + offenders.push(`${path.relative(ROOT, file)} — ${name}.send(), but the module exports {Mail}`); + } + } + } + } + expect(offenders).toEqual([]); +}); + +// The VoIP.ms REST API is a GET against voip.ms/api/v1/rest.php with +// api_username/api_password and method=sendSMS. `api.voip.ms/v1.0/sms/send` +// (which test-sms used to POST to with Basic auth) does not exist -- it +// returned an HTML page, so response.json() threw +// `Unexpected token '<', " { + const offenders = []; + for (const dir of SCAN_DIRS) { + for (const file of walk(path.join(ROOT, dir))) { + // Comments stripped: the note in routes/api_conf.js explaining this + // very bug names the bad host, and describing a mistake is not + // making it. + const src = stripComments(fs.readFileSync(file, 'utf8')); + src.split('\n').forEach((line, i) => { + if (line.includes('api.voip.ms')) { + offenders.push(`${path.relative(ROOT, file)}:${i + 1}`); + } + }); + } + } + expect(offenders).toEqual([]); +}); diff --git a/nodejs/tests/proxmox_interfaces.test.js b/nodejs/tests/proxmox_interfaces.test.js new file mode 100644 index 0000000..4bfbfd9 --- /dev/null +++ b/nodejs/tests/proxmox_interfaces.test.js @@ -0,0 +1,78 @@ +'use strict'; + +const { _Interfaces: Interfaces } = require('../plugins/discovery/proxmox'); + +// Regression coverage for the MAC/IP mismatch: the plugin used to collect MACs +// and IPs into two flat lists and zip them by index, so on a multi-NIC guest +// -- or any guest where one NIC had no address -- the directory recorded an IP +// against the wrong MAC. Interfaces keys by MAC so a pairing can only come from +// the source that observed both together. +describe('proxmox Interfaces', () => { + test('keeps each IP on the NIC it was observed on', () => { + const i = new Interfaces(); + i.add('AA:BB:CC:00:00:01', ['10.0.0.5'], 'eth0'); + i.add('AA:BB:CC:00:00:02', ['192.168.9.7'], 'eth1'); + + expect(i.toArray()).toEqual([ + { mac: 'aa:bb:cc:00:00:01', ip: '10.0.0.5', ips: ['10.0.0.5'], name: 'eth0' }, + { mac: 'aa:bb:cc:00:00:02', ip: '192.168.9.7', ips: ['192.168.9.7'], name: 'eth1' }, + ]); + }); + + test('a NIC with no address does not steal the next NIC\'s IP', () => { + const i = new Interfaces(); + i.add('AA:BB:CC:00:00:01', [], 'eth0'); // stopped/unconfigured + i.add('AA:BB:CC:00:00:02', ['10.0.0.9'], 'eth1'); + + const byMac = Object.fromEntries(i.toArray().map(x => [x.mac, x.ip])); + expect(byMac['aa:bb:cc:00:00:01']).toBeNull(); + expect(byMac['aa:bb:cc:00:00:02']).toBe('10.0.0.9'); + }); + + test('merges the config MAC with the agent-reported address for the same NIC', () => { + const i = new Interfaces(); + i.add('aa:bb:cc:00:00:01', ['10.0.0.5'], 'eth0'); // guest agent + i.add('AA:BB:CC:00:00:01', [], 'net0'); // VM config, same NIC + expect(i.toArray()).toHaveLength(1); + expect(i.toArray()[0]).toMatchObject({ mac: 'aa:bb:cc:00:00:01', ip: '10.0.0.5' }); + }); + + test('collects multiple addresses on one NIC without inventing a second NIC', () => { + const i = new Interfaces(); + i.add('aa:bb:cc:00:00:01', ['10.0.0.5', '10.0.0.6'], 'eth0'); + expect(i.toArray()).toHaveLength(1); + expect(i.toArray()[0].ips).toEqual(['10.0.0.5', '10.0.0.6']); + expect(i.primaryIp()).toBe('10.0.0.5'); + }); + + test('ignores placeholder and malformed MACs', () => { + const i = new Interfaces(); + i.add('00:00:00:00:00:00', [], 'eth0'); + i.add('not-a-mac', [], 'eth1'); + i.add('', [], 'eth2'); + expect(i.toArray()).toEqual([]); + expect(i.primaryMac()).toBeNull(); + }); + + test('keeps an address that arrived without a usable MAC', () => { + const i = new Interfaces(); + i.add(null, ['10.0.0.5'], 'eth0'); + expect(i.primaryIp()).toBe('10.0.0.5'); + expect(i.primaryMac()).toBeNull(); + }); + + test('primary values prefer a NIC that actually has an address', () => { + const i = new Interfaces(); + i.add('aa:bb:cc:00:00:01', [], 'eth0'); + i.add('aa:bb:cc:00:00:02', ['10.0.0.9'], 'eth1'); + expect(i.primaryIp()).toBe('10.0.0.9'); + expect(i.primaryMac()).toBe('aa:bb:cc:00:00:02'); + }); + + test('a fully unaddressed guest still reports its MAC', () => { + const i = new Interfaces(); + i.add('aa:bb:cc:00:00:01', [], 'net0'); + expect(i.primaryIp()).toBeNull(); + expect(i.primaryMac()).toBe('aa:bb:cc:00:00:01'); + }); +}); diff --git a/nodejs/tests/vault_broker.test.js b/nodejs/tests/vault_broker.test.js index c8548d7..2f8e446 100644 --- a/nodejs/tests/vault_broker.test.js +++ b/nodejs/tests/vault_broker.test.js @@ -15,8 +15,36 @@ jest.mock('redis', () => ({ }) })); +// In-memory stand-ins for the ORM-backed models so mintAppToken/renewAppTokens +// can run without a database. +jest.mock('../models/shared_secret', () => ({ + SharedSecret: { list: jest.fn().mockResolvedValue([]) }, +})); +jest.mock('../models/shared_secret_grant', () => ({ + SharedSecretGrant: { listForGrantee: jest.fn().mockResolvedValue([]) }, +})); +jest.mock('../models/vault_app_token', () => { + const rows = []; + const VaultAppToken = { + _rows: rows, + list: jest.fn(async () => rows), + getByName: jest.fn(async (name) => rows.find(r => r.name === name) || null), + create: jest.fn(async (data) => { + const row = { + ...data, + update: jest.fn(async function (patch) { Object.assign(this, patch); }), + delete: jest.fn(async function () { rows.splice(rows.indexOf(this), 1); }), + }; + rows.push(row); + return row; + }), + }; + return { VaultAppToken }; +}); + const baoConf = require('@simpleworkjs/bao-conf'); const vaultBroker = require('../utils/vault_broker'); +const { VaultAppToken } = require('../models/vault_app_token'); describe('vault_broker admin policy', () => { beforeEach(() => { @@ -29,8 +57,8 @@ describe('vault_broker admin policy', () => { return { status: 404, text: async () => '' }; } if (method === 'PUT' && path === 'sys/policies/acl/sso-admin') { - expect(body.policy).toContain('path "secret/metadata" { capabilities = ["list", "read", "delete"] }'); - expect(body.policy).toContain('path "secret/metadata/" { capabilities = ["list", "read", "delete"] }'); + expect(body.policy).toContain('path "secret/metadata" { capabilities = ["create", "read", "update", "delete", "list"] }'); + expect(body.policy).toContain('path "secret/metadata/" { capabilities = ["create", "read", "update", "delete", "list"] }'); return { status: 204, ok: true }; } if (method === 'POST' && path === 'auth/token/create/sso-broker') { @@ -49,3 +77,128 @@ describe('vault_broker admin policy', () => { })); }); }); + +describe('app token lifecycle (accessor storage + renewal)', () => { + beforeEach(() => { + baoConf.request.mockReset(); + VaultAppToken._rows.length = 0; + }); + + function mockBao({ mintAccessor = 'acc-1', renewOk = true } = {}) { + baoConf.request.mockImplementation(async (method, path, body) => { + if (path.startsWith('sys/policies/acl/')) { + if (method === 'GET') return { status: 404, text: async () => '' }; + return { status: 204, ok: true }; + } + if (path === 'auth/token/create/sso-app') { + return { ok: true, json: async () => ({ auth: { client_token: 'app-tok', accessor: mintAccessor, lease_duration: 2764800 } }) }; + } + if (path === 'auth/token/renew-accessor') { + return renewOk ? { ok: true, json: async () => ({}) } : { ok: false, status: 400, text: async () => 'invalid accessor' }; + } + if (path === 'auth/token/revoke-accessor') { + return { ok: true, status: 204, text: async () => '' }; + } + return { status: 200, ok: true, json: async () => ({}) }; + }); + } + + test('mintAppToken stores the accessor; re-mint revokes the old accessor and replaces the row', async () => { + mockBao({ mintAccessor: 'acc-old' }); + await vaultBroker.mintAppToken('demo', 'adminuser'); + expect(VaultAppToken._rows).toHaveLength(1); + expect(VaultAppToken._rows[0]).toMatchObject({ name: 'demo', accessor: 'acc-old', created_by: 'adminuser' }); + + mockBao({ mintAccessor: 'acc-new' }); + await vaultBroker.mintAppToken('demo', 'adminuser'); + expect(baoConf.request).toHaveBeenCalledWith('POST', 'auth/token/revoke-accessor', { accessor: 'acc-old' }); + expect(VaultAppToken._rows).toHaveLength(1); + expect(VaultAppToken._rows[0].accessor).toBe('acc-new'); + }); + + test('renewAppTokens renews each accessor and stamps lastRenewedAt', async () => { + mockBao(); + await vaultBroker.mintAppToken('demo', 'adminuser'); + VaultAppToken._rows[0].lastRenewedAt = 0; + await vaultBroker.renewAppTokens(); + expect(baoConf.request).toHaveBeenCalledWith('POST', 'auth/token/renew-accessor', { accessor: 'acc-1' }); + expect(VaultAppToken._rows[0].lastRenewedAt).toBeGreaterThan(0); + expect(VaultAppToken._rows[0].lastError).toBeNull(); + }); + + test('renewAppTokens records the failure on the row without throwing', async () => { + mockBao({ renewOk: false }); + await vaultBroker.mintAppToken('demo', 'adminuser'); + await vaultBroker.renewAppTokens(); + expect(VaultAppToken._rows[0].lastError).toMatch(/renew failed \(400\)/); + }); +}); + +// Real HTTP round-trip through vaultProxy() against an in-process fake OpenBao. +// This exists because the proxy once shipped with a hook shape the installed +// http-proxy-middleware version ignored (v3 `on: { proxyReq }` vs v2 +// `onProxyReq`), so NO X-Vault-Token was ever injected and every /api/vault +// request 403'd. A unit test on options can't catch that — only a wire test can. +describe('vaultProxy wire behavior', () => { + const http = require('http'); + const express = require('express'); + + let target; // fake OpenBao + let seen; // last request the fake OpenBao received + let app; // sso app fragment: scopeGuard stub + vaultProxy + let server; + + beforeAll((done) => { + target = http.createServer((req, res) => { + let body = ''; + req.on('data', (c) => { body += c; }); + req.on('end', () => { + seen = { method: req.method, url: req.url, headers: req.headers, body }; + res.setHeader('content-type', 'application/json'); + res.end('{"ok":true}'); + }); + }); + target.listen(0, '127.0.0.1', () => { + process.env.VAULT_ADDR = `http://127.0.0.1:${target.address().port}`; + jest.resetModules(); + const broker = require('../utils/vault_broker'); + app = express(); + app.use(express.json()); + app.use('/api/vault', (req, res, next) => { req.vaultToken = 'scoped-token-123'; next(); }, broker.vaultProxy()); + server = app.listen(0, '127.0.0.1', done); + }); + }); + + afterAll((done) => { + server.close(() => target.close(done)); + }); + + function call(path, opts = {}) { + const port = server.address().port; + return fetch(`http://127.0.0.1:${port}${path}`, opts); + } + + test('GET list rewrites /api/vault -> /v1, injects X-Vault-Token, strips sso auth headers', async () => { + const res = await call('/api/vault/secret/metadata/users/alice?list=true', { + headers: { 'auth-token': 'sso-session-token', authorization: 'Bearer sso_x_y', 'content-type': 'application/json' }, + }); + expect(res.status).toBe(200); + expect(seen.url).toBe('/v1/secret/metadata/users/alice?list=true'); + expect(seen.headers['x-vault-token']).toBe('scoped-token-123'); + expect(seen.headers['auth-token']).toBeUndefined(); + expect(seen.headers['authorization']).toBeUndefined(); + }); + + test('POST body survives the express.json + fixRequestBody round-trip', async () => { + const res = await call('/api/vault/secret/data/users/alice/foo', { + method: 'POST', + headers: { 'content-type': 'application/json', 'auth-token': 'sso-session-token' }, + body: JSON.stringify({ data: { hello: 'world' } }), + }); + expect(res.status).toBe(200); + expect(seen.method).toBe('POST'); + expect(seen.url).toBe('/v1/secret/data/users/alice/foo'); + expect(seen.headers['x-vault-token']).toBe('scoped-token-123'); + expect(JSON.parse(seen.body)).toEqual({ data: { hello: 'world' } }); + }); +}); diff --git a/nodejs/utils/agent_auth.js b/nodejs/utils/agent_auth.js new file mode 100644 index 0000000..303ef0e --- /dev/null +++ b/nodejs/utils/agent_auth.js @@ -0,0 +1,26 @@ +'use strict'; + +// Authenticate an agent from a Bearer token (the same token the agent presents +// on its WSS channel). Used by agent-facing REST endpoints (secrets, IAM) that +// are NOT admin-gated — the caller is the agent itself, not an admin session. + +const { Agent } = require('../models/agent'); + +// Resolve a Bearer token to its (non-revoked) Agent, or null. Every failure +// collapses to null so a probing caller learns nothing about which part was +// wrong. +async function authenticateAgent(req) { + const auth = req.headers['authorization'] || ''; + const m = /^Bearer\s+(.+)$/i.exec(auth); + if (!m) return null; + const token = String(m[1]).trim(); + if (!token) return null; + try { + const agent = await Agent.authenticate(token); + return agent || null; + } catch (_) { + return null; + } +} + +module.exports = { authenticateAgent }; diff --git a/nodejs/utils/agent_keys.js b/nodejs/utils/agent_keys.js new file mode 100644 index 0000000..3d91d93 --- /dev/null +++ b/nodejs/utils/agent_keys.js @@ -0,0 +1,99 @@ +'use strict'; + +// The Ed25519 key pair the SSO signs high-risk agent commands with, stored in +// OpenBao at `secret/agent/signing-key`. +// +// This used to be generated in the AgentManager constructor and kept only in +// memory, which made the whole signing scheme decorative: every SSO restart +// produced a new key, so the `public_key` pinned in an agent's agent.yml stopped +// matching and the agent either rejected everything or (because it skips +// verification when no key is configured) executed everything unverified. A +// trust anchor that changes on restart is not a trust anchor. +// +// Requires the sso-broker OpenBao policy to grant `secret/agent/*` +// (theta-suite setup.sh). Without it the load fails and signing is reported as +// unavailable -- we deliberately do NOT fall back to an ephemeral key, because +// signing with a key no agent has ever seen is worse than refusing: it looks +// like it worked. + +const crypto = require('crypto'); +const baoConf = require('@simpleworkjs/bao-conf'); + +const PATH = 'agent/signing-key'; // baoConf adds the secret/data prefix + +let cached = null; // { privateKeyPem, publicKeyPem, publicKeyBase64 } +let loadError = null; + +// Agents pin the raw 32-byte Ed25519 public key, base64-encoded (see the Go +// client's verifySignature, which base64-decodes cfg.public_key and expects +// ed25519.PublicKeySize bytes). Node hands us SPKI PEM, so strip the 12-byte +// DER prefix to get the raw key the agent actually wants. +function rawPublicKeyBase64(publicKeyPem) { + const der = crypto.createPublicKey(publicKeyPem).export({ type: 'spki', format: 'der' }); + return Buffer.from(der.subarray(der.length - 32)).toString('base64'); +} + +function generate() { + const { privateKey, publicKey } = crypto.generateKeyPairSync('ed25519', { + privateKeyEncoding: { type: 'pkcs8', format: 'pem' }, + publicKeyEncoding: { type: 'spki', format: 'pem' } + }); + return { privateKeyPem: privateKey, publicKeyPem: publicKey }; +} + +// Load the stored key pair, generating and persisting one on first run. +// Idempotent and safe to call repeatedly; the result is cached in-process. +async function load() { + if (cached) return cached; + + let stored = null; + try { + stored = await baoConf.get(PATH); + } catch (err) { + loadError = `could not read ${PATH} from OpenBao: ${err.message}`; + console.error(`[agent_keys] ${loadError}`); + return null; + } + + if (stored && stored.privateKeyPem && stored.publicKeyPem) { + cached = { + privateKeyPem: stored.privateKeyPem, + publicKeyPem: stored.publicKeyPem, + publicKeyBase64: rawPublicKeyBase64(stored.publicKeyPem) + }; + loadError = null; + return cached; + } + + // First run: mint one and persist it before use, so a crash between + // generating and storing can't leave agents pinned to a key we forgot. + const fresh = generate(); + try { + await baoConf.set(PATH, fresh); + } catch (err) { + loadError = `could not persist a signing key to ${PATH}: ${err.message}. ` + + 'Re-run ./setup.sh so the sso-broker policy grants secret/agent/*.'; + console.error(`[agent_keys] ${loadError}`); + return null; + } + + cached = { + ...fresh, + publicKeyBase64: rawPublicKeyBase64(fresh.publicKeyPem) + }; + loadError = null; + console.log('[agent_keys] generated and stored a new agent signing key'); + return cached; +} + +function status() { + return { available: !!cached, error: loadError }; +} + +// Test seam: drop the in-process cache. +function _reset() { + cached = null; + loadError = null; +} + +module.exports = { load, status, rawPublicKeyBase64, _reset, PATH }; diff --git a/nodejs/utils/agent_manager.js b/nodejs/utils/agent_manager.js index 93c6f23..9cd39c7 100644 --- a/nodejs/utils/agent_manager.js +++ b/nodejs/utils/agent_manager.js @@ -1,26 +1,19 @@ 'use strict'; const crypto = require('crypto'); +const agentKeys = require('./agent_keys'); +const { Agent } = require('../models/agent'); +// Tracks the live WebSocket for each enrolled agent and brokers commands to it. +// +// The durable facts about an agent (identity, host binding, last seen, last +// discovery/telemetry) live in the Agent table; this class holds only what +// cannot be persisted -- the open socket. That split is what makes an installed +// -but-offline agent visible, and what stops a restart from erasing the fleet. class AgentManager { constructor() { - this.agents = new Map(); // token -> agentRecord - this.privateKeyPem = null; - this.publicKeyPem = null; - this.initKeyPair(); - } - - initKeyPair() { - try { - const { privateKey, publicKey } = crypto.generateKeyPairSync('ed25519', { - privateKeyEncoding: { type: 'pkcs8', format: 'pem' }, - publicKeyEncoding: { type: 'spki', format: 'pem' } - }); - this.privateKeyPem = privateKey; - this.publicKeyPem = publicKey; - } catch (err) { - console.error('[AgentManager] Failed to generate Ed25519 key pair:', err); - } + // agentId -> { ws, ipAddress, connectedAt, lastResponse, pending } + this.live = new Map(); } /** @@ -28,63 +21,103 @@ class AgentManager { * Sort keys alphabetically, remove whitespace, omit 'signature' key. */ canonicalize(payload) { - const cleanObj = {}; - const sortedKeys = Object.keys(payload).filter(k => k !== 'signature').sort(); - for (const key of sortedKeys) { - cleanObj[key] = payload[key]; - } - return JSON.stringify(cleanObj); + const sortObj = (val) => { + if (val === null || typeof val !== 'object') return val; + if (Array.isArray(val)) return val.map(sortObj); + const sorted = {}; + const keys = Object.keys(val).filter(k => k !== 'signature').sort(); + for (const k of keys) { + sorted[k] = sortObj(val[k]); + } + return sorted; + }; + return JSON.stringify(sortObj(payload)); } /** - * Sign payload using Ed25519 private key. - * Returns base64 encoded signature. + * Sign payload using the persisted Ed25519 private key. Throws when no key is + * available rather than minting a throwaway one -- an agent verifies against + * the key pinned in its agent.yml, so a signature from a key it has never + * seen is not a weaker signature, it is a broken command that looks fine from + * this side. */ - signPayload(payload) { - if (!this.privateKeyPem) { - throw new Error('Ed25519 private key is not initialized'); + async signPayload(payload) { + const keys = await agentKeys.load(); + if (!keys) { + const { error } = agentKeys.status(); + throw new Error(`agent command signing is unavailable: ${error || 'no signing key'}`); } const canonicalBytes = Buffer.from(this.canonicalize(payload), 'utf8'); - const signature = crypto.sign(null, canonicalBytes, this.privateKeyPem); - return signature.toString('base64'); + return crypto.sign(null, canonicalBytes, keys.privateKeyPem).toString('base64'); } - registerAgent(token, ws, remoteAddress) { - const existing = this.agents.get(token); + async publicKeyBase64() { + const keys = await agentKeys.load(); + return keys ? keys.publicKeyBase64 : null; + } + + async publicKeyPem() { + const keys = await agentKeys.load(); + return keys ? keys.publicKeyPem : null; + } + + // Bind a freshly authenticated socket to an enrolled agent. `agent` is an + // Agent row that Agent.authenticate() has already vouched for -- this method + // never sees a raw token and must never be called with an unauthenticated one. + // Synchronous by design. The caller must attach its `message` listener in the + // same tick as the connection is accepted: `ws` drops events emitted before a + // listener exists, and the agent sends `discovery` immediately on open, so + // awaiting a database round-trip here silently lost every agent's first + // discovery frame. The connect timestamp is persisted in the background. + registerAgent(agent, ws, remoteAddress) { + const existing = this.live.get(agent.id); if (existing && existing.ws && existing.ws !== ws) { try { existing.ws.close(4002, 'Superseded by new connection'); } catch (e) {} } - const agentRecord = { - token, + this.live.set(agent.id, { ws, ipAddress: remoteAddress, - hostname: 'unknown', connectedAt: new Date().toISOString(), - lastSeen: new Date().toISOString(), - discovery: {}, - telemetry: {}, - pendingResponses: new Map() - }; + lastResponse: null + }); - this.agents.set(token, agentRecord); - return agentRecord; + agent.update({ + last_seen: Math.floor(Date.now() / 1000), + last_ip: remoteAddress || null + }).catch(err => console.error(`[AgentManager] could not record connect for ${agent.id}:`, err.message)); } - unregisterAgent(token, ws) { - const record = this.agents.get(token); - if (record && record.ws === ws) { - this.agents.delete(token); - } + unregisterAgent(agentId, ws) { + const state = this.live.get(agentId); + if (state && state.ws === ws) this.live.delete(agentId); } - handleDiscovery(token, payload) { - const agent = this.agents.get(token); - if (!agent) return; + // Drop an agent's live socket now. Revocation that only takes effect on the + // next reconnect is not revocation -- a connected agent would keep receiving + // commands indefinitely. + disconnect(agentId, code = 4003, reason = 'Disconnected by server') { + const state = this.live.get(agentId); + if (!state || !state.ws) return false; + try { state.ws.close(code, reason); } catch (e) {} + this.live.delete(agentId); + return true; + } - agent.lastSeen = new Date().toISOString(); - agent.hostname = payload.hostname || agent.hostname; - agent.discovery = { + isConnected(agentId) { + const state = this.live.get(agentId); + return !!(state && state.ws && state.ws.readyState === 1); + } + + async touch(agent, extra = {}) { + await agent.update({ + last_seen: Math.floor(Date.now() / 1000), + ...extra + }).catch(err => console.error(`[AgentManager] could not persist agent ${agent.id}:`, err.message)); + } + + async handleDiscovery(agent, payload) { + const discovery = { hostname: payload.hostname || '', ip_addresses: Array.isArray(payload.ip_addresses) ? payload.ip_addresses : [], os: payload.os || '', @@ -92,30 +125,124 @@ class AgentManager { cpu: payload.cpu || '', ram_total_gb: payload.ram_total_gb || 0, disk_total_gb: payload.disk_total_gb || 0, - location: payload.location || 'default' + location: payload.location || 'default', + // The agent's enabled capabilities (from its local agent.yml). The agent + // is the authoritative source for what it will actually do. + capabilities: payload.capabilities || {} }; + await this.touch(agent, { lastDiscovery: discovery }); + await this.applyDiscoveryToDirectory(agent, discovery); } - handleTelemetry(token, payload) { - const agent = this.agents.get(token); - if (!agent) return; + // An agent runs ON the host it describes, which makes it the most + // authoritative source the directory has -- more so than a hypervisor API or + // a network scan. It previously updated nothing at all: the facts sat on an + // in-memory record and were lost on disconnect. + // + // When the agent is bound to a resource we write that row directly; guessing + // is only for an unbound agent, and then we let the shared reconciler do the + // matching (same MAC/IP/name rules every other source goes through) rather + // than inventing a second matcher here. + async applyDiscoveryToDirectory(agent, discovery) { + try { + const { Resource } = require('../models/resource'); + const metadata = { + os: discovery.os || undefined, + kernel: discovery.kernel || undefined, + cpu: discovery.cpu || undefined, + ram_total_gb: discovery.ram_total_gb || undefined, + disk_total_gb: discovery.disk_total_gb || undefined, + ip: (discovery.ip_addresses || [])[0] || undefined, + public_ip: discovery.public_ip || undefined, + agentId: agent.id, + last_seen: Date.now() + }; + // Drop undefined so a field the agent could not determine never + // overwrites a good value already in the directory. + for (const k of Object.keys(metadata)) if (metadata[k] === undefined) delete metadata[k]; - agent.lastSeen = new Date().toISOString(); - agent.telemetry = { - cpu_usage_percent: payload.cpu_usage_percent || 0, - ram_usage_percent: payload.ram_usage_percent || 0, - disk_usage_percent: payload.disk_usage_percent || 0, - zfs_health: payload.zfs_health || 'N/A', - gpu_usage_percent: payload.gpu_usage_percent ?? -1, - timestamp: payload.timestamp || new Date().toISOString() - }; - } + if (agent.resourceId) { + const resource = await Resource.get(agent.resourceId); + if (!resource) return; + const merged = { ...(resource.metadata || {}), ...metadata }; + const sources = new Set(merged.discovery_sources || []); + sources.add('theta-agent'); + merged.discovery_sources = [...sources]; + await resource.update({ metadata: merged, updated_on: Math.floor(Date.now() / 1000) }); + return; + } - handleHeartbeat(token, payload, ws) { - const agent = this.agents.get(token); - if (agent) { - agent.lastSeen = new Date().toISOString(); + if (!discovery.hostname) return; + const { DiscoveryReconciler } = require('../services/discovery_reconciler'); + const { ResourceEdge } = require('../models/resource'); + + const hostSlug = `host-${discovery.hostname.toLowerCase().replace(/[^a-z0-9_-]/g, '-')}`; + await DiscoveryReconciler.reconcile('theta-agent', { + resources: [{ + kind: 'host', + name: discovery.hostname, + slug: hostSlug, + metadata: { ...metadata, subType: 'linux', managed: true } + }], + edges: [] + }); + + // Find the matched or created host resource + const allHosts = await Resource.list({ where: { kind: 'host' } }); + const hostRes = allHosts.find(r => + r.name.toLowerCase() === discovery.hostname.toLowerCase() || + r.slug === hostSlug || + r.metadata?.agentId === agent.id + ); + + if (hostRes) { + // Bind the agent to its Host resource + await agent.update({ resourceId: hostRes.id }).catch(() => {}); + + // Attach host to matching Site by Public IP if not already parented + const existingEdges = await ResourceEdge.list({ where: { childId: hostRes.id } }); + if (existingEdges.length === 0) { + const sites = await Resource.list({ where: { kind: 'site' } }); + let targetSite = null; + if (discovery.public_ip) { + targetSite = sites.find(s => { + const siteIp = (s.metadata?.public_ip || s.metadata?.ip || s.metadata?.address || '').trim(); + return siteIp && (siteIp === discovery.public_ip || siteIp.includes(discovery.public_ip)); + }); + } + if (!targetSite) targetSite = sites[0]; + + if (targetSite) { + await ResourceEdge.create({ + id: crypto.randomUUID(), + parentId: targetSite.id, + childId: hostRes.id, + relation: 'hosts' + }).catch(() => {}); + } + } + } + } catch (err) { + // Never let a directory write break the agent connection. + console.error(`[AgentManager] discovery -> directory failed for agent ${agent.id}:`, err.message); } + } + + async handleTelemetry(agent, payload) { + await this.touch(agent, { + lastTelemetry: { + cpu_usage_percent: payload.cpu_usage_percent || 0, + ram_usage_percent: payload.ram_usage_percent || 0, + disk_usage_percent: payload.disk_usage_percent || 0, + zfs_health: payload.zfs_health || 'N/A', + gpu_usage_percent: payload.gpu_usage_percent ?? -1, + timestamp: payload.timestamp || new Date().toISOString() + } + }); + } + + async handleHeartbeat(agent, payload, ws) { + await this.touch(agent); try { ws.send(JSON.stringify({ type: 'heartbeat_ack', @@ -124,57 +251,60 @@ class AgentManager { } catch (e) {} } - handleResponse(token, payload) { - const agent = this.agents.get(token); - if (agent) { - agent.lastSeen = new Date().toISOString(); - agent.lastResponse = { + async handleResponse(agent, payload) { + const state = this.live.get(agent.id); + if (state) { + state.lastResponse = { status: payload.status || 'ok', message: payload.message || '', output: payload.output || '', timestamp: new Date().toISOString() }; } + await this.touch(agent); } - sendCommand(token, commandType, payload = {}, isHighRisk = false) { - const agent = this.agents.get(token); - if (!agent || !agent.ws || agent.ws.readyState !== 1) { - throw new Error(`Agent with token "${token}" is not connected`); + async sendCommand(agent, commandType, payload = {}, isHighRisk = false) { + const state = this.live.get(agent.id); + if (!state || !state.ws || state.ws.readyState !== 1) { + throw new Error(`Agent "${agent.name}" is not connected`); } const finalPayload = { ...payload }; - if (isHighRisk) { - finalPayload.signature = this.signPayload(finalPayload); - } + if (isHighRisk) finalPayload.signature = await this.signPayload(finalPayload); - const message = { - type: commandType, - payload: finalPayload - }; - - agent.ws.send(JSON.stringify(message)); + const message = { type: commandType, payload: finalPayload }; + state.ws.send(JSON.stringify(message)); return message; } - getConnectedAgents() { - const list = []; - const now = new Date(); - for (const [token, agent] of this.agents.entries()) { - list.push({ - token, - hostname: agent.hostname, - ipAddress: agent.ipAddress, - connectedAt: agent.connectedAt, - lastSeen: agent.lastSeen, - discovery: agent.discovery, - telemetry: agent.telemetry, - lastResponse: agent.lastResponse || null, - isOnline: (now - new Date(agent.lastSeen)) < 90000 - }); - } - return list; + // Live view for one agent, for merging into its row. + liveState(agentId) { + const state = this.live.get(agentId); + if (!state) return { connected: false, lastResponse: null }; + return { + connected: !!(state.ws && state.ws.readyState === 1), + ipAddress: state.ipAddress, + connectedAt: state.connectedAt, + lastResponse: state.lastResponse || null + }; + } + + // Find connected/enrolled agent bound to a resource ID. + async getAgentForResource(resourceId) { + if (!resourceId) return null; + const rows = await Agent.list().catch(() => []); + const agent = rows.find(a => a.resourceId === resourceId); + if (!agent) return null; + return agent.toPublic(this.liveState(agent.id)); + } + + // Every enrolled agent, connected or not. + async listAgents() { + const rows = await Agent.list(); + return rows.map(a => a.toPublic(this.liveState(a.id))); } } module.exports = new AgentManager(); +module.exports.AgentManager = AgentManager; diff --git a/nodejs/utils/groups.js b/nodejs/utils/groups.js new file mode 100644 index 0000000..c00574c --- /dev/null +++ b/nodejs/utils/groups.js @@ -0,0 +1,141 @@ +'use strict'; + +// Theta42 group & permission model. +// +// Canonical spec: theta-suite/docs/GROUPS.md. Group names follow a fixed, +// parseable structure. The structural delimiter is `_`; site/host/app slugs +// never contain it. Aggregates use the plural kind (hosts/apps); per-resource +// uses the singular (host/app). +// +// god_admin global — everything, everywhere +// {site}_super_admin everything on the site +// {site}_hosts_ admin/access/capability on ALL hosts at the site +// {site}_hosts_ +// {site}_host__ admin/access/capability on ONE host +// {site}_apps_ ... on ALL apps at the site +// {site}_app__ ... on ONE app +// {site}_everyone / everyone meta groups (implicit membership) +// +// `level` is 'admin', 'access', or an opaque ``. `admin` implies +// `access`; capabilities are explicit and never implied by `admin`. Groups are +// `groupOfNames` (RBAC) — no gidNumber; hosts map GIDs on the fly (SSSD). +// +// This module is pure logic (no LDAP/DB) so it is fully unit-testable. Callers +// supply the user's group memberships (e.g. from Group.list(user.dn)). + +const GOD_ADMIN = 'god_admin'; +const KNOWN_LEVELS = ['admin', 'access']; +const KINDS = ['host', 'app']; + +// Normalize a site/host/app slug: lowercase; runs of non-alnum -> '-'; never +// contains '_' (the structural delimiter), so group names parse unambiguously. +function slugify(name) { + return String(name || '') + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, ''); +} + +// Validate a kind (host/app) — throw on anything else. +function assertKind(kind) { + if (!KINDS.includes(kind)) throw new Error(`invalid resource kind: ${kind} (must be host or app)`); +} + +// Strip the kind prefix a directory resource slug may carry (`host_theta-env` -> +// `theta-env`), leaving the resource's name slug. Services are stored bare +// (`sso-manager`), so this is a no-op for them. +function resourceNameSlug(slug) { + return String(slug || '').replace(/^(site|host|app)_/, ''); +} + +// {site}_{kind}_{nameSlug}_{level} — the per-resource group for ONE resource. +// Matches docs/GROUPS.md §2 (`S_host__` / `S_app__`): +// `site` is the site resource's slug verbatim (`site_local`), `kind` is the +// group-model kind (`host`/`app`), `nameSlug` is the resource's name (kind +// stripped, e.g. `theta-env` from `host_theta-env`). So a host `host_theta-env` +// yields `site_local_host_theta-env_access` and a service `sso-manager` yields +// `site_local_app_sso-manager_access`. +function resourceGroupCns(site, kind, nameSlug, level) { + assertKind(kind); + return `${site}_${kind}_${slugify(nameSlug)}_${level}`; +} + +// {site}_hosts_ / {site}_apps_ (plural kind — the aggregate). +function aggregateGroupCns(site, kind, level) { + assertKind(kind); + return `${site}_${kind}s_${level}`; +} + +// {site}_super_admin +function siteSuperAdminCns(site) { + return `${site}_super_admin`; +} + +// {site}_everyone +function siteEveryoneCns(site) { + return `${site}_everyone`; +} + +// True if `level` is a known admin/access level (not an opaque capability). +function isKnownLevel(level) { + return KNOWN_LEVELS.includes(level); +} + +// True if holding `level` grants `wanted` (admin implies access). +function levelGrants(level, wanted) { + if (level === wanted) return true; + return level === 'admin' && wanted === 'access'; +} + +// Resolve whether a user (given `memberOf` — the group cns they belong to) has +// `level` on a resource. Applies the inheritance lattice: +// god_admin ⊇ {site}_super_admin ⊇ aggregate ⊇ specific; admin ⊇ access. +// +// memberOf: array of group cns the user is a member of. +// resource: { site, kind: 'host'|'app', slug }. +// level: 'admin' | 'access' | an opaque capability token. +// +// Meta-group grants (`everyone` / `{site}_everyone`) are NOT handled here — they +// are resource-level grants, resolved by the caller against the resource's own +// granted groups (see permission.onResource). This keeps the function pure over +// the user's membership only. +function hasPermission(memberOf, resource, level) { + // `site` is used verbatim (`site_local`); `kind` maps the directory `service` + // kind onto the group model's `app` (docs/GROUPS.md §11 — consoles/services are + // apps); `nameSlug` is the resource name with any kind prefix stripped. + const site = resource && resource.site; + const rawKind = resource && resource.kind; + const kind = rawKind === 'service' ? 'app' : rawKind; + const nameSlug = resourceNameSlug(resource && resource.slug); + const set = new Set(memberOf || []); + + if (set.has(GOD_ADMIN)) return true; + if (set.has(siteSuperAdminCns(site))) return true; + + if (isKnownLevel(level)) { + // admin / access + if (set.has(aggregateGroupCns(site, kind, level))) return true; + if (set.has(resourceGroupCns(site, kind, nameSlug, level))) return true; + if (level === 'access' && hasPermission(memberOf, resource, 'admin')) return true; + return false; + } + // Opaque capability — exact aggregate or specific grant only. + if (set.has(aggregateGroupCns(site, kind, level))) return true; + if (set.has(resourceGroupCns(site, kind, nameSlug, level))) return true; + return false; +} + +module.exports = { + GOD_ADMIN, + KNOWN_LEVELS, + KINDS, + slugify, + resourceNameSlug, + resourceGroupCns, + aggregateGroupCns, + siteSuperAdminCns, + siteEveryoneCns, + isKnownLevel, + levelGrants, + hasPermission, +}; diff --git a/nodejs/utils/ldap_tunnel.js b/nodejs/utils/ldap_tunnel.js new file mode 100644 index 0000000..e404d8c --- /dev/null +++ b/nodejs/utils/ldap_tunnel.js @@ -0,0 +1,84 @@ +'use strict'; + +// LDAP byte-pump relay (DESIGN.md §4). The agent forwards raw LDAP bytes from a +// local socket (SSSD) over the WSS channel as `ldap_tunnel` messages; this +// module relays them into the SSO's real OpenLDAP and pipes the responses back. +// The SSO does not parse LDAP either — it is a transparent socket relay. + +const net = require('net'); +const conf = require('@simpleworkjs/conf').ldap; + +// Parse host:port from an ldap:// or ldaps:// URL. The relay connects plaintext +// to the SSO's own slapd (which is plaintext on localhost); an ldaps:// URL +// would need TLS termination here and is not supported yet (DESIGN.md §9.5). +function ldapTarget() { + const url = conf.url || 'ldap://localhost:389'; + const m = /^ldaps?:\/\/([^:/]+)(?::(\d+))?/.exec(url); + const host = m ? m[1] : 'localhost'; + const port = m && m[2] ? Number(m[2]) : 389; + return { host, port }; +} + +// Per-agent relay state: agentId -> Map(conn_id -> LDAP socket). +const relays = new Map(); + +function relayFor(agentId) { + if (!relays.has(agentId)) relays.set(agentId, new Map()); + return relays.get(agentId); +} + +// Handle one ldap_tunnel message from an agent. +function handleTunnel(agentId, ws, payload) { + const connId = payload.conn_id; + if (!connId) return; + const conns = relayFor(agentId); + + // End of connection: close the relay socket. + if (payload.close) { + const sock = conns.get(connId); + if (sock) { sock.destroy(); conns.delete(connId); } + return; + } + + const data = Buffer.from(payload.data || '', 'base64'); + if (data.length === 0) return; + + let sock = conns.get(connId); + if (!sock) { + const { host, port } = ldapTarget(); + sock = net.connect(port, host); + conns.set(connId, sock); + + // Relay OpenLDAP's responses back to the agent. + sock.on('data', (chunk) => { + if (ws.readyState === 1) { + ws.send(JSON.stringify({ + type: 'ldap_tunnel', + payload: { conn_id: connId, data: chunk.toString('base64') } + })); + } + }); + sock.on('close', () => { + conns.delete(connId); + if (ws.readyState === 1) { + ws.send(JSON.stringify({ + type: 'ldap_tunnel', + payload: { conn_id: connId, close: true } + })); + } + }); + sock.on('error', () => { sock.destroy(); }); + } + sock.write(data); +} + +// Drop every relay socket for an agent (on WSS disconnect). +function cleanup(agentId) { + const conns = relays.get(agentId); + if (conns) { + for (const sock of conns.values()) sock.destroy(); + relays.delete(agentId); + } +} + +module.exports = { handleTunnel, cleanup }; diff --git a/nodejs/utils/permission.js b/nodejs/utils/permission.js index 15fc2e0..079da48 100644 --- a/nodejs/utils/permission.js +++ b/nodejs/utils/permission.js @@ -1,10 +1,27 @@ 'use strict'; const {Group} = require('../models/group_ldap'); +const groups = require('./groups'); -const SUPER_ADMIN_GROUP = 'app_super_admin'; +// The group nested into every resource's _admin group by api_directory_admin +// (cross-resource super-admin administration). This is `god_admin` -- the global +// super group of the new model (docs/GROUPS.md), seeded by docker-entrypoint.sh. +// It used to be the legacy `app_super_admin`, which existed while god_admin +// didn't; now that god_admin is created at boot, the provisioning nests it. +// LEGACY_SUPER_ADMIN_ALIASES still recognizes a `app_super_admin` that predates +// the migration, so an existing deployment isn't stripped of rights until it's +// rebuilt. +const SUPER_ADMIN_GROUP = 'god_admin'; +const LEGACY_SUPER_ADMIN_ALIASES = ['app_super_admin']; -let byGroup = async function(user, groups, ownerOf){ +// True if the user (by resolved member cns) is a global god/super admin. +// Recognizes BOTH the new schema's `god_admin` and the legacy `app_super_admin`. +async function isSuperAdmin(memberOfCns) { + return memberOfCns.includes(groups.GOD_ADMIN) || + memberOfCns.some((cn) => LEGACY_SUPER_ADMIN_ALIASES.includes(cn)); +} + +let byGroup = async function(user, checkGroups, ownerOf){ // Membership is resolved once, transitively: a user placed in an admin group // through a nested group is as much a member as one listed on it directly. // Checking `group.member.includes(user.dn)` per group -- as this used to -- @@ -17,9 +34,9 @@ let byGroup = async function(user, groups, ownerOf){ // they still catch direct membership if the resolver is unavailable. } - if(memberOfCns.includes(SUPER_ADMIN_GROUP)) return true; + if(await isSuperAdmin(memberOfCns)) return true; - for(let group of groups){ + for(let group of checkGroups){ if(memberOfCns.includes(group)) return true; } @@ -42,4 +59,46 @@ let byGroup = async function(user, groups, ownerOf){ throw error; } -module.exports = {byGroup, SUPER_ADMIN_GROUP}; +// Resolve whether a user has `level` on a directory resource under the group +// model (see utils/groups.js). Applies the inheritance lattice and the +// `everyone`/`{site}_everyone` meta grants when the resource grants them. +// +// user: the auth user ({ dn, isMachine }). +// resource:{ site, kind: 'host'|'app', slug }. +// level: 'admin' | 'access' | an opaque capability token. +// grantedGroups: optional array of the resource's granted group cns (used only +// for meta `everyone` handling). Omit to skip meta grants. +async function onResource(user, resource, level, grantedGroups) { + let memberOfCns = []; + try { memberOfCns = await Group.list(user.dn); } catch (e) { /* ignore */ } + + if (await isSuperAdmin(memberOfCns)) return true; + if (groups.hasPermission(memberOfCns, resource, level)) return true; + + // Meta grants: `everyone` / `{site}_everyone` confer access to any + // authenticated (non-machine) user when the resource grants them. + if (level === 'access' && !user.isMachine && Array.isArray(grantedGroups)) { + const siteEveryone = groups.siteEveryoneCns(resource.site); + if (grantedGroups.includes('everyone') || grantedGroups.includes(siteEveryone)) return true; + } + return false; +} + +// Like onResource but throws Insufficient Permission when denied — for guards. +async function requireResource(user, resource, level, grantedGroups) { + if (await onResource(user, resource, level, grantedGroups)) return; + const error = new Error('Insufficient Permission'); + error.name = 'Insufficient Permission'; + error.status = 401; + throw error; +} + +module.exports = { + byGroup, + onResource, + requireResource, + isSuperAdmin, + SUPER_ADMIN_GROUP, + LEGACY_SUPER_ADMIN_ALIASES, + ...groups, // group schema builders (slugify, resourceGroupCns, ...) +}; diff --git a/nodejs/utils/ui.js b/nodejs/utils/ui.js index 6f27c75..525982f 100644 --- a/nodejs/utils/ui.js +++ b/nodejs/utils/ui.js @@ -41,12 +41,8 @@ module.exports = { // Catalog requires login - it's the end-user view of their accessible resources. {href: '/', icon: 'fa-solid fa-compass', label: 'Catalog', groups: ['login']}, {href: '/users', icon: 'fa-solid fa-users', label: 'Users', groups: ['app_sso_admin', 'admin']}, - {href: '/groups', icon: 'fas fa-users-cog', label: 'Groups', groups: ['app_sso_admin']}, {href: '/conf', icon: 'fas fa-cogs', label: 'Configuration', groups: ['app_sso_admin']}, {href: '/directory', icon: 'fa-solid fa-server', label: 'Directory', groups: ['app_sso_admin', 'app_sso_directory_admin', 'admin']}, - {href: '/plugins', icon: 'fa-solid fa-plug', label: 'Plugins', groups: ['app_sso_admin', 'app_sso_directory_admin', 'admin']}, - // Vault requires login - per-user secrets at secret/users//*. - {href: '/vault', icon: 'fa-solid fa-vault', label: 'Vault', groups: ['login']}, {href: '/overview', icon: 'fa-solid fa-gauge-high', label: 'Overview', groups: ['app_sso_admin', 'admin']}, ], }; diff --git a/nodejs/utils/vault_broker.js b/nodejs/utils/vault_broker.js index 927f41b..d44d9b0 100644 --- a/nodejs/utils/vault_broker.js +++ b/nodejs/utils/vault_broker.js @@ -4,15 +4,25 @@ // external apps, using the SSO_VAULT_TOKEN (policy `sso-broker`) and the // `sso-broker` token role created by theta-env/setup.sh. // -// secret/users//* per-user personal KV (user- policy) -// secret/apps//* per-external-app namespace (app- policy) -// secret/* admin UI sessions (sso-admin policy) +// secret/users//* per-user personal KV (user- policy) +// secret/shared//* user-owned shared KV (user- policy) +// secret/apps//* per-external-app namespace (app- policy) +// secret/shared// granted read (added to grantee's policy) +// secret/* admin UI sessions (sso-admin policy) // // The sso-broker policy grants update on auth/token/create/sso-broker and on // sys/policies/acl/user-*, app-*, sso-admin — exactly what this module needs to // create the per-subject policies and mint their tokens. Per-user/admin tokens // are cached in Redis for the token's lifetime and re-minted on miss; per-app // tokens are returned ONCE (displayed in the UI, never stored retrievably). +// +// Policy reconciliation is the load-bearing part: OpenBao parses policy CONTENT +// live at token use (only the SET of policy names on a token is fixed at mint), +// so we ALWAYS reconcile a subject's policy content BEFORE returning any token +// — cached or freshly minted. That way a stale cached token immediately gains +// corrected/revoked capabilities, and a new shared-secret grant takes effect for +// an existing grantee token with no re-mint. The Redis cache only short-circuits +// token MINTING, never policy reconciliation. const baoConf = require('@simpleworkjs/bao-conf'); const { createClient } = require('redis'); @@ -20,6 +30,9 @@ const express = require('express'); const { createProxyMiddleware, fixRequestBody } = require('http-proxy-middleware'); const conf = require('@simpleworkjs/conf'); const permission = require('./permission'); +const { SharedSecret } = require('../models/shared_secret'); +const { SharedSecretGrant } = require('../models/shared_secret_grant'); +const { VaultAppToken } = require('../models/vault_app_token'); const ROLE = 'sso-broker'; const DEFAULT_TTL = 24 * 60 * 60; // matches the role's token_period (24h) @@ -54,52 +67,90 @@ async function bao(method, path, body) { return res; } -// Ensure an ACL policy exists AND carries the latest HCL. Always (re)writes — -// `bao policy write` is an idempotent overwrite — so policy edits (e.g. adding -// a list grant on a directory path) propagate on the next vault-page visit -// without an operator re-running setup.sh. Skipping on an existing policy -// would strand the old, narrower HCL forever. +// Ensure an ACL policy carries exactly `hcl`. Compare-and-skip: read the current +// content and only PUT when it differs. `bao policy write` is an idempotent +// overwrite, so this is safe to call on every token fetch — edits (e.g. adding a +// grant) propagate immediately because OpenBao parses policy content at use. async function ensurePolicy(name, hcl) { - const existing = await baoConf.request('GET', `sys/policies/acl/${name}`); - if (existing.status !== 200 && existing.status !== 404) { - const t = await existing.text().catch(() => ''); - throw new Error(`OpenBao policy read ${name} failed (${existing.status}) ${t}`); + try { + const existing = await baoConf.request('GET', `sys/policies/acl/${name}`); + if (existing.status === 200) { + const body = await existing.json().catch(() => null); + if (body && typeof body.policy === 'string' && body.policy.trim() === hcl.trim()) return; // unchanged + } + } catch (e) { + console.warn(`[VaultBroker] policy GET ${name} warning:`, e.message); + } + try { + await bao('PUT', `sys/policies/acl/${name}`, { policy: hcl }); + } catch (err) { + console.warn(`[VaultBroker] policy PUT ${name} warning:`, err.message); } - await bao('PUT', `sys/policies/acl/${name}`, { policy: hcl }); } -// Mint a token through the sso-broker role with the given policies. Returns -// { token, ttl } (ttl = lease_duration seconds, falls back to DEFAULT_TTL). -async function mintToken(policies) { - const res = await bao('POST', 'auth/token/create/sso-broker', { policies }); +// Mint a token through a token role with the given policies. Returns +// { token, accessor, ttl } (ttl = lease_duration seconds, falls back to +// DEFAULT_TTL). Roles: sso-broker (24h period — user/admin tokens, re-minted +// from cache) and sso-app (768h period — long-lived external-app credentials, +// kept alive via their stored accessor by the renewal loop below). +async function mintToken(policies, role = ROLE) { + const res = await bao('POST', `auth/token/create/${role}`, { policies }); const json = await res.json(); const token = json && json.auth && json.auth.client_token; if (!token) throw new Error(`OpenBao token mint returned no client_token: ${JSON.stringify(json)}`); const ttl = (json.auth && json.auth.lease_duration) || DEFAULT_TTL; - return { token, ttl }; + return { token, accessor: json.auth.accessor, ttl }; +} + +// ── Shared-secret policy rules ─────────────────────────────────────────────── +// Returns the HCL rules granting `read` on every shared secret the given +// grantee (a user uid or an app name) has been granted. Enforcement is +// OpenBao ACL policy CONTENT — live-evaluated at token use, so these rules take +// effect for the grantee's existing token immediately (no re-mint). +async function sharedPolicyRules(granteeType, granteeId) { + const grants = await SharedSecretGrant.listForGrantee(granteeType, granteeId); + if (!grants.length) return ''; + const secretIds = [...new Set(grants.map(g => g.secretId))]; + const secrets = secretIds.length + ? await SharedSecret.list({ where: { id: { in: secretIds } } }) : []; + const byId = new Map(secrets.map(s => [s.id, s])); + const rules = []; + for (const g of grants) { + const sec = byId.get(g.secretId); + if (!sec) continue; + const p = sec.path(); // shared// + rules.push(`path "secret/data/${p}" { capabilities = ["read"] }`); + rules.push(`path "secret/metadata/${p}" { capabilities = ["read", "list"] }`); + } + return rules.join('\n'); } // ── Per-user token ────────────────────────────────────────────────────────── -function userPolicyHcl(uid) { - // uid is an LDAP uid (alphanumeric + a few separators); it is interpolated - // into a policy path, so reject anything but a safe charset. - // The bare `secret/metadata/users/` grant is required to LIST the - // contents of the namespace: `.../*` covers nested paths but NOT the - // directory itself, so without it the /vault secrets list 403s. - return `path "secret/data/users/${uid}/*" { capabilities = ["create", "read", "update", "delete", "list"] } -path "secret/metadata/users/${uid}" { capabilities = ["list", "read", "delete"] } -path "secret/metadata/users/${uid}/" { capabilities = ["list", "read", "delete"] } -path "secret/metadata/users/${uid}/*" { capabilities = ["list", "read", "delete"] }`; +async function userPolicyHcl(uid) { + const granted = await sharedPolicyRules('user', uid); + return `path "secret/data/users/${uid}" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/data/users/${uid}/*" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/users/${uid}" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/users/${uid}/" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/users/${uid}/*" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/data/shared/${uid}" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/data/shared/${uid}/*" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/shared/${uid}" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/shared/${uid}/" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/shared/${uid}/*" { capabilities = ["create", "read", "update", "delete", "list"] } +${granted}`.trim(); } -// Mint (or return the cached) per-user token confined to secret/users//*. -// Re-minted when the cache entry expires (a little before the token's own TTL). +// Mint (or return the cached) per-user token. The policy is ALWAYS reconciled +// (compare-and-skip) before the cache is consulted, so a cached token can never +// outlive a policy change; the cache only short-circuits re-minting. Re-minted +// when the cache entry expires (a little before the token's own TTL). async function getOrCreateUserToken(uid) { if (!/^[A-Za-z0-9._-]{1,64}$/.test(uid)) throw new Error(`invalid uid for vault token: ${uid}`); + await ensurePolicy(`user-${uid}`, await userPolicyHcl(uid)); const cacheKey = `vault_token:${uid}`; const cached = await cacheGet(cacheKey); if (cached) return cached; - await ensurePolicy(`user-${uid}`, userPolicyHcl(uid)); const { token, ttl } = await mintToken([`user-${uid}`]); await cacheSet(cacheKey, token, Math.max(ttl - 60, 60)); return token; @@ -107,47 +158,164 @@ async function getOrCreateUserToken(uid) { // ── Admin token (read/write all of secret/) ───────────────────────────────── function adminPolicyHcl() { - // The bare `secret/metadata` / `secret/metadata/` grants let an admin LIST - // the KV mount root (the top-level dirs); `secret/metadata/*` covers nested - // paths but NOT the root itself, so without it the /vault secrets list 403s. - return `path "secret/data/*" { capabilities = ["create", "read", "update", "delete", "list"] } -path "secret/metadata" { capabilities = ["list", "read", "delete"] } -path "secret/metadata/" { capabilities = ["list", "read", "delete"] } -path "secret/metadata/*" { capabilities = ["list", "read", "delete"] }`; + return `path "secret/*" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/data/*" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/data" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/*" { capabilities = ["create", "read", "update", "delete", "list"] }`; } async function getOrCreateAdminToken(uid) { + await ensurePolicy('sso-admin', adminPolicyHcl()); const cacheKey = `vault_token:admin:${uid || 'global'}`; const cached = await cacheGet(cacheKey); if (cached) return cached; - await ensurePolicy('sso-admin', adminPolicyHcl()); const { token, ttl } = await mintToken(['sso-admin']); await cacheSet(cacheKey, token, Math.max(ttl - 60, 60)); return token; } // ── Per-app token (minted ONCE, returned to the caller, never cached) ─────── -function appPolicyHcl(name) { - // The bare `secret/metadata/apps/` grant lets an app LIST its own - // namespace root (see userPolicyHcl for why `/*` alone isn't enough). - return `path "secret/data/apps/${name}/*" { capabilities = ["create", "read", "update", "delete", "list"] } -path "secret/metadata/apps/${name}" { capabilities = ["list", "read", "delete"] } -path "secret/metadata/apps/${name}/*" { capabilities = ["list", "read", "delete"] }`; +async function appPolicyHcl(name) { + const granted = await sharedPolicyRules('app', name); + return `path "secret/data/apps/${name}" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/data/apps/${name}/*" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/apps/${name}" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/apps/${name}/" { capabilities = ["create", "read", "update", "delete", "list"] } +path "secret/metadata/apps/${name}/*" { capabilities = ["create", "read", "update", "delete", "list"] } +${granted}`.trim(); } // Create the app- policy + mint a token for it. Returns the token ONCE // (the admin UI shows it with a copy button); it is not stored retrievably, so // a later compromise of an admin session cannot recover previously-minted app -// tokens. The caller must record it in the external app immediately. -async function mintAppToken(name) { +// tokens. The caller must record it in the external app immediately. Later +// grants to the app edit app- policy content (live-applied to this token). +// +// What IS stored is the token's ACCESSOR (VaultAppToken row): an accessor +// cannot authenticate, but it lets the renewal loop below keep the (periodic) +// token alive and lets a re-mint revoke the app's previous token so exactly +// one credential per app is ever live. +async function mintAppToken(name, actorUid) { if (!/^[a-z0-9][a-z0-9-]{0,62}$/.test(name)) { throw new Error('invalid app name (lowercase letters, digits, hyphens; max 63 chars)'); } - await ensurePolicy(`app-${name}`, appPolicyHcl(name)); - const { token, ttl } = await mintToken([`app-${name}`]); + await ensurePolicy(`app-${name}`, await appPolicyHcl(name)); + // App tokens are long-lived credentials: mint via the sso-app role (768h + // period) so a renewal inside every 32-day window keeps them alive forever. + // Fall back to the broker's own 24h role on deployments whose setup.sh + // predates the sso-app role (re-running setup.sh creates it). + let minted; + try { + minted = await mintToken([`app-${name}`], 'sso-app'); + } catch (e) { + console.warn(`vault_broker: sso-app token role unavailable (${e.message}); falling back to sso-broker (24h period). Re-run theta-env setup.sh to create the sso-app role.`); + minted = await mintToken([`app-${name}`]); + } + const { token, accessor, ttl } = minted; + // Replace the app's accessor row; revoke the superseded token (best-effort — + // it may already be expired) so re-minting never leaves a zombie credential. + try { + const existing = await VaultAppToken.getByName(name); + if (existing) { + await baoConf.request('POST', 'auth/token/revoke-accessor', { accessor: existing.accessor }); + await existing.delete(); + } + if (accessor) { + await VaultAppToken.create({ + name, accessor, + lastRenewedAt: Date.now(), + created_by: actorUid, created_on: Date.now(), + }); + } + } catch (e) { + // Accessor bookkeeping must never block handing the token out; without a + // row the token simply isn't auto-renewed (it still lives one full period). + console.error(`vault_broker: could not store accessor for app-${name}:`, e.message); + } return { token, ttl, policy: `app-${name}`, path: `secret/apps/${name}/` }; } +// ── App-token renewal loop ────────────────────────────────────────────────── +// Walks the stored accessors and renews each token (auth/token/renew-accessor), +// resetting its periodic clock. Runs at boot and then every RENEW_INTERVAL_MS — +// far inside both possible periods (24h fallback and 768h), so a downstream +// app's token stays valid for as long as sso is running. Failures are recorded +// on the row (visible to admins in the DB / future UI) and never throw. +const RENEW_INTERVAL_MS = 6 * 60 * 60 * 1000; // 6h — several chances per 24h period +let renewTimer; + +async function renewAppTokens() { + let rows; + try { rows = await VaultAppToken.list(); } + catch (e) { console.error('vault_broker: app-token renewal: could not list accessors:', e.message); return; } + for (const row of rows) { + try { + const res = await baoConf.request('POST', 'auth/token/renew-accessor', { accessor: row.accessor }); + if (res.ok) { + await row.update({ lastRenewedAt: Date.now(), lastError: null }); + } else { + const text = await res.text().catch(() => ''); + // 400 "invalid accessor" = token expired or was revoked out-of-band; + // keep the row + error so the admin can see the app needs a re-mint. + await row.update({ lastError: `renew failed (${res.status}) ${text}` }); + console.warn(`vault_broker: renew of app token '${row.name}' failed (${res.status}) — re-mint it from the vault UI if the app is still in use.`); + } + } catch (e) { + try { await row.update({ lastError: e.message }); } catch (e2) { /* best-effort */ } + console.error(`vault_broker: renew of app token '${row.name}' errored:`, e.message); + } + } +} + +// Start the loop (idempotent). unref() so an open handle never blocks exit. +function startAppTokenRenewal() { + if (renewTimer) return renewTimer; + renewAppTokens().catch((e) => console.error('vault_broker: initial app-token renewal failed:', e.message)); + renewTimer = setInterval(() => { + renewAppTokens().catch((e) => console.error('vault_broker: app-token renewal failed:', e.message)); + }, RENEW_INTERVAL_MS); + if (renewTimer.unref) renewTimer.unref(); + return renewTimer; +} + +// ── Grant / revoke shared-secret access ───────────────────────────────────── +// Creating a grant writes the DB row and then edits the grantee's policy content +// to add read on the shared path; revoking removes both. Because OpenBao parses +// policy content live, the change applies to the grantee's existing token +// immediately — no token re-mint, no cache invalidation needed. +async function grantSharedSecret(secretId, granteeType, granteeId, actorUid) { + const grant = await SharedSecretGrant.create({ + secretId, granteeType, granteeId, capability: 'read', + created_by: actorUid, created_on: Date.now(), + updated_by: actorUid, updated_on: Date.now(), + }); + await reconcileGrantee(granteeType, granteeId); + return grant; +} + +async function revokeSharedSecret(grantId, actorUid) { + const grant = await SharedSecretGrant.get(grantId); + if (!grant) return null; + const { granteeType, granteeId } = grant; + await grant.delete(); + await reconcileGrantee(granteeType, granteeId); + return grant; +} + +// Recompute and rewrite a grantee's policy content after a grant/revoke. +async function reconcileGrantee(granteeType, granteeId) { + if (granteeType === 'user') { + await ensurePolicy(`user-${granteeId}`, await userPolicyHcl(granteeId)); + } else if (granteeType === 'app') { + await ensurePolicy(`app-${granteeId}`, await appPolicyHcl(granteeId)); + } else { + throw new Error(`invalid granteeType: ${granteeType}`); + } +} + // ── /api/vault proxy: scope guard + token-injecting proxy ─────────────────── // Replaces the old bare pass-through (which sent no X-Vault-Token and gated // nothing). The guard mints a server-side token for the user (per-user or @@ -156,11 +324,12 @@ async function mintAppToken(name) { // client's sso auth headers so OpenBao never sees them. const VAULT_ADDR = process.env.VAULT_ADDR || 'http://openbao:8200'; +const ADMIN_GROUPS = ['app_sso_admin', 'app_super_admin', 'app_sso_directory_admin']; const ADMIN_GROUP = 'app_sso_admin'; async function isAdmin(user) { try { - await permission.byGroup(user, [ADMIN_GROUP]); + await permission.byGroup(user, ADMIN_GROUPS); return true; } catch (e) { return false; @@ -189,17 +358,13 @@ async function scopeGuard(req, res, next) { return res.status(503).json({ error: 'vault broker unavailable', detail: e.message }); } - // Defense-in-depth: confirm the requested path is within the subject's - // namespace. Admins roam all of secret/; users are confined to - // secret/users//. (The token's own policy enforces the same at the - // OpenBao layer; this catches a buggy/malicious client early with a clear - // 403 instead of an opaque OpenBao denial.) const norm = normalizeVaultPath(req.path); if (norm === null) { return res.status(403).json({ error: 'vault paths must be under /secret/' }); } - const base = `/secret/users/${uid}`; - const allowed = admin || norm === base || norm.startsWith(base + '/'); + const userBase = `/secret/users/${uid}`; + const sharedBase = `/secret/shared`; + const allowed = admin || norm === userBase || norm.startsWith(userBase + '/') || norm === sharedBase || norm.startsWith(sharedBase + '/'); if (!allowed) { return res.status(403).json({ error: 'path outside your vault namespace' }); } @@ -214,15 +379,20 @@ function vaultProxy() { target: VAULT_ADDR, changeOrigin: true, pathRewrite: { '^/api/vault': '/v1' }, - on: { - proxyReq(proxyReq, req, res, options) { - fixRequestBody(proxyReq, req, res, options); - // Inject ONLY the server-minted scoped token; strip the client's - // sso session/api auth so it never reaches OpenBao. - proxyReq.setHeader('X-Vault-Token', req.vaultToken); - proxyReq.removeHeader('auth-token'); - proxyReq.removeHeader('authorization'); - }, + // http-proxy-middleware v2 API: hooks are top-level onProxyReq/onError, + // NOT the v3 `on: { proxyReq }` shape. v2 silently ignores an `on` key, + // which shipped this proxy with NO token injection — every /api/vault + // call reached OpenBao unauthenticated and 403'd. + onProxyReq(proxyReq, req, res, options) { + // Header ops MUST precede fixRequestBody: it write()s the parsed body + // onto proxyReq, which flushes headers — setHeader after that throws + // (swallowed upstream), silently dropping the token on every write. + // Inject ONLY the server-minted scoped token; strip the client's + // sso session/api auth so it never reaches OpenBao. + proxyReq.setHeader('X-Vault-Token', req.vaultToken); + proxyReq.removeHeader('auth-token'); + proxyReq.removeHeader('authorization'); + fixRequestBody(proxyReq, req, res, options); }, }); } @@ -236,7 +406,7 @@ mintAppRouter.post('/', async (req, res, next) => { await permission.byGroup(req.user, [ADMIN_GROUP]); const name = (req.body && req.body.name || '').trim(); if (!name) return res.status(400).json({ error: 'name is required' }); - const result = await mintAppToken(name); + const result = await mintAppToken(name, req.user && req.user.uid); res.json(result); } catch (e) { if (e.status === 401) return res.status(403).json({ error: 'admin only' }); @@ -244,6 +414,27 @@ mintAppRouter.post('/', async (req, res, next) => { } }); +// List the minted external-app tokens (metadata only — the token itself is shown +// once at mint and never stored; the accessor is a renewal/revoke handle and is +// never exposed). Lets the Apps tab show what has been minted instead of a +// credential vanishing into the void. +mintAppRouter.get('/', async (req, res, next) => { + try { + await permission.byGroup(req.user, [ADMIN_GROUP]); + const rows = await VaultAppToken.list(); + res.json({ apps: rows.map((r) => ({ + name: r.name, + createdBy: r.created_by, + createdOn: r.created_on, + lastRenewedAt: r.lastRenewedAt || null, + lastError: r.lastError || null, + })) }); + } catch (e) { + if (e.status === 401) return res.status(403).json({ error: 'admin only' }); + next(e); + } +}); + module.exports = { getOrCreateUserToken, getOrCreateAdminToken, @@ -252,4 +443,16 @@ module.exports = { scopeGuard, vaultProxy, mintAppRouter, -}; \ No newline at end of file + // app-token lifecycle + renewAppTokens, + startAppTokenRenewal, + VaultAppToken, + // sharing + SharedSecret, + SharedSecretGrant, + userPolicyHcl, + appPolicyHcl, + grantSharedSecret, + revokeSharedSecret, + reconcileGrantee, +}; diff --git a/nodejs/views/conf.ejs b/nodejs/views/conf.ejs index a95939b..b21d08f 100644 --- a/nodejs/views/conf.ejs +++ b/nodejs/views/conf.ejs @@ -1,11 +1,17 @@ <%- include('top') %> + -
-
-
-
-

System Configuration

-

- Manage runtime configuration such as SMTP, SMS, OAuth, and Terms of Service - settings. These are stored securely in OpenBao and take effect immediately. - Secret fields (the SMTP password, OAuth JWT secret, and VoIP.ms API password) - are masked — leave them unchanged to keep the stored value. -

-
-
- - -
-
-
- - - -
- -
-
-
-
SMTP Settings
+
+
+
+
+ +
+ +
+ + +
-
-
- - -
-
- - -
-
- - -
-
- -
- - -
-
Leave unchanged to keep the current password stored in OpenBao. Clear and type a new value to replace it.
-
-
- -
- - -
-
Send a test SMS to verify your VoIP.ms configuration is working.
-
-
-
-
- -
- - -
-
Send a test SMS to verify your VoIP.ms configuration is working.
-
-
- - -
-
- - -
-
-
- -
- - + +
+
+ + +
+
OAuth 2.0 & JWT Settings
+

Configure OIDC issuer URLs, token lifetimes, and JWT signing keys. Stored in OpenBao.

+
+ + +
+
+ +
+ + +
+
Stored in OpenBao. Leave unchanged to preserve stored value.
+
+
+
+ + +
+
+ + +
-
Send a test email to verify your SMTP configuration is working.
-
-
-
- -
-
-
-
OAuth & JWT Settings
-
-
-
- - -
-
- -
- - + +
+
SMTP Server Settings
+

System mail server credentials for password resets, notifications, and verification emails.

+
+
+ + +
+
+ + +
+
+
+
+ + +
+
+ +
+ + +
+
+
+
+ + +
+
+ + +
+ +
+
Send Test Email
+
+ + +
+
Saves current SMTP config and sends a test message.
+
-
Leave unchanged to keep the current secret stored in OpenBao. Clear and type a new value to replace it.
-
-
- - -
-
- - -
-
-
-
- -
-
-
-
SMS (VoIP.ms)
-
-
-

Used to deliver SMS 2FA login codes. The API password is stored in OpenBao and masked below.

-
- - -
-
- - -
-
- -
- - + +
+
VoIP.ms SMS Integration
+

Configure VoIP.ms API credentials for delivering SMS 2FA codes.

+
+
+ + +
+
+ + +
+
+
+ +
+ + +
+
+ +
+
Send Test SMS
+
+ + +
+
+ +
+ +
+
Messaging Plugins & Webhooks
+ +
+
-
Leave unchanged to keep the current password stored in OpenBao. Clear and type a new value to replace it.
-
-
- -
- - + + +
+
OpenBao Proxy Integration
+

Secrets stored directly in OpenBao (secret/proxy/conf) and consumed by Proxy at boot.

+ +
OAuth / OIDC Client
+
+ + +
+
+
+ + +
+
+ +
+ + +
+
+
+ +
LDAP Bind Account
+
+ +
+ + +
+
+ +
-
Send a test SMS to verify your VoIP.ms configuration is working.
-
-
-
-
- -
-
-
-
Proxy Secrets (OpenBao)
-
-
-

These secrets are stored directly in OpenBao (`secret/proxy/conf`) and read by the Proxy at boot.

- -
OAuth / OIDC Integration
-
- - -
-
- - -
-
- -
- - + +
+
External App Tokens (OpenBao)
+

Mint scoped OpenBao tokens for external microservices, scripts, and third-party tools (scoped to secret/apps/<name>/*).

+
+
+
+
Mint New App Token
+
+

Mints a periodic OpenBao token. The token will be displayed once.

+
+ + +
Use lowercase letters, numbers, and hyphens.
+
+ +
+
+
+
+
+
+
+
Generated Token
+ +
+
+

Include this token in HTTP header X-Vault-Token:

+ 

+										

+									

+									

+										

+											
 Active App Tokens

+											
+										

+										

+											
Loading apps…

+										

+									

+								

+							

 						

-					

 
-					
LDAP Integration

-					

-						
-						

-							
-							
+						
+						

+							

+								
 Terms of Service Editor

+								
+							

+							

+								
+								
+							

+							

+								
+								
+							

+							
+							
 						

-						
Password for the Proxy's LDAP service account.

-					

 
-					
-				

-			

-		

-
-		
-		

-			

-				

-					
 Terms of Service

-					
-				

-				

-					

-						
-						
 					

-					

-						
-						
-					

-					
-					
 				

 			

 		

diff --git a/nodejs/views/directory.ejs b/nodejs/views/directory.ejs
index 4afd120..4101e7a 100644
--- a/nodejs/views/directory.ejs
+++ b/nodejs/views/directory.ejs
@@ -1,5 +1,16 @@
 <%- include('top') %>
 
+
+
 

   

     

@@ -13,7 +24,12 @@
             
             
+            
           
@@ -25,8 +41,17 @@
 				

 				    

     					 Directory Management
+    					
     				

 					

+                        

+                            
+                            
+                        

+                        

+                            
+                            
+                        

                         
                         
+        
+        
Derived from the name; read-only.

       

     

 
@@ -442,6 +511,95 @@
     

   `;
 
+  var secretsTabHtml = `
+    

+      

+        

+          
Resource Secrets (OpenBao KV Engine)

+          Encrypted key-value secrets stored in OpenBao under secret/data/resources/<slug>/conf
+        

+        
+      

+
+      
+
+      

+        
+          
+            
+              
+              
+              
+            
+          
+          
+            
+          
+        
Secret KeyStatus / SecurityActions
Loading secrets...

+      

+
+      
+      

+        
 Add or Generate Secret Key

+        

+          

+            
+            
+            
Only letters, numbers, and underscores allowed (e.g. DB_PASSWORD).

+          

+          

+            
+            
+          

+          

+            
+          

+        

+
+        

+          

+            
+            
+          

+          

+            
+          

+        

+
+        
+      

+
+      
+      

+        
 Inherit Secret from Parent Resource

+        

+          

+            
+            
+          

+          

+            
+            
+          

+          

+            
+          

+        

+      

+    

+  `;
+
   // Shared by openAddModal/openEditModal: builds the tabbed/footer/(optionally
   // URL-tracked) modal DOM. Callers then populate fields via .val() and hide
   // the Groups/Children tabs in add-mode (no resource id to scope them to).
@@ -454,6 +612,8 @@
         {id: 'details', label: 'Details', bodyHtml: detailsTabHtml},
         {id: 'groups', label: 'Associated LDAP Groups', bodyHtml: groupsTabHtml},
         {id: 'children', label: 'Children', bodyHtml: childrenTabHtml},
+        {id: 'secrets', label: 'Secrets & OpenBao', bodyHtml: secretsTabHtml},
+        {id: 'agent', label: 'Agent', bodyHtml: agentTabHtml(resourcesById[id] && resourcesById[id].agent)},
       ],
       footer: {
         metaHtml: id ? app.modal.formatAudit(resourcesById[id], {formatDate: function(ms){ return moment(ms).format('YYYY-MM-DD HH:mm'); }}) : '',
@@ -461,7 +621,7 @@
       },
       url: id ? {path: '/directory/' + resourcesById[id].slug} : null,
     });
-    $('#sw-modal-tab-groups-btn, #sw-modal-tab-children-btn').closest('li').toggle(!!id);
+    $('#sw-modal-tab-groups-btn, #sw-modal-tab-children-btn, #sw-modal-tab-secrets-btn').closest('li').toggle(!!id);
   }
 
   function refreshChildrenUI(resourceId) {
@@ -487,8 +647,8 @@
   var allGroups = [];
   var allEdges = [];
   var rawResources = [];
-  // resourceId -> { groups: [{cn, accessLevel, exists, memberCount}], memberCount }
   var accessSummary = {};
+  var promoteSlug = null;
 
   $(document).ready(async function() {
     await loadResources();
@@ -499,31 +659,38 @@
     }
   });
   
+  var agentsByHost = {};
+  var agentsByResource = {};
+  var agentsById = {};
+  var agentsUnavailable = false;
+
   async function loadResources() {
     try {
-      const [resResources, resGroups, resEdges, resAccess] = await Promise.all([
+      const [resResources, resGroups, resEdges, resAccess, resAgents] = await Promise.all([
         app.api.get('directory-admin/resources'),
         app.api.get('directory-admin/groups'),
         app.api.get('directory-admin/edges'),
-        // Access counts are a nicety, not load-bearing: if the LDAP join fails
-        // the table still renders, just without the Access column populated.
-        app.api.get('directory-admin/access-summary').catch(function(){ return {results: {}}; })
+        app.api.get('directory-admin/access-summary').catch(function(){ return {results: {}}; }),
+        app.api.get('agent/nodes')
+          .then(function(res){ agentsUnavailable = false; return res; })
+          .catch(function(){ agentsUnavailable = true; return {agents: []}; })
       ]);
 
       accessSummary = (resAccess && resAccess.results) || {};
       resourcesById = {};
-      
+
       for (const r of resResources.results) {
         r.metadata = r.metadata || {};
         resourcesById[r.id] = r;
       }
+
+      indexAgents((resAgents && resAgents.agents) || []);
       
       allGroups = resGroups.results;
       allEdges = resEdges.results;
 
       rawResources = [];
       for (const r of resResources.results) {
-        // Compute hostName from edges
         r.hostName = '—';
         r.parentId = null;
         const parentEdge = allEdges.find(e => e.childId === r.id);
@@ -537,19 +704,285 @@
       
       renderTable();
 
-      // Type-ahead for the "what can this user reach" lookup. Non-blocking: the
-      // input accepts a free-typed uid whether or not the list ever arrives.
       loadDirectoryUsers().then(function(users) {
         $('#access-uid-list').html(users.map(function(u) {
           return '';
         }).join(''));
-      }).catch(function(){ /* datalist is a convenience only */ });
+      }).catch(function(){});
     } catch (err) {
       console.error(err);
       app.messages.toast('Failed to load data', 'danger');
     }
   }
 
+  function indexAgents(agents) {
+    agentsByHost = {};
+    agentsByResource = {};
+    agentsById = {};
+    for (const a of agents || []) {
+      agentsById[a.id] = a;
+      if (a.resourceId) agentsByResource[a.resourceId] = a;
+      const hn = ((a.lastDiscovery && a.lastDiscovery.hostname) || a.name || '').toLowerCase();
+      if (hn && !agentsByHost[hn]) agentsByHost[hn] = a;
+    }
+  }
+
+  function esc(s) { return s == null ? '' : app.util.escapeHtml(String(s)); }
+  function timeAgo(iso) { if (!iso) return ''; var m = moment(iso); return m.isValid() ? m.fromNow() : ''; }
+
+  function attachAgentStatus(n) {
+    n.isHost = true;
+    const name = (n.name || '').toLowerCase();
+    const slug = (n.slug || '').replace(/^host_/, '').toLowerCase();
+    const a = agentsByResource[n.id] || agentsByHost[name] || (slug && agentsByHost[slug]);
+    n.agent = a || null;
+    if (resourcesById[n.id]) resourcesById[n.id].agent = a || null;
+    if (!a) {
+      if (agentsUnavailable) { n.agentColor = '#adb5bd'; n.agentStatusTitle = 'Agent service unreachable'; return; }
+      n.agentColor = '#adb5bd'; n.agentStatusTitle = 'No theta-agent enrolled'; return;
+    }
+    if (a.revoked) { n.agentColor = '#6c757d'; n.agentStatusTitle = 'Agent enrollment revoked'; return; }
+    if (!a.isOnline) {
+      const seen = (a.lastSeen || a.last_seen) ? ' — last seen ' + timeAgo(a.lastSeen || new Date(a.last_seen * 1000).toISOString()) : '';
+      n.agentColor = '#dc3545'; n.agentStatusTitle = 'Agent enrolled but offline' + seen; return;
+    }
+    const t = a.lastTelemetry || {};
+    const high = (t.cpu_usage_percent > 80) || (t.ram_usage_percent > 80) || (t.disk_usage_percent > 90);
+    n.agentColor = high ? '#ffc107' : '#198754';
+    n.agentStatusTitle = high ? 'Connected — high load' : 'Connected — healthy';
+  }
+
+  // Agent tab body for the resource modal.
+  function agentTabHtml(agent) {
+    if (!agent) {
+      return '
No theta-agent connected
Install the agent on this host to see live telemetry and issue control commands.
';
+    }
+    const d = agent.lastDiscovery || {};
+    const t = agent.lastTelemetry || {};
+    const fmtNum = (n) => (n == null || isNaN(n)) ? '0.00' : Number(n).toFixed(2);
+    const fmtSize = (bytes) => {
+      if (!bytes || bytes <= 0) return '0.00 B';
+      const gib = bytes / (1024 * 1024 * 1024);
+      if (gib >= 1) return fmtNum(gib) + ' GiB';
+      const mib = bytes / (1024 * 1024);
+      return fmtNum(mib) + ' MiB';
+    };
+
+    const bar = (val) => `
`;
+    const online = agent.isOnline ? 'Online' : 'Offline';
+    const gpu = (t.gpu_usage_percent != null && t.gpu_usage_percent >= 0) ? fmtNum(t.gpu_usage_percent) + '%' : 'N/A';
+    const lastSeenIso = agent.lastSeen || (agent.last_seen ? new Date(agent.last_seen * 1000).toISOString() : '');
+    const lastSeenStr = timeAgo(lastSeenIso) || 'never';
+
+    // RAM details
+    const ram = t.ram_details || d.ram_details || {};
+    const totalRamBytes = ram.total_bytes || (d.ram_total_gb ? d.ram_total_gb * 1024 * 1024 * 1024 : 0);
+    const usedRamBytes = ram.used_bytes || (totalRamBytes * (t.ram_usage_percent || 0) / 100);
+    const bufRamBytes = ram.buffers_cache_bytes || 0;
+    const freeRamBytes = ram.free_bytes || Math.max(0, totalRamBytes - usedRamBytes - bufRamBytes);
+
+    const usedRamPct = ram.used_percent != null ? ram.used_percent : (t.ram_usage_percent || 0);
+    const bufRamPct = ram.buffers_cache_percent != null ? ram.buffers_cache_percent : 0;
+    const freeRamPct = ram.free_percent != null ? ram.free_percent : Math.max(0, 100 - usedRamPct - bufRamPct);
+
+    // CPU details
+    const cpuDet = t.cpu_details || d.cpu_details || {};
+    const cpuModel = cpuDet.model || d.cpu || 'Unknown CPU';
+    const cpuCores = cpuDet.cores || 'N/A';
+    const cpuThreads = cpuDet.threads || 'N/A';
+    const cpuSpeed = cpuDet.mhz ? (cpuDet.mhz >= 1000 ? (cpuDet.mhz / 1000).toFixed(2) + ' GHz' : cpuDet.mhz.toFixed(0) + ' MHz') : '';
+
+    // Disks
+    const disks = t.disks || d.disks || [];
+
+    let disksHtml = '';
+    if (disks.length > 0) {
+      disksHtml = `
` +
+        disks.map(dk => `
+          
+          
+          
+          
+          
+        `).join('') + `
MountTypeFSUsageTotal
${esc(dk.mountpoint)}${esc(dk.drivetype || 'Disk')}${esc(dk.fstype || 'N/A')}${fmtNum(dk.usage_percent)}%${fmtSize(dk.total_bytes)}
`;
+    } else {
+      disksHtml = `
Disk Usage: ${fmtNum(t.disk_usage_percent ?? 0)}% ${bar(t.disk_usage_percent)}
`;
+    }
+
+    return `

+      

+        
${esc(agent.name || agent.hostname || 'unknown')} ${online}

+        Last seen ${lastSeenStr}
+      

+
+      
+      

+        

+          
 Memory

+        

+        

+          

+            Total: ${fmtSize(totalRamBytes)}
+          

+          

+            Used: ${fmtSize(usedRamBytes)} ${fmtNum(usedRamPct)}%
+          

+          

+            Buffer+Cache: ${fmtSize(bufRamBytes)} ${fmtNum(bufRamPct)}%
+          

+          

+            Free: ${fmtSize(freeRamBytes)} ${fmtNum(freeRamPct)}%
+          

+          

+            

+            

+            

+          

+        

+      

+
+      
+      

+        

+          
 CPU

+        

+        

+          
Model: ${esc(cpuModel)}

+          
Specs: ${esc(cpuCores)} Cores / ${esc(cpuThreads)} Threads ${cpuSpeed ? '@ ' + esc(cpuSpeed) : ''}

+          
Usage: ${fmtNum(t.cpu_usage_percent ?? 0)}% ${bar(t.cpu_usage_percent)}

+        

+      

+
+      
+      

+        

+          
 Disks & Storage

+        

+        

+          ${disksHtml}
+        

+      

+
+      

+        
GPU ${gpu}

+        
ZFS ${esc(t.zfs_health || 'N/A')}

+      

+
+      
+      

+        

+          
 Host Power Operations

+        

+        

+          
+          
+        

+      

+
+      
+      

+        

+          
 Systemd Service Manager

+        

+        

+          

+            Service Name
+            
+            
+            
+            
+            
+          

+          

+            

+          

+        

+      

+
+      
Discovery & Metadata

+      

+        
OS: ${esc(d.os || '')}

+        
Kernel: ${esc(d.kernel || '')}

+        
IPs: ${esc((d.ip_addresses || []).join(', '))}

+        
Location: ${esc(d.location || '')}

+      

+      
Capabilities

+      
${capabilitiesHtml(d.capabilities)}

+    
`;
+  }
+
+  // Render the agent's enabled capabilities (reported in its discovery frame) as
+  // green/gray badges. service_control is a list, so it renders as its own line.
+  function capabilitiesHtml(caps) {
+    caps = caps || {};
+    const badge = (name, on) => `${esc(name)}`;
+    const bools = [
+      ['Telemetry', caps.telemetry],
+      ['LDAP config', caps.configure_ldap],
+      ['LDAP tunnel', caps.ldap_tunnel],
+      ['Secrets', caps.secrets],
+      ['IAM', caps.iam],
+      ['Reboot', caps.reboot],
+      ['Bash', caps.arbitrary_bash],
+    ];
+    const sc = Array.isArray(caps.service_control) ? caps.service_control : [];
+    const scLine = sc.length
+      ? `
Service control: ${esc(sc.join(', '))}
`
+      : '';
+    return bools.map(([n, on]) => badge(n, !!on)).join('') + scLine;
+  }
+
+  async function agentReboot(agentId) {
+    const ok = await app.messages.confirm('Are you sure you want to REBOOT this host?');
+    if (!ok) return;
+    app.api.post(`agent/nodes/${agentId}/command`, { command: 'reboot', isHighRisk: true }, function(err, res) {
+      if (err) return app.messages.toast('Reboot failed: ' + (err.message || err), 'danger');
+      app.messages.toast('Reboot command sent to host', 'success');
+    });
+  }
+
+  async function agentShutdown(agentId) {
+    const ok = await app.messages.confirm('Are you sure you want to SHUTDOWN this host?');
+    if (!ok) return;
+    app.api.post(`agent/nodes/${agentId}/command`, { command: 'shutdown', isHighRisk: true }, function(err, res) {
+      if (err) return app.messages.toast('Shutdown failed: ' + (err.message || err), 'danger');
+      app.messages.toast('Shutdown command sent to host', 'success');
+    });
+  }
+
+  function manageService(agentId, action) {
+    const service = ($(`#sysd-service-${agentId}`).val() || '').trim();
+    if (!service) return app.messages.toast('Service name required', 'warning');
+    const outBox = $(`#sysd-output-${agentId}`);
+    const outText = $(`#sysd-text-${agentId}`);
+    outBox.removeClass('d-none');
+    outText.text(`Executing systemctl ${action} ${service}...`);
+    app.api.post(`agent/nodes/${agentId}/command`, {
+      command: 'systemd_action',
+      payload: { action, service },
+      isHighRisk: action !== 'status'
+    }, function(err, res) {
+      if (err) {
+        outText.text('Error: ' + (err.message || JSON.stringify(err)));
+        return;
+      }
+      outText.text(`Command '${action}' sent for service '${service}'. Check output or logs.`);
+    });
+  }
+
+  // Re-fetch agents (every 30s + on socket events) so status dots stay live.
+  async function refreshAgents() {
+    try {
+      const res = await app.api.get('agent/nodes');
+      indexAgents((res && res.agents) || []);
+      agentsUnavailable = false;
+      renderTable();
+    } catch (e) {
+      agentsUnavailable = true;
+      renderTable(); // re-render so dots flip to neutral, not stale green
+    }
+  }
+
   // "Who can reach this?" at a glance. A resource with no linked group is not a
   // locked-down resource -- it is an unreachable one, and a group whose LDAP
   // entry has been deleted grants nothing, so both get called out rather than
@@ -608,8 +1041,15 @@
   function renderTable() {
     const filter = $('#search-filter').val().toLowerCase();
     const sort = $('#sort-by').val();
+    const showPlumbing = $('#toggle-plumbing').is(':checked');
 
     let filtered = rawResources.filter(r => {
+      if (!showPlumbing && !filter) {
+        const sub = (r.metadata?.subType || '').toLowerCase();
+        if (r.kind === 'container' || r.kind === 'oauth' || sub === 'sidecar' || sub === 'container' || sub === 'openresty') {
+          return false;
+        }
+      }
       if (!filter) return true;
       return (r.name || '').toLowerCase().includes(filter) ||
              (r.slug || '').toLowerCase().includes(filter) ||
@@ -646,6 +1086,10 @@
       }
     });
 
+    // Rows are emitted depth-first, so a node's descendants are exactly the
+    // rows that follow it until depth drops back to its own. `data-depth` is
+    // what applyTreeCollapse() below walks -- that ordering is the whole
+    // mechanism, so keep the traversal depth-first if you change this.
     const flatten = (nodes, depth) => {
        nodes.forEach(n => {
          let indentHtml = '';
@@ -656,7 +1100,18 @@
             indentHtml += '';
          }
          n.indentHtml = indentHtml;
+         n.depth = depth;
+         // A leaf gets a spacer of the same width, so names stay aligned down
+         // the column instead of jittering by whether a row has children.
+         n.caretHtml = n.children.length
+           ? ''
+           : '';
+         n.childCount = n.children.length;
          n.accessHtml = accessCellHtml(n.id);
+         if (n.kind === 'host') attachAgentStatus(n);
          finalRenderList.push(n);
          if (n.children.length > 0) {
            flatten(n.children, depth + 1);
@@ -670,8 +1125,110 @@
     for (const r of finalRenderList) {
       $.scope.resources.push(r);
     }
+
+    applyTreeCollapse();
   }
   
+  // ── Collapsible tree ───────────────────────────────────────────────────────
+  // Which nodes are collapsed, by resource id. Persisted so the shape of the
+  // tree survives a refresh (and the Directory self-heal reload that follows
+  // most edits) -- a tree that re-expands every time is worse than no tree.
+  var TREE_COLLAPSE_KEY = 'directory.collapsedNodes';
+
+  function loadCollapsed() {
+    try {
+      const raw = localStorage.getItem(TREE_COLLAPSE_KEY);
+      return new Set(raw ? JSON.parse(raw) : []);
+    } catch (e) { return new Set(); }
+  }
+
+  function saveCollapsed(set) {
+    try { localStorage.setItem(TREE_COLLAPSE_KEY, JSON.stringify([...set])); } catch (e) {}
+  }
+
+  function escapeHtmlAttr(s) {
+    return String(s).replace(/[&<>"']/g, c => ({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c]));
+  }
+
+  // Hide every row beneath a collapsed node and point its caret sideways.
+  // Rows are in depth-first order, so "beneath" is the run of following rows
+  // with a greater depth. A node inside an already-hidden run stays hidden
+  // regardless of its own state, which is what makes nesting work.
+  function applyTreeCollapse() {
+    const $rows = $('#resources-list tr');
+
+    // While a search is active every match must be visible, even one sitting
+    // under a collapsed ancestor -- otherwise searching silently returns
+    // nothing and looks broken. The collapsed set is left untouched, so the
+    // tree springs back to its saved shape as soon as the box is cleared.
+    if (($('#search-filter').val() || '').trim()) {
+      $rows.show();
+      $rows.find('.tree-caret i').removeClass('fa-chevron-right').addClass('fa-chevron-down');
+      return;
+    }
+
+    const collapsed = loadCollapsed();
+    let hideBelowDepth = null;
+
+    $rows.each(function() {
+      const $row = $(this);
+      const depth = parseInt($row.attr('data-depth') || '0', 10);
+      const id = ($row.attr('id') || '').replace('resource-row-', '');
+
+      if (hideBelowDepth !== null && depth > hideBelowDepth) {
+        $row.hide();
+        return; // still inside a collapsed subtree; its own state is moot
+      }
+      hideBelowDepth = null;
+      $row.show();
+
+      // Visual state lives on the .tree-caret BUTTON, rotated by CSS, and the
+      // hide decision is made from `collapsed` alone.
+      //
+      // This used to read `.tree-caret i` and bail out when it found nothing.
+      // Font Awesome runs in SVG-with-JS mode here: its mutation observer
+      // rewrites every  into an , so moments after a
+      // render that selector matches nothing, the function returned early
+      // WITHOUT setting hideBelowDepth, and collapsing silently did nothing at
+      // all. Never make the collapse logic depend on an element another library
+      // is free to replace.
+      const $caret = $row.find('.tree-caret');
+      if (!$caret.length) return;   // leaf row: nothing to collapse
+      if (collapsed.has(id)) {
+        $caret.addClass('tree-caret-collapsed');
+        hideBelowDepth = depth;
+      } else {
+        $caret.removeClass('tree-caret-collapsed');
+      }
+    });
+  }
+
+  function toggleTreeNode(id) {
+    const collapsed = loadCollapsed();
+    if (collapsed.has(id)) collapsed.delete(id); else collapsed.add(id);
+    saveCollapsed(collapsed);
+    applyTreeCollapse();
+  }
+
+  function expandAllTree() {
+    saveCollapsed(new Set());
+    applyTreeCollapse();
+  }
+
+  // Collapse every row that has children. Reads the ids out of the rendered
+  // rows rather than the resource list so it can only ever collapse something
+  // that is actually on screen and actually has a caret.
+  function collapseAllTree() {
+    const collapsed = new Set();
+    $('#resources-list tr').each(function() {
+      const $row = $(this);
+      if (!$row.find('.tree-caret').length) return;
+      collapsed.add(($row.attr('id') || '').replace('resource-row-', ''));
+    });
+    saveCollapsed(collapsed);
+    applyTreeCollapse();
+  }
+
   function toggleFormFields() {
     const kind = $('#res-kind').val();
     if (kind === 'host') {
@@ -1047,10 +1604,229 @@
     refreshGroupsUI(r.id);
     refreshEdgesUI(r.id);
     refreshChildrenUI(r.id);
+    loadResourceSecrets(r.id);
     await loadLdapGroups();
   }
+
+  var currentResourceSecretsList = [];
+  var currentParentSecretsList = [];
+  var rawResourceSecretsMap = {};
+
+  const SECRET_KEY_REGEX = /^[A-Za-z0-9_]+$/;
+
+  function validateSecretKeyInput(el) {
+    const $el = $(el);
+    const val = $el.val().trim();
+    if (val && !SECRET_KEY_REGEX.test(val)) {
+      $el.addClass('is-invalid');
+      return false;
+    } else {
+      $el.removeClass('is-invalid');
+      return true;
+    }
+  }
+
+  function generateRandomString(len) {
+    const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*()_+-=[]{}|;:,.<>?';
+    const bytes = new Uint8Array(len);
+    window.crypto.getRandomValues(bytes);
+    let str = '';
+    for (let i = 0; i < len; i++) {
+      str += chars[bytes[i] % chars.length];
+    }
+    return str;
+  }
+
+  async function loadResourceSecrets(id) {
+    const $tbody = $('#secrets-table-body').empty();
+    $tbody.append('
Loading secrets from OpenBao...');
+    currentResourceSecretsList = [];
+    currentParentSecretsList = [];
+    rawResourceSecretsMap = {};
+
+    try {
+      const res = await app.api.get(`directory-admin/resources/${id}/secrets`);
+      currentResourceSecretsList = (res && res.secrets) || [];
+      currentParentSecretsList = (res && res.parentSecrets) || [];
+      renderSecretsTable();
+      populateParentSecretsDropdown();
+    } catch (err) {
+      $tbody.empty().append(`Failed to load secrets: ${esc(err.message || 'Unknown error')}`);
+    }
+  }
+
+  function populateParentSecretsDropdown() {
+    const $card = $('#inherit-secret-card');
+    const $select = $('#inherit-parent-select').empty();
+    const $btn = $('#btn-inherit-secret');
+
+    $card.show();
+
+    if (!currentParentSecretsList || currentParentSecretsList.length === 0) {
+      $select.append('');
+      $btn.prop('disabled', true);
+      return;
+    }
+
+    $btn.prop('disabled', false);
+    $select.append('');
+    currentParentSecretsList.forEach(p => {
+      const valStr = `INHERIT:${p.parentSlug}:${p.key}`;
+      const labelStr = `${p.parentName || p.parentSlug} → ${p.key}`;
+      $select.append(``);
+    });
+  }
+
+  function renderSecretsTable() {
+    const $tbody = $('#secrets-table-body').empty();
+
+    if (currentResourceSecretsList.length === 0) {
+      $tbody.append('No secrets configured for this resource yet.');
+      return;
+    }
+
+    currentResourceSecretsList.forEach((s, idx) => {
+      const $row = $(`
+        
+          ${esc(s.key)}
+          
+            ${s.isInherited
+              ? `Inherited from ${esc(s.parentSlug || 'Parent')} (${esc(s.parentKey || s.key)})`
+              : `Configured in OpenBao Secret Value Hidden`
+            }
+          
+          
+            
+            
+          
+        
+      `);
+      $tbody.append($row);
+    });
+  }
+
+  function generateSecretValue() {
+    let key = $('#new-secret-key').val().trim();
+    if (!key) {
+      key = 'SECRET_KEY';
+      $('#new-secret-key').val(key);
+    }
+    const len = parseInt($('#gen-secret-length').val(), 10) || 32;
+    const randomSecret = generateRandomString(len);
+    $('#new-secret-val').val(randomSecret);
+    $('#gen-secret-notice').show();
+  }
+
+  function editSecretKey(key) {
+    $('#new-secret-key').val(key);
+    $('#new-secret-val').val('').focus();
+    $('#gen-secret-notice').hide();
+  }
+
+  async function addSecretRow() {
+    const keyEl = $('#new-secret-key')[0];
+    const key = $('#new-secret-key').val().trim();
+    const val = $('#new-secret-val').val();
+    const resourceId = $('#res-id').val();
+
+    if (!key) {
+      app.messages.action('Please enter a secret key name (e.g. DB_PASSWORD).', $('#secrets-tab-container'), 'warning');
+      return;
+    }
+    if (!validateSecretKeyInput(keyEl)) {
+      app.messages.action('Invalid secret key format. Only uppercase/lowercase letters, numbers, and underscores are allowed (e.g. DB_PASSWORD).', $('#secrets-tab-container'), 'danger');
+      return;
+    }
+    if (!resourceId) return;
+
+    try {
+      app.messages.action('Saving secret to OpenBao...', $('#secrets-tab-container'), 'info');
+      await app.api.post(`directory-admin/resources/${resourceId}/secrets`, { secrets: { [key]: val || '' } });
+      app.messages.action(`Secret '${key}' saved to OpenBao successfully!`, $('#secrets-tab-container'), 'success');
+      $('#new-secret-key').val('');
+      $('#new-secret-val').val('');
+      $('#gen-secret-notice').hide();
+      loadResourceSecrets(resourceId);
+    } catch (err) {
+      app.messages.action(err.message || 'Failed to save secret to OpenBao', $('#secrets-tab-container'), 'danger');
+    }
+  }
+
+  async function inheritParentSecret() {
+    const childKey = $('#inherit-child-key').val().trim();
+    const inheritVal = $('#inherit-parent-select').val();
+    const resourceId = $('#res-id').val();
+
+    if (!childKey) {
+      app.messages.action('Please enter a child secret key name (e.g. DB_HOST).', $('#secrets-tab-container'), 'warning');
+      return;
+    }
+    if (!SECRET_KEY_REGEX.test(childKey)) {
+      app.messages.action('Invalid child key name. Only letters, numbers, and underscores allowed.', $('#secrets-tab-container'), 'danger');
+      return;
+    }
+    if (!inheritVal) {
+      app.messages.action('Select a parent secret to inherit from.', $('#secrets-tab-container'), 'warning');
+      return;
+    }
+    if (!resourceId) return;
+
+    try {
+      app.messages.action('Saving inherited secret to OpenBao...', $('#secrets-tab-container'), 'info');
+      await app.api.post(`directory-admin/resources/${resourceId}/secrets`, { secrets: { [childKey]: inheritVal } });
+      app.messages.action(`Inherited secret '${childKey}' saved successfully!`, $('#secrets-tab-container'), 'success');
+      $('#inherit-child-key').val('');
+      loadResourceSecrets(resourceId);
+    } catch (err) {
+      app.messages.action(err.message || 'Failed to save inherited secret', $('#secrets-tab-container'), 'danger');
+    }
+  }
+
+  async function deleteSecretKey(key) {
+    const confirmed = await app.messages.confirm(`Delete secret '${key}' from OpenBao?`, $('#secrets-tab-container'), 'danger');
+    if (!confirmed) return;
+    const resourceId = $('#res-id').val();
+    if (!resourceId) return;
+
+    try {
+      app.messages.action(`Deleting secret '${key}' from OpenBao...`, $('#secrets-tab-container'), 'info');
+      await app.api.post(`directory-admin/resources/${resourceId}/secrets`, { action: 'delete', key });
+      app.messages.action(`Secret '${key}' deleted successfully from OpenBao.`, $('#secrets-tab-container'), 'success');
+      loadResourceSecrets(resourceId);
+    } catch (err) {
+      app.messages.action(err.message || 'Failed to delete secret from OpenBao', $('#secrets-tab-container'), 'danger');
+    }
+  }
+
+  function refreshResourceSecrets() {
+    const resourceId = $('#res-id').val();
+    if (resourceId) loadResourceSecrets(resourceId);
+  }
   
   async function saveResource() {
+    // Promote path: the modal was opened from a discovered inventory row, so
+    // Save confirms promotion (creates LDAP groups + marks managed) rather than
+    // a normal resource create/update.
+    if (promoteSlug) {
+      const slug = promoteSlug;
+      promoteSlug = null;
+      try {
+        const res = await new Promise((resolve, reject) => {
+          app.api.post('discovery/promote/' + slug, {}, function(err, r) {
+            if (err) reject(err); else resolve(r);
+          });
+        });
+        await loadResources();
+        loadDiscoveryResources();
+        app.modal.close();
+        app.messages.toast('Promoted ' + slug + (res && res.groups ? ' — created groups: ' + res.groups.join(', ') : ''), 'success');
+      } catch (err) {
+        promoteSlug = slug;
+        app.messages.action('Failed to promote: ' + (err.message || err), app.modal.body(), 'danger');
+      }
+      return;
+    }
+
     const id = $('#res-id').val();
     const data = {
       name: $('#res-name').val(),
@@ -1149,6 +1925,7 @@
       allGroups.push(res.results);
       refreshGroupsUI(resourceId);
       $('#new-group-cn').val('');
+      await loadResources(); // keep the Access column in sync
     } catch (err) {
       console.error(err);
       app.messages.action('Failed to add group', app.modal.body(), 'danger');
@@ -1160,6 +1937,7 @@
       await app.api.delete('directory-admin/groups/' + id);
       allGroups = allGroups.filter(g => g.id !== id);
       refreshGroupsUI($('#res-id').val());
+      await loadResources(); // keep the Access column in sync
     } catch (err) {
       console.error(err);
       app.messages.action('Failed to remove group', app.modal.body(), 'danger');
@@ -1188,6 +1966,7 @@
       allEdges.push(res.results);
       refreshEdgesUI(resourceId);
       $('#new-edge-target').val('');
+      await loadResources();
     } catch (err) {
       console.error(err);
       app.messages.action('Failed to add edge', app.modal.body(), 'danger');
@@ -1199,6 +1978,7 @@
       await app.api.delete('directory-admin/edges/' + id);
       allEdges = allEdges.filter(e => e.id !== id);
       refreshEdgesUI($('#res-id').val());
+      await loadResources();
     } catch (err) {
       console.error(err);
       app.messages.action('Failed to remove edge', app.modal.body(), 'danger');
@@ -1238,14 +2018,23 @@
   function renderDiscoveryTable() {
     const search = $('#discovery-search-filter').val().toLowerCase();
     const filtered = allDiscoveryResources.filter(r => {
-      if(search && !r.name.toLowerCase().includes(search) && !r.slug.toLowerCase().includes(search)) return false;
-      const isManaged = !!(r.metadata && r.metadata.managed);
-      if(isManaged) return false;
+      if (search && !r.name.toLowerCase().includes(search) && !r.slug.toLowerCase().includes(search)) return false;
+      // Directory contains managed items; Discovered Inventory only shows unmanaged/pending items awaiting promotion
+      const isExplicitManaged = r.metadata && (r.metadata.managed === true || r.metadata.managed === 'true');
+      if (isExplicitManaged || r.kind === 'site' || r.kind === 'service') return false;
       return true;
     });
 
     $.scope.discoveryResources.empty();
     for(const r of filtered) {
+      // "Unknown IP" was shown for every device whose address is known per-NIC
+      // rather than in metadata.ip -- which is most of them, since a source
+      // that enumerates interfaces (UniFi, Proxmox guest agent) fills
+      // `interfaces[].ip`. Resolve a display address from the NICs so the
+      // column agrees with the interface list right beneath it.
+      const meta = r.metadata || {};
+      const fromNic = (meta.interfaces || []).map(i => i && i.ip).find(Boolean) || null;
+      r.displayIp = meta.ip || fromNic;
       $.scope.discoveryResources.push(r);
     }
     
@@ -1258,42 +2047,63 @@
     }
   }
 
+  // Promoting a discovered resource opens the resource form pre-filled with the
+  // discovered data so it can be reviewed before the resource is marked managed
+  // (and its LDAP groups created). The modal's Save (saveResource) sees
+  // promoteSlug set and calls the promote endpoint instead of a normal save.
   function promoteResource(slug) {
-    app.api.post('discovery/promote/' + slug, {}, function(err, res) {
-      if(err) {
-        app.messages.toast("Error promoting resource: " + (err.message || err), 'danger');
-        return;
-      }
-      const resource = allDiscoveryResources.find(r => r.slug === slug);
-      if(resource) {
-        resource.metadata = resource.metadata || {};
-        resource.metadata.managed = true;
-      }
-      $('.actionMessage').html('
Successfully promoted! Created groups: ' + res.groups.join(', ') + '
').show();
-      renderDiscoveryTable();
-      loadResources(); // Also update directory tab
-    });
+    const r = allDiscoveryResources.find(x => x.slug === slug);
+    if (!r) { app.messages.toast('Discovered resource not found', 'danger'); return; }
+    promoteSlug = slug;
+    openResourceModal('Promote Resource', null); // add-mode: groups/children tabs hidden
+    const m = r.metadata || {};
+    $('#res-name').val(r.name || '');
+    $('#res-slug').val(r.slug || '');
+    $('#res-kind').val(r.kind || 'host');
+    $('#res-description').val(r.description || '');
+    $('#res-ip').val(m.ip || '');
+    $('#res-address').val(m.address || '');
+    $('#res-subtype').val(m.subType || '');
+    $('#res-mac').val(m.macAddress || '');
+    $('#res-port').val(m.port || '');
+    $('#res-external-port').val(m.externalPort || '');
+    $('#res-icon').val(m.icon || '');
+    $('#res-tagline').val(m.tagline || '');
+    updateIconPreview();
+    toggleFormFields();
+    loadLdapGroups();
   }
 
   // --- THETA AGENT INSTALL MODAL & WIZARD ---
-  function generateRandomHexToken(byteLen) {
-    const arr = new Uint8Array(byteLen || 16);
-    (window.crypto || window.msCrypto).getRandomValues(arr);
-    return Array.from(arr, b => b.toString(16).padStart(2, '0')).join('');
-  }
-
-  function regenerateAgentToken(inputId) {
-    const newToken = generateRandomHexToken(16);
-    $('#' + inputId).val(newToken);
-    if (inputId === 'agent-quick-token') $('#agent-custom-token').val(newToken);
-    else $('#agent-quick-token').val(newToken);
-    updateAgentCommands();
-  }
+  // Agent tokens are no longer generated here. The browser minting a token the
+  // server had never heard of is exactly what made /api/agent/ws unauthenticated:
+  // there was nothing to validate against. Tokens now come from
+  // POST /api/agent/enroll (see enrollAgent).
 
   function updateAgentCommands() {
     const quickUrl = ($('#agent-quick-url').val() || window.location.origin).replace(/\/+$/, '');
     const quickToken = $('#agent-quick-token').val() || '';
-    const quickCmd = `curl -fsSL ${quickUrl}/resources/theta-agent/install.sh | sh -s -- --url "${quickUrl}" --token "${quickToken}"`;
+    // public_key must reach the host: without it the agent refuses every
+    // high-risk command. It was never emitted before, which is why signed
+    // commands only ever "worked" while verification was being skipped.
+    // Join-key command. Only a key we just minted can appear here -- the list
+    // endpoint deliberately never returns key values.
+    const joinUrl = ($('#agent-quick-url').val() || window.location.origin).replace(/\/+$/, '');
+    const selectedKeyId = $('#agent-join-key-select').val();
+    let joinCmd;
+    if (mintedJoinKey) {
+      joinCmd = `curl -fsSL ${joinUrl}/resources/theta-agent/install.sh | sh -s -- --url "${joinUrl}" --join-key "${mintedJoinKey}"`;
+    } else if (selectedKeyId) {
+      const k = agentJoinKeys.find(x => x.id === selectedKeyId);
+      joinCmd = `curl -fsSL ${joinUrl}/resources/theta-agent/install.sh | sh -s -- --url "${joinUrl}" --join-key "${k ? k.keyPrefix : ''}…"\n\n# Paste the full value of this key -- it was only shown when created.\n# If you no longer have it, create a new key above.`;
+    } else {
+      joinCmd = '# Create a join key above, or select one you already have the value for.';
+    }
+    $('#agent-join-command').text(joinCmd);
+
+    const pubKey = (pendingEnrollment && pendingEnrollment.publicKey) || '';
+    const quickCmd = `curl -fsSL ${quickUrl}/resources/theta-agent/install.sh | sh -s -- --url "${quickUrl}" --token "${quickToken}"`
+      + (pubKey ? ` --public-key "${pubKey}"` : '');
     $('#agent-quick-command').text(quickCmd);
 
     const customUrl = ($('#agent-custom-url').val() || window.location.origin).replace(/\/+$/, '');
@@ -1314,6 +2124,7 @@
     const yamlStr = [
       `server_url: "${customUrl}"`,
       `auth_token: "${customToken}"`,
+      `public_key: "${pubKey}"`,
       `location: "${customLocation}"`,
       `capabilities:`,
       `  telemetry: ${telemetry}`,
@@ -1349,9 +2160,14 @@
     });
   }
 
+  // Enrollment state for the open install modal. The token exists only here,
+  // in memory, between the enroll call and the operator copying it: the server
+  // stores a hash and cannot show it again.
+  var pendingEnrollment = null;
+
   function openAgentInstallModal() {
     const currentOrigin = window.location.origin;
-    const initialToken = generateRandomHexToken(16);
+    pendingEnrollment = null;
 
     const bodyHtml = `
       

@@ -1364,6 +2180,111 @@
         

       

 
+      
    
+        
+        
+      

+
+      

+        
+        

+          

+            

+               Install with a join key
+            

+            

+              

+                Run this on any host and it enrolls itself. The SSO issues that host its own
+                token and public key on first connect, and the agent writes both into its
+                agent.yml — nothing to copy back and forth. One key works for as
+                many hosts as you like; each still gets its own revocable identity.
+              

+              

+                

+                  
+                  
+                  
A key's value is shown only when it is created — mint a new one if you don't have it saved.

+                

+                
+              

+              
+
+              
+              

+              

+                
+              

+            

+          

+
+          

+            

+               Manage join keys
+            

+            

+              
+                
+                  
+                    
+                    
+                    
+                    
+                    
+                    
+                  
+                
+                
+              
LabelPrefixCreatedHosts joinedStatusActions

+              
+            

+          

+        

+
+        
+        

+
+      

+        

+           1. Enroll this host
+        

+        

+          

+            Use this when you want the agent bound to a specific Directory host from the start.
+            The SSO issues the token here and you copy it onto the machine yourself.
+          

+          

+            

+              
+              
+            

+            

+              
+              
+              
Links the agent to a Directory host, so its status and metrics attach to that resource.

+            

+            

+              
+            

+          

+          
+        

+      

+
+      
             

-              
+              
               

-                
-                
+                
               

             

           

@@ -1415,12 +2333,9 @@
               
             

             

-              
+              
               

-                
-                
+                
               

             

             

@@ -1490,23 +2405,624 @@
           

         

       

+      

+        

+      

     `;
 
     app.modal.open({
-      title: ' Install Theta Agent',
+      title: 'Install Theta Agent',
       bodyHtml: bodyHtml,
       size: 'lg'
     });
 
+    loadAgentJoinKeys();
+
+    // Only hosts can carry an agent -- the API rejects anything else, so don't
+    // offer it here.
+    const $sel = $('#agent-enroll-resource').empty();
+    $sel.append('');
+    rawResources
+      .filter(r => r.kind === 'host')
+      .sort((a, b) => (a.name || '').localeCompare(b.name || ''))
+      .forEach(r => {
+        const taken = agentsByResource[r.id] ? ' — already has an agent' : '';
+        $sel.append($('');
+      } else {
+        $sel.append('');
+        agentJoinKeys.forEach(k => {
+          const used = k.use_count ? `${k.use_count} host${k.use_count === 1 ? '' : 's'}` : 'unused';
+          $sel.append($('