sso-manager-node/directory_spec.md

Home-Lab Directory / Inventory — Design Spec

Status: Implemented (v1.2.1+: model, admin API, UI; v1.3.x: automatic registration from theta-env + ldap-client). §9 adds the planned-consumer readiness review. Owner: wmantly Last updated: 2026-07-23

---

1. Overview & goals

The SSO app manages LDAP groups that gate access across the home lab:

• app_* — applications (Home Assistant, Gitea, Emby, …)
• host_* — machines (Proxmox nodes, LXC containers, VMs, bare metal)

A bootstrap script joins each Debian container to LDAP and grants SSH by group membership. Access control works — but the groups are bare. In LDAP a group (models/group_ldap.js) only carries cn, description, member, owner. There is:

• No metadata — no URL, IP, FQDN, port, icon, or notes per app/host.
• No relationships — nothing links a Proxmox host → the container running on it → the service the container provides → the LDAP group that gates access.

Consequences:

• Users can't see what they have access to or how to reach it. Today views/profile.ejs hardcodes a static "Services" list (Emby, Git, Proxmox…) with fixed URLs that are unrelated to the viewer's actual group membership.
• CI/CD and service discovery have no source of truth. Pipelines and tooling can't answer "which host runs service X", "what's the IP of host_ct101", or "which group grants access to this".

Goals

1. A machine-readable discovery API (build this first) that exposes apps/hosts, their metadata, and the infra graph — for CI/CD, dynamic inventory, and dashboards.
2. Answer, per user, "what can I access and how do I reach it" (later dashboard, powered by the same API).
3. Model the full graph: Proxmox host ← container/VM ← service/app.

Non-goals (v1)

• No write/admin UI yet (management comes in a later phase).
• No auto-provisioning/sync from Proxmox yet (manual population first).
• No change to how access is granted — LDAP remains the access-control truth.

---

2. Architecture

Two stores, one join key.

            ┌──────────────────────────────────────────────┐
            │                 SSO app (this repo)           │
            │                                               │
  Identity  │   LDAP  ──────────────►  access-control truth │
  & access  │   (users, app_*/host_* groups, membership)    │
            │        │                                      │
            │        │  join on group cn                    │
            │        ▼                                      │
  Inventory │   SQL  ──────────────►  metadata + graph      │
            │   (resources, edges, resource↔group links)    │
            │        │                                      │
            │        ▼                                      │
            │   /api/discovery/*  (read-first API)          │
            └──────────────────────────────────────────────┘

• LDAP = access truth. Who is in app_homeassistant / host_ct101 stays in LDAP, unchanged. Read via Group.list(user.dn) (models/group_ldap.js).
• SQL = inventory truth. Metadata and the host→container→service graph.
• Join key = group cn. A SQL resource references the LDAP group(s) that gate it by name (app_* / host_*). "Who can access resource X" is a join: LDAP membership ∩ resource_group.

Tech choice (to confirm)

• PostgreSQL recommended (JSONB for flexible metadata, real relational edges). SQLite is a lighter alternative acceptable for a single-node home lab.
• Thin query layer: node-postgres (pg) directly, or knex for migrations + query building. This is the app's first SQL dependency (today it uses LDAP + Redis via model-redis); the directory is a self-contained module and should not disturb the existing stores.

---

3. Data model (full graph)

Three tables. Metadata is a JSONB bag so fields can evolve without migrations; common query fields can be promoted to columns later.

resource — a node in the graph

column	type	notes
id	uuid / pk
kind	enum	site | host | service
name	text	display name ("Home Assistant", "ct101")
slug	text unique	url-safe id used by the API
description	text	free text
metadata	jsonb	{ subType, ip, macAddress, address, vmid, port, externalPort, gitRepo, installPath, systemdService, os, kernel, isProduction, isExternalReachable, isPublic }
created_at / updated_at	timestamptz

Parent Enforcement Rules:

• A Host MUST have a parent Site or Host.
• A Service MUST have a parent Host.
• An OAuth Integration MUST have a parent Service.

LDAP Group Auto-Creation: When a Host or Service is created, the system will automatically create two LDAP groups in the directory (if they do not already exist):

• <slug>_access (for standard user access)
• <slug>_admin (for administrative access) Additional groups can still be linked manually.

resource_edge — directed relationships (the graph)

column	type	notes
parent_id	fk → resource
child_id	fk → resource
relation	enum	runs_on | hosts | exposes | depends_on

Represents site←host←service (hosts/runs_on) and service→service (depends_on). Directed edges (not a single parent_id column) so a node can have multiple parents/children and multiple relation types.

resource_group — link a resource to the LDAP group(s) that gate it

column	type	notes
resource_id	fk → resource
group_cn	text	LDAP group name, app_* / host_*
access_level	enum	user | admin | owner

This is the bridge to auth. It lets one resource be gated by several groups (e.g. app_gitea for users, host_ct_gitea for shell/admin).

Naming: the app_* / host_* prefix convention is retained in LDAP; in SQL the distinction is captured explicitly by resource.kind + resource_group, so the API never has to parse group-name prefixes.

Example

resource: pve1 (proxmox_node)  ── hosts ──▶  ct101 (container)  ── exposes ──▶  gitea (service)
                                                                                   │
resource_group: gitea ↔ app_gitea (user), ct101 ↔ host_ct101 (user), pve1 ↔ host_pve1 (admin)

A user in app_gitea sees the Gitea service + how to reach it; a user in host_ct101 additionally sees SSH to the container; host_pve1 sees the node.

---

4. Discovery API (v1 — read-first)

JSON. Mounted at /api/discovery behind middleware.auth (see §5).

method & path	purpose
GET /api/discovery/resources?kind=&tag=&group=&parent=	Filtered list of nodes + metadata.
GET /api/discovery/resources/:slug	One node with its edges (parents + children).
GET /api/discovery/graph[?root=<slug>]	Whole graph, or the subtree under a root.
GET /api/discovery/me	Only the resources the caller is entitled to — LDAP membership (Group.list(user.dn)) ∩ resource_group. Powers the future dashboard and the "what can I access" question.

CI/CD-friendly output (optional formats, same data)

• Ansible dynamic inventory shape (?format=ansible): groups of hosts with ansible_host/vars pulled from metadata.
• Dashboard shape (Homepage/Glance): flat list of services with href, icon, description.

Write endpoints (POST/PUT/DELETE) are out of scope for v1; population is manual (SQL seed / migration) until the admin UI phase.

---

5. AuthN / AuthZ

• Interactive users: existing session auth — middleware.auth validating the auth-token header (an AuthToken, models/token.js). No change.
• CI/CD (machine) access: scripts and external integrations (like jump hosts) will use the existing ApiToken system (Personal Access Tokens) passed in the Authorization: Bearer sso_... header. The ApiToken inherits the exact LDAP group permissions of the user who created it, seamlessly mapping to existing access controls.
• Read visibility (decision to confirm): either (a) any authenticated user may read all resource metadata and only /me is filtered, or (b) list endpoints are themselves filtered to entitlement. Recommend (a) for a home lab — simpler, and infra metadata isn't secret — with /me as the personalized view.
• Management (later): gate write/admin endpoints behind a new app_sso_directory_admin LDAP group, mirroring the existing app_sso_oauth_admin pattern (routes/oauth_client.js + utils/permission.js).

---

6. Integration points with the existing app

• Replace the hardcoded Services list in views/profile.ejs (lines ~82–126) with a render of GET /api/discovery/me.
• Reuse the group selector app.ui.groupSelect (public/js/app.js) for linking resources ↔ LDAP groups in the future admin UI — no new group-picker needed.
• Join convention: resource_group.group_cn must equal an LDAP group cn exactly; the discovery layer never invents groups, it only references existing ones.
• Reuse utils/permission.byGroup for the admin gate in the management phase.

---

7. Roadmap

1. v1 — Discovery API (this spec's focus): SQL schema + migrations, read models, /api/discovery/* endpoints, ApiToken for CI/CD.
2. v2 — "My Access" dashboard: swap profile.ejs's static list for /me.
3. v3 — Admin CRUD UI: manage resources/edges/group links (reusing app.ui widgets and the oauth_clients.ejs card+modal pattern); gated by app_sso_directory_admin.
4. v4 — Sync from Proxmox (optional): auto-populate nodes/containers/VMs from the Proxmox API so inventory stays current without manual entry.

---

8. Open questions

1. DB engine: PostgreSQL (recommended) vs SQLite for a single-node lab.
2. Population: manual seed vs Proxmox pull for v1 (spec assumes manual).
3. Metadata mirroring: should any metadata be written back to the LDAP group description so LDAP-only external consumers see it? Default: no — keep LDAP for auth, SQL for inventory.
4. Read-visibility policy: confirm option (a) vs (b) in §5.
5. Service token scope: Currently ApiToken shares the creator's full permissions. A future enhancement could scope tokens specifically to the Directory API.

---

9. Planned consumers — data-model & API readiness

Five consumers the directory data should be able to power. None are being built yet; this section records what each needs, what already exists, and the gaps to close so the model/API never paints us into a corner.

The recurring theme: the graph model itself (Resource / ResourceEdge / ResourceGroup + LDAP groups) is sufficient for all five. The gaps are (a) one new model (access requests), (b) machine-to-machine auth for the read API, (c) documented metadata conventions instead of new columns, and (d) change detection for the drift/sync consumers.

9.1 End-user exploration ("Netflix-style" catalog + request access)

A user browses everything that exists — part advertisement, part documentation — sees what they already have, and requests access to the rest.

Already there:

• /api/discovery/me (getMyAccess) — the "My Services" half.
• Resource.owner + <slug>_access / <slug>_admin ResourceGroup links — who approves, and which group an approval means joining.
• The Notification model — the approval-request delivery mechanism.

Gaps:

1. Catalog projection with metadata privacy. /api/discovery/resources returns full metadata to any authenticated user — including the OAuth kind's client_secret_hash, and operator notes that may name internal IPs. Needed: a per-kind public projection (name, description, kind, subType, icon, address, hasAccess, requestable) and a private-key convention for the rest (e.g. only app_sso_directory_admin sees full metadata). This is a fix worth doing before any catalog UI exists.
2. AccessRequest model — the one genuinely new model: {id, uid, resourceId, groupCn, status: pending|approved|denied, note, requestedOn, decidedBy, decidedOn}. Approval = LDAP group add + notify. Endpoints: user POST/GET own; resource owner / directory admin list/approve/deny.
3. Catalog metadata conventions: icon, tagline (card-length blurb), requestable: false for resources that shouldn't be advertised.

9.2 SSH jump host (username_-_{hostname-or-ip}@publicHost)

A public jump host parses the target out of the SSH username, checks the user may reach that host, and proxies the connection (WinSCP-friendly: one username string, no interactive menu needed — though an interactive picker on plain username@ login is the same query).

Already there:

• Hosts carry ip (and host_<hostname> slugs to resolve by name).
• Access is already group-based (<slug>_access), checkable via LDAP alone — the jump host can run entirely off LDAP (SSSD) + one directory query.
• User SSH keys are in LDAP (openssh-lpk) — the jump host authenticates the real user without local accounts.

Gaps:

1. Machine auth for the access query. The jump host must ask "may user X reach host Y" / "list hosts user X may reach" about another user. getMyAccess only answers for the calling user. Needed: a service-token-authenticated endpoint (GET /api/discovery/access/:uid[/:slug]). ServiceToken already exists and is even linked to a resource (resource_id) — what's missing is an auth middleware that accepts it and a permission rule ("service tokens may read access info, scoped read-only").
2. Connection metadata conventions on hosts: sshPort (default 22), optional fqdn (when IP is dynamic), optional jumpVia edge relation if multi-hop topologies ever appear.
3. Document the username grammar ({uid}_-_{host-slug-or-ip}) here so the seed/ldap-client keep host slugs DNS-safe (they already are: slugify strips everything but [a-z0-9-]).

9.3 Firewall port-forward rules (build / update / drift-test)

An automation renders the public firewall's forwarding table from the directory, applies it, and alerts on drift in either direction.

Already there:

• metadata.port / metadata.externalPort / metadata.ip / metadata.isExternalReachable — the core mapping data, already seeded for the stack's own services.

Gaps:

1. Port-mapping convention is too thin for real rules: no protocol, no multi-port services. Adopt metadata.portMappings: [{proto: "tcp"|"udp", external: n, internal: n, comment}] as the authoritative form (port/externalPort stay as the simple single-mapping case).
2. Drift detection needs cheap change polling: an updated_on timestamp on resources surfaced in the graph API, or a graph-level etag/hash, so the runner can poll without diffing full payloads. (The ORM already publishes create/update events internally — a future push feed can ride that; polling comes first.)
3. Same service-token read auth as 9.2 — automation must not run on a human's session token.

9.4 Local DNS / mDNS

A DNS (or mDNS advertiser) zone is generated from the directory: hosts get A records from metadata.ip, services get CNAMEs/records from their addresses, sites map to zones.

Already there:

• host_<hostname> + ip covers A records; site_<name> is a natural zone boundary; service address yields names.

Gaps:

1. Name conventions: metadata.dnsNames: [] for extra aliases, and a documented rule for which name wins (slug vs address hostname). TTL only if someone actually needs per-record TTLs — default is fine.
2. Same change detection as 9.3 (poll updated_on / etag; push later).
3. Nothing else — this consumer is nearly free once 9.3's conventions land.

9.5 Access control for hosts

Who may log in to / sudo on which machine, driven by the directory.

Already there — this is the original point of the system:

• <slug>_access / <slug>_admin groups are auto-provisioned per host; ldap-client configures SSSD/PAM against the directory; sudoRole and openssh-lpk schemas cover sudo and SSH keys.

Gaps:

1. Close the loop in ldap-client: joined hosts should set an SSSD access filter (access_provider = ldap, filter on host_<hostname>_access membership) so directory group membership is login permission, not just identity. Today the registration exists but enforcement is host-side convention.
2. accessLevel granularity: ResourceGroup's member/owner maps to login/admin today; if finer roles emerge (e.g. login vs sudo vs admin), extend the enum — the join-table shape already supports it.

9.6 Consolidated work list (model/API only, no consumers)

Ordered by how much they unblock:

1. Metadata privacy projection on the read API (blocks 9.1; fixes the client_secret_hash exposure regardless of any consumer).
2. Service-token auth for /api/discovery/* + access/:uid endpoint (blocks 9.2, 9.3; ServiceToken model already exists).
3. AccessRequest model + endpoints (blocks 9.1's request half).
4. Metadata conventions doc entries (sshPort, portMappings, dnsNames, icon, tagline, requestable) in docs/directory.md — conventions, not schema changes; the json column already holds them.
5. updated_on in graph output / graph etag (blocks drift/DNS freshness; trivial once surfaced).

---

10. Subtype Management & Metrics Drivers Architecture

The Directory incorporates a 4-tier Driver Resolution Engine (services/driver_registry.js) that binds resource subType metadata to specific telemetry, log streaming, and operational management protocols.

Subtype Matrix & Drivers

Subtype Category	Supported Subtypes	Primary Driver	Management Capabilities	Telemetry & Metrics
Service Managers	systemd, openrc, windows_service	ThetaAgentDriver / Systemd	start, stop, restart, reload	CPU, Memory, Active PID, SubState
Containers & Stacks	docker, docker_compose	DockerSocketDriver / Agent	start, stop, restart, pause	CPU %, Memory Limit/Usage, Net/Block I/O
Virtualization & Hypervisors	proxmox, lxc, kvm, esxi, libvirt_kvm, vps_generic	ProxmoxDriver / Hypervisor	start, stop, shutdown, reboot	Guest VMID CPU/RAM/Disk, Parent Hypervisor status
Networking & Appliances	wireguard, unifi_ap, unifi_switch, pfsense	NetworkDriver	restart, locate, sync	Connected Clients, Handshakes, Gateway RTT, Channels
Databases & Vaults	postgresql, redis, openbao_vault	DbDriver	flush, seal, unseal	DB Size, Connections, Hit Rates, Active Leases
Orchestration	k8s_pod, k8s_deployment	K8sDriver	scale, restart, rollout_restart	Desired/Ready Replicas, Pod Phase, IP
Workstations	desktop_linux, desktop_windows	ThetaAgentDriver	reboot, shutdown, Display Manager	CPU, Memory, GPU, Active Sessions

(Note: Reverse Proxy subtypes like Nginx/HAProxy/Caddy/Traefik are excluded per environment configuration).

4-Tier Driver Resolution Engine

1. Direct Agent Execution: If theta-agent is connected directly to the target resource.
2. Subtype-Specific Driver: Executes specialized protocol driver (e.g. Proxmox API, Docker Engine API, DB Driver).
3. Ancestor / Hypervisor Fallback: If an LXC/KVM guest lacks a direct agent, queries its parent Proxmox hypervisor node for metrics and power controls.
4. Unmanaged Fallback: Reports unmanaged status cleanly without breaking UI/API contracts.

Subtype Operations Endpoints

• GET /api/directory-admin/resources/:id/driver-metrics — Real-time telemetry payload
• POST /api/directory-admin/resources/:id/driver-action — Execute management action ({ action, params })
• GET /api/directory-admin/resources/:id/driver-logs — Tail log output (?lines=100)

11. Multi-Site

This directory can run across multiple sites (one master with write authority, any number of spoke read-only replicas that stay live-synced after joining), coordinate master promotion, and share the agent-signing key across sites. Full design and operational detail: docs/site-join.md and, at the suite level, theta-suite's docs/MULTI_SITE_SPEC.md.