Compare commits

...

31 Commits

Author SHA1 Message Date
wmantly 7efd3fd6bd Merge pull request #210 from theta42/release-v2.8.0
release(v2.8.0): promotion/LDAP-orphan fix, site-slug unification, LDAP status UI
2026-08-10 20:59:31 -07:00
wmantly 9f285c960b release(v2.8.0): promotion/LDAP-orphan fix, site-slug unification, LDAP status UI 2026-08-10 23:57:04 -04:00
wmantly d75daf81b7 Merge pull request #209 from theta42/feat-ldap-status-ui
feat(directory): LDAP replication status + per-spoke detail on the Multi-Site modal
2026-08-10 20:56:32 -07:00
wmantly 6861a113d2 fix(directory): unify the Directory's site slug with the multi-site replication identity
Two previously-unrelated "site slug" concepts existed: the Directory
catalog's site Resource (site.slug, what group names and the resource
tree actually use -- e.g. "E2E Site" / site_e2e) vs. site_config.js's
siteSlug (the multi-site replication identity shown on the Multi-Site
modal's "Local Site Slug" row, sourced only from a separately-set
SITE_SLUG env var). Nothing ever kept them in sync -- a real
deployment could show a real site name in the Directory tree and the
literal "site-default" fallback on the Multi-Site modal for the exact
same node, which is exactly what a live demo of the modal surfaced.

Synced at the source: POST /resources now sets site_config's siteSlug
to match, the moment this node's own site Resource is first created
(bootstrap.js's initial call). Only for a still-default master --
never overwrites a real multi-site identity a join/promote has
already established, and never touches a spoke's identity (the
master's to assign via registration, not this node's own resource
creation to decide).

Verified against a real running container via
docker-compose.multisite-e2e.yml's existing site-creation step.
2026-08-10 23:54:05 -04:00
wmantly 4542c055bb feat(directory): LDAP replication status + per-spoke detail on the Multi-Site modal
The modal previously showed zero LDAP replication status -- no
ServerID, no MMR active/inactive indicator, nothing -- and only
aggregate counts, never per-spoke detail (no-inbound flag, relay
note, assigned ldapServerId). An operator had no way to tell whether
replication was actually configured/working without SSHing in.

Added utils/ldap_replication.js's currentSlapdServerId(), which reads
the ACTUAL running ServerID straight from this node's own slapd.conf
-- distinct from what GET /ldap-peers / /ldap-replication-config
currently ADVERTISE for it, which can genuinely disagree right after a
promotion or a new spoke joining (OpenLDAP's static config only
reloads at process start). GET /directory-admin/site-status now
surfaces both plus a `stale` flag, and a full spokes list (not just a
count) with each one's endpoint, LDAP ServerID, and relay path.

directory.ejs renders this as an "LDAP Replication (MMR)" status row
(ServerID, peer count, a "needs setup.sh re-run" warning when stale)
and a Registered Spokes table.

Verified two ways: real running containers via
docker-compose.multisite-e2e.yml (new site-status assertions), and an
actual browser session against the promoted node -- screenshotted the
rendered modal showing the real registered spoke with its assigned
ldapServerId and the correctly-surfaced "not configured (standalone)"
MMR state (this test node never ran site-ldap-register.js against it,
so the mismatch itself is the expected, documented behavior).
2026-08-10 23:48:20 -04:00
wmantly bd2205f1a9 Merge pull request #208 from theta42/docs-site-join-staleness
docs(site-join): fix stale claims about relay/mesh routing/LDAP
2026-08-10 20:31:52 -07:00
wmantly d5e0d61546 docs(site-join): fix stale "not yet built" claims about relay/mesh routing/LDAP
Three items listed as unbuilt had actually shipped: cross-component
routing over the mesh, no-inbound relay automation, and (as of this
session) OpenLDAP MMR auto-config. Replaced with what's actually true
today, plus the two known LDAP-replication caveats around promotion
(covered in the code comments added alongside the fix for the
promotion/demote SiteSpoke-orphaning bug).
2026-08-10 23:27:45 -04:00
wmantly 0dfb69cc9a Merge pull request #207 from theta42/fix-promotion-ldap-orphan
fix(multi-site): promotion no longer orphans the demoted old master's LDAP replication
2026-08-10 20:24:29 -07:00
wmantly dae0361e82 fix(multi-site): promotion no longer orphans the demoted old master's LDAP replication
Found while auditing the new LDAP MMR auto-config for gaps: neither
POST /site-promote nor POST /demote ever touched SiteSpoke. Two real
problems:

1. The demoted old master got a fresh masterJoinKey but was never
   registered as a spoke of the new master -- no SiteSpoke row, no
   ldapServerId, invisible to GET /ldap-peers's peer list. It also
   structurally could not self-heal: POST /join refuses re-join for a
   node that's already a spoke, and separately requires a fresh
   install (siteIsFresh()) -- neither true for a former master with
   real users/agents. Fixed: /demote now registers itself with the new
   master immediately (POST /spokes), the same way a real join does,
   deriving its own endpoint from stack.selfUrl (override) or
   https://stack.ssoHost (the normal case).

2. The promoted node's live OpenLDAP ServerID doesn't change --
   GET /ldap-replication-config starts advertising 1 for it
   immediately (derived purely from cfg.isMaster), but nothing
   restarts slapd with that value (OpenLDAP's static slapd.conf only
   reloads at process start, and this app has no safe way to restart
   its own container). Can't be fixed in-process; surfaced instead --
   /site-promote's response now includes ldapReplicationNote telling
   the operator to re-run setup.sh promptly.

Verified against real running containers (docker-compose.multisite-e2e.yml):
after promotion, the demoted old master correctly appears in the new
master's LDAP peer list with a real assigned ldapServerId.
2026-08-10 23:21:59 -04:00
wmantly eef7852b69 Merge pull request #206 from theta42/release-v2.7.0
release(v2.7.0): auto-assigned LDAP ServerID + replication hosts
2026-08-10 20:07:42 -07:00
wmantly 0ac0c045ec release(v2.7.0): auto-assigned LDAP ServerID + replication hosts 2026-08-10 23:03:33 -04:00
wmantly dfb819a715 Merge pull request #205 from theta42/feat-ldap-mmr-auto-config
feat(multi-site): auto-assign LDAP ServerID + replication hosts at join time
2026-08-10 19:53:42 -07:00
wmantly d486fb946b feat(multi-site): auto-assign LDAP ServerID + replication hosts at join time
OpenLDAP N-way multi-master replication (docs/replication.md) required
an operator to hand-set LDAP_SERVER_ID (unique per site) and
LDAP_REPLICATION_HOSTS (every OTHER site's LDAP URL, kept in sync by
hand across every node) -- real coordination work, and easy to get
wrong or let drift as sites are added.

Automates the coordination the master is already in a position to do:
- SiteSpoke gets ldapServerId, auto-assigned (next free from 2 upward,
  1 reserved for the master) at registration and reused across
  re-registrations -- same pattern as jump-host's mesh index.
- ldapHost is derived from each site's already-known HTTP(S) endpoint
  (same hostname, port 636) rather than a separately-configured field
  that could drift from it.
- New utils/ldap_replication.js (nextFreeLdapServerId, ldapHostFor),
  shared between the spoke-facing GET /api/site/ldap-peers (Bearer
  site join key, returns this caller's own ID + every peer) and the
  master-local GET /directory-admin/ldap-replication-config (computes
  its own config directly from SiteSpoke, no HTTP round-trip needed).

Verified against real running containers (docker-compose.multisite-e2e.yml):
after a real join, the master's computed config correctly includes the
spoke as a peer with an assigned ID, and the spoke's own fetched
config matches that ID and correctly excludes itself from its own
peer list.

Known limitation, documented in docs/replication.md: the master's own
LDAP_REPLICATION_HOSTS only gets recomputed when ITS setup.sh is
re-run (or an admin re-applies it directly) -- there's no live push to
an already-running master when a new spoke joins. A spoke's own config
is re-checked on every setup.sh run, which is the common/recurring
event; the master side is a documented manual step for now rather than
a live hot-reload (which would need OpenLDAP's dynamic cn=config
backend -- a bigger change, deliberately out of scope here to avoid
risking a live directory's LDAP replication on undertested config).
2026-08-10 22:51:04 -04:00
wmantly 39db290265 Merge pull request #204 from theta42/release-v2.6.0
release(v2.6.0): dedupe promotion groups, resources API perf, nmap RTTVAR fix, real gateway count
2026-08-10 19:37:04 -07:00
wmantly 964ca6dd02 release(v2.6.0): dedupe promotion groups, resources API perf, nmap RTTVAR fix, real gateway count 2026-08-10 22:34:26 -04:00
wmantly 4e388a411e Merge pull request #203 from theta42/feat-real-gateway-count-and-service-wiring
feat(directory): real mesh-gateway count on the Multi-Site modal
2026-08-10 19:29:37 -07:00
wmantly e9310a1e5c Merge remote-tracking branch 'origin/master' into feat-real-gateway-count-and-service-wiring
# Conflicts:
#	nodejs/package.json
2026-08-10 22:27:22 -04:00
wmantly f70419bbc4 Merge pull request #202 from theta42/fix-nmap-rttvar-false-failure
fix(discovery): recover nmap scans that succeed despite a benign RTTVAR stderr warning
2026-08-10 19:26:36 -07:00
wmantly 358231532f Merge remote-tracking branch 'origin/master' into feat-real-gateway-count-and-service-wiring
# Conflicts:
#	nodejs/package.json
2026-08-10 22:24:55 -04:00
wmantly d8bd338814 Merge remote-tracking branch 'origin/master' into fix-nmap-rttvar-false-failure
# Conflicts:
#	nodejs/package.json
2026-08-10 22:24:20 -04:00
wmantly 0a3bdbef06 Merge pull request #201 from theta42/fix-group-dup-and-resources-perf
fix(directory): dedupe access/admin groups; stop self-healing groups on every GET
2026-08-10 19:23:17 -07:00
wmantly 611f1a3318 feat(directory): real mesh-gateway count on the Multi-Site modal; docs links
"Theta Gateways: N active gateways" was counting this app's own
unrelated WireGuard roaming-client/exit-node Resources
(metadata.subType === 'wireguard') -- a completely different subsystem
from the gateway-to-gateway mesh the modal is actually about, and it
never queried jump-host's mesh registry at all (so it couldn't show
the local self-entry either, since there was nothing mesh-related
being counted in the first place).

Added utils/jump_client.js (same pattern as utils/proxy_client.js:
reuses jump-host's existing self-service jmp_ API token system rather
than inventing a new credential) to query jump-host's real
GET /api/mesh/gateways. Reports a null count (not misleading 0) when
the integration isn't configured/reachable, surfaced distinctly in the
UI. Also added help links to the published multi-site/mesh docs on the
modal.

Includes docs links + count only -- this session also discovered that
utils/proxy_client.js's PROXY_INTERNAL_URL, and now JUMP_INTERNAL_URL,
were never actually wired into theta-suite's docker-compose.yml, so
both service-to-service integrations were unreachable in every real
deployment despite existing in code (fixed in theta-suite separately).
2026-08-10 22:17:57 -04:00
wmantly e313697bfd fix(discovery): recover an nmap scan that succeeded despite a benign stderr warning
node-nmap (the vendored library, not our code) treats ANY stderr
output from the nmap binary as a fatal scan error -- including nmap's
own harmless RTT-calibration warning ("RTTVAR has grown to over N
seconds, decreasing to M"), which it prints *during* a scan that goes
on to complete normally. That meant a real, successful scan (valid XML
already sitting in the library's rawData) got discarded and reported
as a failed run with zero hosts discovered -- not just log noise as
initially assumed.

Recover in our own plugin code by detecting this specific known-benign
message and manually re-running node-nmap's own XML-parse-then-complete
path when there's actually output to parse. A genuine parse failure or
any other error message still rejects exactly as before -- this only
widens the recovery path.
2026-08-10 22:14:13 -04:00
wmantly 2c3ec4e967 fix(directory): dedupe access/admin groups on repeated promotion; stop self-healing on every GET
Three independent copies of the same bug: routes/discovery.js's
POST /discovery/promote/:slug (the actual "Promote" button in the UI)
and services/discovery_reconciler.js's autoPromote path both called
ResourceGroup.create() directly with no existence check -- unlike
routes/api_directory_admin.js's own ensureResourceGroup, which already
carried a comment describing this exact "groups appear 3x" bug and
fixing it, just not everywhere it occurred. ResourceGroup has no DB
unique constraint on (resourceId, groupCn), so a resource promoted
more than once (retried UI click, or the same LXC discovered from
multiple Proxmox cluster nodes) silently accumulated duplicate
access/admin rows every time. Added ResourceGroup.ensure() (the
existing check-then-create pattern, now on the model) and switched all
three call sites to it. New regression test in tests/reconciler.test.js.

Also: GET /api/directory-admin/resources ran a full group-model
self-heal fan-out (ensureSiteGroups per site + provisionResourceGroups
per resource, each several sequential LDAP round-trips) unconditionally
on every single list -- confirmed via code read as the actual
bottleneck once a directory has more than a handful of resources, not
data volume. Moved healing to where resources actually change instead
(POST/PUT /resources, POST /discovery/promote/:slug -- PUT had none at
all before this), and added POST /resources/heal-groups as an explicit
on-demand equivalent for backfilling a directory seeded before this
change.
2026-08-10 22:08:04 -04:00
wmantly b6a82d58d5 Merge pull request #200 from theta42/release-v2.5.0
release(v2.5.0): no-inbound relay automation, mesh-preferred replication
2026-08-10 18:28:27 -07:00
wmantly 18da6582ed release(v2.5.0): no-inbound relay automation, mesh-preferred replication 2026-08-10 21:25:38 -04:00
wmantly b1739ec965 Merge pull request #199 from theta42/feat-mesh-routing
feat(multi-site): route replication over the mesh when available
2026-08-10 18:09:28 -07:00
wmantly 7bc6f47070 feat(multi-site): forward noInbound/meshIp/publicHost through /api/site/join
/api/site/spokes already accepted these and drove proxy_client's relay
automation, but nothing on the real operator join path ever passed
them through -- a no-inbound spoke calling /join had no way to trigger
its own relay route. Forward them into the internal /spokes call and
surface the resulting relay note in /join's response.
2026-08-10 20:52:35 -04:00
wmantly a0964ce350 feat(multi-site): route replication traffic over the mesh when available
Cross-component routing TODO item: a spoke's resync push now prefers
its WG mesh IP (reported via the noInbound/meshIp fields added for the
relay automation) over the public endpoint, falling back to the public
endpoint if the mesh attempt fails for any reason (tunnel not actually
up between these two particular gateways yet, transient failure,
etc.) -- a mesh-routing preference must never turn into "spoke stops
getting updates."

Plain HTTP over the mesh IP, not HTTPS: the WG tunnel is already
encrypted, same reasoning already applied to the no-inbound relay
terminating at the master.

A spoke with no meshIp on file behaves exactly as before (public
endpoint only) -- this is additive, not a behavior change for spokes
that haven't opted into mesh registration.
2026-08-10 20:39:27 -04:00
wmantly 0d3ee3e2ee Merge pull request #198 from theta42/feat-no-inbound-relay
feat(multi-site): no-inbound relay automation via theta-proxy's existing API tokens
2026-08-10 17:36:03 -07:00
wmantly b95cb08c41 feat(multi-site): no-inbound relay automation via theta-proxy's existing API tokens
Implements TODO items "service-to-service auth model" and "no-inbound
relay automation" together -- the first was never really "no credential
type exists", it was that nothing wired one of the credential types that
ALREADY exist (theta-proxy, jump-host, and this app each already have
their own self-service API token system, models/api_token.js) into an
actual inter-service call. This is that wiring, not a new invented
credential type.

- utils/proxy_client.js: ensureRelayRoute({host, ip, targetPort}) calls
  theta-proxy's real Host API (GET/POST/PUT /api/host) using a `prx_...`
  token an operator mints on theta-proxy and stores in OpenBao
  (secret/integrations/theta-proxy), same pattern as agent_keys.js.
  Idempotent and best-effort -- never fails the caller if the token/URL
  isn't configured, since this is an enhancement on top of a working
  join, not a join requirement.
- POST /api/site/spokes accepts optional noInbound/meshIp/publicHost
  fields; when a spoke reports itself no-inbound, the master
  best-effort creates/updates the matching relay route automatically.
  SiteSpoke gained the fields + a relayNote for visibility.

Verified against a REAL running theta-proxy container (not mocked):
booted it standalone, logged in as the local admin, minted a real
`prx_` token via its actual API, and drove ensureRelayRoute() against
it for real. Caught a real bug doing this: GET /api/host/:item wraps
the record in `{item, results}`, not flat -- the mocked unit tests
(which I wrote first) all had the flat shape baked in and passed
cleanly, so this only surfaced against the real API. Fixed in both the
implementation and the unit tests' mocked response shape.
2026-08-10 20:31:42 -04:00
24 changed files with 1178 additions and 101 deletions
+29
View File
@@ -1,3 +1,32 @@
# v2.8.0 - 2026-08-11
### Fixed
- **Promotion no longer orphans the demoted old master's LDAP replication.** Neither `/site-promote` nor `/demote` touched `SiteSpoke` -- the demoted old master got a fresh join key but no `SiteSpoke` entry on the new master (no `ldapServerId`, invisible to the peer list), and structurally could never self-heal via `/join` (refuses re-join for a node that's already a spoke). `/demote` now registers itself with the new master immediately, deriving its own endpoint from `stack.selfUrl`/`stack.ssoHost`. `/site-promote`'s response also now surfaces that the promoted node's own OpenLDAP ServerID needs a `setup.sh` re-run to actually apply.
- **The Directory's site slug and the multi-site replication identity are unified.** These were two unrelated values that happened to share a name -- a deployment could show a real site name in the Directory catalog and the literal `site-default` fallback on the Multi-Site modal for the same node. `POST /resources` now syncs `site_config`'s `siteSlug` to match the moment this node's own site Resource is first created, for a still-default master only.
### Added
- **LDAP replication status + per-spoke detail on the Multi-Site modal.** New `utils/ldap_replication.js`'s `currentSlapdServerId()` reads the actual running ServerID from this node's own `slapd.conf` -- distinct from what the API currently advertises, which can genuinely disagree right after a promotion or a new spoke joining. `GET /directory-admin/site-status` now surfaces both plus a `stale` flag and a full spokes list (endpoint, assigned `ldapServerId`, relay path), not just an aggregate count.
# v2.7.0 - 2026-08-11
### Added
- **OpenLDAP N-way multi-master replication auto-config.** `SiteSpoke.ldapServerId` is now auto-assigned at registration (next free from 2 upward, 1 reserved for the master -- same pattern as jump-host's mesh index), and each site's LDAP URL is derived from its already-known HTTP(S) endpoint rather than a separately-configured field. New `utils/ldap_replication.js`, `GET /api/site/ldap-peers` (spoke-facing, Bearer site join key) and `GET /directory-admin/ldap-replication-config` (master-local). Operators no longer hand-maintain `LDAP_SERVER_ID`/`LDAP_REPLICATION_HOSTS` for a `theta-suite`-joined cluster (see `theta-suite`'s `bootstrap/site-ldap-register.js`). Verified against real running containers (`docker-compose.multisite-e2e.yml`).
# v2.6.0 - 2026-08-11
### Fixed
- **Duplicate access/admin groups on repeated resource promotion.** Three independent copies of the same bug (`routes/discovery.js`'s `POST /discovery/promote/:slug` -- the actual "Promote" button in the UI -- and `services/discovery_reconciler.js`'s `autoPromote` path both called `ResourceGroup.create()` directly with no existence check, unlike `routes/api_directory_admin.js`'s own `ensureResourceGroup`, which already carried a comment describing this exact bug). A resource promoted more than once (a retried click, or the same LXC discovered from multiple Proxmox cluster nodes) silently accumulated duplicate rows every time. Consolidated into `ResourceGroup.ensure()` on the model, used everywhere.
- **`GET /api/directory-admin/resources` ran a full LDAP group self-heal fan-out on every single list** (`ensureSiteGroups` per site + `provisionResourceGroups` per resource, each several sequential LDAP round-trips), unconditionally -- confirmed as the actual bottleneck once a directory has more than a handful of resources, not data volume. Moved healing to where resources actually change instead (`POST`/`PUT /resources`, `POST /discovery/promote/:slug` -- `PUT` had none at all before this), and added `POST /resources/heal-groups` as an explicit on-demand equivalent for backfilling a directory seeded before this change.
- **An nmap discovery scan that completed successfully could be reported as a failed run with zero hosts found.** `node-nmap` (the vendored library) treats any stderr output from the nmap binary as fatal -- including nmap's own harmless RTT-calibration warning ("RTTVAR has grown to over N seconds..."), which it prints *during* a scan that goes on to complete normally, discarding valid results already sitting in the library's `rawData`. Our plugin now recognizes this specific benign message and manually completes the scan from the data that's already there; any other error still rejects as before.
- **The Multi-Site modal's "Theta Gateways" count was measuring the wrong subsystem.** It counted this app's own unrelated WireGuard roaming-client/exit-node Resources, not jump-host's actual gateway-to-gateway mesh registry. New `utils/jump_client.js` (same self-service-token pattern as `utils/proxy_client.js`) queries jump-host's real `GET /api/mesh/gateways`, reporting a distinct "unknown" state instead of a misleading 0 when the integration isn't configured. Also added help links to the published multi-site/mesh docs on the modal.
# v2.5.0 - 2026-08-10
### Added
- **No-inbound relay automation.** A spoke with no public IP of its own can now register as such (`noInbound`/`meshIp`/`publicHost` on `POST /api/site/spokes`, forwarded through `POST /api/site/join` for the real operator join flow), and the master auto-creates/updates the relay route on its own `theta-proxy` via `utils/proxy_client.js` — a new self-service `prx_...` API token client, reusing `theta-proxy`'s existing token system rather than inventing a new credential type. Verified against a real running `theta-proxy` container (`GET /api/host/:item`'s actual `{item, results: {...}}` response shape, not the flat shape first assumed).
- **Replication traffic prefers the mesh.** `utils/site_replicate.js`'s fire-and-forget resync push now tries a registered spoke's `meshIp` first (falling back to its public `endpoint` on failure) — cross-component routing over the gateway-to-gateway WireGuard mesh instead of the open internet, for any spoke that's registered one.
- `POST /api/site/join` surfaces the resulting relay status in its response (`relay.note`), and `theta-suite`'s bootstrap flow (`CFG_SPOKE_NO_INBOUND`/`CFG_SPOKE_PUBLIC_HOST`, `bootstrap/site-relay-register.js`) drives all of this from the real operator-facing setup script, not just the API.
# v2.4.0 - 2026-08-10 # v2.4.0 - 2026-08-10
### Added ### Added
+6
View File
@@ -25,6 +25,11 @@ services:
- LDAP_ADMIN_PASS=secret - LDAP_ADMIN_PASS=secret
- ORG_NAME=E2E Master - ORG_NAME=E2E Master
- app_oauth__jwtSecret=e2e-multisite-master-jwt-secret - app_oauth__jwtSecret=e2e-multisite-master-jwt-secret
# POST /demote's self-registration (routes/api_site.js) needs a real
# reachable endpoint for this container; stack.selfUrl overrides the
# normal https://<stack.ssoHost> derivation, which isn't reachable
# here (plain HTTP, no TLS/proxy in front, non-443 port).
- app_stack__selfUrl=http://master:3001
healthcheck: healthcheck:
test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"] test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"]
interval: 2s interval: 2s
@@ -42,6 +47,7 @@ services:
- LDAP_ADMIN_PASS=secret - LDAP_ADMIN_PASS=secret
- ORG_NAME=E2E Spoke - ORG_NAME=E2E Spoke
- app_oauth__jwtSecret=e2e-multisite-spoke-jwt-secret - app_oauth__jwtSecret=e2e-multisite-spoke-jwt-secret
- app_stack__selfUrl=http://spoke:3001
healthcheck: healthcheck:
test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"] test: ["CMD-SHELL", "wget -qO- http://localhost:3001/health >/dev/null 2>&1"]
interval: 2s interval: 2s
+27 -8
View File
@@ -23,32 +23,51 @@ In an N-Way Multi-Master setup, every site runs a fully active OpenLDAP server (
## Configuration ## Configuration
To enable replication, you must pass two environment variables to the `sso-manager` container: The container's entrypoint reads two environment variables to configure this
-- `LDAP_SERVER_ID` (a unique integer for this node) and
`LDAP_REPLICATION_HOSTS` (a space-separated list of every **other** node's
LDAP URL) -- and, when both are set, automatically loads the `syncprov`
module, enables `mirrormode`, and generates the necessary `syncrepl` blocks
in `/etc/openldap/slapd.conf`.
1. `LDAP_SERVER_ID`: A unique integer for this node (e.g., `1`, `2`, `3`). This MUST be unique across the cluster. **If you're using `theta-suite`'s `setup.sh`, you don't set these by hand.**
2. `LDAP_REPLICATION_HOSTS`: A space-separated list of the LDAP URLs of all **other** nodes in the cluster. The master assigns each spoke a unique `LDAP_SERVER_ID` at join time (the
same way it assigns a WireGuard mesh index), and `LDAP_REPLICATION_HOSTS` is
derived automatically from every site's already-known HTTPS endpoint
(`ldaps://<same-host>:636`) -- see `GET /api/site/ldap-peers` (spoke) and
`GET /api/directory-admin/ldap-replication-config` (master), and
`theta-suite`'s `bootstrap/site-ldap-register.js`, which re-checks on every
`setup.sh` run since the peer list changes as new spokes join.
### Example using `theta-env` / Docker Compose Setting the two env vars directly still works (e.g. a non-`theta-suite`
deployment) -- example using three manually-configured nodes:
**Site 1 (`setup.env` or `docker-compose.yml`)** **Site 1**
```env ```env
LDAP_SERVER_ID=1 LDAP_SERVER_ID=1
LDAP_REPLICATION_HOSTS="ldaps://sso.site2.com:636 ldaps://sso.site3.com:636" LDAP_REPLICATION_HOSTS="ldaps://sso.site2.com:636 ldaps://sso.site3.com:636"
``` ```
**Site 2 (`setup.env` or `docker-compose.yml`)** **Site 2**
```env ```env
LDAP_SERVER_ID=2 LDAP_SERVER_ID=2
LDAP_REPLICATION_HOSTS="ldaps://sso.site1.com:636 ldaps://sso.site3.com:636" LDAP_REPLICATION_HOSTS="ldaps://sso.site1.com:636 ldaps://sso.site3.com:636"
``` ```
**Site 3 (`setup.env` or `docker-compose.yml`)** **Site 3**
```env ```env
LDAP_SERVER_ID=3 LDAP_SERVER_ID=3
LDAP_REPLICATION_HOSTS="ldaps://sso.site1.com:636 ldaps://sso.site2.com:636" LDAP_REPLICATION_HOSTS="ldaps://sso.site1.com:636 ldaps://sso.site2.com:636"
``` ```
Once configured, the container's entrypoint will automatically load the `syncprov` module, enable `mirrormode`, and generate the necessary `syncrepl` blocks in `/etc/openldap/slapd.conf`. **A known limitation of the automatic path**: the *master's* own
`LDAP_REPLICATION_HOSTS` only gets recomputed when its `setup.sh` is
re-run (or the operator re-applies it directly) -- there's no live push
telling the master's already-running container about a spoke that joined
five minutes ago. A spoke's own config, by contrast, is re-checked and
applied on every `setup.sh` run there, which is the common/recurring event.
Re-run `setup.sh` on the master after bringing up a new spoke to pick up the
new peer and restart replication with it.
## User Locations ## User Locations
+28 -6
View File
@@ -135,11 +135,33 @@ already joined reports "already a spoke" and setup continues (idempotent).
## Not yet built ## Not yet built
- Traffic between sites (join/export/resync) still goes over the open
network path that already reaches the target — it does not route over the
WireGuard mesh `theta-gateway` can now establish (see `MULTI_SITE_SPEC.md`).
- A no-inbound spoke (no public IP at all) still can't join — the mechanism
for a master to relay through the mesh to such a spoke is verified as
working, but nothing automates creating that route yet.
- OpenBao secret replication covers only the agent-signing key; LDAP admin - OpenBao secret replication covers only the agent-signing key; LDAP admin
creds, JWT secret, and other per-deployment secrets aren't synced. creds, JWT secret, and other per-deployment secrets aren't synced.
- A promoted spoke's own OpenLDAP `ServerID` doesn't apply live -- `POST
/site-promote` starts advertising `1` for it immediately
(`GET /directory-admin/ldap-replication-config`), but nothing restarts
`slapd` with that value automatically (its static `slapd.conf` is only
read at process start). Re-run `setup.sh` on the newly-promoted node
promptly after promotion to actually apply it.
- The master's own `LDAP_REPLICATION_HOSTS` peer list only recomputes on
its next `setup.sh` run, not live the instant a new spoke joins -- same
re-run-`setup.sh` caveat as above, just triggered by a join instead of a
promotion.
## Shipped since the above was last stale
- Traffic between sites (`utils/site_replicate.js`'s resync push) prefers a
registered spoke's WireGuard mesh IP over the open internet when one's on
file, falling back to the public endpoint on failure.
- A no-inbound spoke (no public IP at all) CAN join: `noInbound`/`meshIp`/
`publicHost` on `POST /api/site/join` drive `utils/proxy_client.js`, which
auto-creates/updates the relay route on the master's own `theta-proxy`.
Mesh peering between the two jump-hosts is still a manual, one-time step
(see `theta-suite`'s `spoke.env.example` for the operator-facing side).
A spoke with zero inbound *and* zero outbound path still can't join --
the join itself needs to reach the master's API directly.
- OpenLDAP N-way multi-master replication now auto-configures on join --
the master assigns each spoke a unique `LDAP_SERVER_ID` and derives every
site's `ldaps://` URL automatically (`GET /api/site/ldap-peers`,
`GET /directory-admin/ldap-replication-config`). See
`docs/replication.md`.
+13
View File
@@ -1,3 +1,4 @@
const crypto = require('crypto');
const { Model } = require('@simpleworkjs/orm'); const { Model } = require('@simpleworkjs/orm');
const { Group } = require('./group_ldap'); const { Group } = require('./group_ldap');
@@ -228,6 +229,18 @@ class ResourceGroup extends Model {
groupCn: { type: 'string', isRequired: true }, groupCn: { type: 'string', isRequired: true },
accessLevel: { type: 'string', isRequired: true } accessLevel: { type: 'string', isRequired: true }
}; };
// No DB-level unique constraint on (resourceId, groupCn) exists, so callers
// MUST check-then-create rather than relying on a constraint violation to
// catch a dupe. A caller that skips this (raw ResourceGroup.create()) and
// runs more than once for the same resource -- e.g. discovery reconciling
// the same LXC from multiple Proxmox cluster nodes -- silently accumulates
// duplicate access/admin rows every pass, with no error to notice it by.
static async ensure(resourceId, groupCn, accessLevel) {
const existing = await this.list({ where: { resourceId, groupCn } });
if (existing.length) return existing[0];
return this.create({ id: crypto.randomUUID(), resourceId, groupCn, accessLevel });
}
} }
module.exports = { module.exports = {
+19 -1
View File
@@ -29,7 +29,25 @@ class SiteSpoke extends Model {
siteSlug: { type: 'string' }, siteSlug: { type: 'string' },
pushToken: { type: 'string', isRequired: true }, pushToken: { type: 'string', isRequired: true },
created_on: { type: 'integer' }, created_on: { type: 'integer' },
last_seen_on: { type: 'integer' } last_seen_on: { type: 'integer' },
// No-inbound relay (MULTI_SITE_SPEC.md): a spoke with no public IP of
// its own reports its WG mesh IP + the public hostname it wants
// reached at; the master then best-effort creates a matching relay
// route on its own theta-proxy (utils/proxy_client.js). relayNote
// records what happened for visibility in the UI -- this automation
// is optional/best-effort, never a join requirement.
noInbound: { type: 'boolean', default: false },
meshIp: { type: 'string' },
publicHost: { type: 'string' },
relayNote: { type: 'string' },
// OpenLDAP multi-master replication (docs/replication.md): a unique
// small integer this spoke's slapd.conf ServerID must use. Assigned
// once at registration (see api_site.js's nextFreeLdapServerId),
// reused on re-registration -- a spoke that re-registers after a
// restart must not get bumped to a new ID, same reasoning as
// jump-host's meshIndex. The master reserves 1 for itself, never
// assigned here.
ldapServerId: { type: 'integer' }
}; };
toPublic() { toPublic() {
+2 -2
View File
@@ -1,6 +1,6 @@
{ {
"name": "t42-theta-directory", "name": "t42-theta-directory",
"version": "2.4.0", "version": "2.8.0",
"description": "A very simple LDAP management and SSO system", "description": "A very simple LDAP management and SSO system",
"author": [ "author": [
{ {
@@ -11,7 +11,7 @@
"scripts": { "scripts": {
"start": "node ./bin/www", "start": "node ./bin/www",
"dev": "npx nodemon --ignore public/ ./bin/www", "dev": "npx nodemon --ignore public/ ./bin/www",
"test": "NODE_ENV=test jest tests/groups.test.js tests/subtypes.test.js tests/site_join.test.js tests/site_config.test.js tests/site_replicate.test.js --forceExit" "test": "NODE_ENV=test jest tests/groups.test.js tests/subtypes.test.js tests/site_join.test.js tests/site_config.test.js tests/site_replicate.test.js tests/proxy_client.test.js tests/reconciler.test.js tests/nmap_plugin.test.js tests/jump_client.test.js tests/ldap_replication.test.js --forceExit"
}, },
"jest": { "jest": {
"testEnvironment": "node", "testEnvironment": "node",
+21 -2
View File
@@ -88,9 +88,28 @@ module.exports = {
var msg = (error && error.message) || String(error); var msg = (error && error.message) || String(error);
if (/nmap.*not found|command location/i.test(msg)) { if (/nmap.*not found|command location/i.test(msg)) {
reject(new Error('nmap binary not installed in the container image (rebuild with Dockerfile.openldap, which apk-adds nmap)')); reject(new Error('nmap binary not installed in the container image (rebuild with Dockerfile.openldap, which apk-adds nmap)'));
} else { return;
reject(error);
} }
// node-nmap (node_modules/node-nmap/index.js) treats ANY stderr
// output from the nmap binary as a fatal scan error -- including
// nmap's own benign RTT timing-calibration warnings ("RTTVAR has
// grown to over N seconds, decreasing to M"), which it prints
// *during* a scan that goes on to complete normally. That means a
// scan that actually succeeded (valid XML already sitting in
// scan.rawData) got thrown away and reported as a failed run with
// zero hosts discovered -- not just a noisy log line. Recover by
// manually re-running node-nmap's own XML-parse-then-complete path
// (rawDataHandler -> scanComplete -> the 'complete' listener above)
// when the "error" is this specific known-benign nmap message and
// there's actually output to parse. A genuine XML parse failure
// re-emits 'error' with a different message, which falls through to
// reject() below same as before -- this only widens the recovery
// path, it doesn't swallow real failures.
if (/RTTVAR has grown/i.test(msg) && scan.rawData) {
scan.rawDataHandler(scan.rawData);
return;
}
reject(error);
}); });
scan.startScan(); scan.startScan();
+155 -37
View File
@@ -11,6 +11,7 @@ const { projectResources } = require('@simpleworkjs/directory-schema');
const SUPER_ADMIN_GROUP = permission.SUPER_ADMIN_GROUP; const SUPER_ADMIN_GROUP = permission.SUPER_ADMIN_GROUP;
const groups = require('../utils/groups'); const groups = require('../utils/groups');
const meshReplicate = require('../utils/site_replicate'); const meshReplicate = require('../utils/site_replicate');
const jumpClient = require('../utils/jump_client');
// Make `childCn` a member of `parentCn`, i.e. everyone in the child is // Make `childCn` a member of `parentCn`, i.e. everyone in the child is
// transitively in the parent. Idempotent and non-fatal: "already a member" is // transitively in the parent. Idempotent and non-fatal: "already a member" is
@@ -68,10 +69,10 @@ async function ensureGroup(name, ownerDn, description) {
// naive create on every Directory self-heal (which runs ensureSiteGroups / // naive create on every Directory self-heal (which runs ensureSiteGroups /
// provisionResourceGroups on each load) was accumulating duplicate links -- the // provisionResourceGroups on each load) was accumulating duplicate links -- the
// "groups appear 3x under a resource" bug. Always check first. // "groups appear 3x under a resource" bug. Always check first.
// (services/discovery_reconciler.js's autoPromote path had the same bug via
// its own raw ResourceGroup.create() -- both now share ResourceGroup.ensure().)
async function ensureResourceGroup(resourceId, groupCn, accessLevel) { async function ensureResourceGroup(resourceId, groupCn, accessLevel) {
const existing = await ResourceGroup.list({ where: { resourceId, groupCn } }); return ResourceGroup.ensure(resourceId, groupCn, accessLevel);
if (existing.length) return existing[0];
return ResourceGroup.create({ resourceId, groupCn, accessLevel });
} }
// Provision the site-level groups + the aggregates the per-resource groups nest // Provision the site-level groups + the aggregates the per-resource groups nest
@@ -215,35 +216,17 @@ router.get('/resources', async (req, res, next) => {
}); });
// Even admins never receive secret metadata (e.g. client_secret_hash) over // Even admins never receive secret metadata (e.g. client_secret_hash) over
// the wire; projectResources strips it unconditionally. // the wire; projectResources strips it unconditionally.
//
// Self-heal the group model (docs/GROUPS.md): ensure every site has its // Group-model self-heal (docs/GROUPS.md) used to run here, on every GET --
// site-level groups (S_super_admin, S_hosts_*, S_apps_*, S_everyone) + the // idempotent per-call, but the fan-out (ensureSiteGroups per site +
// aggregates, and every host/app resource has its per-resource groups nested // provisionResourceGroups per resource, each several sequential LDAP
// into them. Idempotent, so this is a cheap no-op once present -- it's what // round-trips) ran unconditionally on every single list, which is what
// backfills a directory seeded by an older release without a rebuild. // made this route slow/unresponsive once a directory had more than a
// Never fails the list. // handful of resources. Healing now happens where resources actually
const sites = resources.filter(r => r.kind === 'site'); // change instead: POST /resources, PUT /resources/:id (see below), and
await Promise.all(sites.map(site => // POST /discovery/promote/:slug. See POST /resources/heal-groups for an
ensureSiteGroups(site.slug, req.user.dn, site.name, site.id) // on-demand equivalent of what this GET used to do implicitly, for
.catch(err => console.error(`ensureSiteGroups(${site.slug}) failed:`, err.message)) // backfilling a directory seeded before this change.
));
const siteByResource = new Map();
for (const site of sites) siteByResource.set(site.id, site.slug);
const siteOf = async (r) => {
const direct = siteByResource.get(r.id);
if (direct) return direct;
// findAncestorSiteSlug returns the site's full slug (`site_local`) -- the
// group-model builders take it verbatim, so do NOT strip the `site_` prefix.
return await Resource.findAncestorSiteSlug(r.id).catch(() => null);
};
await Promise.all(resources.map(async (r) => {
const gKind = groupKind(r);
if (!gKind) return;
const siteSlug = await siteOf(r);
if (!siteSlug) return;
await provisionResourceGroups(r, gKind, siteSlug, req.user.dn)
.catch(err => console.error(`provisionResourceGroups(${r.slug}) failed:`, err.message));
}));
const projected = projectResources(resources, { fullMetadata: true }).map(r => { const projected = projectResources(resources, { fullMetadata: true }).map(r => {
r.hasSecret = !!(r.metadata?.hasSecret || (r.metadata?.secretKeys && r.metadata.secretKeys.length > 0)); r.hasSecret = !!(r.metadata?.hasSecret || (r.metadata?.secretKeys && r.metadata.secretKeys.length > 0));
@@ -290,6 +273,40 @@ router.use((req, res, next) => {
next(); next();
}); });
// On-demand equivalent of the group-model self-heal that GET /resources used
// to run implicitly on every list (see the comment there). Same fan-out,
// same idempotent ensure()-based helpers -- just explicit and admin-
// triggered instead of hidden in every page load, for backfilling a
// directory whose resources predate write-time healing.
router.post('/resources/heal-groups', async (req, res, next) => {
try {
const resources = await Resource.list();
const sites = resources.filter(r => r.kind === 'site');
await Promise.all(sites.map(site =>
ensureSiteGroups(site.slug, req.user.dn, site.name, site.id)
.catch(err => console.error(`ensureSiteGroups(${site.slug}) failed:`, err.message))
));
const siteByResource = new Map();
for (const site of sites) siteByResource.set(site.id, site.slug);
const siteOf = async (r) => {
const direct = siteByResource.get(r.id);
if (direct) return direct;
return await Resource.findAncestorSiteSlug(r.id).catch(() => null);
};
let healed = 0;
await Promise.all(resources.map(async (r) => {
const gKind = groupKind(r);
if (!gKind) return;
const siteSlug = await siteOf(r);
if (!siteSlug) return;
await provisionResourceGroups(r, gKind, siteSlug, req.user.dn)
.then(() => { healed += 1; })
.catch(err => console.error(`provisionResourceGroups(${r.slug}) failed:`, err.message));
}));
res.json({ status: 'ok', sitesHealed: sites.length, resourcesHealed: healed });
} catch (err) { next(err); }
});
router.post('/resources', async (req, res, next) => { router.post('/resources', async (req, res, next) => {
try { try {
if (!req.body.hostId && req.body.parentSlug) { if (!req.body.hostId && req.body.parentSlug) {
@@ -348,6 +365,25 @@ router.post('/resources', async (req, res, next) => {
const ancestorSite = await Resource.findAncestorSiteSlug(r.id); const ancestorSite = await Resource.findAncestorSiteSlug(r.id);
if (r.kind === 'site') { if (r.kind === 'site') {
await ensureSiteGroups(r.slug, req.user.dn, r.name, r.id); await ensureSiteGroups(r.slug, req.user.dn, r.name, r.id);
// Two previously-unrelated "site slug" concepts: this Resource's own
// slug (the Directory catalog's site container -- what every group
// name and the resource tree actually use) vs. site_config.js's
// siteSlug (the multi-site replication identity shown on the
// Multi-Site modal, sourced only from a separately-set SITE_SLUG env
// var). They coincidentally share the name "site slug" but nothing
// ever kept them in sync -- a real deployment could show "E2E Site"
// in the Directory tree and "site-default" on the Multi-Site modal
// for the exact same node. Sync them here, the moment this node's own
// site Resource is created (bootstrap.js's first call), so there's
// one real identity instead of two that can drift apart. Only for a
// still-default master: never overwrite a real multi-site identity a
// join/promote has already established, and a spoke's replication
// identity is the master's to assign, not this node's own resource
// creation to decide.
const cfg = siteConfig.get();
if (cfg.isMaster && cfg.siteSlug === 'site-default') {
siteConfig.save({ siteSlug: r.slug });
}
} else if (gKind && ancestorSite) { } else if (gKind && ancestorSite) {
await ensureSiteGroups(ancestorSite, req.user.dn, r.name); // backfill site tier if missing await ensureSiteGroups(ancestorSite, req.user.dn, r.name); // backfill site tier if missing
await provisionResourceGroups(r, gKind, ancestorSite, req.user.dn); await provisionResourceGroups(r, gKind, ancestorSite, req.user.dn);
@@ -407,7 +443,24 @@ router.put('/resources/:id', async (req, res, next) => {
await ResourceEdge.create({ parentId: req.body.hostId, childId: r.id, relation: updated.kind === 'oauth' ? 'oauth' : 'hosts' }); await ResourceEdge.create({ parentId: req.body.hostId, childId: r.id, relation: updated.kind === 'oauth' ? 'oauth' : 'hosts' });
} }
} }
// Group provisioning (docs/GROUPS.md), same as POST /resources -- an
// update can be what first makes a resource group-eligible (e.g. a
// manual `metadata.managed` edit, or a reparent moving it under a
// different site), and this route never provisioned groups at all
// before. Never fails the update: groups are repairable via
// POST /resources/heal-groups if this best-effort attempt fails.
const gKind = groupKind(updated);
if (gKind) {
const ancestorSite = await Resource.findAncestorSiteSlug(updated.id).catch(() => null);
if (ancestorSite) {
await ensureSiteGroups(ancestorSite, req.user.dn, updated.name)
.catch(err => console.error(`ensureSiteGroups(${ancestorSite}) failed:`, err.message));
await provisionResourceGroups(updated, gKind, ancestorSite, req.user.dn)
.catch(err => console.error(`provisionResourceGroups(${updated.slug}) failed:`, err.message));
}
}
res.json({ results: updated }); res.json({ results: updated });
} catch (err) { } catch (err) {
next(err); next(err);
@@ -928,6 +981,7 @@ const siteConfig = require('../utils/site_config');
const { siteIsFresh } = require('../utils/site_join'); const { siteIsFresh } = require('../utils/site_join');
const { Agent } = require('../models/agent'); const { Agent } = require('../models/agent');
const { SiteSpoke } = require('../models/site_spoke'); const { SiteSpoke } = require('../models/site_spoke');
const { ldapHostFor, currentSlapdServerId } = require('../utils/ldap_replication');
// probeMasterHealth checks whether this (spoke) node can reach its master over // probeMasterHealth checks whether this (spoke) node can reach its master over
// the site join key. The master's /api/site/ping is deliberately lightweight. // the site join key. The master's /api/site/ping is deliberately lightweight.
@@ -954,8 +1008,6 @@ async function probeMasterHealth(cfg) {
router.get('/site-status', async (req, res, next) => { router.get('/site-status', async (req, res, next) => {
try { try {
const sites = await Resource.list({ where: { kind: 'site' } }); const sites = await Resource.list({ where: { kind: 'site' } });
const allResources = await Resource.list();
const gateResources = allResources.filter(r => r.metadata && r.metadata.subType === 'wireguard');
const cfg = siteConfig.get(); const cfg = siteConfig.get();
const wanConnected = await probeMasterHealth(cfg); const wanConnected = await probeMasterHealth(cfg);
@@ -969,7 +1021,29 @@ router.get('/site-status', async (req, res, next) => {
// via an older bootstrap, or the UI form before it grew the field) is // via an older bootstrap, or the UI form before it grew the field) is
// fully joined but silently stuck on the one-time snapshot, which was // fully joined but silently stuck on the one-time snapshot, which was
// otherwise invisible anywhere in the UI. // otherwise invisible anywhere in the UI.
const registeredSpokesCount = cfg.isMaster ? await SiteSpoke.list().then(l => l.length).catch(() => 0) : 0; const allSpokes = cfg.isMaster ? await SiteSpoke.list().catch(() => []) : [];
const registeredSpokesCount = allSpokes.length;
// Real gateway-to-gateway mesh peer count from jump-host's own registry
// (utils/jump_client.js), not this app's unrelated WireGuard
// roaming-client Resources. count is null (not 0) when the query
// couldn't run at all -- the UI distinguishes "0 gateways" from "can't
// tell" instead of showing a misleading zero.
const gateways = await jumpClient.getGatewayCount();
// LDAP MMR status (docs/replication.md): configuredServerId is read
// straight from THIS node's own live slapd.conf; advertisedServerId is
// what GET /ldap-peers / /ldap-replication-config currently hand out for
// it. These can genuinely disagree -- a promotion or a newly-joined
// spoke changes the advertised value immediately, but OpenLDAP's static
// config only reloads at process start, so a mismatch means "re-run
// setup.sh here" rather than "something's broken". Only computed for the
// master (a spoke's advertised ID lives on the master, not locally, and
// querying it here would mean another WAN round-trip on every page load).
const configuredServerId = currentSlapdServerId();
const ldap = cfg.isMaster
? { configuredServerId, advertisedServerId: 1, stale: configuredServerId !== null && configuredServerId !== 1, peersCount: allSpokes.filter(s => s.ldapServerId).length }
: { configuredServerId, advertisedServerId: null, stale: null, peersCount: null };
res.json({ res.json({
status: 'ok', status: 'ok',
config: { config: {
@@ -984,11 +1058,45 @@ router.get('/site-status', async (req, res, next) => {
}, },
sitesCount: sites.length, sitesCount: sites.length,
sites: sites.map(s => ({ id: s.id, name: s.name, slug: s.slug })), sites: sites.map(s => ({ id: s.id, name: s.name, slug: s.slug })),
gatewaysCount: gateResources.length gatewaysCount: gateways.count,
gatewaysNote: gateways.note,
ldap,
// Per-spoke detail (master only) -- endpoint/siteSlug/noInbound/
// relayNote/ldapServerId, not just an aggregate count, so an operator
// can actually see what's registered instead of only "N spokes".
spokes: allSpokes.map(s => ({
siteSlug: s.siteSlug, endpoint: s.endpoint, noInbound: !!s.noInbound,
relayNote: s.relayNote || null, ldapServerId: s.ldapServerId || null,
lastSeenOn: s.last_seen_on || null
}))
}); });
} catch (err) { next(err); } } catch (err) { next(err); }
}); });
// OpenLDAP multi-master replication config for THIS node (docs/replication.md).
// Master-only: the master already has every registered spoke's info locally
// (SiteSpoke), so it can compute its own ServerID (always 1) + full peer
// list without an HTTP round-trip. A spoke gets its config from the master
// directly instead (GET /api/site/ldap-peers -- see bootstrap/
// site-ldap-register.js in theta-suite, which calls whichever of the two
// applies to this node's role).
router.get('/ldap-replication-config', async (req, res, next) => {
try {
const cfg = siteConfig.get();
if (!cfg.isMaster) {
return res.status(400).json({ status: 'error', message: 'this node is a spoke -- fetch replication config from the master via GET /api/site/ldap-peers instead' });
}
const spokes = await SiteSpoke.list();
const peers = [];
for (const s of spokes) {
if (!s.ldapServerId) continue;
const host = ldapHostFor(s.endpoint);
if (host) peers.push({ ldapServerId: s.ldapServerId, ldapHost: host });
}
res.json({ status: 'ok', ldapServerId: 1, peers });
} catch (err) { next(err); }
});
router.post('/site-promote', async (req, res, next) => { router.post('/site-promote', async (req, res, next) => {
try { try {
// god_admin privilege check. This used to read req.user.groups, which // god_admin privilege check. This used to read req.user.groups, which
@@ -1052,6 +1160,16 @@ router.post('/site-promote', async (req, res, next) => {
status: 'ok', status: 'ok',
message: 'Node successfully promoted to Master Site', message: 'Node successfully promoted to Master Site',
handoff: handoffNote, handoff: handoffNote,
// This node's own OpenLDAP ServerID stays whatever it was as a spoke
// (e.g. 2) until `setup.sh` is re-run here -- GET
// /ldap-replication-config will immediately start advertising 1 for
// this node (the master's reserved ID) since that's derived purely
// from cfg.isMaster, but nothing restarts slapd with the new value
// automatically (OpenLDAP's static slapd.conf is only read at process
// start, and this app has no safe way to restart its own container).
// Surfaced here + on the Multi-Site modal so an operator promoting a
// site knows to re-run setup.sh promptly, not just assume it's done.
ldapReplicationNote: 'Re-run setup.sh on this node to apply its new LDAP ServerID (1) and pick up the current spoke peer list -- OpenLDAP config only reloads at process start.',
config: { config: {
isMaster: true, isMaster: true,
masterUrl: '', masterUrl: '',
+120 -10
View File
@@ -42,6 +42,8 @@ function logAudit(action, details) {
console.log(JSON.stringify({ timestamp: new Date().toISOString(), component: 'site', action, ...details })); console.log(JSON.stringify({ timestamp: new Date().toISOString(), component: 'site', action, ...details }));
} }
const { nextFreeLdapServerId, ldapHostFor } = require('../utils/ldap_replication');
// slurpLdif dumps the local LDAP tree with slapcat (the sso-manager container // slurpLdif dumps the local LDAP tree with slapcat (the sso-manager container
// carries an OpenLDAP build with slapcat on PATH). // carries an OpenLDAP build with slapcat on PATH).
async function slurpLdif() { async function slurpLdif() {
@@ -120,27 +122,85 @@ router.post('/spokes', async (req, res, next) => {
const key = await SiteJoinKey.authenticate(rawKey); const key = await SiteJoinKey.authenticate(rawKey);
if (!key) return res.status(401).json({ status: 'error', message: 'invalid or revoked site join key' }); if (!key) return res.status(401).json({ status: 'error', message: 'invalid or revoked site join key' });
const { endpoint, siteSlug } = req.body || {}; const { endpoint, siteSlug, noInbound, meshIp, publicHost } = req.body || {};
if (!endpoint || !/^https?:\/\//.test(endpoint)) { if (!endpoint || !/^https?:\/\//.test(endpoint)) {
return res.status(400).json({ status: 'error', message: 'a valid http(s) endpoint is required' }); return res.status(400).json({ status: 'error', message: 'a valid http(s) endpoint is required' });
} }
const now = Math.floor(Date.now() / 1000); const now = Math.floor(Date.now() / 1000);
let spoke = (await SiteSpoke.list({ where: { endpoint } }))[0]; let spoke = (await SiteSpoke.list({ where: { endpoint } }))[0];
const patch = { siteSlug: siteSlug || (spoke && spoke.siteSlug) || null, last_seen_on: now, noInbound: !!noInbound, meshIp: meshIp || '', publicHost: publicHost || '' };
if (spoke) { if (spoke) {
await spoke.update({ siteSlug: siteSlug || spoke.siteSlug, last_seen_on: now }); await spoke.update(patch);
} else { } else {
spoke = await SiteSpoke.create({ spoke = await SiteSpoke.create({
id: crypto.randomUUID(), id: crypto.randomUUID(),
endpoint, endpoint,
siteSlug: siteSlug || null,
pushToken: SiteSpoke.generatePushToken(), pushToken: SiteSpoke.generatePushToken(),
created_on: now, created_on: now,
last_seen_on: now ldapServerId: await nextFreeLdapServerId(),
...patch
}); });
} }
logAudit('spoke_registered', { endpoint, siteSlug: spoke.siteSlug });
res.json({ status: 'ok', pushToken: spoke.pushToken }); // No-inbound relay automation: best-effort, never blocks registration.
// See utils/proxy_client.js for why this reuses theta-proxy's existing
// API token system rather than a new credential type.
let relayNote = 'not applicable (spoke has inbound access)';
if (noInbound) {
if (meshIp && publicHost) {
const proxyClient = require('../utils/proxy_client');
const result = await proxyClient.ensureRelayRoute({ host: publicHost, ip: meshIp, targetPort: 3001 });
relayNote = result.note;
} else {
relayNote = 'skipped: noInbound set but meshIp/publicHost missing';
}
await spoke.update({ relayNote });
}
logAudit('spoke_registered', { endpoint, siteSlug: spoke.siteSlug, noInbound: !!noInbound, relayNote });
res.json({ status: 'ok', pushToken: spoke.pushToken, relay: { note: relayNote } });
} catch (e) { next(e); }
});
// ── LDAP replication peer list (SPOKE-callable, Bearer site join key) ───────
// OpenLDAP multi-master replication (docs/replication.md) needs each site to
// know its own ServerID plus every OTHER site's LDAPS URL. The master
// coordinates ID assignment (nextFreeLdapServerId, above); this is how a
// spoke asks "what's my ID, and who are my peers" -- called by
// theta-suite's bootstrap/site-ldap-register.js on every setup.sh run, not
// just once at join time, since the peer list changes as other spokes join.
// Same join-key auth as /spokes (a spoke already has this stored from its
// own join). `endpoint` identifies the CALLER so it can be excluded from its
// own peer list -- same identity SiteSpoke.list() keys registration on.
router.get('/ldap-peers', async (req, res, next) => {
try {
const auth = req.headers.authorization || '';
const rawKey = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';
const key = await SiteJoinKey.authenticate(rawKey);
if (!key) return res.status(401).json({ status: 'error', message: 'invalid or revoked site join key' });
const callerEndpoint = req.query.endpoint;
if (!callerEndpoint) {
return res.status(400).json({ status: 'error', message: 'endpoint query param is required' });
}
const cfg = siteConfig.get();
const masterHost = ldapHostFor(cfg.masterUrl || req.protocol + '://' + req.get('host'));
const spokes = await SiteSpoke.list();
const caller = spokes.find((s) => s.endpoint === callerEndpoint);
if (!caller || !caller.ldapServerId) {
return res.status(404).json({ status: 'error', message: 'this endpoint is not a registered spoke -- register via POST /api/site/spokes first' });
}
const peers = [{ ldapServerId: 1, ldapHost: masterHost }];
for (const s of spokes) {
if (s.endpoint === callerEndpoint || !s.ldapServerId) continue;
const host = ldapHostFor(s.endpoint);
if (host) peers.push({ ldapServerId: s.ldapServerId, ldapHost: host });
}
res.json({ status: 'ok', ldapServerId: caller.ldapServerId, peers });
} catch (e) { next(e); } } catch (e) { next(e); }
}); });
@@ -200,7 +260,44 @@ router.post('/demote', async (req, res, next) => {
const base = String(newMasterUrl).replace(/\/+$/, ''); const base = String(newMasterUrl).replace(/\/+$/, '');
siteConfig.save({ isMaster: false, masterUrl: base, masterJoinKey: newJoinKey }); siteConfig.save({ isMaster: false, masterUrl: base, masterJoinKey: newJoinKey });
logAudit('demoted', { demotedBy: key.keyPrefix, newMasterUrl: base }); logAudit('demoted', { demotedBy: key.keyPrefix, newMasterUrl: base });
res.json({ status: 'ok', message: 'Demoted to spoke of ' + base });
// Register with the new master immediately, the same way a real /join
// does (POST /spokes) -- without this, a demoted former master was
// orphaned: it had a masterJoinKey but no SiteSpoke entry on the new
// master (so no ldapServerId, no live replication push target), and
// structurally could never self-heal via /join (which refuses re-join
// for a node that's already a spoke, and requires a fresh install --
// neither true for a former master with real users/agents). Best-effort:
// failing to register here must not fail the demotion itself, same
// reasoning as a normal join's optional live-replication registration.
let registrationNote = 'not attempted (no stack.ssoHost/stack.selfUrl configured to register with)';
// stack.selfUrl is a full-URL override (scheme + port) for environments
// where "https://<ssoHost>" isn't the real reachable address -- the
// multisite e2e test harness (plain HTTP, docker-network hostnames,
// no TLS/proxy in front) is exactly that case; every real deployment
// just relies on the ssoHost derivation.
const selfUrl = (conf.stack && conf.stack.selfUrl) || (conf.stack && conf.stack.ssoHost && `https://${conf.stack.ssoHost}`);
if (selfUrl) {
try {
const regResp = await fetch(base + '/api/site/spokes', {
method: 'POST',
headers: { Authorization: 'Bearer ' + newJoinKey, 'Content-Type': 'application/json' },
body: JSON.stringify({ endpoint: selfUrl, siteSlug: cfg.siteSlug })
});
if (regResp.ok) {
const regBody = await regResp.json();
if (regBody.pushToken) siteConfig.save({ replicationPushToken: regBody.pushToken });
registrationNote = 'registered as a spoke of the new master';
} else {
registrationNote = 'registration failed: HTTP ' + regResp.status;
}
} catch (e) {
registrationNote = 'registration failed: ' + e.message;
}
}
logAudit('demoted_self_registered', { newMasterUrl: base, registrationNote });
res.json({ status: 'ok', message: 'Demoted to spoke of ' + base, registration: { note: registrationNote } });
} catch (e) { next(e); } } catch (e) { next(e); }
}); });
@@ -354,7 +451,7 @@ async function adoptFromMaster({ masterUrl, joinKey }) {
router.post('/join', async (req, res, next) => { router.post('/join', async (req, res, next) => {
try { try {
const { masterUrl, joinKey, selfUrl } = req.body || {}; const { masterUrl, joinKey, selfUrl, noInbound, meshIp, publicHost } = req.body || {};
if (!masterUrl || !joinKey) { if (!masterUrl || !joinKey) {
return res.status(400).json({ status: 'error', message: 'masterUrl and joinKey are required' }); return res.status(400).json({ status: 'error', message: 'masterUrl and joinKey are required' });
} }
@@ -388,17 +485,29 @@ router.post('/join', async (req, res, next) => {
// snapshot for that spoke, not a hard failure). // snapshot for that spoke, not a hard failure).
let replicationPushToken = null; let replicationPushToken = null;
let replicationNote = 'not registered (no selfUrl given)'; let replicationNote = 'not registered (no selfUrl given)';
let relayNote = noInbound ? 'not attempted (registration did not run)' : 'not applicable (this spoke has inbound access)';
if (selfUrl) { if (selfUrl) {
try { try {
const regResp = await fetch(base + '/api/site/spokes', { const regResp = await fetch(base + '/api/site/spokes', {
method: 'POST', method: 'POST',
headers: { Authorization: 'Bearer ' + joinKey, 'Content-Type': 'application/json' }, headers: { Authorization: 'Bearer ' + joinKey, 'Content-Type': 'application/json' },
body: JSON.stringify({ endpoint: selfUrl, siteSlug: exportData.siteSlug || cfg.siteSlug }) // noInbound/meshIp/publicHost: this spoke has no public IP of its
// own; forwarded so the master can best-effort auto-create a relay
// route on its own theta-proxy (utils/proxy_client.js). Previously
// accepted by /spokes but never actually reachable from here --
// nothing forwarded them, so the automation existed but no real
// join flow could ever trigger it.
body: JSON.stringify({
endpoint: selfUrl,
siteSlug: exportData.siteSlug || cfg.siteSlug,
...(noInbound ? { noInbound: true, meshIp, publicHost } : {})
})
}); });
if (regResp.ok) { if (regResp.ok) {
const regBody = await regResp.json(); const regBody = await regResp.json();
replicationPushToken = regBody.pushToken; replicationPushToken = regBody.pushToken;
replicationNote = 'registered for live replication'; replicationNote = 'registered for live replication';
if (regBody.relay) relayNote = regBody.relay.note;
} else { } else {
replicationNote = 'registration failed: HTTP ' + regResp.status; replicationNote = 'registration failed: HTTP ' + regResp.status;
} }
@@ -438,7 +547,8 @@ router.post('/join', async (req, res, next) => {
resources: { created: imp.created, updated: imp.updated, edges: imp.edgeCount }, resources: { created: imp.created, updated: imp.updated, edges: imp.edgeCount },
ldap: { note: ldapNote }, ldap: { note: ldapNote },
signingKey: { note: signingKeyNote }, signingKey: { note: signingKeyNote },
replication: { note: replicationNote, live: !!replicationPushToken } replication: { note: replicationNote, live: !!replicationPushToken },
relay: { note: relayNote }
}); });
} catch (e) { next(e); } } catch (e) { next(e); }
}); });
+8 -14
View File
@@ -169,20 +169,14 @@ router.post('/promote/:slug', async (req, res, next) => {
else throw e; else throw e;
} }
// Link them // Link them. ensure(), not create(): a re-submitted/retried Promote
const crypto = require('crypto'); // click (or the modal being saved twice) had no existence check here,
await ResourceGroup.create({ // so repeated promotion attempts on the same resource accumulated
id: crypto.randomUUID(), // duplicate access/admin group rows -- see ResourceGroup.ensure()'s
resourceId: resource.id, // comment on models/resource.js for why this can't rely on a DB
groupCn: accessGroup, // constraint instead.
accessLevel: 'user' await ResourceGroup.ensure(resource.id, accessGroup, 'user');
}); await ResourceGroup.ensure(resource.id, adminGroup, 'admin');
await ResourceGroup.create({
id: crypto.randomUUID(),
resourceId: resource.id,
groupCn: adminGroup,
accessLevel: 'admin'
});
const meta = resource.metadata || {}; const meta = resource.metadata || {};
meta.managed = true; meta.managed = true;
+8 -2
View File
@@ -320,8 +320,14 @@ class DiscoveryReconciler {
await Group.get(adminGroup).catch(async (e) => { await Group.get(adminGroup).catch(async (e) => {
if (e.status === 404) await Group.add({ name: adminGroup, description: `Admin access to ${res.name}`, owner: 'cn=admin' }); if (e.status === 404) await Group.add({ name: adminGroup, description: `Admin access to ${res.name}`, owner: 'cn=admin' });
}); });
await ResourceGroup.create({ id: crypto.randomUUID(), resourceId: res._actualId, groupCn: accessGroup, accessLevel: 'user' }).catch(() => {}); // ensure(), not create(): reconcile() runs on every discovery pass
await ResourceGroup.create({ id: crypto.randomUUID(), resourceId: res._actualId, groupCn: adminGroup, accessLevel: 'admin' }).catch(() => {}); // (e.g. once per Proxmox cluster node reporting the same LXC), and
// a raw create() here had no existence check, so a resource ended
// up with the same access/admin group rows duplicated once per
// pass -- see ResourceGroup.ensure()'s comment for why this can't
// rely on a DB constraint instead.
await ResourceGroup.ensure(res._actualId, accessGroup, 'user').catch(() => {});
await ResourceGroup.ensure(res._actualId, adminGroup, 'admin').catch(() => {});
} catch (err) { } catch (err) {
console.error(`[DiscoveryReconciler] autoPromote failed for ${res.slug}:`, err.message); console.error(`[DiscoveryReconciler] autoPromote failed for ${res.slug}:`, err.message);
} }
+80
View File
@@ -0,0 +1,80 @@
'use strict';
let mockBaoStore = new Map();
jest.mock('@simpleworkjs/bao-conf', () => ({
get: jest.fn(async (path) => mockBaoStore.get(path) || null),
set: jest.fn(async (path, value) => { mockBaoStore.set(path, value); })
}));
describe('jump_client', () => {
let jumpClient;
let originalFetch;
let mockFetchImpl;
let calls;
beforeEach(() => {
jest.resetModules();
mockBaoStore = new Map();
calls = [];
mockFetchImpl = async () => ({ ok: true, status: 200, json: async () => ({ status: 'ok', gateways: [] }) });
originalFetch = global.fetch;
global.fetch = (...args) => { calls.push(args); return mockFetchImpl(...args); };
jumpClient = require('../utils/jump_client');
jumpClient._reset();
delete process.env.JUMP_INTERNAL_URL;
});
afterEach(() => {
global.fetch = originalFetch;
});
test('reports null count (not zero) when JUMP_INTERNAL_URL is not configured', async () => {
const result = await jumpClient.getGatewayCount();
expect(result.count).toBeNull();
expect(result.note).toMatch(/JUMP_INTERNAL_URL/);
expect(calls.length).toBe(0);
});
test('reports null count when no token is stored in OpenBao', async () => {
process.env.JUMP_INTERNAL_URL = 'http://jump-host.internal';
const result = await jumpClient.getGatewayCount();
expect(result.count).toBeNull();
expect(result.note).toMatch(/no jump-host API token/);
expect(calls.length).toBe(0);
});
test('returns the real gateway count on success', async () => {
process.env.JUMP_INTERNAL_URL = 'http://jump-host.internal';
mockBaoStore.set('integrations/theta-jump', { token: 'jmp_test_token' });
mockFetchImpl = async () => ({
ok: true, status: 200,
json: async () => ({ status: 'ok', gateways: [{ siteSlug: '(self)' }, { siteSlug: 'site-b' }] })
});
const result = await jumpClient.getGatewayCount();
expect(result.count).toBe(2);
expect(result.note).toBe('ok');
expect(calls[0][0]).toBe('http://jump-host.internal/api/mesh/gateways');
expect(calls[0][1].headers.Authorization).toBe('Bearer jmp_test_token');
});
test('reports null count on a non-2xx response', async () => {
process.env.JUMP_INTERNAL_URL = 'http://jump-host.internal';
mockBaoStore.set('integrations/theta-jump', { token: 'jmp_test_token' });
mockFetchImpl = async () => ({ ok: false, status: 403 });
const result = await jumpClient.getGatewayCount();
expect(result.count).toBeNull();
expect(result.note).toMatch(/HTTP 403/);
});
test('reports a network failure without throwing', async () => {
process.env.JUMP_INTERNAL_URL = 'http://jump-host.internal';
mockBaoStore.set('integrations/theta-jump', { token: 'jmp_test_token' });
mockFetchImpl = async () => { throw new Error('connection refused'); };
const result = await jumpClient.getGatewayCount();
expect(result.count).toBeNull();
expect(result.note).toMatch(/failed: connection refused/);
});
});
+44
View File
@@ -0,0 +1,44 @@
require('./setup');
const { SiteSpoke } = require('../models/site_spoke');
const { nextFreeLdapServerId, ldapHostFor } = require('../utils/ldap_replication');
describe('ldap_replication', () => {
beforeEach(async () => {
const all = await SiteSpoke.list();
for (const s of all) await s.delete();
});
describe('ldapHostFor', () => {
test('derives ldaps://<host>:636 from an http(s) endpoint, ignoring its own port', () => {
expect(ldapHostFor('https://sso.site2.example.com')).toBe('ldaps://sso.site2.example.com:636');
expect(ldapHostFor('https://sso.site2.example.com:8443')).toBe('ldaps://sso.site2.example.com:636');
expect(ldapHostFor('http://sso.site3.example.com')).toBe('ldaps://sso.site3.example.com:636');
});
test('returns null for an unparseable endpoint', () => {
expect(ldapHostFor('not-a-url')).toBeNull();
expect(ldapHostFor('')).toBeNull();
});
});
describe('nextFreeLdapServerId', () => {
test('starts at 2 (1 is reserved for the master) when no spokes are registered', async () => {
await expect(nextFreeLdapServerId()).resolves.toBe(2);
});
test('picks the lowest free id, not just the next highest', async () => {
const now = Math.floor(Date.now() / 1000);
await SiteSpoke.create({ id: 'a', endpoint: 'https://a.example.com', pushToken: 'tok-a', created_on: now, ldapServerId: 2 });
await SiteSpoke.create({ id: 'b', endpoint: 'https://b.example.com', pushToken: 'tok-b', created_on: now, ldapServerId: 4 });
await expect(nextFreeLdapServerId()).resolves.toBe(3);
});
test('ignores spokes with no ldapServerId assigned yet', async () => {
const now = Math.floor(Date.now() / 1000);
await SiteSpoke.create({ id: 'c', endpoint: 'https://c.example.com', pushToken: 'tok-c', created_on: now });
await expect(nextFreeLdapServerId()).resolves.toBe(2);
});
});
});
+48
View File
@@ -10,6 +10,7 @@ jest.mock('node-nmap', () => {
this.targetRange = targetRange; this.targetRange = targetRange;
this.customFlags = customFlags; this.customFlags = customFlags;
this.command = ['-oX', '-', ...(customFlags || []), targetRange]; this.command = ['-oX', '-', ...(customFlags || []), targetRange];
this.rawData = '';
} }
startScan() { startScan() {
setImmediate(() => { setImmediate(() => {
@@ -18,6 +19,15 @@ jest.mock('node-nmap', () => {
]); ]);
}); });
} }
// Real node-nmap's rawDataHandler XML-parses this.rawData then calls
// this.scanComplete(results), which emits 'complete' -- the mock skips
// straight to emitting the same shape so the RTTVAR-recovery test below
// exercises the exact call our plugin code makes.
rawDataHandler() {
this.emit('complete', [
{ ip: '192.168.1.20', hostname: 'host-20', openPorts: [] }
]);
}
} }
return { return {
NmapScan: MockNmapScan, NmapScan: MockNmapScan,
@@ -43,4 +53,42 @@ describe('nmap discovery plugin', () => {
expect(result.resources[0].name).toBe('host-10'); expect(result.resources[0].name).toBe('host-10');
expect(result.edges).toHaveLength(1); expect(result.edges).toHaveLength(1);
}); });
test('recovers a scan that completed despite nmap\'s benign RTTVAR stderr warning', async () => {
// Regression: node-nmap treats ANY stderr output as fatal, including
// nmap's own harmless RTT-calibration message -- which discards a scan
// that actually succeeded. Simulate that by emitting 'error' with the
// RTTVAR text instead of 'complete', with rawData present.
const nmapModule = require('node-nmap');
const originalStartScan = nmapModule.NmapScan.prototype.startScan;
nmapModule.NmapScan.prototype.startScan = function () {
this.rawData = '<nmaprun>...</nmaprun>';
setImmediate(() => {
this.emit('error', new Error('RTTVAR has grown to over 2.3 seconds, decreasing to 2.0'));
});
};
try {
const result = await nmapPlugin.discover({ targetRange: '192.168.1.0/24' });
expect(result.resources.some((r) => r.name === 'host-20')).toBe(true);
} finally {
nmapModule.NmapScan.prototype.startScan = originalStartScan;
}
});
test('still rejects a genuine error even when the message differs from RTTVAR', async () => {
const nmapModule = require('node-nmap');
const originalStartScan = nmapModule.NmapScan.prototype.startScan;
nmapModule.NmapScan.prototype.startScan = function () {
setImmediate(() => {
this.emit('error', new Error('nmap: permission denied'));
});
};
try {
await expect(nmapPlugin.discover({ targetRange: '192.168.1.0/24' })).rejects.toThrow('permission denied');
} finally {
nmapModule.NmapScan.prototype.startScan = originalStartScan;
}
});
}); });
+98
View File
@@ -0,0 +1,98 @@
'use strict';
let mockBaoStore = new Map();
jest.mock('@simpleworkjs/bao-conf', () => ({
get: jest.fn(async (path) => mockBaoStore.get(path) || null),
set: jest.fn(async (path, value) => { mockBaoStore.set(path, value); })
}));
describe('proxy_client', () => {
let proxyClient;
let originalFetch;
let mockFetchImpl;
let calls;
beforeEach(() => {
jest.resetModules();
mockBaoStore = new Map();
calls = [];
mockFetchImpl = async () => ({ ok: true, status: 404 });
originalFetch = global.fetch;
global.fetch = (...args) => { calls.push(args); return mockFetchImpl(...args); };
proxyClient = require('../utils/proxy_client');
proxyClient._reset();
delete process.env.PROXY_INTERNAL_URL;
});
afterEach(() => {
global.fetch = originalFetch;
});
test('skips cleanly when required fields are missing', async () => {
const result = await proxyClient.ensureRelayRoute({ host: '', ip: '', targetPort: 0 });
expect(result.note).toMatch(/required/);
expect(calls.length).toBe(0);
});
test('skips cleanly when PROXY_INTERNAL_URL is not configured', async () => {
const result = await proxyClient.ensureRelayRoute({ host: 'sso-a.example.com', ip: '172.24.5.1', targetPort: 3001 });
expect(result.note).toMatch(/PROXY_INTERNAL_URL/);
expect(calls.length).toBe(0);
});
test('skips cleanly when no token is stored in OpenBao', async () => {
process.env.PROXY_INTERNAL_URL = 'https://proxy.internal';
const result = await proxyClient.ensureRelayRoute({ host: 'sso-a.example.com', ip: '172.24.5.1', targetPort: 3001 });
expect(result.note).toMatch(/no proxy API token/);
expect(calls.length).toBe(0);
});
test('creates the route when the host does not already exist', async () => {
process.env.PROXY_INTERNAL_URL = 'https://proxy.internal';
mockBaoStore.set('integrations/theta-proxy', { token: 'prx_test_token' });
mockFetchImpl = async (url, opts) => {
if (opts.method === undefined) return { ok: true, status: 404 }; // GET lookup
if (opts.method === 'POST') return { ok: true, status: 200 };
throw new Error('unexpected method ' + opts.method);
};
const result = await proxyClient.ensureRelayRoute({ host: 'sso-a.example.com', ip: '172.24.5.1', targetPort: 3001 });
expect(result.note).toBe('created');
const postCall = calls.find((c) => c[1].method === 'POST');
expect(postCall[0]).toBe('https://proxy.internal/api/host');
expect(postCall[1].headers.Authorization).toBe('Bearer prx_test_token');
expect(JSON.parse(postCall[1].body)).toEqual({ host: 'sso-a.example.com', ip: '172.24.5.1', targetPort: 3001 });
});
test('updates the route when it exists but points somewhere else', async () => {
process.env.PROXY_INTERNAL_URL = 'https://proxy.internal';
mockBaoStore.set('integrations/theta-proxy', { token: 'prx_test_token' });
mockFetchImpl = async (url, opts) => {
if (!opts.method) return { ok: true, status: 200, json: async () => ({ results: { ip: '172.24.9.9', targetPort: 3001 } }) };
if (opts.method === 'PUT') return { ok: true, status: 200 };
throw new Error('unexpected method ' + opts.method);
};
const result = await proxyClient.ensureRelayRoute({ host: 'sso-a.example.com', ip: '172.24.5.1', targetPort: 3001 });
expect(result.note).toBe('updated');
});
test('is a no-op when the route already matches', async () => {
process.env.PROXY_INTERNAL_URL = 'https://proxy.internal';
mockBaoStore.set('integrations/theta-proxy', { token: 'prx_test_token' });
mockFetchImpl = async () => ({ ok: true, status: 200, json: async () => ({ results: { ip: '172.24.5.1', targetPort: 3001 } }) });
const result = await proxyClient.ensureRelayRoute({ host: 'sso-a.example.com', ip: '172.24.5.1', targetPort: 3001 });
expect(result.note).toBe('already up to date');
});
test('reports a network failure without throwing', async () => {
process.env.PROXY_INTERNAL_URL = 'https://proxy.internal';
mockBaoStore.set('integrations/theta-proxy', { token: 'prx_test_token' });
mockFetchImpl = async () => { throw new Error('connection refused'); };
const result = await proxyClient.ensureRelayRoute({ host: 'sso-a.example.com', ip: '172.24.5.1', targetPort: 3001 });
expect(result.note).toMatch(/failed: connection refused/);
});
});
+24 -1
View File
@@ -1,5 +1,5 @@
require('./setup'); require('./setup');
const { Resource } = require('../models/resource'); const { Resource, ResourceGroup } = require('../models/resource');
const { DiscoveryReconciler } = require('../services/discovery_reconciler'); const { DiscoveryReconciler } = require('../services/discovery_reconciler');
describe('DiscoveryReconciler', () => { describe('DiscoveryReconciler', () => {
@@ -72,4 +72,27 @@ describe('DiscoveryReconciler', () => {
expect(merged.metadata.interfaces).toHaveLength(1); expect(merged.metadata.interfaces).toHaveLength(1);
expect(merged.metadata.interfaces[0].ip).toBe('10.0.0.6'); // Updated IP expect(merged.metadata.interfaces[0].ip).toBe('10.0.0.6'); // Updated IP
}); });
it('does not duplicate access/admin groups across repeated autoPromote passes', async () => {
// Regression: autoPromote used to call ResourceGroup.create() directly
// with no existence check, so reconciling the same managed resource
// more than once (e.g. a Proxmox cluster reporting one LXC from
// multiple nodes) accumulated duplicate access/admin rows every pass.
const payload = {
resources: [{
kind: 'host',
name: 'LXC 127',
slug: 'lxc-127',
metadata: { interfaces: [{ mac: '00:11:22:33:44:99', ip: '10.0.0.99' }] }
}]
};
await DiscoveryReconciler.reconcile('plugin-A', payload, { autoPromote: true });
await DiscoveryReconciler.reconcile('plugin-A', payload, { autoPromote: true });
await DiscoveryReconciler.reconcile('plugin-A', payload, { autoPromote: true });
const resource = (await Resource.list()).find((r) => r.slug === 'lxc-127');
const groups = await ResourceGroup.list({ where: { resourceId: resource.id } });
expect(groups.map((g) => g.groupCn).sort()).toEqual(['lxc-127_access', 'lxc-127_admin']);
});
}); });
+39
View File
@@ -55,6 +55,45 @@ describe('site_replicate', () => {
expect(JSON.parse(optsA.body).reason).toBe('catalog-changed'); expect(JSON.parse(optsA.body).reason).toBe('catalog-changed');
}); });
test('prefers the mesh IP over the public endpoint when the spoke reported one', async () => {
SiteSpoke._seed([
{ endpoint: 'https://spoke-a.example.com:8443', pushToken: 'token-a', meshIp: '172.24.5.1' }
]);
await siteReplicate.replicateToSpokes('catalog-changed');
await new Promise((r) => setImmediate(r));
expect(mockFetchCalls.length).toBe(1);
expect(mockFetchCalls[0][0]).toBe('http://172.24.5.1:8443/api/site/resync');
});
test('falls back to the public endpoint if the mesh attempt fails', async () => {
SiteSpoke._seed([
{ endpoint: 'https://spoke-a.example.com', pushToken: 'token-a', meshIp: '172.24.5.1' }
]);
mockFetchImpl = async (url) => {
if (url.startsWith('http://172.24.5.1')) throw new Error('mesh unreachable');
return { ok: true, status: 200 };
};
await siteReplicate.replicateToSpokes('catalog-changed');
await new Promise((r) => setImmediate(r));
expect(mockFetchCalls.length).toBe(2);
expect(mockFetchCalls[0][0]).toMatch(/^http:\/\/172\.24\.5\.1/);
expect(mockFetchCalls[1][0]).toBe('https://spoke-a.example.com/api/site/resync');
});
test('a spoke with no meshIp only ever tries the public endpoint', async () => {
SiteSpoke._seed([{ endpoint: 'https://spoke-a.example.com', pushToken: 'token-a' }]);
await siteReplicate.replicateToSpokes('catalog-changed');
await new Promise((r) => setImmediate(r));
expect(mockFetchCalls.length).toBe(1);
expect(mockFetchCalls[0][0]).toBe('https://spoke-a.example.com/api/site/resync');
});
test('no known spokes: resolves cleanly, no fetch calls', async () => { test('no known spokes: resolves cleanly, no fetch calls', async () => {
await siteReplicate.replicateToSpokes('catalog-changed'); await siteReplicate.replicateToSpokes('catalog-changed');
await new Promise((r) => setImmediate(r)); await new Promise((r) => setImmediate(r));
+81
View File
@@ -0,0 +1,81 @@
'use strict';
// Service-to-service client for jump-host's mesh registry -- used by the
// Directory's Multi-Site & Network Gateway Status modal to show the real
// number of gateway-to-gateway WireGuard mesh peers (see MULTI_SITE_SPEC.md),
// instead of counting the unrelated older WireGuard roaming-client/exit-node
// Resources in this app's own catalog (a different subsystem entirely --
// api_directory_admin.js used to filter Resource.list() for
// metadata.subType === 'wireguard', which has nothing to do with the mesh).
//
// Same pattern as utils/proxy_client.js: reuses jump-host's existing
// self-service API token system (models/api_token.js, `jmp_<id>_<secret>`
// bearer tokens) rather than inventing a new credential type. The token must
// be minted by a jump-admin user (GET /api/mesh/gateways requires
// requireJumpAdmin, which checks the token's creator's username/groups, not
// anything the token itself carries) and stored in OpenBao.
const baoConf = require('@simpleworkjs/bao-conf');
const PATH = 'integrations/theta-jump'; // baoConf adds the secret/data prefix
const REQUEST_TIMEOUT_MS = 10000;
let cachedToken = null;
async function loadToken() {
if (cachedToken) return cachedToken;
let stored;
try {
stored = await baoConf.get(PATH);
} catch (err) {
console.error(`[jump_client] could not read ${PATH} from OpenBao: ${err.message}`);
return null;
}
if (!stored || !stored.token) return null;
cachedToken = stored.token;
return cachedToken;
}
function jumpBaseUrl() {
// Not OpenBao -- this is where jump-host's admin API lives, not a secret.
return process.env.JUMP_INTERNAL_URL || '';
}
// Returns { count, note }. count is null (not 0) when the query couldn't run
// at all (not configured, unreachable, unauthorized) -- the modal shows a
// count of gateways it could actually see, not a misleading "0" that reads
// as "you have no mesh peers" when the truth is "this isn't wired up yet".
async function getGatewayCount() {
const base = jumpBaseUrl();
if (!base) {
return { count: null, note: 'skipped: JUMP_INTERNAL_URL not configured' };
}
const token = await loadToken();
if (!token) {
return { count: null, note: `skipped: no jump-host API token at OpenBao ${PATH} -- mint one on jump-host (as a jump-admin user) and store it there` };
}
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
const resp = await fetch(base.replace(/\/+$/, '') + '/api/mesh/gateways', {
headers: { Authorization: 'Bearer ' + token },
signal: controller.signal
});
if (!resp.ok) {
return { count: null, note: `failed: HTTP ${resp.status}` };
}
const body = await resp.json();
const gateways = Array.isArray(body.gateways) ? body.gateways : [];
return { count: gateways.length, note: 'ok' };
} catch (err) {
return { count: null, note: `failed: ${err.message}` };
} finally {
clearTimeout(timer);
}
}
// Test seam.
function _reset() { cachedToken = null; }
module.exports = { getGatewayCount, _reset, PATH };
+68
View File
@@ -0,0 +1,68 @@
'use strict';
const fs = require('fs');
// OpenLDAP multi-master replication (docs/replication.md) config derivation,
// shared between routes/api_site.js (the spoke-facing side: assigns a
// ServerID at registration, serves GET /api/site/ldap-peers) and
// routes/api_directory_admin.js (the master-local side: GET
// /directory-admin/ldap-replication-config computes the master's own
// replication config from the same SiteSpoke registry, no HTTP round-trip
// needed since it already has the data).
const { SiteSpoke } = require('../models/site_spoke');
// The master reserves ServerID 1 for itself; every spoke gets the lowest
// free ID from 2 upward, assigned once at registration and reused across
// re-registrations (SiteSpoke.ldapServerId is only ever set on first
// create). Small max, matching mesh_gateway.js's mesh index -- nothing in
// the OpenLDAP protocol requires a small ServerID, but this deployment's
// docs/examples always have.
const MAX_LDAP_SERVER_ID = 4094;
async function nextFreeLdapServerId() {
const spokes = await SiteSpoke.list();
const used = new Set(spokes.map((s) => s.ldapServerId).filter(Boolean));
for (let i = 2; i <= MAX_LDAP_SERVER_ID; i++) {
if (!used.has(i)) return i;
}
throw new Error(`LDAP server ID space exhausted (max ${MAX_LDAP_SERVER_ID} spokes)`);
}
// A site's LDAP replication URL, derived from its already-known HTTP(S)
// endpoint rather than requiring a separately-configured field: same
// hostname, LDAPS port 636 -- exactly the convention docs/replication.md's
// own worked examples already use (ldaps://sso.site2.com:636 alongside
// https://sso.site2.com). No new config an operator has to keep in sync.
function ldapHostFor(endpoint) {
try {
const host = new URL(endpoint).hostname;
return `ldaps://${host}:636`;
} catch (e) {
return null;
}
}
const SLAPD_CONF_PATH = process.env.SLAPD_CONF_PATH || '/etc/openldap/slapd.conf';
// The ServerID this node's OpenLDAP is ACTUALLY running with right now, read
// straight from slapd.conf (the same file docker-entrypoint.sh writes
// `ServerID <n>` into). This can genuinely differ from what
// GET /ldap-peers / /ldap-replication-config currently ADVERTISE for this
// node -- OpenLDAP's static slapd.conf is only read at process start, so a
// promotion or a new spoke joining doesn't retroactively change what's
// already running until `setup.sh` restarts the container. Surfaced on the
// Multi-Site modal so an operator can see "configured X, but slapd is still
// running Y" instead of assuming replication is live because the API says so.
function currentSlapdServerId() {
let contents;
try {
contents = fs.readFileSync(SLAPD_CONF_PATH, 'utf8');
} catch (e) {
return null;
}
const m = contents.match(/^ServerID\s+(\d+)/m);
return m ? Number(m[1]) : null;
}
module.exports = { MAX_LDAP_SERVER_ID, nextFreeLdapServerId, ldapHostFor, currentSlapdServerId };
+105
View File
@@ -0,0 +1,105 @@
'use strict';
// Service-to-service client for theta-proxy's Host management API --
// MULTI_SITE_SPEC.md's "no-inbound relay automation" (a master creating a
// relay route so a spoke with zero inbound path of its own is reachable).
//
// Deliberately does NOT invent a new credential type. theta-proxy already
// has a self-service API token system (models/api_token.js, `prx_<id>_<secret>`
// bearer tokens that authenticate as their creator's user + group snapshot --
// same pattern this app and jump-host both already have their own copy of).
// The "service-to-service auth" gap was never "no credential type exists" --
// it's that nothing wired one of these tokens into an actual inter-service
// call. This is that wiring, using the credential type that was already
// there. The token itself is operator-provisioned (minted on theta-proxy by
// an admin with Host-management rights) and stored in OpenBao, same as the
// agent-signing key in agent_keys.js.
const baoConf = require('@simpleworkjs/bao-conf');
const PATH = 'integrations/theta-proxy'; // baoConf adds the secret/data prefix
const REQUEST_TIMEOUT_MS = 10000;
let cachedToken = null;
async function loadToken() {
if (cachedToken) return cachedToken;
let stored;
try {
stored = await baoConf.get(PATH);
} catch (err) {
console.error(`[proxy_client] could not read ${PATH} from OpenBao: ${err.message}`);
return null;
}
if (!stored || !stored.token) return null;
cachedToken = stored.token;
return cachedToken;
}
function proxyBaseUrl() {
// Not OpenBao -- this is where the proxy's admin API lives, not a secret.
// No safe default: relaying to a guessed host would be worse than
// refusing, so this must be explicitly configured.
return process.env.PROXY_INTERNAL_URL || '';
}
// Creates the relay Host route if missing, updates its target IP if it
// already exists and points somewhere else. Idempotent -- safe to call
// again for the same host on every spoke resync.
//
// Returns { note } describing what happened (created/updated/skipped/failed)
// rather than throwing on a missing token or base URL -- callers (spoke
// registration) must never fail the whole registration just because this
// automation isn't configured yet; it's an enhancement layered on top of a
// working join, not a requirement of one.
async function ensureRelayRoute({ host, ip, targetPort }) {
if (!host || !ip || !targetPort) {
return { note: 'skipped: host, ip, and targetPort are all required' };
}
const base = proxyBaseUrl();
if (!base) {
return { note: 'skipped: PROXY_INTERNAL_URL not configured' };
}
const token = await loadToken();
if (!token) {
return { note: `skipped: no proxy API token at OpenBao ${PATH} -- mint one on theta-proxy and store it there` };
}
const headers = { Authorization: 'Bearer ' + token, 'Content-Type': 'application/json' };
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
const existing = await fetch(base.replace(/\/+$/, '') + '/api/host/' + encodeURIComponent(host), {
headers, signal: controller.signal
});
if (existing.status === 200) {
// GET /api/host/:item wraps the record in { item, results }, not
// flat -- confirmed against a real running proxy (this check
// silently always "updated" instead of no-op'ing until fixed).
const body = await existing.json();
const current = body.results || body;
if (current.ip === ip && Number(current.targetPort) === Number(targetPort)) {
return { note: 'already up to date' };
}
const put = await fetch(base.replace(/\/+$/, '') + '/api/host/' + encodeURIComponent(host), {
method: 'PUT', headers, body: JSON.stringify({ ip, targetPort }), signal: controller.signal
});
return put.ok ? { note: 'updated' } : { note: `update failed: HTTP ${put.status}` };
}
const create = await fetch(base.replace(/\/+$/, '') + '/api/host', {
method: 'POST', headers, body: JSON.stringify({ host, ip, targetPort }), signal: controller.signal
});
return create.ok ? { note: 'created' } : { note: `create failed: HTTP ${create.status}` };
} catch (err) {
return { note: `failed: ${err.message}` };
} finally {
clearTimeout(timer);
}
}
// Test seam.
function _reset() { cachedToken = null; }
module.exports = { ensureRelayRoute, _reset, PATH };
+42 -15
View File
@@ -37,21 +37,48 @@ function replicateToSpokes(reason) {
})(); })();
} }
async function pingOne(spoke, reason) { // Cross-component routing (MULTI_SITE_SPEC.md): if this spoke reported a WG
const url = String(spoke.endpoint).replace(/\/+$/, '') + '/api/site/resync'; // mesh IP when it registered (utils/proxy_client.js's no-inbound relay path
const controller = new AbortController(); // populates the same field), prefer sending the resync push over the mesh
const timer = setTimeout(() => controller.abort(), RESYNC_TIMEOUT_MS); // tunnel instead of the open internet -- plain HTTP is fine here since the
try { // WG tunnel itself is already encrypted, same reasoning as the no-inbound
const resp = await fetch(url, { // relay terminating at the master. Falls back to the spoke's public endpoint
method: 'POST', // if the mesh attempt fails (mesh IP set but that particular tunnel isn't
headers: { Authorization: 'Bearer ' + spoke.pushToken, 'Content-Type': 'application/json' }, // actually up yet, or unreachable for any other reason) -- never let a
body: JSON.stringify({ reason: reason || 'catalog-changed' }), // mesh-routing preference turn into "spoke never gets updates."
signal: controller.signal function resyncUrls(spoke) {
}); const urls = [];
if (!resp.ok) throw new Error('status ' + resp.status); if (spoke.meshIp) {
} finally { let port = '3001';
clearTimeout(timer); try { port = new URL(spoke.endpoint).port || port; } catch (_) { /* keep default */ }
urls.push(`http://${spoke.meshIp}:${port}/api/site/resync`);
} }
urls.push(String(spoke.endpoint).replace(/\/+$/, '') + '/api/site/resync');
return urls;
} }
module.exports = { replicateToSpokes }; async function pingOne(spoke, reason) {
const urls = resyncUrls(spoke);
let lastErr;
for (const url of urls) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), RESYNC_TIMEOUT_MS);
try {
const resp = await fetch(url, {
method: 'POST',
headers: { Authorization: 'Bearer ' + spoke.pushToken, 'Content-Type': 'application/json' },
body: JSON.stringify({ reason: reason || 'catalog-changed' }),
signal: controller.signal
});
if (!resp.ok) throw new Error('status ' + resp.status);
return; // success -- don't try the next (fallback) URL
} catch (err) {
lastErr = err;
} finally {
clearTimeout(timer);
}
}
throw lastErr;
}
module.exports = { replicateToSpokes, resyncUrls };
+60 -3
View File
@@ -3319,6 +3319,55 @@
} catch (e) { console.error('Failed to fetch site status:', e); } } catch (e) { console.error('Failed to fetch site status:', e); }
} }
// ldap: { configuredServerId, advertisedServerId, stale, peersCount } from
// GET /directory-admin/site-status (routes/api_directory_admin.js).
// configuredServerId is read from THIS node's live slapd.conf;
// advertisedServerId (master only) is what the API currently hands spokes.
// They can genuinely disagree right after a promotion or a new spoke
// joining -- OpenLDAP's static config only reloads at process start.
function renderLdapStatus(ldap) {
if (!ldap) return '<span class="text-muted">unknown</span>';
if (ldap.configuredServerId == null) {
return '<span class="badge bg-secondary"><i class="fa-solid fa-circle-minus me-1"></i> Not configured (standalone)</span>';
}
let html = '<span class="badge bg-dark">ServerID ' + esc(ldap.configuredServerId) + '</span>';
if (ldap.peersCount != null) {
html += ' <span class="badge bg-primary">' + ldap.peersCount + ' peer' + (ldap.peersCount === 1 ? '' : 's') + '</span>';
}
if (ldap.stale) {
html += ' <span class="badge bg-warning text-dark" title="This node now advertises ServerID ' + esc(ldap.advertisedServerId) +
' (e.g. after a promotion), but slapd is still running with ' + esc(ldap.configuredServerId) +
' -- re-run setup.sh here to apply it."><i class="fa-solid fa-triangle-exclamation me-1"></i> Needs setup.sh re-run</span>';
}
return html;
}
// spokes: [{siteSlug, endpoint, noInbound, relayNote, ldapServerId, lastSeenOn}]
// Per-spoke detail (master only) so an operator can see what's actually
// registered instead of only an aggregate count.
function renderSpokesTable(spokes) {
if (!spokes || !spokes.length) return '';
const rows = spokes.map(function(s) {
return '<tr>' +
'<td><code>' + esc(s.siteSlug || '?') + '</code></td>' +
'<td class="small">' + esc(s.endpoint) + '</td>' +
'<td>' + (s.ldapServerId != null ? '<span class="badge bg-dark">' + esc(s.ldapServerId) + '</span>' : '<span class="text-muted small">unassigned</span>') + '</td>' +
'<td>' + (s.noInbound
? '<span class="badge bg-info text-dark" title="' + esc(s.relayNote || '') + '"><i class="fa-solid fa-diagram-project me-1"></i> Relayed</span>'
: '<span class="text-muted small">direct</span>') + '</td>' +
'</tr>';
}).join('');
return '<div class="card mt-3">' +
'<div class="card-header py-2 fw-bold small"><i class="fa-solid fa-diagram-project me-1"></i> Registered Spokes</div>' +
'<div class="card-body p-0">' +
'<table class="table table-sm table-hover mb-0">' +
'<thead><tr><th>Site</th><th>Endpoint</th><th>LDAP ServerID</th><th>Path</th></tr></thead>' +
'<tbody>' + rows + '</tbody>' +
'</table>' +
'</div>' +
'</div>';
}
async function openSiteStatusModal() { async function openSiteStatusModal() {
try { try {
const res = await app.api.get('directory-admin/site-status'); const res = await app.api.get('directory-admin/site-status');
@@ -3332,7 +3381,11 @@
'<span>' + (isMaster ? '👑 <strong>Master Site Node</strong>' : '⚡ <strong>Spoke Site Node</strong>') + '</span>' + '<span>' + (isMaster ? '👑 <strong>Master Site Node</strong>' : '⚡ <strong>Spoke Site Node</strong>') + '</span>' +
'<span class="badge bg-' + (isMaster ? 'warning text-dark' : 'info text-dark') + '">' + esc(cfg.siteMode || 'master') + '</span>' + '<span class="badge bg-' + (isMaster ? 'warning text-dark' : 'info text-dark') + '">' + esc(cfg.siteMode || 'master') + '</span>' +
'</h5>' + '</h5>' +
'<p class="card-text text-muted small mb-2">Multi-site directory & replication state for this node.</p>' + '<p class="card-text text-muted small mb-2">Multi-site directory & replication state for this node. ' +
'<a href="https://theta42.github.io/theta-suite/sso/multi-site.html" target="_blank" rel="noopener"><i class="fa-solid fa-book me-1"></i>Directory join docs</a>' +
' &middot; ' +
'<a href="https://theta42.github.io/theta-suite/jump-host/mesh.html" target="_blank" rel="noopener"><i class="fa-solid fa-book me-1"></i>Gateway mesh docs</a>' +
'</p>' +
'<table class="table table-sm text-start mb-0">' + '<table class="table table-sm text-start mb-0">' +
'<tr><th>Local Site Slug:</th><td><code>' + esc(cfg.siteSlug || 'site-default') + '</code></td></tr>' + '<tr><th>Local Site Slug:</th><td><code>' + esc(cfg.siteSlug || 'site-default') + '</code></td></tr>' +
'<tr><th>Master Authority URL:</th><td>' + (cfg.masterUrl ? ('<code>' + esc(cfg.masterUrl) + '</code>') : '<em>(This Node is Master)</em>') + '</td></tr>' + '<tr><th>Master Authority URL:</th><td>' + (cfg.masterUrl ? ('<code>' + esc(cfg.masterUrl) + '</code>') : '<em>(This Node is Master)</em>') + '</td></tr>' +
@@ -3344,13 +3397,17 @@
: '<span class="badge bg-warning text-dark"><i class="fa-solid fa-triangle-exclamation me-1"></i> Snapshot only (re-join to register for live updates)</span>') + '</td></tr>' : '') + : '<span class="badge bg-warning text-dark"><i class="fa-solid fa-triangle-exclamation me-1"></i> Snapshot only (re-join to register for live updates)</span>') + '</td></tr>' : '') +
(isMaster ? '<tr><th>Registered Spokes:</th><td><span class="badge bg-success">' + (cfg.registeredSpokesCount || 0) + ' receiving live updates</span></td></tr>' : '') + (isMaster ? '<tr><th>Registered Spokes:</th><td><span class="badge bg-success">' + (cfg.registeredSpokesCount || 0) + ' receiving live updates</span></td></tr>' : '') +
'<tr><th>Registered Sites:</th><td><span class="badge bg-primary">' + (res.sitesCount || 0) + ' sites</span></td></tr>' + '<tr><th>Registered Sites:</th><td><span class="badge bg-primary">' + (res.sitesCount || 0) + ' sites</span></td></tr>' +
'<tr><th>Theta Gateways:</th><td><span class="badge bg-dark">' + (res.gatewaysCount || 0) + ' active gateways</span></td></tr>' + '<tr><th>Theta Gateways:</th><td>' + (res.gatewaysCount == null
? '<span class="badge bg-secondary" title="' + esc(res.gatewaysNote || 'not configured') + '"><i class="fa-solid fa-question me-1"></i> Unknown (jump-host integration not configured)</span>'
: '<span class="badge bg-dark">' + res.gatewaysCount + ' active gateway' + (res.gatewaysCount === 1 ? '' : 's') + '</span>') + '</td></tr>' +
'<tr><th>LDAP Replication (MMR):</th><td>' + renderLdapStatus(res.ldap) + '</td></tr>' +
'</table>' + '</table>' +
'</div>' + '</div>' +
'</div>' + '</div>' +
'<div class="alert alert-secondary small mb-3">' + '<div class="alert alert-secondary small mb-3">' +
'<i class="fa-solid fa-network-wired me-1"></i> <strong>WireGuard Gateway Mesh & NETMAP</strong>: Inter-site routing operates via <code>theta-gateway</code> subnets (<code>10.x.0.0/16</code>) with default NETMAP shadow translations (<code>10.x.168.0/24 &rarr; 192.168.1.0/24</code>).' + '<i class="fa-solid fa-network-wired me-1"></i> <strong>WireGuard Gateway Mesh & NETMAP</strong>: Inter-site routing operates via <code>theta-gateway</code> subnets (<code>10.x.0.0/16</code>) with default NETMAP shadow translations (<code>10.x.168.0/24 &rarr; 192.168.1.0/24</code>).' +
'</div>'; '</div>' +
(isMaster ? renderSpokesTable(res.spokes) : '');
// Fresh install (no users/resources yet): offer to JOIN an existing // Fresh install (no users/resources yet): offer to JOIN an existing
// master site instead of seeding a new directory. // master site instead of seeding a new directory.
+53
View File
@@ -172,6 +172,12 @@ async function main() {
}); });
if (siteRes.status !== 200) fail(`seeding pre-join site on master failed: ${siteRes.status} ${JSON.stringify(siteRes.body)}`); if (siteRes.status !== 200) fail(`seeding pre-join site on master failed: ${siteRes.status} ${JSON.stringify(siteRes.body)}`);
step('Verifying the Directory site Resource\'s slug synced into the multi-site replication identity');
const { body: masterCfgAfterSite } = await api(MASTER_URL, '/api/site/config', { token: masterToken });
if (masterCfgAfterSite.config.siteSlug !== 'site_e2e') {
fail(`expected site_config's siteSlug to sync to the new site Resource's slug (site_e2e), got ${JSON.stringify(masterCfgAfterSite.config.siteSlug)}`);
}
const seedRes = await api(MASTER_URL, '/api/directory-admin/resources', { const seedRes = await api(MASTER_URL, '/api/directory-admin/resources', {
method: 'POST', method: 'POST',
token: masterToken, token: masterToken,
@@ -205,6 +211,43 @@ async function main() {
if (spokeCfg.config.isMaster !== false) fail(`spoke should be isMaster:false after join, got ${JSON.stringify(spokeCfg.config)}`); if (spokeCfg.config.isMaster !== false) fail(`spoke should be isMaster:false after join, got ${JSON.stringify(spokeCfg.config)}`);
if (!spokeCfg.config.masterUrl) fail('spoke should have masterUrl set after join'); if (!spokeCfg.config.masterUrl) fail('spoke should have masterUrl set after join');
step('Verifying the master computed its own LDAP replication config (ServerID 1 + the new spoke as a peer)');
const { body: masterLdapCfg } = await api(MASTER_URL, '/api/directory-admin/ldap-replication-config', { token: masterToken });
if (masterLdapCfg.ldapServerId !== 1) fail(`master's own ldapServerId should be 1, got ${JSON.stringify(masterLdapCfg)}`);
const spokePeer = (masterLdapCfg.peers || []).find(p => p.ldapHost === 'ldaps://spoke:636');
if (!spokePeer || typeof spokePeer.ldapServerId !== 'number') {
fail(`master's peer list should include the spoke at ldaps://spoke:636 with an assigned ldapServerId, got ${JSON.stringify(masterLdapCfg.peers)}`);
}
step('Verifying the spoke can fetch its own assigned LDAP ServerID + peer list from the master');
const spokeLdapPeersResp = await fetch(`${MASTER_URL}/api/site/ldap-peers?endpoint=${encodeURIComponent('http://spoke:3001')}`, {
headers: { Authorization: 'Bearer ' + joinKey }
});
const spokeLdapCfg = await spokeLdapPeersResp.json();
if (spokeLdapPeersResp.status !== 200) fail(`GET /api/site/ldap-peers failed: ${spokeLdapPeersResp.status} ${JSON.stringify(spokeLdapCfg)}`);
if (spokeLdapCfg.ldapServerId !== spokePeer.ldapServerId) {
fail(`spoke's own reported ldapServerId (${spokeLdapCfg.ldapServerId}) should match what the master's peer list assigned it (${spokePeer.ldapServerId})`);
}
const masterAsPeer = (spokeLdapCfg.peers || []).find(p => p.ldapServerId === 1);
if (!masterAsPeer || masterAsPeer.ldapHost !== 'ldaps://master:636') {
fail(`spoke's peer list should include the master (ServerID 1, ldaps://master:636), got ${JSON.stringify(spokeLdapCfg.peers)}`);
}
const selfInOwnPeerList = (spokeLdapCfg.peers || []).some(p => p.ldapServerId === spokeLdapCfg.ldapServerId);
if (selfInOwnPeerList) fail(`spoke's own peer list should not include itself, got ${JSON.stringify(spokeLdapCfg.peers)}`);
step('Verifying the master\'s own site-status surfaces LDAP status + per-spoke detail (Multi-Site modal data)');
const { body: masterStatus } = await api(MASTER_URL, '/api/directory-admin/site-status', { token: masterToken });
if (!masterStatus.ldap || masterStatus.ldap.advertisedServerId !== 1) {
fail(`master's site-status should report ldap.advertisedServerId 1, got ${JSON.stringify(masterStatus.ldap)}`);
}
if (masterStatus.ldap.peersCount !== 1) {
fail(`master's site-status should report exactly 1 LDAP peer (the spoke), got ${JSON.stringify(masterStatus.ldap)}`);
}
const statusSpokeEntry = (masterStatus.spokes || []).find(s => s.endpoint === 'http://spoke:3001');
if (!statusSpokeEntry || typeof statusSpokeEntry.ldapServerId !== 'number') {
fail(`master's site-status spokes list should include the spoke with an ldapServerId, got ${JSON.stringify(masterStatus.spokes)}`);
}
step('Verifying the spoke adopted the master\'s pre-join catalog'); step('Verifying the spoke adopted the master\'s pre-join catalog');
const spokeResources = await api(SPOKE_URL, '/api/directory-admin/resources', { token: spokeToken }); const spokeResources = await api(SPOKE_URL, '/api/directory-admin/resources', { token: spokeToken });
const adopted = (spokeResources.body.results || spokeResources.body.resources || spokeResources.body || []); const adopted = (spokeResources.body.results || spokeResources.body.resources || spokeResources.body || []);
@@ -257,6 +300,16 @@ async function main() {
if (promoteRes.body.handoff !== 'previous master demoted') { if (promoteRes.body.handoff !== 'previous master demoted') {
fail(`expected the old master to be demoted as part of promotion, got handoff=${JSON.stringify(promoteRes.body.handoff)}`); fail(`expected the old master to be demoted as part of promotion, got handoff=${JSON.stringify(promoteRes.body.handoff)}`);
} }
if (!promoteRes.body.ldapReplicationNote) {
fail('expected /site-promote to surface a note that this node\'s LDAP ServerID needs a setup.sh re-run to apply');
}
step('Verifying the demoted old master auto-registered itself as a real spoke of the new master (not orphaned)');
const { body: newMasterLdapCfg } = await api(SPOKE_URL, '/api/directory-admin/ldap-replication-config', { token: spokeToken });
const oldMasterAsPeer = (newMasterLdapCfg.peers || []).find(p => p.ldapHost === 'ldaps://master:636');
if (!oldMasterAsPeer || typeof oldMasterAsPeer.ldapServerId !== 'number') {
fail(`the demoted old master should appear as a registered peer with an assigned ldapServerId, got ${JSON.stringify(newMasterLdapCfg.peers)}`);
}
step('Verifying the newly-promoted node is master'); step('Verifying the newly-promoted node is master');
const { body: newMasterCfg } = await api(SPOKE_URL, '/api/site/config', { token: spokeToken }); const { body: newMasterCfg } = await api(SPOKE_URL, '/api/site/config', { token: spokeToken });