Removes LDAP_SERVER_ID/LDAP_REPLICATION_HOSTS as vars an operator has
to hand-set and keep in sync across every site. bootstrap/
site-ldap-register.js (new) asks sso-manager-node's new
GET /api/site/ldap-peers (spoke) or GET /directory-admin/
ldap-replication-config (master) for this node's assigned ServerID +
current peer list, persists it to /config/ldap-replication.env, and
restarts sso-manager only when the computed config actually changed
(OpenLDAP's static slapd.conf is only read at process start). Runs on
every setup.sh invocation -- both master (peer list grows as spokes
join) and spoke.
CFG_LDAP_MMR_MANUAL=true skips the automatic step entirely, for a
topology outside this theta-suite cluster the script can't derive on
its own -- without this escape hatch, an operator's hand-set
LDAP_SERVER_ID/LDAP_REPLICATION_HOSTS would get silently overwritten
on the next run, since every fresh install starts as a master (the
automatic step always runs by default).
Bumps sso-manager-node to pick up the new endpoints + SiteSpoke.ldapServerId.
A dedicated spoke.env for the join-a-cluster vars (CFG_MASTER_DIRECTORY_URL/
_JOIN_KEY, CFG_SPOKE_NO_INBOUND/_PUBLIC_HOST, CFG_PUBLIC_DOMAIN), split out
of setup.env purely for clarity -- setup.env still has every option and
keeps working as a single file if that's preferred. setup.sh reads both
(setup.env first, spoke.env layered on top so its values win), same
first-run-only rule as setup.env already had.
Also adds CFG_PUBLIC_DOMAIN (documented in MULTI_SITE_SPEC.md §4 but never
actually wired into setup.sh): an inbound spoke/standalone site's own public
web domain, independent of CFG_DOMAIN (the shared LDAP identity namespace,
which must stay identical across every site). Unset behaves exactly as
before -- hostnames derive from CFG_DOMAIN like any standalone install.
Two real gaps found while fixing the Directory's Multi-Site modal:
1. SITE_SLUG was never set anywhere -- site_config.js's own fallback
("site-default") was all a fresh master could ever show, since
nothing in setup.sh/docker-compose.yml passed it a value and
bootstrap.js never generated one. Derived from CFG_SITE_NAME (same
source jump-host's default exit node name already uses) with the
same slugify rule bootstrap.js's own site Resource slug uses,
formatted to match site_config.js's own "site-default" convention.
Only a first-run default -- a real join/promote's persisted
site.json value always wins.
2. PROXY_INTERNAL_URL and JUMP_INTERNAL_URL -- the env vars
utils/proxy_client.js (no-inbound relay automation) and the new
utils/jump_client.js (real gateway-mesh count on the modal) read to
find each service -- were never actually set anywhere in
docker-compose.yml. Both features existed in sso-manager-node's
code but were completely unreachable in every real deployment,
always hitting their "not configured" fallback. Wired both to the
docker network hostnames.
Also documents how to mint + store the two integration API tokens
those features need (self-service tokens each app already has, not a
new credential type -- same reasoning as the relay automation).
Two real bugs found on a live deployment:
1. setup.sh's theta-agent install step sed'd in join_key but never
touched server_url, so /etc/theta42/agent.yml kept
agent.yml.example's literal "https://sso.example.com" placeholder
forever. Fixed for both first install and an already-installed
agent.yml (self-heals server_url only, never touches
join_key/auth_token, which may since have been rewritten by the
agent itself with real issued credentials).
2. `theta-agent update` 404'd downloading
https://sso.../resources/theta-agent/theta-agent-linux-amd64 --
that route never existed server-side (only
/resources/theta-agent/install.sh is static-served); self-update
itself was already fixed upstream to pull from GitHub Releases, but
setup.sh was still installing the binary committed in the
theta-agent submodule checkout, which predated that fix and could
therefore never self-update out of the bug. Switched setup.sh to
download the current release binary from GitHub instead (matching
theta-agent's own install.sh), and bumped the submodule to
theta-agent's latest commit, which removes the stale committed
binaries entirely -- this exact "stale committed binary" bug class
has bitten this repo at least twice before (see theta-agent's
CHANGELOG v1.5.0 entry).
jump-host's local admin login (via @simpleworkjs/oidc-client's shared
router) expects `username`, not `uid` -- sso-manager-node's own
/api/auth/login (used by site-join.js) is the one that expects `uid`.
Caught live: the script's login call to jump-host silently 401'd with
`uid`. Confirmed against a real jump-host container that `username`
succeeds.
sso-manager-node/jump-host already had the relay-automation mechanism
(noInbound/meshIp/publicHost -> theta-proxy route via proxy_client.js,
GET /api/mesh/self on jump-host) but nothing in the actual operator
bring-up flow could ever reach it -- setup.sh, bootstrap/site-join.js,
and setup.env.example had zero wiring for it.
Add bootstrap/site-relay-register.js: reads this spoke's own role from
/config/site.json, logs into the local jump-host as its bootstrap
admin to discover its mesh IP, and registers it with the master. Mesh
peering itself stays a manual step (mint/paste a join token, same
pattern as the site join key), so this runs on every setup.sh
invocation via CFG_SPOKE_NO_INBOUND/CFG_SPOKE_PUBLIC_HOST and is a
no-op ("not meshed yet") until an operator has actually meshed the two
jump-hosts.
Also updates MULTI_SITE_SPEC.md's status table/TODO and the published
mesh.md docs page, which still described this as "designed but not
automated" after the API-level work had already shipped.
Rolls up theta-agent v2.2.0: Windows hosts override (CRLF-aware, ipconfig
/flushdns), /32 host-route pinning so the WireGuard tunnel can't swallow the
direct LAN path, and a prompt WS reconnect on apply/revert. Marks Windows
local-discovery shipped in MULTI_SITE_SPEC.md; macOS remains the one unbuilt
piece (in progress on a macOS VM).
Service-to-service auth is a prerequisite for both cross-component
routing and no-inbound relay automation (both need a real credential
between sso-manager-node and theta-proxy/theta-gateway) -- reordered so
that's not buried as item 5. Also notes that Windows/macOS mDNS is being
built by a separate session rather than silently dropping it with no
explanation.
Everything shipped this pass (live replication, master/spoke join,
gateway-to-gateway WireGuard mesh) had real spec docs in the repo
(docs/MULTI_SITE_SPEC.md, sso-manager-node's docs/site-join.md) but
nothing on the actual published docs site (theta42.github.io/theta-suite/)
-- a reader landing there would find no mention of it at all beyond a
vague, unlinked "multi-site replication" bullet on the homepage.
- New docs/sso/multi-site.md: the operator-facing master/spoke join guide
(why, how, promoting a spoke, what replicates, current limits), with an
explicit section distinguishing it from the pre-existing N-way LDAP MMR
replication page (replication.html) -- two different mechanisms that
were at real risk of being conflated with nothing to tell them apart.
- New docs/jump-host/mesh.md: the gateway-to-gateway WireGuard mesh guide,
linked from a "WireGuard mesh routing" bullet that already existed on
the jump-host homepage but pointed nowhere.
- docs/sso/index.md, docs/jump-host/index.md: link the new pages from
each component's Features list.
- docs/index.md: replaced the oversold, unlinked "multi-site replication
running in seconds" homepage copy with an accurate, linked claim.
v2.1.2's release build failed on the Windows CI leg (test-only issue --
TestApplyHostsOverride_* didn't skip on non-Linux, where
applyHostsOverride() correctly refuses). v2.1.2's actual code was never
functionally broken, but v2.1.3 is the release whose CI run is actually
green, so that's what theta-suite should point at.
Rolls up theta-directory v2.4.0, jump-host v2.1.0, theta-agent v2.1.2.
Multi-site directory sync stops being a one-time snapshot (live
fire-and-forget replication, identical agent-signing keys, coordinated
master promotion), and site-to-site networking becomes real
infrastructure (gateway-to-gateway WireGuard mesh, kernel-first with a
userspace wireguard-go fallback, real two-container-verified tunnels)
instead of a documented-but-unbuilt design. Linux mDNS local-discovery
also lands end to end (announcer + agent listener).
See CHANGELOG.md for the full rollup and docs/MULTI_SITE_SPEC.md for
the architecture + explicit TODO list of what's still open.
Announce (theta-gateway) + discover/apply/revert (theta-agent) confirmed
working end-to-end over real multicast between real containers, including
two real bugs found and fixed along the way (IPv6 query abort, EBUSY on
rename over a bind-mounted /etc/hosts).
Windows/macOS mDNS is now the ONLY unbuilt piece of the original design
this session set out to implement -- and it's blocked on platform access
this environment doesn't have, not on missing design or effort.
Confirmed the core idea (master terminates a connection, relays over a
spoke's WG mesh IP to a spoke with zero published/inbound ports of its
own) with a standalone test: an external client hit the master's public
port and got a response that could only have come from the spoke,
which had no reachable port except over the tunnel.
Deliberately did NOT wire this into theta-proxy's actual Lua/Redis
routing engine -- that needs its own dedicated pass to do safely, plus a
real service-to-service credential between sso-manager-node and
theta-proxy/theta-gateway that doesn't exist yet. Recorded as verified
mechanism / unbuilt automation, not conflated with either "done" or
"unknown whether it would even work."
Updates the status table and top-of-doc callout to reflect what actually
shipped this pass: live catalog replication, identical-directory signing
key, coordinated master promotion (with two real bugs found + fixed along
the way), and a real, tested gateway-to-gateway WireGuard mesh.
Explicitly calls out what's still NOT true despite all of the above: the
mesh exists as its own transport layer but sso-manager-node's join/
replicate traffic doesn't route over it yet, so the no-inbound-spoke
relay scenario still isn't solved end-to-end. mDNS remains unbuilt.
Extends the shipped CFG_MASTER_DIRECTORY_URL/JOIN_KEY join flow with the
selfUrl a spoke needs to register itself for live catalog replication
(theta-directory v2.4.0's POST /api/site/spokes) -- without this, every
spoke was permanently limited to the one-time join snapshot even after
the master gained the ability to push live updates.
setup.sh already computes CFG_SSO_HOST before this point in the script;
passes https://$CFG_SSO_HOST as bootstrap/site-join.js's third argument,
which forwards it as `selfUrl` in the POST /api/site/join body.
MULTI_SITE_SPEC.md described a WireGuard-mesh + live-replication design as
if unbuilt-but-planned; meanwhile theta-directory v2.2.0-v2.3.0 (rolled up
in theta-suite v2.2.0) already shipped a simpler, real join mechanism
(one-time LDIF/catalog export over a site join key, read-only spoke
enforcement, setup.sh wiring) that this doc didn't mention at all. Added a
callout pointing at docs/site-join.md as the actual current behavior, and
corrected the status table so it no longer implies unbuilt features are
implemented.
Also adds AGENT_LOCAL_DISCOVERY_SPEC.md, a standalone handoff spec for the
mDNS "prefer local discovered directory" optimization -- confirmed not
implemented anywhere in theta-agent. Needs Windows/Mac-native investigation
this environment can't do; written so it can be picked up independently.
Renaming the job broke the master protection rule, which requires a check
named exactly 'Syntax check bootstrap.js'. The job still checks both bootstrap
scripts, just under the protected name.
Multi-site join wiring (server + UI landed in theta-directory v2.3.0):
- bootstrap/site-join.js: runs inside the sso-manager container (same
self-contained rule as bootstrap.js); logs in as the bootstrap admin and calls
/api/site/join. Idempotent: an already-joined node reports 'already a spoke'.
- setup.sh step 5b: if setup.env sets CFG_MASTER_DIRECTORY_URL +
CFG_MASTER_DIRECTORY_JOIN_KEY, run the join after the bootstrap. Only honored
on first run (ensure_config reads setup.env once and ignores it once
./config/ exists), so an already-populated directory can never be merged.
- setup.env.example documents both vars.
- lint.yml also node --check's site-join.js.
- CFG_ORG default was 'SSO Manager', which became the browser tab title / app
name on fresh installs. Default is now 'Theta Directory' (existing deployments
keep their operator-owned ./config/sso-secrets.js name).
- Records theta-agent d937f8d (silent-install server_url + tray autostart +
self-update 404 fixes).
Dockerfile.openldap now pulls ghcr.io/theta42/openldap-nestgroup instead of
compiling from source on every build (~5-6min -> ~1.5min per CI matrix run,
and removes the runtime dependency on git.openldap.org).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Directory tab admitted every kind:'host' resource regardless of promotion
status (every discovery plugin creates its finds as kind:'host'), and
site-status 500'd on a nonexistent Resource.subType column.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Unifies the GitHub Pages docs site: the SSO/Proxy/Jump Host pages, their
nav labels, and each component's own README now consistently say Theta
Directory / Theta Proxy / Theta Gateway, drop marketing sections ("Why this
over the alternatives", "Get it", "Related projects") that don't apply to a
suite component, remove every standalone/bare-metal install path, and link
to theta42.github.io/theta-suite/... instead of the old per-repo Pages sites.
Bumps submodules: theta-directory v2.0.2, proxy v2.0.1, jump-host v2.0.1.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
wmantly