theta42/theta-suite
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(multi-site): mark Linux mDNS local-discovery shipped and verified

Browse Source 
Announce (theta-gateway) + discover/apply/revert (theta-agent) confirmed
working end-to-end over real multicast between real containers, including
two real bugs found and fixed along the way (IPv6 query abort, EBUSY on
rename over a bind-mounted /etc/hosts).

Windows/macOS mDNS is now the ONLY unbuilt piece of the original design
this session set out to implement -- and it's blocked on platform access
this environment doesn't have, not on missing design or effort.
This commit is contained in:
William Mantly
2026-08-10 18:10:13 -04:00
parent 182787f268
commit 9aaa35fa4e
1 changed files with 10 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/MULTI_SITE_SPEC.md
+10 -10
View File
@@ -8,8 +8,8 @@
> ## Shipped today
> - **Join, live replication, promotion** (`sso-manager-node`): a spoke joins via a one-time export over a site join key (`POST /api/site/join-keys` / `/export` / `/join`), then registers its own endpoint so the master can push live resync pings on every catalog write — no longer a one-time snapshot. Promotion (`POST /api/directory-admin/site-promote`) coordinates a real handoff, demoting the old master as one action. Identical agent-signing keys ride the same export/resync path. Read [`sso-manager-node/docs/site-join.md`](https://github.com/theta42/theta-directory/blob/master/docs/site-join.md) and `directory_spec.md` §11 for the endpoint-level detail.
> - **Gateway-to-gateway WireGuard mesh** (`theta-gateway`): real site-to-site tunnels via `POST /api/mesh/register`/`/join`, kernel WireGuard with a userspace `wireguard-go` fallback. Verified with an actual two-container encrypted tunnel passing traffic, not a mock.
> - **Not yet connected to each other**: the mesh is a transport layer that exists on its own; `sso-manager-node`'s HTTPS-based join/replicate calls don't route over it yet. That wiring, plus the no-inbound relay it would enable, is the next layer — see the status table.
> - **Not built at all**: mDNS local-discovery (speced for handoff in Appendix B; genuinely needs Windows/Mac work this environment can't do).
> - **Not yet connected to each other**: the mesh is a transport layer that exists on its own; `sso-manager-node`'s HTTPS-based join/replicate calls don't route over it yet. That wiring, plus the no-inbound relay it would enable (mechanism verified, automation not built — see status table), is the next layer.
> - **mDNS local-discovery, Linux**: shipped and verified end-to-end — `theta-gateway` announces (`services/mdns_announce.js`), `theta-agent` discovers and applies a hosts-file override, cleanly reverts when the announcement disappears. Windows/macOS remain unbuilt — see the TODO list.
Design scale: a handful of sites (dozen max, 254 hard ceiling — see §4), a few hundred users/hosts total. This is a deliberate, small, trusted-operator deployment, not a hyperscale/adversarial-tenant one — several decisions below (fire-and-forget replication, identical directories) trade blast-radius for simplicity *because* the scale allows it. Don't generalize these choices past that scale without re-deriving them.
@@ -262,16 +262,16 @@ See [`AGENT_LOCAL_DISCOVERY_SPEC.md`](./AGENT_LOCAL_DISCOVERY_SPEC.md) — split
| Coordinated master promotion (demote the old master as one action) | **Shipped** — `POST /api/site/demote` + `site-promote`'s handoff logic. Fixed two real pre-existing bugs while wiring this in: `site-promote`'s god_admin check read a `req.user.groups` field nothing ever populated (permanently 403'd for everyone), and the read-only write-gate 403'd `site-promote` itself before the handler could run. |
| WireGuard gateway-to-gateway mesh (`theta-gateway`) | **Shipped** — `POST /api/mesh/register`/`/join` (join-token bootstrap), `utils/wg_iface.js` (kernel WireGuard, falls back to userspace `wireguard-go`). Verified with a real two-container test: actual encrypted tunnel, real ICMP traffic across it, 0% loss. This is the mesh transport layer only — nothing in `sso-manager-node`'s replication yet routes traffic *over* it; today's site-to-site HTTPS calls (join/export/resync) still go over whatever network path already reaches the target, same as before this layer existed. |
| No-inbound-spoke relay (master proxies a spoke with no public IP) | **Mechanism verified, automation not built.** Confirmed with a standalone test (not `theta-proxy`'s actual Lua/Redis engine, which needs its own dedicated pass to wire safely): a spoke with zero published ports, reachable only via its WG mesh IP, served a request that an external client sent to the master's public port — the master terminated the connection and relayed over the tunnel. So the underlying idea works; what's missing is `theta-proxy` automatically creating that relay route when a no-inbound spoke registers (needs a real service-to-service credential between `sso-manager-node` and `theta-proxy`/`theta-gateway` that doesn't exist yet — a new integration, not a small wiring task), and today's HTTPS-based join/replicate still requires the spoke to reach the master's API directly (and vice versa for export), so a spoke with zero inbound *and* zero outbound path still can't join at all. |
| mDNS local-discovery | Not built — speced for handoff, see Appendix B (still applicable regardless of which replication mechanism eventually lands) |
| mDNS local-discovery (Linux) | **Shipped** — `theta-gateway` announces (`services/mdns_announce.js`, opt-in via `THETA_LOCAL_DISCOVERY_HOSTS`), `theta-agent` discovers and applies a hosts-file override (`local_discovery.go`, opt-in via `prefer_local_directory`). Verified end-to-end with real containers over real multicast: announce → discover → apply → clean revert on disappearance, all confirmed. Caught two real bugs along the way (`mdns.Lookup()`'s IPv6 query aborting the whole lookup even after a valid IPv4 response arrived; `rename()` failing with EBUSY over a bind-mounted `/etc/hosts`, common in every container runtime) — see the commit messages in `theta-agent`. |
| mDNS local-discovery (Windows, macOS) | Not built — needs platform-native testing this environment can't do (hosts-file vs. stub-resolver tradeoff, elevation, DNS-cache behavior per OS — see Appendix B §3). This is now the **only unbuilt piece** of the original design. |
### TODO — what's actually left, in rough dependency order
1. **mDNS local-discovery, Linux side** (announcer on `theta-gateway`/`theta-proxy`, listener + override in `theta-agent`). Buildable and testable in this environment — in progress now. See Appendix B for the full design; Windows/macOS are separately gated below.
2. **mDNS local-discovery, Windows + macOS** — needs platform-native testing this Linux environment cannot do (hosts-file vs. stub-resolver tradeoff, elevation, DNS-cache quirks per OS — see Appendix B §3). Blocked on a Windows/Mac dev environment, not on design.
3. **Route `sso-manager-node`'s HTTPS traffic (join/export/resync) over the WireGuard mesh** instead of the open internet, now that the mesh exists as its own transport layer. Currently the two subsystems don't know about each other.
4. **`theta-proxy` automation for the no-inbound relay** — mechanism is verified (see status table), but nothing creates the relay route automatically when a no-inbound spoke registers. Needs a new service-to-service credential between `sso-manager-node` and `theta-proxy`/`theta-gateway` — a real design decision (who mints it, what it authorizes), not just wiring.
5. **OpenBao secret replication beyond the one agent-signing key** — LDAP admin creds, JWT secret, other per-deployment secrets that currently differ per site.
6. **`theta-proxy`/`theta-gateway` service-to-service auth model in general** — items 3 and 4 both need it; worth designing once rather than inventing a credential per integration.
7. **Mesh peer removal cleanup** — `wg_iface.removePeer()` doesn't remove the kernel routes `setPeer()` adds (flagged in code, not yet exercised because nothing removes a mesh peer today).
1. **mDNS local-discovery, Windows + macOS** — needs platform-native testing this Linux environment cannot do (hosts-file vs. stub-resolver tradeoff, elevation, DNS-cache quirks per OS — see Appendix B §3). Blocked on a Windows/Mac dev environment, not on design. The Linux side (announcer + agent listener) is done and verified — this is the only remaining piece of the original design with no Linux-buildable path forward.
2. **Route `sso-manager-node`'s HTTPS traffic (join/export/resync) over the WireGuard mesh** instead of the open internet, now that the mesh exists as its own transport layer. Currently the two subsystems don't know about each other.
3. **`theta-proxy` automation for the no-inbound relay** — mechanism is verified (see status table), but nothing creates the relay route automatically when a no-inbound spoke registers. Needs a new service-to-service credential between `sso-manager-node` and `theta-proxy`/`theta-gateway` — a real design decision (who mints it, what it authorizes), not just wiring.
4. **OpenBao secret replication beyond the one agent-signing key** — LDAP admin creds, JWT secret, other per-deployment secrets that currently differ per site.
5. **`theta-proxy`/`theta-gateway` service-to-service auth model in general** — items 2 and 3 both need it; worth designing once rather than inventing a credential per integration.
6. **Mesh peer removal cleanup** — `wg_iface.removePeer()` doesn't remove the kernel routes `setPeer()` adds (flagged in code, not yet exercised because nothing removes a mesh peer today).
*Committed under [`docs/MULTI_SITE_SPEC.md`](file:///home/william/dev/theta42/theta-env/docs/MULTI_SITE_SPEC.md).*