backup_before_rebuild() stalled on the deploy host after printing
"Snapshotting state to ..." with no LDAP/Redis output, and silently
no-oped on any stack brought up under a different compose project.
Root causes:
- The LDAP slapcat + in-container base-DN read used
`$COMPOSE exec -T sso-manager`, which exits 1 *silently* (no stderr)
when the running container belongs to a different compose project than
the one the superproject resolves (the standalone SSO from the
sso-manager-node submodule is project "sso-manager-node", not
"theta-env"). The snapshot then no-ops with no breadcrumb.
- The host-side `node -e 'require(./config/sso-secrets.js)'` base-DN read
had no timeout, so a malformed secrets.js could block at require() time
and hang the whole rebuild at that line.
- The Redis RDB copy hardcoded `/data/dump.rdb` and used
`$COMPOSE cp` — wrong path on the standalone layout (Redis dir is
/app, not /data) and the same compose-project mismatch as above.
Fix:
- Switch every in-container call to `docker exec <name>` / `docker cp`
by container name, which works regardless of the owning compose project
(matches the already-working Redis BGSAVE/SAVE path).
- Wrap the host-side node read in `timeout 5` and slapcat in `timeout 20`
so no single command can hang the rebuild.
- Ask Redis for its own `CONFIG GET dir` / `dbfilename` and copy from the
real path, so the RDB is found on both the unified (/data) and standalone
(/app) layouts.
- Add per-step progress lines (config/LDAP/each Redis service) so any future
stall is localized to the exact step instead of hanging silently.
Co-authored-by: Claude <noreply@anthropic.com>
Two bugs in backup_before_rebuild() that made every pre-rebuild snapshot
fail once a stack was actually running:
1. Redis "BGSAVE did not finish in 30s" — a race. The code issued BGSAVE and
only THEN captured `before = LASTSAVE`. On a small dataset BGSAVE finishes
in well under a second, so `before` was already the post-save value and the
poll waited 30s for a second advance that never came (both services, every
run). Fix: capture LASTSAVE before BGSAVE. Also add a synchronous SAVE
fallback — BGSAVE can fork-fail when the host has vm.overcommit_memory=0
(this host does: 0), and SAVE can't fork-fail. The brief block is fine
pre-rebuild. Poll shortened to 10s since a small dataset completes in <1s.
2. "could not read ldapBaseDn from sso-secrets.js" — the base DN was read via
`docker compose exec sso-manager node -e 'require("/config/sso-secrets.js")'`,
which fails when the running container predates the ./config bind-mount
(no /config in the container). Fix: read from the host-side
./config/sso-secrets.js first (same require pattern the SSO entrypoint
uses; works regardless of the running container's mounts), falling back to
the in-container read.
Verified live: SAVE advances LASTSAVE (reply "OK"); host-side require returns
stack.ldapBaseDn on the example config.
Co-authored-by: Claude <noreply@anthropic.com>
Part A — lossless upgrades:
- Persist both bundled Redis stores via AOF+RDB on named volumes (sso-data,
proxy-data) so OAuth clients, Host records, perms, DNS creds, and auto-ssl
Let's Encrypt certs survive rebuilds.
- setup.sh: backup_before_rebuild() snapshots ./config/ + LDAP (slapcat) +
both Redis (BGSAVE + compose cp) to ./backups/<ts>/ before each rebuild,
keeps last BACKUP_KEEP (default 5). First run is a no-op.
- Restore runbook (README + docs): full / Redis-only / LDAP-only, with the
AOF-vs-RDB note (delete the AOF before restoring an RDB).
Part B — eliminate .env / proxy.env:
- All config + secrets live in bind-mounted ./config/ (gitignored), read by each
app's @simpleworkjs/conf from a symlinked secrets.js. Compose passes only
NODE_ENV + NODE_PORT (no app_* env, which would override secrets.js).
- ./config/sso-secrets.js: app secrets + orchestrator-only stack/bootstrap/
serviceAccountPass keys (app ignores the ones it doesn't use).
- ./config/proxy-secrets.js: oidc (clientId/clientSecret filled in by the
bootstrap), ldap (bind creds), auth (admin groups/users).
- setup.sh ensure_config(): generates ./config/ with random secrets on first
run (then exits for editing); one-time migration from .env/proxy.env
preserving existing secrets (LDAP admin pass, JWT, OAuth client, service
pass) so a running deployment keeps its directory + tokens + OAuth client.
- bootstrap/bootstrap.js: reads /config/*.js (not process.env), registers the
proxy as an OIDC client, and writes the SSO-generated client id+secret back
into ./config/proxy-secrets.js (sso mounts ./config RW, proxy RO).
- config.example/ holds committed annotated templates for manual reference.
- .gitignore: add config/, backups/, *.rdb, *.ldif.
Bump both gitlinks to the merged submodule tips:
- sso-manager-node -> 6920a9f (PR #34)
- proxy -> 8e78604 (PR #118)
Co-authored-by: Claude <noreply@anthropic.com>
Add a first step to setup.sh that runs `git submodule update --init --remote
--recursive`, so each ./setup.sh builds from the newest sso-manager-node + proxy
upstream rather than whatever was pinned at clone time. --init also populates
the submodules if the repo was cloned without --recursive.
Behavior:
- If the fetch is unreachable (offline), warn and continue building the
currently checked-out code instead of hard-failing.
- SKIP_SUBMODULE_UPDATE=1 locks to the pinned commits (offline rebuild /
deliberate pin).
- Verifies the build contexts (Dockerfile.openldap, Dockerfile) exist and
dies with a clear message if a submodule was never initialized.
- Requires git (added to the Requires line); git absent is fatal unless
SKIP_SUBMODULE_UPDATE=1.
Renumbered the subsequent step headers (env -> 2, sso-manager -> 3, ...).
README repo-layout note updated to say setup.sh auto-updates submodules.
Co-Authored-By: Claude <noreply@anthropic.com>
wmantly