From bbcc235b68d1f12c0954ded6f9beff279cb2101f Mon Sep 17 00:00:00 2001 From: William Mantly Date: Tue, 4 Aug 2026 13:26:25 -0400 Subject: [PATCH] feat: Directory agent status + plugin modal rework, Vault restyle, navbar (v1.24.0) - Merge theta-agent into Directory: remove the Agents page; add green/yellow/red status dots to host rows and a Metrics tab (telemetry + discovery) to the resource modal, joined to hosts by hostname, live via socket.io + 30s refresh. - Discovery Plugins New-plugin modal: slug derived from name (field removed), cron dropdown (hourly/daily/weekly/custom), configSchema-driven settings (Proxmox url/tokenId/tokenSecret) sent as a populated config. - Directory resource slug now read-only + derived from name. - Vault page restyled to match the site. - Navbar: username no longer underlined; only the active link is bold+underlined. - docs/agents.md: document the Directory status/metrics + NAT troubleshooting. Co-Authored-By: Claude --- CHANGELOG.md | 7 ++ docs/agents.md | 44 ++++++++ nodejs/public/css/styles.css | 6 + nodejs/routes/index.js | 6 - nodejs/utils/ui.js | 1 - nodejs/views/agents.ejs | 112 ------------------- nodejs/views/directory.ejs | 206 ++++++++++++++++++++++++++++++++--- nodejs/views/top.ejs | 2 +- nodejs/views/vault.ejs | 59 ++++++---- 9 files changed, 282 insertions(+), 161 deletions(-) delete mode 100644 nodejs/views/agents.ejs diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d240ab..6c9bd15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +# v1.24.0 +- feat: Agents merged into the Directory — removed the standalone Agents page. Host rows show a green/yellow/red theta-agent status dot (healthy / high-load / not connected) and the resource modal gained a Metrics tab with live telemetry + discovery +- feat: Discovery Plugins New-plugin modal — slug is now derived from the name (field removed), the cron field is a dropdown (hourly/daily/weekly + custom), and per-plugin settings are collected from the configSchema (e.g. Proxmox url/tokenId/tokenSecret) instead of an empty config +- feat: Directory resource slug is now read-only and derived from the name +- feat: Vault page restyled to match the rest of the site (bounded container, card + nav-tabs header, h4) +- feat: navbar — the username is no longer underlined; only the active nav link is bold + underlined + # v1.23.0 - fix: /api/vault proxy never injected X-Vault-Token — the true root cause of the recurring vault 403 "permission denied". The proxy declared its hook with http-proxy-middleware v3 syntax (`on: { proxyReq }`), which the installed HPM v2 silently ignores, so every request reached OpenBao unauthenticated (and the client's sso auth headers were never stripped). Rewritten as v2 `onProxyReq`. - fix: vault proxy header injection ordered before `fixRequestBody` — the body write flushes headers, so setting X-Vault-Token after it silently failed on every POST/PUT (writes would still 403 even with the hook fixed) diff --git a/docs/agents.md b/docs/agents.md index d1f020a..c996ed4 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -31,6 +31,25 @@ Every 30 seconds, the agent streams real-time performance metrics: --- +## Viewing in the SSO Manager + +Agent status and telemetry live on the **Directory** page — there is no separate +Agents page. For each **host** resource that has a connected theta-agent, the +Directory shows a status dot in the row: + +| Color | Meaning | +| :--- | :--- | +| **Green** | Connected, healthy (CPU/RAM/disk within limits). | +| **Yellow** | Connected but under high load (CPU > 80% or RAM > 80% or disk > 90%). | +| **Red** | Not connected (no agent, or the agent is offline). | + +Opening a host's resource modal reveals a **Metrics** tab with the agent's live +telemetry (CPU/RAM/disk/ZFS/GPU) and discovery info (OS, kernel, IPs, location). +The agent is joined to its host by hostname (`agent.discovery.hostname` ↔ the +resource name), so name the Directory host the same as the machine's hostname. + +--- + ## Local-First Security & Capability Matrix To protect hosts against unauthorized control, `theta-agent` enforces a **strict, local-first capability matrix** defined in `/etc/theta42/agent.yml`. Central SSO Manager requests are checked against local configuration before execution; permissions cannot be overridden remotely. @@ -90,3 +109,28 @@ capabilities: arbitrary_bash: false ``` +--- + +## Troubleshooting: agent can't connect (`dial tcp ... i/o timeout`) + +If the agent host logs `Dial error: dial tcp :443: i/o timeout` while +connecting to `wss:///api/agent/ws`, the WebSocket path is usually +fine — this is a **network/NAT** problem, not an agent or SSO bug. A host behind +the same NAT that owns the SSO often cannot reach its own **public IP** (no +hairpin/loopback NAT on many home routers), so the TCP dial times out even +though the same address works from outside. + +Fix options: +1. Point `agent.yml` `server_url` at an address the host can reach directly — + e.g. the SSO host's LAN IP (`http://` or `http://:3001` for a + no-TLS direct path). +2. Enable **NAT reflection / hairpin NAT** on the router so LAN hosts can reach + their own public IP:443. +3. Add a local route/firewall rule on the agent host for its public IP. + +> Note: on a deployment where the theta42 proxy fronts `sso.suite.example`, make +> sure the proxy has a **persistent Host record** for the real SSO domain — not +> just the `localtest.me` placeholder — so routing survives a proxy restart +> (an in-memory lookup cache can mask a missing Redis record for up to ~1h). + + diff --git a/nodejs/public/css/styles.css b/nodejs/public/css/styles.css index 7528f11..2775a58 100755 --- a/nodejs/public/css/styles.css +++ b/nodejs/public/css/styles.css @@ -3,6 +3,12 @@ nav.navbar{ padding-right: 1em; } +/* Only the active top-nav link is bold + underlined; the username is plain. */ +.top-nav a.active{ + font-weight: bold; + text-decoration: underline; +} + body { display: flex; flex-direction: column; diff --git a/nodejs/routes/index.js b/nodejs/routes/index.js index 0f07641..144a4a3 100755 --- a/nodejs/routes/index.js +++ b/nodejs/routes/index.js @@ -59,12 +59,6 @@ router.get('/overview', function(req, res) { res.render('overview', {...values}); }); -// Connected theta-agent hosts + live telemetry (admin). Data from -// GET /api/agent/nodes; live updates via socket.io 'agent.*' events. -router.get('/agents', function(req, res) { - res.render('agents', {...values}); -}); - router.get('/admin', (req, res) => res.redirect(301, '/overview')); router.get('/notifications', (req, res) => res.redirect(301, '/overview')); router.get('/dashboard', (req, res) => res.redirect(301, '/overview')); diff --git a/nodejs/utils/ui.js b/nodejs/utils/ui.js index d35af61..4128e3d 100644 --- a/nodejs/utils/ui.js +++ b/nodejs/utils/ui.js @@ -46,7 +46,6 @@ module.exports = { {href: '/directory', icon: 'fa-solid fa-server', label: 'Directory', groups: ['app_sso_admin', 'app_sso_directory_admin', 'admin']}, // Vault requires login - per-user secrets at secret/users//*. {href: '/vault', icon: 'fa-solid fa-vault', label: 'Vault', groups: ['login']}, - {href: '/agents', icon: 'fa-solid fa-microchip', label: 'Agents', groups: ['app_sso_admin', 'admin']}, {href: '/overview', icon: 'fa-solid fa-gauge-high', label: 'Overview', groups: ['app_sso_admin', 'admin']}, ], }; diff --git a/nodejs/views/agents.ejs b/nodejs/views/agents.ejs deleted file mode 100644 index 2f8662d..0000000 --- a/nodejs/views/agents.ejs +++ /dev/null @@ -1,112 +0,0 @@ -<%- include('top') %> - -
-
-

Theta Agents (connected hosts)

- -
- -
-
Connected agents
-
- - - - - - - - - - - - - - - - - -
HostIPStatusCPURAMDiskZFSGPULast seen
Loading agents...
-
-
- -

- Live data from the theta-agent telemetry stream. An agent reports hostname/IP discovery and - CPU/RAM/disk/ZFS/GPU usage every ~60s over the WebSocket; "Online" means seen in the last 90s. -

-
- - - -<%- include('bottom') %> diff --git a/nodejs/views/directory.ejs b/nodejs/views/directory.ejs index dd329cb..25a7125 100644 --- a/nodejs/views/directory.ejs +++ b/nodejs/views/directory.ejs @@ -72,6 +72,7 @@ {{{indentHtml}}} + {{#isHost}}{{/isHost}} {{kind}}{{#metadata.subType}} ({{metadata.subType}}){{/metadata.subType}} {{#metadata.isProduction}}Prod{{/metadata.isProduction}} {{^metadata.isProduction}}Dev{{/metadata.isProduction}} @@ -235,7 +236,8 @@
- + +
Derived from the name; read-only.
@@ -476,6 +478,7 @@ {id: 'details', label: 'Details', bodyHtml: detailsTabHtml}, {id: 'groups', label: 'Associated LDAP Groups', bodyHtml: groupsTabHtml}, {id: 'children', label: 'Children', bodyHtml: childrenTabHtml}, + {id: 'metrics', label: 'Metrics', bodyHtml: metricsTabHtml(resourcesById[id] && resourcesById[id].agent)}, ], footer: { metaHtml: id ? app.modal.formatAudit(resourcesById[id], {formatDate: function(ms){ return moment(ms).format('YYYY-MM-DD HH:mm'); }}) : '', @@ -521,24 +524,34 @@ } }); + // Connected theta-agent join: hostname->agent and token->agent (case-insensitive + // hostname). Populated by loadResources/refreshAgents; host rows + the Metrics + // tab read from these. Agent data comes from /api/agent/nodes (admin-gated). + var agentsByHost = {}; + var agentsByToken = {}; + async function loadResources() { try { - const [resResources, resGroups, resEdges, resAccess] = await Promise.all([ + const [resResources, resGroups, resEdges, resAccess, resAgents] = await Promise.all([ app.api.get('directory-admin/resources'), app.api.get('directory-admin/groups'), app.api.get('directory-admin/edges'), // Access counts are a nicety, not load-bearing: if the LDAP join fails // the table still renders, just without the Access column populated. - app.api.get('directory-admin/access-summary').catch(function(){ return {results: {}}; }) + app.api.get('directory-admin/access-summary').catch(function(){ return {results: {}}; }), + // Agents are a nicety too: never block the directory on them. + app.api.get('agent/nodes').catch(function(){ return {agents: []}; }) ]); accessSummary = (resAccess && resAccess.results) || {}; resourcesById = {}; - + for (const r of resResources.results) { r.metadata = r.metadata || {}; resourcesById[r.id] = r; } + + indexAgents((resAgents && resAgents.agents) || []); allGroups = resGroups.results; allEdges = resEdges.results; @@ -572,6 +585,79 @@ } } + // Build the hostname->agent and token->agent lookup maps from /api/agent/nodes. + function indexAgents(agents) { + agentsByHost = {}; + agentsByToken = {}; + for (const a of agents || []) { + const hn = (a.hostname || (a.discovery && a.discovery.hostname) || '').toLowerCase(); + if (hn) agentsByHost[hn] = a; + if (a.token) agentsByToken[a.token] = a; + } + } + + function esc(s) { return s == null ? '' : app.util.escapeHtml(String(s)); } + function timeAgo(iso) { if (!iso) return ''; var m = moment(iso); return m.isValid() ? m.fromNow() : ''; } + + // Green (online, healthy) / Yellow (online, high load) / Red (not connected + // or offline). Attaches n.isHost + a colored dot + tooltip for host rows, and + // stores the agent on resourcesById so the Metrics tab can find it. + function attachAgentStatus(n) { + n.isHost = true; + const name = (n.name || '').toLowerCase(); + const slug = (n.slug || '').replace(/^host_/, '').toLowerCase(); + const a = agentsByHost[name] || (slug && agentsByHost[slug]); + n.agent = a || null; + if (resourcesById[n.id]) resourcesById[n.id].agent = a || null; + if (!a) { n.agentColor = '#dc3545'; n.agentStatusTitle = 'No theta-agent connected'; return; } + if (!a.isOnline) { n.agentColor = '#dc3545'; n.agentStatusTitle = 'Agent offline (' + (a.hostname || 'unknown') + ')'; return; } + const t = a.telemetry || {}; + const high = (t.cpu_usage_percent > 80) || (t.ram_usage_percent > 80) || (t.disk_usage_percent > 90); + n.agentColor = high ? '#ffc107' : '#198754'; + n.agentStatusTitle = high ? 'Connected — high load' : 'Connected — healthy'; + } + + // Metrics tab body for the resource modal (snapshot of the joined agent). + function metricsTabHtml(agent) { + if (!agent) { + return '
No theta-agent connected

Install the agent on this host to see live metrics.

'; + } + const d = agent.discovery || {}; + const t = agent.telemetry || {}; + const bar = (val) => `
`; + const online = agent.isOnline ? 'Online' : 'Offline'; + const gpu = (t.gpu_usage_percent != null && t.gpu_usage_percent >= 0) ? t.gpu_usage_percent + '%' : 'N/A'; + return `
+
+
${esc(agent.hostname || 'unknown')} ${online}
+ Last seen ${timeAgo(agent.lastSeen)} +
+
+
CPU ${t.cpu_usage_percent ?? 0}%${bar(t.cpu_usage_percent)}
+
RAM ${t.ram_usage_percent ?? 0}%${bar(t.ram_usage_percent)}
+
Disk ${t.disk_usage_percent ?? 0}%${bar(t.disk_usage_percent)}
+
GPU ${gpu}
+
ZFS ${esc(t.zfs_health || 'N/A')}
+
+
Discovery
+
+
OS: ${esc(d.os || '')}
+
Kernel: ${esc(d.kernel || '')}
+
IPs: ${esc((d.ip_addresses || []).join(', '))}
+
Location: ${esc(d.location || '')}
+
+
`; + } + + // Re-fetch agents (every 30s + on socket events) so status dots stay live. + async function refreshAgents() { + try { + const res = await app.api.get('agent/nodes'); + indexAgents((res && res.agents) || []); + renderTable(); + } catch (e) { /* non-fatal */ } + } + // "Who can reach this?" at a glance. A resource with no linked group is not a // locked-down resource -- it is an unreachable one, and a group whose LDAP // entry has been deleted grants nothing, so both get called out rather than @@ -679,6 +765,7 @@ } n.indentHtml = indentHtml; n.accessHtml = accessCellHtml(n.id); + if (n.kind === 'host') attachAgentStatus(n); finalRenderList.push(n); if (n.children.length > 0) { flatten(n.children, depth + 1); @@ -1588,6 +1675,79 @@ var discoveryPluginTypes = []; + // ── Discovery plugin config helpers (ported from plugins.ejs) ───────────── + // Stored value is always a 5-field cron string; the dropdown picks a preset + // and "Custom…" reveals the raw input. Config fields are driven by each + // plugin type's configSchema so per-plugin settings (e.g. Proxmox url / + // tokenId / tokenSecret) are collected at create time. + var DP_CRON_PRESETS = [ + { key: 'hourly', label: 'Hourly', cron: '0 * * * *' }, + { key: 'daily', label: 'Daily (midnight)', cron: '0 0 * * *' }, + { key: 'weekly', label: 'Weekly (Sun)', cron: '0 0 * * 0' }, + { key: 'custom', label: 'Custom…', cron: null }, + ]; + function dpCronKeyFor(cron) { + var m = DP_CRON_PRESETS.filter(function(p){ return p.cron === cron; })[0]; + return m ? m.key : 'custom'; + } + function dpCronSelectHtml(prefix, current) { + current = current || '0 * * * *'; + var key = dpCronKeyFor(current); + var opts = DP_CRON_PRESETS.map(function(p){ + return ''; + }).join(''); + var rawStyle = key === 'custom' ? '' : ' style="display:none"'; + return '' + + ''; + } + function dpOnCronChange(prefix) { + var sel = document.getElementById(prefix + 'cron-select'); + var raw = document.getElementById(prefix + 'cron'); + if (!sel || !raw) return; + if (sel.value === 'custom') { raw.style.display = ''; } + else { + raw.style.display = 'none'; + var preset = DP_CRON_PRESETS.filter(function(p){ return p.key === sel.value; })[0]; + if (preset) raw.value = preset.cron; + } + } + function dpCronFromForm(prefix) { + var sel = document.getElementById(prefix + 'cron-select'); + if (sel && sel.value !== 'custom') { + var preset = DP_CRON_PRESETS.filter(function(p){ return p.key === sel.value; })[0]; + if (preset) return preset.cron; + } + var raw = document.getElementById(prefix + 'cron'); + return (raw && raw.value.trim()) || '0 * * * *'; + } + function dpConfigFormHtml(type, prefix) { + var t = discoveryPluginTypes.filter(function(x){ return x.type === type; })[0]; + var schema = t && t.configSchema; + if (!schema || !schema.length) return '

No configuration fields for this plugin.

'; + var html = ''; + schema.forEach(function(f) { + var inputType = f.type === 'password' ? 'password' : (f.type === 'url' ? 'url' : 'text'); + var req = f.required ? ' required' : ''; + var ph = f.placeholder ? (' placeholder="' + f.placeholder + '"') : ''; + var label = f.label + (f.secret ? ' ' : '') + (f.required ? ' *' : ''); + html += '
' + + '
'; + }); + return html; + } + function dpCollectConfig(type, prefix) { + var t = discoveryPluginTypes.filter(function(x){ return x.type === type; })[0]; + var schema = t && t.configSchema; + var out = {}; + if (!schema) return out; + schema.forEach(function(f) { var el = document.getElementById(prefix + f.key); if (el) out[f.key] = el.value; }); + return out; + } + function dpRenderFields() { + var type = document.getElementById('new-plugin-type').value; + document.getElementById('new-plugin-config-fields').innerHTML = dpConfigFormHtml(type, 'np-'); + } + function openNewDiscoveryPluginModal() { app.api.get('plugins/types', function(err, res) { if (err) { app.messages.toast('Error loading plugin types: ' + err.message, 'danger'); return; } @@ -1601,25 +1761,22 @@ const bodyHtml = `
- +
+
A slug is derived automatically from the name.
- - -
-
- - -
Standard 5-field cron expression (e.g. */15 * * * * for every 15 mins)
+ + ${dpCronSelectHtml('np-', '0 * * * *')}
+
Configuration
${dpConfigFormHtml(discoveryPluginTypes[0].type, 'np-')}
@@ -1629,7 +1786,7 @@ app.modal.open({ title: 'Configure New Discovery Plugin', bodyHtml: bodyHtml, - size: 'md' + size: 'lg' }); }); } @@ -1637,20 +1794,20 @@ async function saveNewDiscoveryPlugin() { const type = $('#new-plugin-type').val(); const name = $('#new-plugin-name').val().trim(); - const slug = $('#new-plugin-slug').val().trim() || name.toLowerCase().replace(/[^a-z0-9]/g, '-'); - const cron = $('#new-plugin-cron').val().trim() || '*/15 * * * *'; + const cron = dpCronFromForm('np-'); const enabled = $('#new-plugin-enabled').is(':checked'); + const config = dpCollectConfig(type, 'np-'); + if (!type) return app.messages.action('Select a plugin type.', app.modal.body(), 'danger'); if (!name) return app.messages.action('Name is required', app.modal.body(), 'danger'); try { await app.api.post('plugins', { pluginType: type, name, - slug, cron, enabled, - config: {} + config }); app.messages.toast('Discovery plugin created successfully!', 'success'); app.modal.close(); @@ -1663,6 +1820,21 @@ $(document).ready(function(){ loadDiscoveryResources(); loadDiscoveryPlugins(); + // Keep the host status dots live: refresh the agent join periodically and on + // socket.io agent.* broadcasts (dedicated socket — the app default is P2PSub). + refreshAgents(); + setInterval(refreshAgents, 30000); + try { + const dirAgentSocket = io({ auth: { token: app.auth.getToken() } }); + dirAgentSocket.on('agent.telemetry', function(msg){ + const a = msg && agentsByToken[msg.token]; + if (a) { a.telemetry = msg.payload; a.isOnline = true; renderTable(); } + }); + dirAgentSocket.on('agent.discovery', function(msg){ + const a = msg && agentsByToken[msg.token]; + if (a) { a.discovery = msg.payload; if (msg.payload && msg.payload.hostname) a.hostname = msg.payload.hostname; a.isOnline = true; renderTable(); } + }); + } catch (e) { /* socket is optional; periodic refresh still runs */ } }); diff --git a/nodejs/views/top.ejs b/nodejs/views/top.ejs index 0ea839c..aaeb960 100755 --- a/nodejs/views/top.ejs +++ b/nodejs/views/top.ejs @@ -49,7 +49,7 @@
<% if(ui.profileUrl){ %> - <% } else { %> diff --git a/nodejs/views/vault.ejs b/nodejs/views/vault.ejs index aa9a7e4..1e0e5ad 100644 --- a/nodejs/views/vault.ejs +++ b/nodejs/views/vault.ejs @@ -1,27 +1,29 @@ <%- include('top') %> -
-
-

My Secrets (personal namespace)

- -
- -
+
+
+
+
+
+ +
+
+
-
- +
+
My Secrets (personal namespace)
+
-
+
+
-
Secrets List
+
Secrets List
Loading...
@@ -29,7 +31,7 @@
-
+
+
-
Mint an app token
+
Mint an app token

Mints a scoped OpenBao token confined to secret/apps/<name>/* for an external app. The token is shown once — record it in the app immediately; it cannot be recovered later.

The token is periodic: it stays valid as long as the app renews it within its period (POST /v1/auth/token/renew-self). If it lapses, mint a new one here — the app's policy and stored secrets are kept.

@@ -68,7 +72,7 @@
-
+
App token
@@ -83,15 +87,17 @@ curl "$VAULT_ADDR/v1/secret/data/apps//conf"
+
-
+
+
-
+
My shared secrets
@@ -102,7 +108,7 @@ curl "$VAULT_ADDR/v1/secret/data/apps//conf"
-
Shared with me
+
Shared with me
Loading...
@@ -110,6 +116,11 @@ curl "$VAULT_ADDR/v1/secret/data/apps//conf"
+
+
+
+
+