azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

wiki: compile gururmm (full) — fold 07-04/05 batches (v0.3.97-99, agent 0.6.78), policy-core deployed (mig 067), BUG-003 closed, BUG-023 fixed, BUG-024 linux glibc OPEN/quarantined

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Howard Enos
2026-07-05 15:42:55 -07:00
parent fa4867580f
commit a737c4ab24
2 changed files with 191 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files

276
wiki/projects/gururmm.md
View File

@@ -2,11 +2,22 @@
type: project
name: gururmm
display_name: GuruRMM
last_compiled: 2026-07-04
last_compiled: 2026-07-05
compiled_by: Howard-Home/claude-main
aliases:
- guru-rmm
sources:
- "gururmm@origin/main 01d962a: migrations 066-067, policy-core merge e7e90c2, FEATURE_ROADMAP BUG-003/018/023/024"
- "gururmm@origin/main: server/migrations/066_visible_agents_view.sql, 067_policy_folders.sql"
- "gururmm@origin/main: server/src/api/mod.rs (route surface at 01d962a — policy-folders endpoints, 39 modules)"
- "gururmm@origin/main: docs/specs/policy-core/{shape,plan,references,standards}.md, docs/specs/policy-ui/{shape,plan,references,standards}.md"
- "gururmm@origin/main: commit e7e90c2 (policy-core merge), 6d2830d (066->067 renumber), 01d962a (policy-ui spec)"
- "gururmm@origin/main: commit 76a6bd2 (repair batch 2 merge), 6ccd35e (bug-fix batch 3 merge), c34194a (batch 4 merge)"
- "gururmm@origin/main: commit 519eef2/f2d02dd (easy-bugs+preauth), 238aad7 (batch 2 review hardening), 27aaab7 (batch 3 review hardening)"
- "projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md"
- "projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-easy-bugs-batch-deploy.md (incl. 00:45/01:00 PT Update sections — repair batch 2/v0.3.97/agent 0.6.78)"
- "projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-05-howard-bugfix-batch3-channel-race.md (incl. 11:05 PT Update section — batch 4/v0.3.99/BUG-024)"
- "projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md (BUG-003/BUG-018/BUG-023/BUG-024 sections)"
- "gururmm@main: server/src/api/*.rs (REST API surface, ~40 route modules; integrations.rs new)"
- "gururmm@main: server/src/api/mod.rs (route surface diff, 2026-06-25..2026-07-04 — integrations CRUD only new surface)"
- "gururmm@main: server/migrations/*.sql (65 migrations — feature checkpoints through 065_integrations_framework)"
@@ -131,25 +142,27 @@ backlinks:
## Summary
GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 359 enrolled agents (228 typically fresh/online) across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints.
GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 360 enrolled agents (~229 fresh/online at last sample) across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints.
**Current version:** agent 0.6.77 / server v0.3.96 as of 2026-07-04 (was 0.6.75 / 0.3.87 on 2026-06-25). Active `server_error` alerts reduced to 0 fleet-wide after the 2026-07-04/05 fix batch.
**Current version (2026-07-05):** agent 0.6.78 fleet-wide (Windows amd64, ~229+ agents, promoted stable 2026-07-05 after a channel-race incident — see BUG-023) / server v0.3.99 / dashboard v0.2.83 beta. Migrations end at `067_policy_folders.sql`. Active `server_error` alerts: 0.
**Easy-bugs batch + alert-triage fixes deployed 2026-07-05 (server v0.3.95 + v0.3.96):** Fixed three long-standing silent-failure classes surfaced by a full 24h alert triage: (1) the update **scanner** dropped macOS/Linux agent binaries during its self-check because they cannot execute on the Linux build host — every macOS/Linux agent had **never** been offered an update; fixed with host-native-vs-foreign self-check regimes (foreign platforms trust the filename version, host-native binaries stay a hard `--version` gate). (2) **Hardware inventory** silently failed to persist for 7+ fleet machines because Windows registry strings routinely carry embedded NUL bytes that Postgres jsonb/text rejects; fixed with a new shared `db/sanitize.rs` (NUL + C0/C1 control-char strip, collision-safe). (3) **WebSocket log noise** — benign disconnects and pre-auth connection churn (sleeping laptops, half-open NAT, scanners) were logged as `ERROR` and minted `server_error` alerts; reclassified to `INFO`, with genuine auth rejections (bad key, malformed auth) staying `ERROR`. Also fixed in the same batch: `create_or_update_alert` now returns `Option<Alert>` and survives an agent cascade-delete mid-flight (FK violation -> `Ok(None)` instead of aborting the offline sweep); `move_agent` maps a duplicate-site-device conflict to 409 instead of a raw 500. Went through a 19-agent high-effort code review before merge (9 confirmed findings, all fixed). See Recent Work for detail.
**Policy folders + inheritance ("policy-core") deployed 2026-07-05 (Mike, migration 067, merge `e7e90c2`):** the biggest architectural change of this compile window. An org-rooted **policy folder tree** now sits on top of the existing policy engine — each client gets a folder tree (exactly one root), any folder optionally references a library policy, and a device sits in exactly one folder, inheriting the merged policies of its folder ancestry (root->leaf) plus an optional per-asset override. This replaces the fixed System->Client->Site->Agent stack with a per-client tree while preserving identical effective-policy output during migration (the backfill is a deliberate, gated, dry-run-verified step — landing the migration changes zero behavior, since `agents.folder_id` is NULL fleet-wide until backfilled). Ships folder-tree CRUD + placement + version history REST API, recompute-and-push on policy edit (scoped to affected agents only), and policy versioning (`policy_versions`, immutable snapshot per edit, for diff/rollback). Migration renumbered 066->067 after colliding with `066_visible_agents_view.sql`, which landed on main in the same window from an unrelated bug-fix arc. Explicitly out of scope for this phase: module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI, and any agent-side wire-format change (none — this is server+dashboard only). A follow-on **policy-ui** shape spec (folder management dashboard UI) was added the same day (`01d962a`) — implementation not started.
**Four bug-fix batches shipped 2026-07-04..05 (server v0.3.95 -> v0.3.99, agent 0.6.77 -> 0.6.78):** a dense back-to-back sequence of alert-triage, easy-bugs, repair, and issue-hunt passes. Highlights (full detail in Recent Work / History Highlights): update-scanner platform-coverage + channel-default fixes (macOS/Linux agents were never offered updates; then a scanner/build-pipeline race briefly mass-offered unsoaked beta to the stable fleet — **BUG-023**, fixed); inventory NUL-byte storage failures fixed for 7+ machines; WS log noise reclassified off ERROR; **BUG-003** (unsafe deploy pipeline) closed with a build lock + gated git reset + pre-stop migration validation; **BUG-018** (agent delete race) got its permanent altitude fix via a `visible_agents` DB view + schema trigger; **BUG-001** (Windows ACPI thermal) shipped and was proven live on 112+ agents; comms-durability Phase 1 completed (half-open WS eviction sweep); a pending-command TTL, an agent-status reconciler, and update-supersede logic shipped in batch 4; and a **critical Linux glibc incident (BUG-024, still OPEN/mitigated)** crash-looped every older-distro Linux agent that took the newly-enabled Linux update offer.
**Cross-site agent duplication fixed 2026-07-04:** Root cause of ~40 duplicate agent rows (24 at Dataforth D2->D1 alone): both WS enrollment paths deduped only within `(site_id, device_id)`, so re-running a different site's installer on an already-enrolled machine created a second row instead of finding the existing one. Fixed to match by hardware `device_id` **across all sites** and re-home via `move_agent_to_site` — but a same-day follow-up (`1c1cac2`) stopped the auto-re-home on every reconnect (the agent re-presents its install-site code as its API key on every beat, which was bouncing admin-moved agents back to their original site). Net effect: dedup by device_id, but site placement stays admin-controlled and persists.
**Legacy (Server 2008R2/Win7, Rust 1.77) build variant disabled 2026-07-04:** The unpinned legacy dependency re-resolve pulled edition2024 crates (`chacha20`/`rand_core 0.10.1`) that Rust 1.77 cannot parse — a BUG-021 recurrence that fail-closed the entire Windows build (blocking modern amd64/x86/tray artifacts too). Gated behind `LEGACY_ENABLED=false`; the last-built legacy artifact (agent 0.6.76) is preserved and frozen for existing 2008R2/Win7 endpoints. 2008R2 support returns via a separate legacy-only rewrite (C++/.NET or a clean minimal Rust core), not by re-enabling the 1.77 toolchain — captured as **RMM_THOUGHTS Feature 12** (Raw, Mike's requirement).
**Policy-Controlled Tray Deployment shipped 2026-07-04 (code merged, policy default OFF):** The tray binary is now deployed to every Windows endpoint via the agent-update flow (previously only fresh MSI installs got it — fleet census found ~111/113 sampled hosts with no tray at all) but its **launch is gated on `tray.enabled` policy** — no process, no icon, until a client/site/agent policy turns it on. Authenticode-verified + version-locked to the agent. Built via a rejected-then-corrected design (17-agent code review + Gemini + Grok adversarial critique caught 9 defects in the first attempt — see Recent Work). Fleet-wide policy enablement (Task 4) is **not yet flipped**.
**Policy-Controlled Tray Deployment shipped 2026-07-04 (code merged, policy default OFF):** The tray binary is now deployed to every Windows endpoint via the agent-update flow (previously only fresh MSI installs got it — fleet census found ~111/113 sampled hosts with no tray at all) but its **launch is gated on `tray.enabled` policy** — no process, no icon, until a client/site/agent policy turns it on. Authenticode-verified + version-locked to the agent. Fleet-wide policy enablement (Task 4) is **not yet flipped**.
**VSS Policy Configurator shipped 2026-07-03 (SPEC-016 redesign, agent v0.6.76):** The agent no longer runs its own scheduled shadow-copy create+prune loop — the MITRE T1490 "Inhibit System Recovery" surface that got the agent blocked by CrowdStrike Falcon. It now **configures** two kernel-enforced retention governors (VSS diff-area size cap + `MaxShadowCopies` count) and registers a scheduled task whose action is a native-COM **create only** (never a delete) — retention becomes Windows' own FIFO eviction, not agent code. Works uniformly on server AND client Windows SKUs (previously `vssadmin create shadow` blocked workstations). A 23-agent code review caught a data-loss-class defect pre-merge (writing the machine-global `MaxShadowCopies` would have evicted OTHER products' shadows) — dropped in favor of the size cap as sole governor. Live-verified on the T1490 canary (NEPTUNE, Falcon present): legacy scheduled task removed, no T1490 trip, 17 existing shadows preserved. Deployed to the **beta** channel only (0.6.76); stable fleet remains on 0.6.66 pending a deliberate promotion.
**VSS Policy Configurator shipped 2026-07-03 (SPEC-016 redesign, agent v0.6.76):** The agent no longer runs its own scheduled shadow-copy create+prune loop — the MITRE T1490 "Inhibit System Recovery" surface that got the agent blocked by CrowdStrike Falcon. It now **configures** two kernel-enforced retention governors (VSS diff-area size cap + `MaxShadowCopies` count evaluated-but-not-machine-globally-written) and registers a scheduled task whose action is a native-COM **create only** (never a delete). A 23-agent code review caught a data-loss-class defect pre-merge (writing the machine-global `MaxShadowCopies` would have evicted OTHER products' shadows) — dropped in favor of the size cap as sole governor. Deployed to the **beta** channel only (0.6.76 at the time of that compile); stable-channel promotion of the VSS behavior riding on 0.6.78 has not been separately re-verified this compile.
**Vendor-agnostic Integrations Framework shipped 2026-07-01..02 (migration 065, CrowdStrike Falcon first plugin):** A generic per-partner plugin framework (`integrations`, `integration_credentials`, `integration_client_mappings`, `integration_sync_state`, `integration_agent_links` tables) so a new EDR/security vendor is a new server module, not a new schema — modeled on the proven MSPBackups per-partner shape. CrowdStrike Falcon is the first plugin: API client, RMM-client-to-Falcon-CID mapping, and a periodic sync worker pulling sensor state (version/status/RFM) into `integration_agent_links`. REST CRUD at `/api/integrations`; no dedicated dashboard UI yet (API/backend only).
**Vendor-agnostic Integrations Framework shipped 2026-07-01..02 (migration 065, CrowdStrike Falcon first plugin):** A generic per-partner plugin framework so a new EDR/security vendor is a new server module, not a new schema — modeled on the proven MSPBackups per-partner shape. CrowdStrike Falcon is the first plugin. REST CRUD at `/api/integrations`; no dedicated dashboard UI yet.
**SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA — code merged + deployed 2026-06-23..24, NOT guaranteed to work; Route B = server-orchestrated):** The code is on main and deployed, but the **feature is still beta and unreliable** — it succeeds on cleanly-uninstallable programs and is explicitly NOT guaranteed for the hard cases (most managed/protected AV, WiX/Burn bundles, uninstallers that need UI or spawn lingering child processes — Launchy/AIMP-class, drivers/system components). Treat results as best-effort, verify removals, and do not rely on it for production removals yet. See the Capabilities section for full detail + the open gaps.
**SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA, not guaranteed to work; Route B = server-orchestrated):** unchanged status this compile — best-effort, not reliable for the hard cases. See Capabilities.
**Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+):** One install URL/script self-detects arch/OS/legacy and pulls the correct agent; the dashboard install UI now points at it. P2 (native i386 bootstrapper EXE for locked-down boxes) pending.
**Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+):** unchanged this compile.
**Physical server migration completed 2026-06-11:** New physical box at 172.16.3.30 running Ubuntu 26.04 + PostgreSQL 18. Old VM at .46 decommissioned 2026-06-12.
@@ -163,7 +176,51 @@ GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer G
## Recent Work
### 2026-07-04..05 — Easy-Bugs Batch + Alert Triage: Update-Scanner, Inventory NUL, WS Noise (Howard-Home)
### 2026-07-05 — Policy Folders + Inheritance ("policy-core") Deployed (Mike, merge e7e90c2, migration 067)
Replaced the fixed System->Client->Site->Agent policy stack with a per-client **folder tree**: each client gets exactly one root folder; any folder may optionally reference a library policy; a device sits in exactly one folder and inherits the merged policies of its folder ancestry (root->leaf) plus an optional per-asset override. Nine tasks: folder-tree schema + inheritance resolution + versioning (migration 067, renumbered from 066 after a same-window collision with `066_visible_agents_view.sql`); recompute+push on policy edit scoped to affected agents only (not a fleet-wide recompute); folder tree REST API (`/clients/:id/policy-folders`, `/policy-folders/:id[/rename|/policy|/parent]`, `/agents/:id/folder`); migration backfill job with a verification gate + dry-run mode (`/policy-folders/backfill`); integration tests. `policy_versions` gives every policy edit an immutable snapshot for diff/rollback and per-device "what changed."
Landing migration 067 changes **zero behavior** by construction: every new column/table is nullable and `agents.folder_id` is NULL fleet-wide until the backfill runs, so effective-policy resolution stays on the legacy Client->Site->Agent path until a deliberate, gated flip. Explicitly out of scope for this phase: module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI, and any agent-side change (wire format unchanged — server+dashboard only). A follow-on **policy-ui** shape spec (folder management dashboard UI: tree view, agent placement, effective-policy viewer with source attribution, version history, admin backfill trigger) was added the same day (`01d962a`) as an MVP largely assembling existing dashboard components (ContextTree, PolicySectionEditor, EffectivePolicyTable) — not yet implemented.
---
### 2026-07-05 — Bug-Fix Batch 3: BUG-023 Channel Race, BUG-003 Closed, BUG-018 Altitude Fix (Howard-Home, merge 6ccd35e, server v0.3.98)
Post-deploy verification of the prior night's repair batch 2 came back clean, but the new T2 half-open-eviction sweep surfaced an anomaly: 127 evictions, 79 of them one agent (DESKTOP-3USU20B, Safesite Glendale) cycling connect->auth->silent-90s->evict->reconnect every ~6 minutes, plus 3 other chronic loopers. Root-causing why the loopers never got 0.6.78 uncovered **BUG-023**: a publish/tag race in the build pipeline. Builds are tagged `beta` by a late "MARK AS BETA" phase minutes *after* binaries land in the downloads directory, and the scanner defaulted a missing `.channel` sidecar to `stable` — so the 07:38 scan classified the untagged 0.6.78 as stable and mass-offered it to 209 stable-channel agents before the 07:43 scan reclassified it beta, splitting the fleet (214 agents on unsoaked-beta 0.6.78, ~31 stragglers pinned with no path forward). Fixed in three layers: the scanner now **fails closed** (missing/malformed `.channel` => beta, with a regression test) so the race is harmless even if a future pipeline step crashes between publish and tag; `build-windows.sh`'s `deploy_and_sign` writes the beta `.channel` sidecar atomically with each published artifact, closing the timing window at the source; and `promote_version` was rewritten to key on agent **binaries** rather than pre-existing sidecar files (the old logic only rewrote files whose content was exactly `beta`, making fail-closed sidecar-less binaries permanently unpromotable) — it now creates/overwrites `stable` sidecars and a partial write fails the request (500) instead of recording a `stable` rollout row that disagrees with disk. With Howard's go, 0.6.78 (already soaked 8+ hours on 214 agents) was manually promoted to stable, healing the split fleet (229 fresh agents converged by session end).
Also shipped in the same batch: the **BUG-018 altitude fix** — migration `066_visible_agents_view.sql` (a `visible_agents` DB view filtering `status<>'deleting'`, replacing nine scattered inline filters including a previously-missed agent-detail-page query) plus a `BEFORE UPDATE` trigger pinning `status='deleting'` at the schema level, and all three WS `authenticate_agent` paths now reject a mid-purge row instead of binding a live session to an agent about to be hard-deleted. A 15-agent code review (22 verified, 9 findings, 1 refuted) drove a hardening commit: `set_ignore_missing(true)` on the migrator (so a rollback binary tolerates an ahead DB instead of `VersionMissing`-crash-looping — the review's top finding), and a pre-existing `matches_pattern` panic on no-wildcard patterns was fixed (previously hidden among "19 known" test failures — really 18 DB-env + 1 real bug).
**BUG-003 closed:** `deploy/build-pipeline/build-server.sh` (confirmed this session to auto-sync to `/opt/gururmm` on every push) gained a `flock` build lock, `PIPESTATUS`-gated `git fetch`/`reset`, and pre-stop migration validation via a new `--migrate-only` server flag (validates+applies migrations against the live DB while the OLD binary keeps serving, then exits without binding a port). The gate fired correctly on its **first real run**: the merge webhook collided with the CI version-bump job's `index.lock` and aborted cleanly with nothing deployed — where the old script would have silently built a stale tree. Deployed as server v0.3.98 (16:46 UTC), 0 ERROR lines, view/trigger verified live in a rolled-back test transaction. DESKTOP-3USU20B's one-way-WS loop was fingerprinted as per-site network behavior (agent->server frames eaten post-auth; server->agent and REST both fine; reproduces on 0.6.66, 0.6.77, and 0.6.78 alike) — 4 chronic loopers total, telemetry exhausted server-side, packet capture is next. RMM live temperature reading was also spot-verified against ACG-Tech03L's (Snapdragon WoA) local sensor panel: zone-for-zone match, values within tenths of a degree.
---
### 2026-07-05 — Bug-Fix Batch 4: Command TTL, Status Reconciler, Update Supersede; BUG-024 Linux glibc Incident (Howard-Home, merge c34194a, server v0.3.99)
A live-system "what's not working as expected" sweep found and shipped three fixes, twice review-hardened (16 findings addressed across two workflow passes):
1. **Pending-command 7-day TTL:** commands could sit `pending` indefinitely and execute on reconnect regardless of age (May-dated commands would have fired on a stale agent's reconnect). The reaper now expires anything past 7 days with an honest distinct message for never-sent vs. unconfirmed-delivery rows; the first tick expired 10 stale rows.
2. **Agent-status reconciler:** `agents.status` was never reconciled against `last_seen`, leaving zombie `'online'` rows (11 found, one stranded since June 2). A new sweep flips status-only (never touching `last_seen`, after a review finding caught an earlier version fabricating freshness and auto-resolving offline alerts), respects the shared 2-hour update-hold window, skips agents with a live dispatch-map connection, and broadcasts the change via event + SSE. First post-warm-up sweep flipped 4 stranded rows.
3. **Stale pending-update supersede:** re-dispatching a pending update at auth time now supersedes it when a newer version exists or the target binary was pruned from disk (all 13 previously-pruned targets were being blindly re-offered); guarded to only fire on `pending` rows over an hour old with a non-empty platform index.
**BUG-024 (NEW, OPEN, mitigated) — Linux glibc incident:** Linux agent binaries >=0.6.77 were built against the build host's glibc 2.43 (the box is now Ubuntu 26.04), and the 2026-07-04 scanner fix enabled Linux update offers for the **first time ever** — so every older-distro Linux agent that took the offer crash-looped on `GLIBC_2.39` not found: ix.azcomputerguru.com (CloudLinux 9, 3898 restarts), the Jupiter agent container (Debian 12, 623 restarts), a Lonestar container, and presumed SL-SERVER (Scileppi). The Unix rollback watchdog never fired on either affected box (a distinct sub-bug, uninvestigated), and the 0.6.59 container agent still performs the BUG-019 ephemeral in-place update, so the bad binary persisted in the writable layer across restarts. **Mitigated:** Linux binaries quarantined out of the downloads directory (scanner now offers nothing for Linux); ix and the Jupiter container were manually revived from their pre-update backups and reconnected healthy. SL-SERVER and the Lonestar container host remain down pending manual recovery (no vault SSH creds for either). **Real fix pending:** a glibc compatibility floor for Linux builds (old-glibc build container, e.g. rockylinux:8/debian:11, or a switch to musl static linking) plus an `objdump -T` max-GLIBC build gate — tracked as roadmap BUG-024. Also this pass: 4 orphaned duplicate agent rows deleted (zombies with fresher same-site siblings); `.30`'s `gururmm-agent.service` found `disabled` and re-enabled; RECEPTIONIST-PC's two rows at Cascades confirmed to be two real machines sharing a hostname (rename recommended, not a bug). Operational flags relayed to Howard/Mike: LAB-SVR offline 17.7 days with an active critical alert; KSTEENBB2025 peaking 83.8C cpu_temp; a stale repo-root `build-server.sh` lineage is a deletion candidate (the canonical script lives under `deploy/build-pipeline/`).
---
### 2026-07-04..05 — Repair Batch 2: Comms-Durability Phase 1 Complete, BUG-001 WMI Thermal, Uninstall Teardown (Howard-Home, merge 76a6bd2, server v0.3.97, agent 0.6.78)
**Comms-durability Phase 1 completed:** a gap analysis found T1 (30s agent heartbeat) and T3 (ACK'd durable delivery, migrations 058/059) already live; this batch implemented the missing T2 — a half-open-connection eviction sweep (`evict_half_open_connections`, 90s no-inbound threshold, 30s sweeper cadence) — closing out the phase. `SO_KEEPALIVE` was deliberately skipped (the TCP peer is the NPM proxy, not the agent).
**BUG-001 (Windows WMI ACPI thermal) SHIPPED and PROVEN:** `collect_temps_windows_acpi()` reads `MSAcpi_ThermalZoneTemperature` via WMI (`root\WMI`), converts tenths-Kelvin, applies a -20..125C plausibility filter, and reports the hottest zone as `cpu_temp` with a `sysinfo` fallback. Verified live on 112+ Windows agents reporting `cpu_temp` within 10 minutes of deploy (was 0 fleet-wide since LibreHardwareMonitor's CVE-2020-14979 removal on 2026-05-27); sensor-for-sensor matched a local panel on a Snapdragon WoA machine (ACG-Tech03L) including correctly filtering frozen 0-Kelvin zones. A review finding moved the WMI query to `spawn_blocking` with a 5-second deadline so a wedged `winmgmt` service cannot stall heartbeats.
**BUG-020 uninstall teardown fixed:** the real gap (policy-disable/swap `terminate_all` wiring was already correct; the roadmap note calling it broken was stale) was the uninstall path leaving the watchdog service and `gururmm-tray.exe` behind. Fixed ordering: `remove_watchdog_service()` first, then `TrayLauncher::terminate_all().await`, then service delete, plus removal of the tray binary from disk.
**BUG-018 delete UX:** `delete_agent` now marks `status='deleting'` synchronously before the 202 response; all listing/stat queries exclude 'deleting' rows; the dashboard does an optimistic cache removal. Three resurrection paths that could flip a mid-purge row back to `'online'`/`'offline'` (heartbeat, metrics, and offline-grace writes) were closed by making `update_agent_status` refuse to overwrite `'deleting'`. A zombie-connection check in the WS read loop now closes any connection no longer matching the agent's currently-registered one (fixes a permanent-orphan class: agent shown online in the DB, absent from the dispatch map, commands pending forever). Startup reconciliation re-runs the purge for any row stuck `'deleting'` after a mid-purge restart. (Full altitude fix via the `visible_agents` view followed the next day — see Bug-Fix Batch 3.) **MSP360 sync noise:** transient 5xx from MSP360 now logs WARN, escalating to ERROR only after 4 consecutive failed cycles (~1h), so real outages still alert without every cloud-maintenance blip minting a `server_error`.
A 26-candidate, 10-finding code review preceded merge; all 8 findings attributed to this batch were fixed pre-merge (`238aad7`). All 4 build targets went green: server v0.3.97, Windows agent 0.6.78 (wmi crate compiled clean), Linux 0.6.78, dashboard v0.2.83 beta.
---
### 2026-07-04..05 — Easy-Bugs Batch + Alert Triage: Update-Scanner, Inventory NUL, WS Noise (Howard-Home, merges 519eef2/f2d02dd, server v0.3.95/v0.3.96)
Full 24h alert triage (`GET /api/alerts`) surfaced 5 `backup_failed` (client-side, unresolved — see Active State), 2 active `server_error` (product bugs), and 1 resolved `mass_offline_fleet` (36 agents, transient network blip). Root-caused both server errors against the server's own self-log (pseudo-agent `00000000-...-001`):
@@ -176,9 +233,7 @@ A concurrent same-branch session absorbed a prior session's in-progress inventor
Continued into the last stale alert: "Authentication timeout" WS churn (2-6/day baseline, 57-spike correlated with the same day's fleet-dedup work). Profiled 7 days of journal, concluded pre-auth connection noise (peer connects, never authenticates — sleeping laptops, scanners, half-open NAT), added `is_preauth_connection_churn` to classify it INFO while keeping genuine rejections (bad API key, malformed auth) at ERROR. Deployed as **server v0.3.96** (06:23 UTC). Active `server_error` alert count: **0**.
BUG-009 (`Logs.tsx` isError handling) and BUG-011 (`: any` occurrences) were verified already fixed in-tree during the same pass — only the stale `FEATURE_ROADMAP.md` status lines needed correcting, not the code.
Remaining/pending from this session: the **machine-side backup failures** (the user's actual priority) — GND-SERVER (Grabb & Durando, chronic, 42 nightly failures since 2026-06-17), AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial) — investigation started but not completed; all say "no error detail reported" (MSP360 sync doesn't capture failure detail). 13-pair duplicate-agent cleanup still open. WS auth-timeout source (~1/min, unauthenticated) still unidentified (needs Feature 5, server-side public-IP capture — still unapproved).
BUG-009 (`Logs.tsx` isError handling) and BUG-011 (`: any` occurrences) were verified already fixed in-tree during the same pass — only the stale `FEATURE_ROADMAP.md` status lines needed correcting, not the code. Remaining/pending from this session, superseded by later batches where noted: machine-side backup failures (GND-SERVER, AD1, LAB-SVR, DESKTOP-EVA4H1A) stayed open; 13-pair duplicate-agent cleanup (4 orphaned rows cleaned in batch 4, full pass still open); WS auth-timeout narrowed to pre-auth churn here, but 4 specific chronic loopers were only fingerprinted, not fixed, in batch 3.
---
@@ -279,7 +334,7 @@ Five tasks, one per day-slice: (1) framework schema migration; (2) plugin trait
- **Cutover:** New physical box took over 172.16.3.30 (PG 18, Ubuntu 26.04); production down <27 min; old VM decommissioned 2026-06-12.
- **Durable agent identity Phase 1 Task 1:** Multi-location `device_id` (registry+ProgramData on Windows, /etc+/var/lib on Linux, /Library+mirror on macOS) with self-heal; `cleanup.ps1` whitelists identity files.
- **Agent Comms Durability Phase 1 (agent v0.6.63 / server v0.3.68):** Root cause = NAT conntrack asymmetry (server->agent pushes black-holed in idle conntrack gaps). Fix: agent `CommandAck` on receipt + dedup cache; server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via a partial index so old agents keep legacy behavior.
- **Agent Comms Durability Phase 1 (agent v0.6.63 / server v0.3.68):** Root cause = NAT conntrack asymmetry (server->agent pushes black-holed in idle conntrack gaps). Fix: agent `CommandAck` on receipt + dedup cache; server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via a partial index so old agents keep legacy behavior. (Phase 1 fully completed 2026-07-05, see Repair Batch 2 above — the T2 half-open eviction sweep was the missing piece.)
---
@@ -294,13 +349,13 @@ Five tasks, one per day-slice: (1) framework schema migration; (2) plugin trait
## Capabilities / Feature Set
*Synthesized from authoritative artifacts (API routes, agent modules, 65 migrations through migration 065, roadmap, commit log) — not from session logs alone. See Compilation Notes.*
*Synthesized from authoritative artifacts (API routes, agent modules, 67 migrations, roadmap, commit log) — not from session logs alone. See Compilation Notes.*
Agent<->server communication is a persistent authenticated WebSocket with auto-reconnect + heartbeat; on reconnect, in-flight commands flip to `interrupted`. Platform-parity rule: agent features ship on Windows/Linux/macOS in the same change (stub + TODO where a real impl isn't yet feasible).
### Monitoring & Telemetry
- Core metrics per interval (policy-tunable): CPU %, memory %/bytes, disk %/bytes, network rx/tx deltas, uptime, logged-in user, user idle time, public/WAN IP. Cross-platform via `sysinfo`.
- Hardware sensor telemetry: CPU/GPU temps + full sensor array. Linux via `/sys/class/thermal`; `sysinfo::Components` fallback. Windows LibreHardwareMonitor removed (CVE-2020-14979); replacement pending.
- **Windows CPU/thermal (BUG-001, shipped + PROVEN 2026-07-05, server v0.3.97):** `collect_temps_windows_acpi()` reads `MSAcpi_ThermalZoneTemperature` via WMI (`root\WMI`, `spawn_blocking` + 5s deadline so a wedged `winmgmt` can't stall heartbeats), tenths-Kelvin conversion, -20..125C plausibility filter, hottest zone reported as `cpu_temp`, `sysinfo::Components` fallback. Verified fleet-wide: 112+ agents reporting within 10 minutes of deploy (was 0 since LibreHardwareMonitor's CVE-2020-14979 removal 2026-05-27); sensor-for-sensor match against a local panel on Snapdragon WoA hardware (ACG-Tech03L, 20 valid zones, frozen 0-K zones correctly filtered). GPU/NVAPI thermal still open.
- Process drill-down: top 10 by CPU + top 10 by memory in every metrics payload.
- Network state: per-interface IPv4/IPv6, MAC, derived CIDR subnets — sent on change only.
- **Windows BSOD/kernel-crash detection:** `agent/src/bsod.rs` polls `C:\Windows\Minidump`, parses kernel dump header at fixed offsets, cross-references System event log, deduplicates via a watermark file. Always-Critical alert deduplicated by `(agent_id, bugcheck_code)` (changed from dump-hash 2026-06-15 so a mute silences all recurrences).
@@ -310,15 +365,16 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
- Command types: `shell`, `powershell`, `python`, raw `script{interpreter}`, `claude_task`, and `cmd` (alias for `shell`). Options: `timeout_seconds`, `elevated`. Unknown/unparseable commands NAK immediately.
- **Execution context** (`041`): `system` (default) or `user_session` (WTS token impersonation, Windows-only).
- Commands individually cancellable, aborted on disconnect. Auditable history. Script library (`017`) for stored scripts.
- **Agent Comms Durability (Phase 1, shipped 2026-06-11 — agent v0.6.63 / server v0.3.68):** Agent sends `CommandAck` on receipt (migration 058 `acked_at`); dedup via a recent-results cache. Server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via `idx_commands_agent_acked` partial index so pre-durability agents keep legacy fail-on-timeout behavior. Phase 2 (live TTY) and Phase 3 (adaptive keepalive, bulk->HTTPS, half-open eviction) remain PLANNED, not shipped.
- **Agent Comms Durability — Phase 1 COMPLETE as of 2026-07-05 (agent v0.6.63.., server v0.3.68->v0.3.97):** T1 (30s agent heartbeat) and T3 (`CommandAck` on receipt, migration 058 `acked_at`, dedup via a recent-results cache, reaper re-delivers un-acked commands past a 60s deadline) shipped 2026-06-11; T2 (server-side half-open-connection eviction sweep, 90s no-inbound threshold, 30s cadence, `conn_id`-guarded) shipped 2026-07-05 closing out the phase. Capability-gated via `idx_commands_agent_acked` partial index so pre-durability agents keep legacy fail-on-timeout behavior. Phase 2 (live TTY) and Phase 3 (adaptive keepalive, bulk->HTTPS) remain PLANNED, not shipped.
- **Pending-command TTL + status hygiene (batch 4, server v0.3.99):** commands stranded `pending` past 7 days are expired by the reaper (previously unbounded — a May-dated command could still fire on a stale agent's reconnect) with distinct never-sent vs. unconfirmed-delivery messaging. `agents.status` is now reconciled against `last_seen` by a periodic sweep (status-only conditional flip, preserves real `last_seen`, respects the 2h update-hold window, skips live dispatch-map connections, full event+SSE broadcast) — fixes zombie `'online'` rows left by unobserved disconnects.
### Volume Shadow Copy / VSS (Windows) — Policy Configurator (SPEC-016 redesign, shipped 2026-07-03, agent v0.6.76, beta channel only)
### Volume Shadow Copy / VSS (Windows) — Policy Configurator (SPEC-016 redesign, shipped 2026-07-03, agent v0.6.76+)
- **The agent no longer schedules its own create+prune loop.** It is a **configurator**: on each policy apply it idempotently sets two kernel-enforced retention governors — VSS diff-area **size cap** (native `IVssDifferentialSoftwareSnapshotMgmt`) and `MaxShadowCopies` count (evaluated, but NOT machine-globally written — see below) — and registers a scheduled task whose action is the agent's own native-COM **create** (`IVssBackupComponents::DoSnapshotSet`, `VSS_CTX_CLIENT_ACCESSIBLE`), never a delete. Retention is enforced by Windows' own FIFO eviction (`volsnap.sys`) when a governor is hit, eliminating the MITRE T1490 "Inhibit System Recovery" surface that had gotten the agent blocked by CrowdStrike Falcon.
- **Works uniformly on server AND client SKUs** (previously `vssadmin create shadow`, Server-only, gated client support) — the underlying VSS COM API and `volsnap` cap eviction are not SKU-restricted, only the `vssadmin.exe` CLI is.
- **`MaxShadowCopies` is NOT set machine-globally** (pre-merge code review caught this as a data-loss defect — it would FIFO-evict OTHER VSS consumers' shadows, e.g. System Restore or third-party backup, below the configured count). The per-volume size cap is the sole enforced governor; `retention_count`/`retention_max_age_days` policy fields are advisory/ignored (a warning fires if `retention_max_age_days` is still set, since age-based pruning requires deletion).
- On first policy apply, the legacy `GuruRMM-VSS-Snapshot` scheduled task (present fleet-wide) is removed and replaced with `GuruRMM-VSS-Create`, supporting both daily-time and every-N-hours interval triggers.
- On-demand admin operations (create/delete/browse/restore via `dispatch_operation`, `GET /agents/:id/vss/restores`) are unaffected — this redesign only removes the *scheduled* create+prune loop.
- **Not yet promoted to the stable channel** as of this compile (beta head 0.6.76; stable pinned at 0.6.66).
- Deployed to beta (0.6.76) at ship time; agent has since advanced to 0.6.78 stable-wide via later bug-fix batches — stable-channel behavior on 0.6.78 not separately re-verified this compile [verify].
### Software Inventory & Remote Uninstall (SPEC-030 — BETA, not guaranteed)
- **Maturity:** code merged to main + deployed (2026-06-23..24), but the feature is **beta and unreliable** — best-effort only.
@@ -328,12 +384,20 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
- **Fleet removal knowledge base (migrations 061-063):** per-agent removal-tracking loop, fleet three-state KB (`silent`/`requires_ui`/`unknown`), self-tuning per-target timeout.
- **Known gaps:** drivers/system components register in ARP and appear in the bulk list (no exclusion/flag yet); Launchy/AIMP-class uninstallers hang the agent on a lingering child process; no list-jobs endpoint (get-by-id only). The **av-removal-recipes spec (P3, removal-only)** is the follow-on.
### Third-Party Integrations Framework (NEW, migration 065, shipped 2026-07-01..02)
### Third-Party Integrations Framework (migration 065, shipped 2026-07-01..02)
- **Vendor-agnostic plugin framework:** `integrations` (one enabled plugin per partner+kind), `integration_credentials` (AES-256-GCM encrypted, same `Encryptor` chokepoint as the `credentials` table), `integration_client_mappings` (RMM client -> external tenant id, e.g. a Falcon CID — single-CID partners map every client to one id, MSSP partners map child CIDs), `integration_sync_state` (per-integration sync health), `integration_agent_links` (RMM agent <-> external agent id + last-synced state, e.g. Falcon AID/sensor version/RFM).
- **API:** `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `DELETE /api/integrations/:id/mappings/:mapping_id`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test`.
- **First plugin: CrowdStrike Falcon.** API client + periodic sync worker (default `sync_interval_seconds=900`) pulls sensor state into `integration_agent_links`. Framework is modeled on the proven per-partner MSPBackups shape (migration 034).
- **No dedicated dashboard UI yet** — this is API/backend-only as of this compile (verify before claiming a management screen exists).
### Policy & Configuration Management
- **Policy folders + inheritance ("policy-core", NEW, deployed 2026-07-05, migration 067, merge e7e90c2):** per-client **folder tree** replacing the fixed global->client->site->agent stack for agents placed in a folder. Exactly one root folder per client; any folder optionally references a library policy (`policy_folders.policy_id`); `agents.folder_id` places a device in the tree; effective policy is the merged ancestry root->leaf plus an optional per-asset override. Recompute+push on policy edit is scoped to affected agents only. `policy_versions` gives every policy edit an immutable snapshot for diff/rollback and per-device "what changed." **Non-breaking migration by construction:** every new column/table is nullable, `folder_id` is NULL fleet-wide until an explicit, verification-gated, dry-run-capable backfill job runs (`POST /policy-folders/backfill`) — until then effective-policy resolution stays on the legacy path. **API:** `GET/POST /clients/:id/policy-folders`, `DELETE /policy-folders/:id`, `PUT /policy-folders/:id/rename`, `PUT /policy-folders/:id/policy`, `PUT /policy-folders/:id/parent` (move), `PUT /agents/:id/folder` (placement), `POST /policy-folders/backfill`. **Explicitly out of scope this phase:** module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI — no agent-side change (wire format unchanged). **No dashboard UI yet** — a follow-on `policy-ui` shape spec (folder tree view, agent placement, effective-policy viewer with source attribution, version history, admin backfill trigger) was added 2026-07-05 (`01d962a`), implementation not started.
- **Legacy inheritance chain (agents not yet placed in a folder):** global -> client -> site -> agent; server computes merged effective policy, pushes via `ConfigUpdate`.
- Checks engine (`018`/`019`): cpu, memory, disk, ping, port, script, service. Policy-attached check templates (`024`).
- Remote registry (Windows): enumerate/read routed; write paths exist in the agent but are not routed yet [verify].
- **Event Log Watch (migration 047):** dashboard management UI at `/event-log-watches`. Policy-clobber class bug fixed (PR #43) by reusing a single rule-shaper across all config-push sites.
- **Tray policy gate (shipped 2026-07-04, code merged, policy default OFF):** `tray.enabled` policy flag is now enforced by the watchdog (previously plumbed end-to-end but not enforced — the tray launched/showed regardless of the flag). Fleet-wide materialize (deploying the binary via agent-update, not just fresh MSI) has shipped in code; the deliberate flip to enable it fleet-wide (spec's "Task 4") has NOT happened yet.
### Inventory & Discovery
- Hardware inventory (mfr/model/serial/BIOS, CPU, memory, disks, NICs, OS), software inventory, service inventory. On-demand refresh. NUL/control-char sanitized before storage (2026-07-05, see Monitoring section).
- VM / hypervisor / container detection (`032/033`).
@@ -343,8 +407,10 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
### Patch / Agent Update Management
- Self-updater: server sends version + URL + SHA256; agent downloads, verifies, atomically swaps, restarts, auto-rolls-back on failed reconnect.
- Auto-update gated on effective policy `auto_update`. Force via `POST /agents/:id/update`.
- **Channel model:** new builds tag **beta** by default; agents default to **stable**; promotion is a deliberate re-tag.
- **Channel model:** new builds tag **beta**; agents default to **stable**; promotion is a deliberate re-tag. **Fail-closed as of 2026-07-05 (BUG-023 fix, server v0.3.98):** a missing or malformed `.channel` sidecar now classifies **beta**, not stable — closing a publish/tag race where a late "MARK AS BETA" pipeline phase left a multi-minute window during which an untagged build could be mass-offered to the entire stable fleet (incident detail in Recent Work / History). `build-windows.sh` now writes the beta sidecar atomically at publish; `promote_version` keys on agent binaries (not pre-existing sidecar files) and creates/overwrites `stable` sidecars, failing the request (500) rather than recording a rollout row that disagrees with disk on a partial write.
- **Update-scanner platform coverage fixed 2026-07-05 (server v0.3.95):** the scanner's self-check previously tried to execute every binary regardless of platform to read its embedded version, dropping macOS/Linux artifacts entirely because they can't run on the Linux build host — **macOS/Linux agents had never been offered an update**. Now uses host-native-vs-foreign self-check regimes: foreign-platform binaries trust the filename-embedded version; host-native binaries stay a hard `--version` gate (with a best-effort exec-bit repair for suspected 0644 CI-copy artifacts). Post-fix: scanner indexes 8 platform/arch combos (was 4, Windows-only).
- **BUG-024 (OPEN, CRITICAL, mitigated 2026-07-05):** enabling Linux update offers for the first time (the fix above) exposed that Linux agent binaries >=0.6.77 are built against the build host's glibc 2.43 (Ubuntu 26.04) with no compatibility floor — every older-distro Linux agent that took the offer crash-looped on `GLIBC_2.39` not found (ix.azcomputerguru.com CloudLinux 9, Jupiter's Debian 12 container, a Lonestar container, presumed SL-SERVER). Mitigated by quarantining Linux binaries out of the downloads directory (scanner currently offers nothing for Linux). Real fix (glibc floor via an old-glibc build container or a musl static target, plus an `objdump -T` build gate) is pending — see History Highlights and FEATURE_ROADMAP BUG-024.
- **Pending-update supersede (batch 4, server v0.3.99):** a stale `pending` update row is now superseded at re-dispatch time when a newer version exists or its target binary was pruned from disk (guarded: only rows over 1h old with a non-empty platform index) — previously all 13 pruned targets were blindly re-offered.
- **Legacy fleet build (Server 2008R2/Win7, Rust 1.77) DISABLED 2026-07-04** (`LEGACY_ENABLED=false`): the unpinned dep re-resolve pulled edition2024 crates Rust 1.77 cannot parse, fail-closing the entire Windows build. Last legacy artifact (agent 0.6.76) frozen and preserved for existing 2008R2/Win7 endpoints; modern builds advance normally. 2008R2 support returns only via a separate legacy-only rewrite (RMM_THOUGHTS Feature 12, Raw — not approved).
- **Universal self-detecting installer (Feature 9, P1 shipped v0.6.71):** one install path self-detects arch/OS and pulls the correct binary.
- **Promotion API:** `POST /api/updates/rollouts/:version/promote` body `{"os","arch"}`; rollback via `.../rollback`.
@@ -352,14 +418,9 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
### Agent Identity & Enrollment
- Per-agent enrollment keys; site-specific MSI with SITEKEY injected. `enrolled_agents` is the authoritative enrollment history log.
- **Durable multi-location `device_id`** (Phase 1 Task 1, 2026-06-11): registry+ProgramData (Windows), /etc+/var/lib (Linux), /Library+mirror (macOS), self-healing; `cleanup.ps1` whitelists identity files.
- **Cross-site dedup fixed 2026-07-04 (commits `2851523`, `1c1cac2`):** Both WS enrollment paths previously deduped only within `(site_id, device_id)`, so re-running a different site's installer against an already-enrolled machine created a cross-site duplicate. Fixed to match by hardware `device_id` **across all sites** (`db::get_agent_by_device_id`) and re-home via `move_agent_to_site` on first match. A same-day follow-up stopped auto-re-homing **on every reconnect** — the agent re-presents its embedded install-site code as its API key on every beat, which was silently bouncing admin-relocated agents back to their original site. Current behavior: dedup by device_id, but site placement is admin-controlled and persists across reconnects. **13 duplicate-hostname agent pairs remain fleet-wide** (pre-fix legacy `win-*`/`hw-*` records) — cleanup pass not yet done; these are the ongoing trigger for `move_agent` 409s.
### Policy & Configuration Management
- Inheritance chain global -> client -> site -> agent; server computes merged effective policy, pushes via `ConfigUpdate`.
- Checks engine (`018`/`019`): cpu, memory, disk, ping, port, script, service. Policy-attached check templates (`024`).
- Remote registry (Windows): enumerate/read routed; write paths exist in the agent but are not routed yet [verify].
- **Event Log Watch (migration 047):** dashboard management UI at `/event-log-watches`. Policy-clobber class bug fixed (PR #43) by reusing a single rule-shaper across all config-push sites.
- **Tray policy gate (NEW, shipped 2026-07-04, code merged, policy default OFF):** `tray.enabled` policy flag is now enforced by the watchdog (previously plumbed end-to-end but not enforced — the tray launched/showed regardless of the flag). Fleet-wide materialize (deploying the binary via agent-update, not just fresh MSI) has shipped in code; the deliberate flip to enable it fleet-wide (spec's "Task 4") has NOT happened yet.
- **Cross-site dedup fixed 2026-07-04 (commits `2851523`, `1c1cac2`):** Both WS enrollment paths previously deduped only within `(site_id, device_id)`, so re-running a different site's installer against an already-enrolled machine created a cross-site duplicate. Fixed to match by hardware `device_id` **across all sites** (`db::get_agent_by_device_id`) and re-home via `move_agent_to_site` on first match. A same-day follow-up stopped auto-re-homing **on every reconnect** — the agent re-presents its embedded install-site code as its API key on every beat, which was silently bouncing admin-relocated agents back to their original site. Current behavior: dedup by device_id, but site placement is admin-controlled and persists across reconnects. 13 duplicate-hostname agent pairs were identified fleet-wide (pre-fix legacy `win-*`/`hw-*` records) — 4 orphaned zombies were cleaned up in batch 4 (2026-07-05); the remaining full cleanup pass is still open and is the ongoing trigger for `move_agent` 409s.
- **Delete race closed to the schema level (BUG-018 altitude fix, migration 066, server v0.3.98):** a `visible_agents` DB view (`status<>'deleting'`) is now the single source of truth for all listing/stat queries (nine call sites, including a previously-missed agent-detail-page query), a `BEFORE UPDATE` trigger pins `status='deleting'` at the schema level, and all three WS `authenticate_agent` paths reject a mid-purge row. **Note for future schema changes:** the view freezes the `agents` column list at `CREATE VIEW` time (`SELECT *`) — any `ALTER TABLE agents ADD COLUMN` migration must also recreate `visible_agents` or listings 500 fleet-wide with `ColumnNotFound` (documented in the migration 066 header).
- **Agent status reconciliation (batch 4, server v0.3.99):** a periodic sweep flips zombie `'online'` rows (agent status stranded by an unobserved disconnect) back to `'offline'` without disturbing the real `last_seen` timestamp, respecting the update-hold window and skipping agents with a live dispatch-map connection.
### Alerting & Watchdog
- Threshold alerts (ack/resolve, per-agent + fleet summary). Alert templates (`022`). Maintenance mode (`021`).
@@ -368,6 +429,7 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
- **`create_or_update_alert` hardened (2026-07-05, server v0.3.95):** now returns `Result<Option<Alert>>` — `Ok(None)` when the target alert row was deleted mid-flight (agent cascade-delete FK violation) instead of aborting the caller. Both UPDATE-by-id branches switched from `fetch_one` to `fetch_optional` with fall-through to INSERT (`refresh_alert_row` helper dedups the two near-identical UPDATE blocks). Fixed a `RowNotFound` that had been aborting the entire offline sweep whenever an alert was concurrently deleted (e.g. during duplicate-agent cleanup).
- **WebSocket log noise reclassified INFO, not ERROR (2026-07-05, server v0.3.96):** benign disconnects (`is_benign_ws_disconnect`) and pre-auth connection churn (`is_preauth_connection_churn` — sleeping laptops, scanners, half-open NAT never sending an auth frame) no longer mint `server_error` alerts. Genuine auth rejections (bad API key, malformed auth message) remain ERROR. Classified at INFO (not DEBUG) so a mass-disconnect anomaly stays visible at the server's default log level.
- **`move_agent` duplicate-key hardened (2026-07-05):** unique-violation on `idx_agents_site_device` now maps to 409 CONFLICT with an actionable message instead of a raw 500.
- **MSP360 transient-5xx noise reduction (batch 2, server v0.3.97):** sync 5xx logs WARN, escalating to ERROR only after 4 consecutive cycles (~1h) so persistent outages still alert without every cloud-maintenance blip minting a `server_error`.
### Credentials Management
- Encrypted credentials vault (`016`): scoped global/client/site, typed, metadata-only by default with a separate `/reveal` endpoint. Known HIGH item: `/reveal` ownership-scope check — [verify current state].
@@ -379,7 +441,7 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
### Backup Integration (MSP360 / MSPBackups)
- Multi-provider config, connection test, scheduled sync, per-agent + fleet coverage report, agent<->MSP360 mapping with confidence scoring.
- **Active chronic failure (unresolved as of 2026-07-04):** GND-SERVER (Grabb & Durando) — 42 `backup_failed` alerts since 2026-06-17, failing nightly ~01:02 UTC, "no error detail reported." Also open: AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial). MSP360 sync doesn't capture failure detail — investigation needs a live PowerShell dispatch to read MSP360/CloudBerry local logs.
- **Active chronic failure (unresolved as of 2026-07-05):** GND-SERVER (Grabb & Durando) — 42+ `backup_failed` alerts since 2026-06-17, failing nightly ~01:02 UTC, "no error detail reported." Also open: AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage, offline 17.7 days with an active critical alert as of batch 4), DESKTOP-EVA4H1A (Gary A Hartman, partial), Blaster2 (x2). MSP360 sync doesn't capture failure detail — investigation needs a live PowerShell dispatch to read MSP360/CloudBerry local logs.
### Remote Access (Tunnel)
- Agent side substantially built. Server side is a dead-code skeleton (no `/tunnel` routes). Live TTY design (agent-comms-durability Phase 2) will supersede the skeleton.
@@ -392,8 +454,9 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
### Platform coverage
- **Cross-platform (Win/Linux/macOS):** metrics, network state, hardware/software/service inventory, user/group + DC/domain detection, checks, discovery, self-update, scripts/commands.
- **Windows-only:** `user_session` command context, remote registry, tray-into-session (now policy-gated), watchdog SCM supervision, BSOD detection, VSS policy configurator, legacy fleet (frozen at 0.6.76).
- **macOS:** agent deployed (Phase 1); tray is a stub; **now correctly included in the update scanner's platform coverage as of 2026-07-05** (previously silently excluded).
- **Windows-only:** `user_session` command context, remote registry, tray-into-session (now policy-gated), watchdog SCM supervision, BSOD detection, VSS policy configurator, WMI ACPI thermal, legacy fleet (frozen at 0.6.76).
- **Linux:** update offers currently QUARANTINED fleet-wide pending the BUG-024 glibc-floor fix (see Patch/Update section) — all other cross-platform capabilities unaffected.
- **macOS:** agent deployed (Phase 1); tray is a stub; included in the update scanner's platform coverage since 2026-07-05.
---
@@ -403,16 +466,16 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
| Component | Location | Tech | State |
|---|---|---|---|
| Server | 172.16.3.30:3001, systemd `gururmm-server.service` | Rust, Axum | deployed, production; v0.3.96 |
| Dashboard | https://rmm.azcomputerguru.com, nginx | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production |
| Agent (Windows) | Endpoints, `GuruRMMAgent` service via WiX MSI | Rust, Windows MSVC | deployed; modern build v0.6.77, legacy (2008R2/Win7) FROZEN at 0.6.76 and disabled |
| Agent (Linux) | Endpoints, systemd `gururmm-agent` | Rust, musl static | deployed |
| Server | 172.16.3.30:3001, systemd `gururmm-server.service` | Rust, Axum | deployed, production; v0.3.99 |
| Dashboard | https://rmm.azcomputerguru.com, nginx | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production; v0.2.83 beta |
| Agent (Windows) | Endpoints, `GuruRMMAgent` service via WiX MSI | Rust, Windows MSVC | deployed; modern build v0.6.78 fleet-wide (promoted stable 2026-07-05), legacy (2008R2/Win7) FROZEN at 0.6.76 and disabled |
| Agent (Linux) | Endpoints, systemd `gururmm-agent` | Rust, musl static | deployed; **update offers QUARANTINED fleet-wide (BUG-024, glibc floor pending)** — existing installs unaffected, no new updates offered |
| Agent (macOS) | Endpoints, LaunchDaemon | Rust, aarch64/x86_64 | Phase 1 deployed 2026-05-12; code-signing issue on Apple Silicon; update-scanner platform gap fixed 2026-07-05 |
| Tray (Windows) | System tray, named pipe IPC | Rust | deployed to disk fleet-wide via agent-update (2026-07-04); **launch policy-gated, default OFF** |
| Tray (Linux) | System tray, Unix socket IPC | Rust, GTK | deployed 2026-05-24 |
| Tray (macOS) | Menu bar | Rust | stub/TODO |
| PostgreSQL DB | localhost:5432 on 172.16.3.30 | PostgreSQL 18 | deployed |
| Build pipeline | 172.16.3.30:9000 webhook + `/opt/gururmm/` scripts | Python, Bash | deployed; builds agents AND server |
| PostgreSQL DB | localhost:5432 on 172.16.3.30 | PostgreSQL 18 | deployed; migrations through 067 |
| Build pipeline | 172.16.3.30:9000 webhook + `/opt/gururmm/` scripts | Python, Bash | deployed; builds agents AND server; build-lock + gated git reset added 2026-07-05 (BUG-003) |
| Windows build — Beast (PRIMARY) | GURU-BEAST-ROG over Tailscale | Rust MSVC, WiX 4.x | primary — two-wave parallel, ~336s |
| Windows build — Pluto (FALLBACK) | 172.16.3.36 | Rust MSVC, WiX v4 | fallback only |
@@ -423,16 +486,20 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
- **Server binary:** `/opt/gururmm/gururmm-server` on 172.16.3.30
- **Server config:** `/opt/gururmm/.env`
- **New shared sanitize module:** `server/src/db/sanitize.rs` (`strip_nul`, `clean_control_chars`) — consumed by `db/logs.rs` and `db/inventory.rs`
- **Visible-agents view + delete-race schema fix:** `server/migrations/066_visible_agents_view.sql` (view + BEFORE UPDATE trigger; column-freeze warning in the header — any `ALTER TABLE agents ADD COLUMN` must recreate the view)
- **Policy folders / policy-core:** `server/migrations/067_policy_folders.sql` (`policy_folders`, `agents.folder_id`, `policy_versions`), `server/src/api/policies.rs` (folder CRUD + placement + backfill), `docs/specs/policy-core/`, `docs/specs/policy-ui/` (dashboard UI spec, not implemented)
- **Integrations framework:** `server/src/api/integrations.rs`, `server/src/integrations/` (plugin trait + registry), migration `065_integrations_framework.sql`
- **VSS configurator:** `agent/src/vss_com.rs`, `agent/src/vss.rs`, `specs/vss-policy-config/`, `specs/vss-native-com/`
- **Tray policy-deploy:** `agent/src/tray_stage.rs`, `agent/src/tray_state.rs`, `agent/src/authenticode.rs`, `specs/tray-policy-deploy/`
- **Downloads dir:** `/var/www/gururmm/downloads/` on 172.16.3.30
- **Windows WMI thermal:** `agent/src/` (BUG-001 `collect_temps_windows_acpi`, `wmi` crate dep)
- **Downloads dir:** `/var/www/gururmm/downloads/` on 172.16.3.30 (Linux binaries currently quarantined to a sibling `quarantine/` dir pending BUG-024)
- **Webhook handler:** `/opt/gururmm/webhook-handler.py`
- **Deploy pipeline (canonical, auto-synced to /opt/gururmm on push):** `deploy/build-pipeline/build-server.sh` (flock lock, PIPESTATUS-gated git reset, `--migrate-only` pre-stop validation — BUG-003), `deploy/build-pipeline/build-windows.sh` (atomic beta `.channel` sidecar write — BUG-023)
- **API (internal):** http://172.16.3.30:3001
- **API (external):** https://rmm-api.azcomputerguru.com
- **Dashboard:** https://rmm.azcomputerguru.com
- **Vault paths:** `infrastructure/gururmm-server.sops.yaml` (API creds, DB password, sudo password), `infrastructure/gururmm-server-physical` (SSH ed25519 key — this fleet key works on 172.16.3.30 currently, not just the future physical box; confirmed 2026-07-04)
- **SSH to prod:** `ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30` (ed25519, key-only; password auth disabled)
- **SSH to prod:** `ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30` (ed25519, key-only; password auth disabled for SSH; sudo over SSH still requires the vault password via `-S`)
- **Pre-merge verification skill:** `bash .claude/skills/gururmm-build/scripts/verify.sh server|agent|dashboard|migrations`
### Repo Structure
@@ -441,36 +508,38 @@ Agent<->server communication is a persistent authenticated WebSocket with auto-r
gururmm/
├── agent/ Rust agent (managed endpoints)
│ └── src/
│ ├── authenticode.rs NEW — Authenticode signature verify (tray-policy-deploy)
│ ├── tray_stage.rs NEW — tray materialize FSM + staging + versioned mutex
│ ├── tray_state.rs NEW — tray policy state
│ ├── authenticode.rs Authenticode signature verify (tray-policy-deploy)
│ ├── tray_stage.rs tray materialize FSM + staging + versioned mutex
│ ├── tray_state.rs tray policy state
│ ├── bsod.rs Windows BSOD/kernel-crash detection
│ ├── vss_com.rs Native VSS COM create/provision (vss-native-com + vss-policy-config)
│ ├── vss_com.rs Native VSS COM create/provision
│ ├── vss.rs VSS policy configurator entry points
│ ├── commands/mod.rs CommandExecutor + recent-results cache (ACK dedup)
│ ├── device_id.rs Durable multi-location device_id
│ ├── transport/
│ │ └── websocket.rs WS transport; execute_command ACK + dedup; cmd-alias NAK
│ ├── tunnel/ TunnelManager state machine
│ ├── metrics/ sysinfo collection + temps
│ ├── metrics/ sysinfo collection + temps; WMI ACPI thermal (BUG-001)
│ ├── registry_ops/ Windows registry read/write
│ ├── updater/ Self-update handler (container guard, BUG-019)
│ ├── updater/ Self-update handler (container guard BUG-019, unix rollback watchdog)
│ └── main.rs
├── server/ Rust/Axum API server
│ └── src/
│ ├── api/
│ │ ├── integrations.rs NEW — integrations framework CRUD (crowdstrike-falcon Task 3)
│ │ └── ... ~40 route modules; software.rs SPEC-030
│ ├── alerts/ Alerting modules (offline.rs: offline_sweep + mass_offline)
│ │ ├── integrations.rs integrations framework CRUD (crowdstrike-falcon)
│ │ ├── policies.rs policy CRUD + folder tree + placement + backfill (policy-core, NEW)
│ │ └── ... 39 route modules; software.rs SPEC-030
│ ├── alerts/ Alerting modules (offline.rs: offline_sweep + mass_offline + status reconciler)
│ ├── db/
│ │ ├── sanitize.rs NEW — shared NUL/control-char sanitize (2026-07-05)
│ │ ├── sanitize.rs shared NUL/control-char sanitize
│ │ ├── alerts.rs create_or_update_alert -> Option<Alert>, refresh_alert_row
│ │ ├── inventory.rs NUL-scrub over jsonb + text before upsert
│ │ ├── agents.rs visible_agents-view reads, status write guards
│ │ └── software_jobs.rs SPEC-030 async removal jobs
│ ├── updates/
│ │ └── scanner.rs host-native vs foreign self-check regimes (2026-07-05)
│ ├── ws/mod.rs WS handler; is_benign_ws_disconnect / is_preauth_connection_churn
│ └── mspbackups/ MSP360 backup integration
│ │ └── scanner.rs host-native vs foreign self-check regimes; fail-closed channel default (BUG-023)
│ ├── ws/mod.rs WS handler; is_benign_ws_disconnect / is_preauth_connection_churn; deleting-row auth rejection; half-open eviction sweep (T2)
│ └── mspbackups/ MSP360 backup integration; transient-5xx WARN escalation
├── dashboard/ React/TypeScript UI
│ ├── pages/
│ │ └── EventLogWatches.tsx
@@ -480,9 +549,10 @@ gururmm/
├── tray/ System tray binary
├── installer/ WiX v4 MSI; cleanup.ps1; universal self-detecting installer
├── deploy/
│ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh (LEGACY_ENABLED=false gate added 2026-07-04)
│ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh (flock+PIPESTATUS+migrate-only, LEGACY_ENABLED=false)
├── scripts/ Build/ops scripts
└── docs/ FEATURE_ROADMAP.md, UI_GAPS.md, RMM_THOUGHTS.md, specs/
docs/specs/ policy-core/, policy-ui/, SPEC-025-policy-compliance-posture.md, ...
specs/ tray-policy-deploy/, vss-policy-config/, vss-native-com/, crowdstrike-falcon/,
av-removal-recipes/, remote-software-uninstall/, agent-comms-durability/, ...
```
@@ -493,20 +563,23 @@ gururmm/
### Current Focus
As of 2026-07-04 (agent 0.6.77 / server v0.3.96):
As of 2026-07-05 (agent 0.6.78 / server v0.3.99):
- **Machine-side backup failures (highest user priority, unresolved):** GND-SERVER chronic 42-failure streak, AD1, LAB-SVR, DESKTOP-EVA4H1A — MSP360 sync reports no error detail; needs a live PowerShell dispatch to read local MSP360/CloudBerry logs or the mspbackups.com console.
- **13 duplicate-hostname agent pairs fleet-wide** (legacy `win-*`/`hw-*` device_ids alongside re-enrolled hardware-UUID records) — the ongoing trigger for `move_agent` 409s; a one-time cleanup pass (delete stale offline duplicates) is not yet done.
- **WS auth-timeout source unidentified:** ~1/min unauthenticated connection attempts, now correctly classified INFO (not alerting), but the source is still unknown — would need connection-level source-IP visibility (Feature 5, server-side public-IP capture, still unapproved) or box-level tcpdump.
- **BUG-024 real fix (glibc floor for Linux builds):** Linux binaries remain quarantined out of downloads — no Linux agent gets update offers until an old-glibc build container (or musl static target) plus an `objdump -T` build gate lands. SL-SERVER (Scileppi) and Lonestar's container host still need manual agent recovery (no vault SSH creds for either).
- **Machine-side backup failures (highest user priority, unresolved):** GND-SERVER chronic 42+-failure streak, AD1, LAB-SVR (offline 17.7 days as of batch 4), DESKTOP-EVA4H1A, Blaster2 (x2) — MSP360 sync reports no error detail; needs a live PowerShell dispatch to read local MSP360/CloudBerry logs or the mspbackups.com console.
- **Duplicate-hostname agent cleanup:** 4 orphaned zombies cleared in batch 4; the remaining pairs of the original 13 fleet-wide are the ongoing trigger for `move_agent` 409s — a full one-time cleanup pass is not yet done.
- **4 chronic one-way-WS loopers (T2 field data):** DESKTOP-3USU20B (Safesite Glendale, 79/127 evictions), DESKTOP-BA45FRS (Safesite Bell), DDDOffice072023 (ACG discovery), DESKTOP-PL2RCGL (Robyn Pittman) — agent->server WS frames eaten post-auth, version-independent, server-side telemetry exhausted; packet capture on an affected box is the next diagnostic step.
- **Policy-ui (folder management dashboard UI):** shape spec added 2026-07-05 (`docs/specs/policy-ui/`); implementation not started. Policy-core backend is deployed but has no dashboard surface yet.
- **Tray policy fleet enablement (Task 4, tray-policy-deploy spec):** materialize code is merged and deployed; flipping `tray.enabled` on for the fleet is a deliberate next step, not yet done.
- **VSS configurator stable-channel promotion:** shipped to beta (0.6.76) only; promotion to stable is a separate deliberate step pending further soak.
- **Legacy (2008R2/Win7) agent rewrite (RMM_THOUGHTS Feature 12, Raw):** not approved; needs a language/runtime decision (.NET Framework 4.x vs C++) and scope decision (trimmed feature set: enroll, check-in, metrics, command exec, self-update only) before a `/shape-spec`.
- **Legacy (2008R2/Win7) agent rewrite (RMM_THOUGHTS Feature 12, Raw):** not approved; needs a language/runtime decision (.NET Framework 4.x vs C++) and scope decision before a `/shape-spec`.
- **Integrations framework dashboard UI:** CrowdStrike Falcon backend + sync worker are live; no management screen exists yet in the dashboard.
- **SPEC-030 software-uninstall follow-on (Route A / recipes):** Launchy/AIMP-class uninstallers still hang the agent; driver/system-component exclusion flag and a list-jobs endpoint remain unbuilt.
- **Agent-comms-durability Phase 2/3 (planned):** live TTY, adaptive keepalive, bulk file transfer -> short-lived HTTPS, server half-open eviction sweeper.
- **Agent-comms-durability Phase 2/3 (planned):** live TTY, adaptive keepalive, bulk file transfer -> short-lived HTTPS. Phase 1 is now fully complete (T1/T2/T3 all shipped).
- **Phase-1 acceptance test:** interactive command dispatch to PST-SERVER / Safesite boxes during an idle window, still pending across three consecutive session logs.
- **Durable agent identity Phase 2-3 (pending soak):** guarded auto-reclaim + an operator merge tool for the pre-fix ghost/duplicate agents.
- **SPEC-028 offboarding wizard:** specification complete; implementation pending.
- **Model routing correction (2026-07-04, logged --correction):** Tier-0 simplify/cleanup passes should route to local Ollama before Claude; a routing miss on this session's easy-bugs batch was logged for tracking.
- **Repo-root `build-server.sh`:** a stale divergent lineage of the canonical `deploy/build-pipeline/build-server.sh` — flagged as a deletion candidate to end confusion about which script is authoritative.
- **KSTEENBB2025:** fleet-max `cpu_temp` peaking 83.8C — worth a glance once thermal data has a wider baseline.
### Patterns & Anti-Patterns
@@ -514,12 +587,18 @@ As of 2026-07-04 (agent 0.6.77 / server v0.3.96):
| Pattern | What Went Wrong |
|---|---|
| Publish-then-tag-later build pipeline ordering (BUG-023) | Binaries landed in the downloads dir minutes before a late "MARK AS BETA" phase tagged them; the scanner's missing-sidecar default was `stable`, so a 5-minute scan window could mass-offer an unsoaked build to the entire stable fleet. Fix: fail-closed scanner default (missing/malformed sidecar => beta) AND atomic sidecar write at publish — defense in depth, not either alone. |
| Promotion logic keyed on pre-existing sidecar file content rather than the binary itself | `promote_version` only rewrote files whose content was exactly `beta`, so a fail-closed (sidecar-less) binary became permanently unpromotable the moment the scanner default flipped. Key promotion on the binary set, create/overwrite the sidecar, and fail loudly (500) on a partial write rather than recording a DB rollout row that disagrees with disk. |
| Building Linux agent binaries against the build host's live glibc with no floor (BUG-024) | The build host's OS upgrade (Ubuntu 26.04, glibc 2.43) silently raised the minimum glibc every Linux agent binary requires; combined with a same-week fix that enabled Linux update offers for the first time, every older-distro Linux agent that took the update crash-looped on a missing GLIBC symbol. Any cross-compiled/host-compiled binary needs an explicit compatibility floor (old-glibc container or static linking) verified by a build-time `objdump -T` gate, not an assumption that "it compiled" means "it runs everywhere." |
| Hand-copied `WHERE status <> 'deleting'` guards scattered per query/UPDATE | The same forgotten-guard class recurred (BUG-018) even after a prior "fix" — nine call sites needed the filter and one (agent detail page) was missed. Fixed at the schema level: a view for reads, a `BEFORE UPDATE` trigger for writes, so no future query/update can forget the guard. Tradeoff accepted: a `SELECT *` view freezes the column list at creation time — documented as a loud warning for future migrations. |
| Update-scanner self-check executing every binary regardless of target platform | macOS/Linux binaries can't run on the Linux build/scan host, so they were silently dropped from the scannable set entirely — those agents never received an update offer. Fix: host-native binaries get a hard `--version` execute-and-check gate; foreign-platform binaries trust the filename-embedded version. |
| Writing a machine-global VSS registry governor (`MaxShadowCopies`) to a per-policy `retention_count` | `MaxShadowCopies` is shared by ALL VSS consumers on the box (System Restore, third-party backup) — lowering it FIFO-evicts THEIR shadows too, a genuine data-loss class caught only by a 23-agent pre-merge code review. Use a per-volume size cap as the sole enforced governor instead. |
| Re-homing an agent's site on every WS reconnect after a device_id-based dedup fix | The agent re-presents its embedded install-site code as its API key on every beat; auto-re-homing on each connect silently bounces an admin-relocated agent back to its original install site. Dedup by device_id, but leave site placement admin-controlled and persistent. |
| `fetch_one` on an UPDATE-by-id inside a concurrently-deleting table | An agent cascade-delete can remove the target row between a SELECT and the UPDATE; `fetch_one` panics `RowNotFound` and aborts the whole caller (here: the entire offline sweep). Use `fetch_optional` with a fall-through path. |
| Two sessions editing the same branch's working tree without a coord lock | A concurrent session's uncommitted edits were silently absorbed into another session's commit. Benign this time (same branch, all changes verified at HEAD) but neither session checked the coord lock list first — take a lock before editing a shared submodule branch. |
| Assuming a spec's `shape.md`/`plan.md` header "Status: not started" reflects current reality | Tasks can ship and merge (git log shows Tasks 1-3+5 done) while the spec doc header is never updated. Verify against `git log`/deployed version, not the spec file's status line. |
| Two sessions editing the same branch's working tree without a coord lock | A concurrent session's uncommitted edits were silently absorbed into another session's commit more than once across this window. Benign each time (same branch, all changes verified at HEAD) but a coord lock was only reliably taken starting with bug-fix batch 3 — take a lock before editing a shared submodule branch. |
| Assuming a spec's `shape.md`/`plan.md` header "Status: not started" reflects current reality | Tasks can ship and merge (git log shows work done) while the spec doc header is never updated. Verify against `git log`/deployed version, not the spec file's status line. |
| Testing a pipe's exit code (`if ! cmd \| tee ...`) instead of the upstream command's | Shell tests the last command in the pipe (`tee`), not `git`; a failed `git fetch`/`reset` could report success. Use the shell's `PIPESTATUS` array explicitly. |
| Mischaracterizing "19 known test failures" as a single homogeneous class | All 19 were assumed to be the same pre-existing `DATABASE_URL`-missing environment gap; one was a genuine `matches_pattern` panic on no-wildcard input hiding in the same failure count. Bucket-count assumptions on a recurring "known failures" number should be re-verified periodically, not re-quoted. |
**Existing anti-patterns (carried forward):**
@@ -536,18 +615,21 @@ As of 2026-07-04 (agent 0.6.77 / server v0.3.96):
| `+1.77` legacy builds without pinned deps | Fresh dep resolution repeatedly pulls edition2024 crates Rust 1.77 can't parse (BUG-021, recurred 2026-07-04 with a different dep pair, led to the variant being disabled outright rather than re-chasing pins). |
| Dead WebSocket write half | WS write fails, send task dies, receive loop keeps agent connected with a dead write half. Fix: `tokio::select!` monitoring both tasks. |
| Using the `minidump` crate for Windows kernel dumps | Only parses Breakpad MDMP, not Windows kernel PAGEDU64. Parse `DUMP_HEADER64`/`32` at fixed offsets directly. |
| Build classification defaulting to stable | New builds default `beta`; promotion is an explicit re-tag. |
| Build classification defaulting to stable | Superseded 2026-07-05 (BUG-023) — the scanner's missing-sidecar default is now beta, not stable; new builds are tagged beta by the pipeline and promotion is an explicit re-tag on top of that. |
| `command_type: "cmd"` in API commands | Not a valid `CommandType` variant originally — agent silently dropped the message. Now aliased to `Shell`. |
| Reading the stale submodule working tree for code | The submodule gitlink intentionally lags `origin/main`. Always read via `git -C ... show origin/main:<path>` or a worktree. |
**Good patterns (carried forward + new):**
- **Shared sanitize chokepoint** (`db/sanitize.rs`, 2026-07-05) — one NUL/control-char scrubber consumed by both `db/logs.rs` and `db/inventory.rs`, replacing duplicated ad hoc scrub logic.
- **Capability-based classification (host-native vs foreign) instead of a single universal check** — the update scanner now branches its self-check strategy per target platform rather than applying one strategy fleet-wide.
- **Multi-AI adversarial review before shipping risky agent-side changes** — the VSS redesign (Grok + Gemini) and tray redesign (17-agent Claude review + Grok + Gemini) both caught genuine defects (a data-loss governor and 9 architecture defects respectively) that a single-model review missed on the first pass.
- **Push guards to the schema, not the call site** (2026-07-05) — the `visible_agents` view + `BEFORE UPDATE` trigger closed a class of forgotten-guard bugs that a second round of hand-copied `WHERE` clauses had already failed to fully close once.
- **Fail-closed defaults on ambiguous/missing state** (BUG-023 scanner fix) — treating a missing `.channel` sidecar as the risky classification (beta) rather than the safe-looking one (stable) makes a future pipeline crash harmless instead of dangerous.
- **Shared sanitize chokepoint** (`db/sanitize.rs`) — one NUL/control-char scrubber consumed by both `db/logs.rs` and `db/inventory.rs`, replacing duplicated ad hoc scrub logic.
- **Capability-based classification (host-native vs foreign) instead of a single universal check** — the update scanner branches its self-check strategy per target platform rather than applying one strategy fleet-wide.
- **Multi-AI adversarial review before shipping risky agent-side changes** — the VSS redesign (Grok + Gemini) and tray redesign (17-agent Claude review + Grok + Gemini) both caught genuine defects (a data-loss governor and 9 architecture defects respectively) that a single-model review missed on the first pass; the 2026-07-04..05 batches ran 15-26-agent high-effort code-review workflows before every fleet-deploy merge and consistently surfaced findings (data-loss guards, race conditions, unpromotable-binary edge cases) that first-pass implementations missed.
- **Platform parity rule** — any agent feature goes on Windows + Linux + macOS in the same commit.
- **`gururmm-build` skill for pre-merge verification** — does NOT trigger production build; merging to main is the only prod trigger.
- **Dashboard beta-before-main rule** — dashboard changes go to beta (auto-built from main) first; promote explicitly.
- **Non-breaking migrations by construction for structural changes** (policy-core, migration 067) — new nullable columns/tables plus a deliberate, verification-gated, dry-run-capable backfill job means landing a large schema change changes zero live behavior until an explicit flip.
- **Two independently-deployable slices for cross-component features, capability-gated** — agent slice and server slice can deploy in any order; old agents/servers keep legacy behavior until the gate flips.
### Build & Deploy
@@ -558,37 +640,47 @@ As of 2026-07-04 (agent 0.6.77 / server v0.3.96):
Gitea push to main
-> webhook-handler.py (172.16.3.30:9000)
-> build-shared.sh (auto-version bump)
-> build-linux.sh
-> build-linux.sh (Linux artifact publish currently QUARANTINED post-build pending BUG-024 glibc floor)
-> build-windows.sh (Beast PRIMARY over Tailscale || Pluto FALLBACK
TWO-WAVE PARALLEL BUILD; legacy wave now gated LEGACY_ENABLED=false)
TWO-WAVE PARALLEL BUILD; legacy wave gated LEGACY_ENABLED=false;
deploy_and_sign writes the beta .channel sidecar ATOMICALLY per artifact — BUG-023 fix)
-> build-mac.sh (stub — no build machine configured yet)
-> build-server.sh (gated on server/ changes; backup+rollback)
-> artifacts -> /var/www/gururmm/downloads/ with sha256 + -latest symlinks + .channel (beta)
-> build-server.sh (gated on server/ changes; flock lock, PIPESTATUS-gated git fetch/reset,
pre-stop --migrate-only migration validation, backup+rollback — BUG-003 fix)
-> artifacts -> /var/www/gururmm/downloads/ with sha256 + -latest symlinks + .channel
(missing/malformed .channel now scans as beta, not stable — BUG-023 fix)
```
**BUG-003 closed 2026-07-05 (`6ccd35e`):** the deploy script confirmed to auto-sync to `/opt/gururmm` on every push. Gained a `flock` build lock, `PIPESTATUS`-gated `git fetch`/`reset` (fired correctly on its first live run — a merge-webhook build collided with the CI version-bump job's `index.lock` and aborted cleanly with nothing deployed, instead of silently building a stale tree), and pre-stop migration validation via a new `server --migrate-only` flag (validates+applies migrations against the live DB while the OLD binary keeps serving; paired with `set_ignore_missing(true)` on the migrator so a rollback binary tolerates an ahead DB instead of re-creating the outage the gate was meant to prevent).
**BUG-023 closed 2026-07-05:** `build-windows.sh`'s `deploy_and_sign` now writes the beta `.channel` sidecar atomically with each published artifact, closing the multi-minute publish-then-tag window that had let an untagged build get scanned as stable.
**Legacy variant disabled 2026-07-04 (`84a82a3`):** `LEGACY_ENABLED=false` in `build-windows.sh` gates the legacy wave/copy/deploy/symlink off entirely; the last-built legacy artifact (0.6.76) is preserved and excluded from cleanup so existing 2008R2/Win7 endpoints keep receiving that frozen build. Modern (amd64/x86/tray/debug) builds are no longer blocked by legacy dep-resolution failures.
**Channel/promotion model, downloads layout, dashboard beta/prod split, DB migration hazards, and Windows build host detail are unchanged from the 2026-06-25 compile** — see prior History Highlights / Compilation Notes for the full mechanics; current migration count is 065 (`065_integrations_framework`).
**Channel/promotion model, downloads layout, dashboard beta/prod split, and Windows build host detail otherwise unchanged from the 2026-07-04 compile** — see History Highlights / Compilation Notes for full prior mechanics; current migration count is 067 (`067_policy_folders`).
---
## Active State
**Fleet (as of 2026-07-04):**
- 359 enrolled agents; 228 fresh/online at last sample
- Agent 0.6.77 modern / 0.6.76 legacy-frozen-and-disabled; server v0.3.96
- Active `server_error` alerts: **0** (cleaned to zero by the 2026-07-05 deploys)
- 13 duplicate-hostname agent pairs outstanding (not yet cleaned up)
- 5 active `backup_failed` alerts (GND-SERVER chronic, AD1, LAB-SVR, DESKTOP-EVA4H1A) — investigation open
- VSS configurator on beta channel only (0.6.76, 8 agents); stable fleet pinned at 0.6.66 (233 agents)
- Beta channel: site "Mike's Car" (`103c10b9-c1de-4dd8-b382-b8362ed3143e`) has `update_channel='beta'` (persists across re-enrollment)
**Fleet (as of 2026-07-05):**
- ~360 enrolled agents; ~229 fresh/online at last sample
- Agent 0.6.78 modern fleet-wide (stable, promoted 2026-07-05) / 0.6.76 legacy-frozen-and-disabled; server v0.3.99; dashboard v0.2.83 beta
- Active `server_error` alerts: **0**
- Duplicate-hostname agent pairs: 4 orphaned zombies cleaned in batch 4; remaining pairs of the original 13 still outstanding
- Active `backup_failed` alerts: GND-SERVER (chronic), AD1, LAB-SVR (offline 17.7 days), DESKTOP-EVA4H1A, Blaster2 (x2) — investigation open
- Linux update offers QUARANTINED fleet-wide (BUG-024) — existing Linux installs unaffected, no new offers until the glibc-floor fix ships
- 4 chronic one-way-WS loopers under active diagnosis (DESKTOP-3USU20B, DESKTOP-BA45FRS, DDDOffice072023, DESKTOP-PL2RCGL) — packet capture pending
- Policy-core (folder tree) deployed but `agents.folder_id` is NULL fleet-wide — no agent is yet on the new inheritance path pending the backfill job
- SL-SERVER (Scileppi) and Lonestar's container host down pending manual agent recovery (no vault SSH creds for either)
**API auth:**
- `POST /api/auth/login` -> JWT (~24h)
- Creds: vault `infrastructure/gururmm-server.sops.yaml`
- Key endpoints: `GET /api/agents`, `POST /api/agents/:id/command`, `GET /api/commands/:id`, `POST /api/agents/:id/update`, `POST /api/updates/rollouts/:version/promote`
- Software (SPEC-030): `GET /api/agents/:id/software`, `POST /api/agents/:id/software/uninstall` (-> job_id), `GET .../uninstall/jobs/:job_id`, `GET .../removal-status`, `GET /api/software/knowledge`
- **Integrations (NEW):** `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test`
- Integrations: `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test`
- **Policy folders (NEW):** `GET/POST /api/clients/:id/policy-folders`, `DELETE /api/policy-folders/:id`, `PUT /api/policy-folders/:id/rename`, `PUT /api/policy-folders/:id/policy`, `PUT /api/policy-folders/:id/parent`, `PUT /api/agents/:id/folder`, `POST /api/policy-folders/backfill`
- Command fields: `command_type` (`shell`/`powershell`/`python`/`script`/`claude_task`/`cmd` alias), `command`, optional `context` (`system`/`user_session`), `timeout_seconds`/`elevated`.
**Dashboard — complete and working:**
@@ -600,7 +692,8 @@ Agents management, Clients/Sites CRUD, Commands execution + terminal, Logs + AI
- Tunnel session management (server skeleton)
- Offboarding wizard UI (SPEC-028 complete; implementation pending)
- Alert mute UI — server endpoints ready, dashboard NOT started
- **Integrations Center UI (NEW gap)** — CrowdStrike Falcon framework + sync worker are live server-side; no management screen in the dashboard yet
- Integrations Center UI — CrowdStrike Falcon framework + sync worker are live server-side; no management screen in the dashboard yet
- **Policy folders / policy-ui (NEW gap)** — policy-core backend + folder-tree API deployed 2026-07-05; the `policy-ui` shape spec exists but no implementation has started, so folder placement/inheritance is API-only today
- **Tray policy toggle surfacing** — `tray.enabled` is enforced server/agent-side; verify whether the policy UI exposes the toggle to operators [verify]
**Security backlog (HIGH):**
@@ -615,7 +708,7 @@ These decisions are locked. Do not reverse without explicit user approval.
1. **Per-agent enrollment keys.**
2. **Site-specific MSI generation.**
3. **No TOML/config for endpoints** — server URL compiled into binary.
4. **Policy inheritance chain** — global -> site -> client -> agent.
4. **Policy inheritance chain** — global -> site -> client -> agent (legacy path); a device may instead be placed in a per-client **policy folder** (added 2026-07-05, decision 14) inheriting its folder-ancestry chain — the two models coexist during migration via `agents.folder_id`.
5. **Platform parity rule** — any agent feature ships on Windows, Linux, and macOS in the same change.
6. **Watchdog as separate process** — reports via REST (`POST /watchdog-alert`), no WebSocket.
7. **Build pipeline is the only path to production.**
@@ -625,6 +718,8 @@ These decisions are locked. Do not reverse without explicit user approval.
11. **Durability-first command delivery** — the durable Postgres `commands` row is the source of truth; the WebSocket is only a delivery hint.
12. **VSS: configure, never delete on a schedule** (added 2026-07-03) — the agent may only set retention governors and create shadows on a schedule; scheduled or automatic deletion reintroduces the MITRE T1490 surface. On-demand admin-initiated delete stays allowed.
13. **Third-party security integrations are a plugin, not a schema** (added 2026-07-01) — a new vendor implements the `integrations` plugin trait against the generic framework tables; it does not get its own bespoke migration set.
14. **Policy scoping is a folder tree, not a fixed level stack** (added 2026-07-05) — a client's policy structure is an arbitrary-depth folder tree per client (policy-core); the legacy fixed global->client->site->agent stack remains the resolution path for any agent not yet placed in a folder, and migration between the two models is backfill-gated, never automatic or silent.
15. **Update-channel classification fails closed** (added 2026-07-05, BUG-023) — a missing or malformed `.channel` sidecar classifies as beta, never stable; only an explicit `stable` sidecar (written only by the atomic-publish path or an operator promotion) offers a build to the default fleet.
---
@@ -649,7 +744,7 @@ These decisions are locked. Do not reverse without explicit user approval.
| 2026-06-02 | Server 0.3.37 + migration 048 deployed. Webhook wired to dispatch build-server.sh. |
| 2026-06-04 | BUG-020 (ghost tray icons) fixed to beta (`137dd85`). |
| 2026-06-07 | Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit all shipped same day. |
| 2026-06-11 | Physical server migration executed (<27 min downtime). Ghost agent root-caused, durable identity Phase 1 Task 1 shipped. Agent-comms-durability Phase 1 shipped (agent v0.6.63/server v0.3.68). |
| 2026-06-11 | Physical server migration executed (<27 min downtime). Ghost agent root-caused, durable identity Phase 1 Task 1 shipped. Agent-comms-durability Phase 1 (T1/T3) shipped (agent v0.6.63/server v0.3.68). |
| 2026-06-12 | Beast parallel Windows build wired in (`cfbdb59`, ~336s). command_type "cmd" root-caused and fixed (`3de9faf`). |
| 2026-06-15 | BSOD dedup key changed to bugcheck code. MSI EXDEV fix. sync.sh Phase 1 auto-heal. logs/analyze switched to Claude API. 500-error-leak closed. |
| 2026-06-18 | sync.sh submodule auto-heal verified fleet-wide. |
@@ -661,13 +756,20 @@ These decisions are locked. Do not reverse without explicit user approval.
| 2026-07-01..02 | Vendor-agnostic Integrations Framework shipped (migration 065) with CrowdStrike Falcon as the first plugin — API client, client mappings, periodic sync worker. No dashboard UI yet. |
| 2026-07-03..04 | VSS Policy Configurator (SPEC-016 redesign) shipped to beta (agent 0.6.76) — replaces scheduled create+prune with a configure-governors-only model, eliminating the T1490 surface; 23-agent code review caught and fixed a data-loss-class `MaxShadowCopies` defect pre-merge; verified live on the T1490 canary NEPTUNE. |
| 2026-07-04 | Legacy (2008R2/Win7, Rust 1.77) build variant disabled (`84a82a3`) after an edition2024 dep recurrence fail-closed the whole Windows build; last legacy artifact frozen at 0.6.76. RMM_THOUGHTS Feature 12 (legacy agent rewrite) added Raw. Cross-site agent dedup fixed (`2851523`, `1c1cac2`) — dedup by device_id across all sites, no re-home on reconnect. Policy-Controlled Tray Deployment merged (`f6a4251`) — materialize shipped fleet-wide, launch stays policy-gated default OFF. |
| 2026-07-04..05 | Easy-bugs batch + full alert triage: update-scanner host-native/foreign self-check regimes (fixed macOS/Linux agents never receiving update offers), inventory NUL/control-char sanitize (`db/sanitize.rs`, fixed 7+ machines' silently-failing inventory), WS benign-disconnect + pre-auth churn reclassified ERROR->INFO, `create_or_update_alert` survives cascade-delete races, `move_agent` maps duplicate-key to 409. Deployed as server v0.3.95 + v0.3.96. Active `server_error` alerts reduced to 0. Machine-side backup-failure investigation (GND-SERVER et al.) started but not completed. |
| 2026-07-04..05 | Easy-bugs batch + full alert triage: update-scanner host-native/foreign self-check regimes (fixed macOS/Linux agents never receiving update offers), inventory NUL/control-char sanitize (`db/sanitize.rs`), WS benign-disconnect + pre-auth churn reclassified ERROR->INFO, `create_or_update_alert` survives cascade-delete races, `move_agent` maps duplicate-key to 409. Deployed as server v0.3.95 + v0.3.96. Active `server_error` alerts reduced to 0. |
| 2026-07-05 (00:45-01:00 PT) | Repair batch 2 merged + deployed (`76a6bd2`, server v0.3.97, agent 0.6.78): comms-durability Phase 1 completed (T2 half-open eviction sweep); BUG-001 Windows WMI ACPI thermal SHIPPED and PROVEN live on 112+ agents; BUG-020 uninstall teardown fixed; BUG-018 delete-race resurrection paths closed; MSP360 transient-5xx downgraded to WARN. |
| 2026-07-05 | BUG-023 discovered and fixed (Howard): update-channel publish/tag race had mass-offered unsoaked beta 0.6.78 to 209 stable agents; fixed with a fail-closed scanner default, an atomic sidecar write at publish, and a binaries-keyed `promote_version`. 0.6.78 manually promoted to stable, healing the split fleet (229 agents converged). |
| 2026-07-05 | BUG-003 closed (Howard, server v0.3.98): deploy pipeline hardened with a build lock, PIPESTATUS-gated git reset, and pre-stop `--migrate-only` migration validation — the gate fired correctly on its first live run (aborted cleanly on a CI index.lock collision instead of building a stale tree). |
| 2026-07-05 | BUG-018 altitude fix shipped (Howard, migration 066, server v0.3.98): `visible_agents` DB view + `BEFORE UPDATE` schema trigger replace nine scattered inline `status<>'deleting'` filters; all three WS auth paths reject mid-purge rows. |
| 2026-07-05 (11:05 PT) | Bug-fix batch 4 merged (Howard, server v0.3.99): pending-command 7-day TTL, agent-status reconciler for zombie 'online' rows, stale pending-update supersede logic. BUG-024 discovered (CRITICAL, OPEN): Linux agent binaries built against the build host's newer glibc crash-looped every older-distro Linux agent that took the newly-enabled update offer; mitigated via quarantine, real glibc-floor fix pending. |
| 2026-07-05 | Policy-core (policy folders + inheritance) deployed to production (Mike, merge `e7e90c2`, migration 067 renumbered from 066): per-client folder tree with inheritance resolution, versioning, scoped recompute+push, and a gated/dry-run backfill — non-breaking by construction. Follow-on policy-ui (dashboard UI) shape spec added same day (`01d962a`), not yet implemented. |
---
## Compilation Notes
- **2026-07-04 recompile (Howard-Home/claude-main):** Delta from 2026-06-25 (migrations 64 -> 65; agent 0.6.75 -> 0.6.77 modern / 0.6.76 legacy-frozen; server 0.3.87 -> 0.3.96; fleet 270 -> 359 enrolled). Added: (1) **Third-Party Integrations Framework** (migration 065, CrowdStrike Falcon plugin) as a new Capabilities subsection — API-only, no dashboard UI yet, flagged as a new UI gap. (2) **VSS Policy Configurator (SPEC-016 redesign)** replacing the prior scheduled create+prune model — new Capabilities subsection, new locked architecture decision #12, data-loss-defect anti-pattern captured. (3) **Policy-Controlled Tray Deployment** — materialize/launch split documented, Task 4 (fleet enablement) flagged as pending in Current Focus. (4) **Legacy (2008R2/Win7) build variant disabled** — Patch/Update section + Build & Deploy + anti-patterns updated; RMM_THOUGHTS Feature 12 noted. (5) **Cross-site agent dedup fix + no-re-home-on-reconnect follow-up** — new Agent Identity & Enrollment subsection; 13 remaining duplicate pairs noted as open work. (6) **2026-07-04/05 easy-bugs + alert-triage batch** (server v0.3.95/v0.3.96) — update-scanner platform regimes, `db/sanitize.rs`, WS noise reclassification, `create_or_update_alert` Option<Alert> hardening, `move_agent` 409 — folded into Monitoring, Patch/Update, and Alerting Capabilities subsections plus a new Recent Work entry. (7) Active State refreshed (fleet count, 0 active server_error, 13 dup pairs, 5 open backup_failed alerts, VSS beta-only channel state). (8) New anti-patterns: scanner host/foreign self-check split, machine-global VSS governor data-loss class, reconnect re-homing footgun, `fetch_one` on a concurrently-deletable row, concurrent-session-no-lock branch collision, stale spec-status-header trap. History Highlights/Patterns preserved verbatim except new rows appended; article length kept in the same range as the 2026-06-25 compile by condensing older Recent Work entries into shorter summaries where the full detail is preserved in git history and prior compiles.
- **2026-07-05 recompile (Howard-Home/claude-main):** Full rebuild. Delta from 2026-07-04 (migrations 65 -> 67; agent 0.6.77 -> 0.6.78 fleet-wide stable; server 0.3.96 -> 0.3.99; dashboard v0.2.83 beta). Major additions: **Policy-core** (migration 067, `e7e90c2`) — new Capabilities subsection, locked decision #14, follow-on policy-ui gap; **four sequential bug-fix batches** (repair batch 2/v0.3.97, batch 3/v0.3.98, batch 4/v0.3.99) folded into Recent Work + Capabilities; **BUG-003 CLOSED** (was open); **BUG-018** corrected from the morning's status-write-guard fix to the full v0.3.98 schema-level view+trigger fix; **BUG-023 (NEW)** — channel-race incident caused by the 07-04 scanner fix, own Recent Work entry + anti-pattern + locked decision #15; **BUG-024 (NEW, OPEN, CRITICAL)** — Linux glibc incident, documented across Recent Work/Capabilities/Active State/Architecture; **comms-durability Phase 1 marked COMPLETE** (was T2/T3-planned). All prior anti-patterns/good-patterns preserved verbatim except one ("build classification defaulting to stable") marked superseded rather than deleted.
- **2026-07-04 recompile (Howard-Home/claude-main):** Delta from 2026-06-25 (migrations 64 -> 65; agent 0.6.75 -> 0.6.77 modern / 0.6.76 legacy-frozen; server 0.3.87 -> 0.3.96; fleet 270 -> 359 enrolled). Added: Third-Party Integrations Framework, VSS Policy Configurator, Policy-Controlled Tray Deployment, legacy build variant disabled, cross-site agent dedup fix, 2026-07-04/05 easy-bugs + alert-triage batch.
- **2026-06-25 recompile (Howard-Home/claude-main):** Delta from 2026-06-22 — SPEC-030 remote software inventory + bulk uninstall added as a major Capabilities subsection; universal self-detecting installer (Feature 9) P1 shipped; Current Focus rewritten; flagged the old "pending PRs #40-#46, migrations 061-063" prediction as superseded because software-removal work consumed 061-064 instead.
- **2026-06-22 recompile (Howard-Home/claude-main):** Delta from 2026-06-11 — fleet ~270/~178, BUG-021/018/019 fixed on main, Event Log Watch UI shipped, BUG-022 dead-code watchdog path removed, Beast parallel two-wave build documented, command_type "cmd" alias + NAK documented, migration count to 60.
- **2026-06-11 recompile (GURU-5070/claude-main):** Full recompile — physical server migration, durable agent identity, agent-comms-durability Phase 1 full detail, migration count to 59.