diff --git a/wiki/index.md b/wiki/index.md index af552627..ef8077f3 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -68,7 +68,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks. | Article | Summary | Last Compiled | |---|---|---| -| [GuruRMM](projects/gururmm.md) | RMM platform, Rust/Axum server + React dashboard + cross-platform agent; **agent v0.6.75 / server v0.3.87**; ~270 enrolled (~178 online); 64 migrations on main (through 064_software_removal_jobs). **Two-wave Windows build** — stable (modern x86_64) + legacy 1.77 wave (l_amd64/l_x86 for Win7/Server 2008 R2). Merging to main = build+deploy; artifacts hit **beta** first, `promote-dashboard.sh --confirm` for prod. Recent (2026-06-23/24): **SPEC-030 remote software inventory + bulk uninstall** — **BETA, merged+deployed but NOT guaranteed** (best-effort; fails on protected AV, WiX bundles, UI-only/lingering uninstallers, drivers) — silent-uninstall engine (`uninstall-engine.ps1`, refuses-to-guess + ARP false-success verification), async removal jobs (mig 064, live progress), three-state fleet knowledge base + learned timing (mig 061-063), `SoftwareManager.tsx`; **universal self-detecting installer (Feature 9)** P1 shipped (v0.6.71). Prior (06-21/22): BUG-021 legacy-build dep-pin (`1dce66d`); BUG-018 reliable delete (202+bg, `cea87d4`); Event Log Watch UI (`0fa65f5`); BUG-022 WatchdogEvent dead-code removed. Watchdog reports via REST `watchdog-alert` only (no WS). NOTE: earlier pending PRs #40-#46 (SPEC-021/MSP360/BUG-018-FK) predicted migrations 061-063 — software-removal merged ahead and took 061-064; verify their live status. Active development | 2026-06-25 | +| [GuruRMM](projects/gururmm.md) | RMM platform, Rust/Axum server + React dashboard + cross-platform agent; **agent v0.6.77 / server v0.3.96**; 359 enrolled (~228 fresh); 65 migrations on main (through 065_integrations_framework). Merging to main = build+deploy; artifacts hit **beta** first, `promote-dashboard.sh --confirm` for prod. Recent (2026-07-04/05): **easy-bugs batch, v0.3.95/v0.3.96** — update scanner no longer drops macOS/Linux binaries (host-native vs foreign self-check; macOS/Linux agents were NEVER offered updates before), hardware-inventory NUL/control-char sanitize (`db/sanitize.rs`, fixed 7+ machines silently failing), WS benign-disconnect + pre-auth churn ERROR->INFO (server_error alerts to 0), alert upsert survives agent cascade-delete, move_agent duplicate -> 409; **cross-site agent dedup** by device_id fleet-wide, no auto-re-home on reconnect (`2851523`+`1c1cac2`); **legacy Rust 1.77 wave DISABLED** (`84a82a3` — 0.6.76 legacy artifact frozen; 2008R2 path = Feature 12 rewrite, Raw); VSS policy configurator merged (`de30b2b`); policy-controlled tray deploy (`f6a4251`); CrowdStrike/integrations framework (mig 065). Prior: SPEC-030 software inventory+bulk uninstall (BETA, best-effort), universal self-detecting installer P1 (v0.6.71). Active development | 2026-07-04 | | [GuruConnect](projects/guruconnect.md) | ACG's proprietary Rust remote-access/remote-support tool (ScreenConnect-class) — Windows agent + web dashboard; **v0.3.0 production** at connect.azcomputerguru.com; versioned integration contract with GuruRMM; co-located on the physical .30 box (shares the PG cluster); next: SPEC-018 session broker / capture worker as SYSTEM | 2026-06-12 | | [GPS -> GuruRMM Coverage Audit](projects/gps-rmm-audit.md) | Verify every GPS-billed client is fully enrolled in GuruRMM with services real (backup/AV/email); enrollment 111/189 (baseline 46); staging auto-enroll pipeline + 30-min harvest loop; found+fixed GuruRMM cross-site dup bug (v6.77); Phase 4 AV matrix (8 clients/20 devices NO-AV); billing findings HELD for Winter/Mike; AV migration BD->EDR (exception: Glaztech only) | 2026-07-04 | | [Dataforth DOS — Test Datasheet Pipeline](projects/dataforth-dos.md) | DOS update system + TestDataDB pipeline (Node.js, PostgreSQL, Hoffman API); 469K records, 458.5K live on website; 2025 crypto attack recovery; security incident 2026-03-27; SCMVAS/SCMHVAS extension; email notifications via Graph API | 2026-05-24 | diff --git a/wiki/projects/gururmm.md b/wiki/projects/gururmm.md index ba36f837..cb23b054 100644 --- a/wiki/projects/gururmm.md +++ b/wiki/projects/gururmm.md @@ -2,17 +2,41 @@ type: project name: gururmm display_name: GuruRMM -last_compiled: 2026-06-25 +last_compiled: 2026-07-04 compiled_by: Howard-Home/claude-main aliases: - guru-rmm sources: - - "gururmm@main: server/src/api/*.rs (REST API surface, ~38 route modules; software.rs SPEC-030)" + - "gururmm@main: server/src/api/*.rs (REST API surface, ~40 route modules; integrations.rs new)" + - "gururmm@main: server/src/api/mod.rs (route surface diff, 2026-06-25..2026-07-04 — integrations CRUD only new surface)" + - "gururmm@main: server/migrations/*.sql (65 migrations — feature checkpoints through 065_integrations_framework)" + - "gururmm@main: server/migrations/065_integrations_framework.sql" + - "gururmm@main: server/src/db/sanitize.rs (NEW — shared NUL/control-char sanitize module)" + - "gururmm@main: server/src/db/inventory.rs, server/src/db/alerts.rs, server/src/updates/scanner.rs, server/src/ws/mod.rs, server/src/api/agents.rs (2026-07-04 easy-bugs + alert-triage batch)" + - "gururmm@main: agent/src/ (agent capabilities; tray_stage.rs, tray_state.rs, authenticode.rs new; device_id.rs, ohw.rs, watchdog/wts.rs, bsod.rs)" + - "gururmm@main: specs/tray-policy-deploy/shape.md + plan.md (policy-controlled tray deployment)" + - "gururmm@main: specs/vss-policy-config/shape.md (VSS policy configurator, SPEC-016 redesign)" + - "gururmm@main: specs/vss-native-com/ (native VSS COM create/provision)" + - "gururmm@main: specs/crowdstrike-falcon/shape.md (integrations framework + Falcon plugin)" + - "gururmm@main: docs/RMM_THOUGHTS.md (header index Features 1-12; Refinement 4a #1 DONE; Feature 12 legacy rewrite Raw)" + - "gururmm@main: docs/FEATURE_ROADMAP.md (BUG-009/BUG-011 verified fixed 2026-07-04; Known Bugs section)" + - "gururmm@main: git log origin/main (2026-06-25..2026-07-04 — VSS, tray, integrations, easy-bugs batches)" + - "gururmm@main: commit 8b75274..5f38ef3 (crowdstrike-falcon Tasks 1-5)" + - "gururmm@main: commit de30b2b (VSS policy configurator merge)" + - "gururmm@main: commit f6a4251 (policy-controlled tray deployment merge)" + - "gururmm@main: commit 84a82a3 (legacy Rust 1.77 variant disabled)" + - "gururmm@main: commit 2851523 + 1c1cac2 (cross-site agent dedup fix + no re-home on reconnect)" + - "gururmm@main: commit 519eef2 + f2d02dd (easy-bugs batch + pre-auth churn noise merges, v0.3.95/v0.3.96)" + - "gururmm@main: session-logs/2026-07/2026-07-04-howard-easy-bugs-batch-deploy.md" + - "gururmm@main: session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md" + - "session-logs/2026-07/2026-07-02-mike-crowdstrike-rollout-365-appsuite.md" + - "session-logs/2026-07/2026-07-03-mike-gururmm-vss-configurator-and-pst-recovery.md" + - "session-logs/2026-07/2026-07-04-mike-gururmm-vss-build-merge-deploy.md" + - ".claude/memory/project_gururmm_legacy_disabled.md" - "gururmm@main: agent/src/ (agent capabilities; transport/CommandContext, ohw.rs, watchdog/wts.rs, bsod.rs, device_id.rs)" - "gururmm@main: agent/scripts/uninstall-engine.ps1 (SPEC-030 Tier-1 silent uninstall engine)" - "gururmm@main: server/src/db/software_jobs.rs (async removal jobs)" - "gururmm@main: dashboard/src/components/SoftwareManager.tsx (installed-programs list + bulk uninstall + live progress)" - - "gururmm@main: server/migrations/*.sql (64 migrations — feature checkpoints through 064_software_removal_jobs)" - "gururmm@main: server/migrations/061_software_removal_attempts.sql" - "gururmm@main: server/migrations/062_software_knowledge.sql" - "gururmm@main: server/migrations/063_software_knowledge_timing.sql" @@ -28,12 +52,9 @@ sources: - "gururmm@main: server/migrations/060_alert_mutes_agent_id_index.sql" - "gururmm@main: agent/src/bsod.rs" - "gururmm@main: server/src/api/updates.rs (promote/rollback endpoints)" - - "gururmm@main: server/src/api/mod.rs (route surface, 2026-06-22)" - "gururmm@main: deploy/build-pipeline/webhook-handler.py" - "gururmm@main: deploy/build-pipeline/build-server.sh" - - "gururmm@main: docs/FEATURE_ROADMAP.md (BUG-018..022 + SPEC-021 status)" - "gururmm@main: docs/BUILD.md (build + pre-merge verification guide)" - - "gururmm@main: git log origin/main -40 (2026-06-22)" - "gururmm@main: commit 1dce66d (BUG-021 dep-pin)" - "gururmm@main: commit cea87d4 (BUG-018 202+bg)" - "gururmm@main: commit 0fa65f5 (Event Log Watch UI)" @@ -110,31 +131,31 @@ backlinks: ## Summary -GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 270 enrolled agents across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints. +GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 359 enrolled agents (228 typically fresh/online) across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints. -**Current version:** agent 0.6.75 / server v0.3.87 as of 2026-06-25 (was 0.6.67 / 0.3.74 on 2026-06-22). Fleet at ~270 enrolled / ~178 typically online; all metrics flowing, alerts firing with 0 legacy null dedup_keys. +**Current version:** agent 0.6.77 / server v0.3.96 as of 2026-07-04 (was 0.6.75 / 0.3.87 on 2026-06-25). Active `server_error` alerts reduced to 0 fleet-wide after the 2026-07-04/05 fix batch. -**SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA — code merged + deployed 2026-06-23..24, NOT guaranteed to work; Route B = server-orchestrated):** The code is on main and deployed, but the **feature is still beta and unreliable** — it succeeds on cleanly-uninstallable programs and is explicitly NOT guaranteed for the hard cases (most managed/protected AV, WiX/Burn bundles, uninstallers that need UI or spawn lingering child processes — Launchy/AIMP-class, drivers/system components). Treat results as best-effort, verify removals, and do not rely on it for production removals yet. Shape: GuruRMM lists installed programs per agent and attempts bulk removal via a PowerShell **silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`, BCU-informed, vendor silent-switch table) that **refuses to guess** (returns `needs_interactive` rather than risk a wrong removal) and **verifies** each reported success via a generic ARP re-check (`Test-StillInstalled`, catches the false-success class). Backed by a fleet **three-state knowledge base** (migration 062: `silent` / `requires_ui` / `unknown`), **self-tuning learned timeouts** (migration 063), a **removal-tracking loop** (migration 061: what still needs removing, flag for a remote GuruConnect session), and **async jobs** (migration 064: POST returns a `job_id` instantly, background per-target worker, dashboard live progress — replaces the synchronous request that died at the proxy's ~100s timeout). See the Capabilities section for full detail + the open gaps. +**Easy-bugs batch + alert-triage fixes deployed 2026-07-05 (server v0.3.95 + v0.3.96):** Fixed three long-standing silent-failure classes surfaced by a full 24h alert triage: (1) the update **scanner** dropped macOS/Linux agent binaries during its self-check because they cannot execute on the Linux build host — every macOS/Linux agent had **never** been offered an update; fixed with host-native-vs-foreign self-check regimes (foreign platforms trust the filename version, host-native binaries stay a hard `--version` gate). (2) **Hardware inventory** silently failed to persist for 7+ fleet machines because Windows registry strings routinely carry embedded NUL bytes that Postgres jsonb/text rejects; fixed with a new shared `db/sanitize.rs` (NUL + C0/C1 control-char strip, collision-safe). (3) **WebSocket log noise** — benign disconnects and pre-auth connection churn (sleeping laptops, half-open NAT, scanners) were logged as `ERROR` and minted `server_error` alerts; reclassified to `INFO`, with genuine auth rejections (bad key, malformed auth) staying `ERROR`. Also fixed in the same batch: `create_or_update_alert` now returns `Option` and survives an agent cascade-delete mid-flight (FK violation -> `Ok(None)` instead of aborting the offline sweep); `move_agent` maps a duplicate-site-device conflict to 409 instead of a raw 500. Went through a 19-agent high-effort code review before merge (9 confirmed findings, all fixed). See Recent Work for detail. -**Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+):** One install URL/script self-detects arch/OS/legacy and pulls the correct agent; the dashboard install UI now points at it (replaces the hardcoded-amd64 one-liner). P2 (native i386 bootstrapper EXE for locked-down boxes) pending. +**Cross-site agent duplication fixed 2026-07-04:** Root cause of ~40 duplicate agent rows (24 at Dataforth D2->D1 alone): both WS enrollment paths deduped only within `(site_id, device_id)`, so re-running a different site's installer on an already-enrolled machine created a second row instead of finding the existing one. Fixed to match by hardware `device_id` **across all sites** and re-home via `move_agent_to_site` — but a same-day follow-up (`1c1cac2`) stopped the auto-re-home on every reconnect (the agent re-presents its install-site code as its API key on every beat, which was bouncing admin-moved agents back to their original site). Net effect: dedup by device_id, but site placement stays admin-controlled and persists. -**BUG-021 (FIXED, commit `1dce66d`, 2026-06-22):** The legacy wave (Rust 1.77 for Win7/Server 2008 R2) was resolving deps fresh and pulling `edition2024` crates. Fixed by pinning `getrandom 0.3.1` + `zeroize 1.8.1` below edition2024 in agent Cargo.toml. NOT a toolchain bump — bumping would drop legacy support. Windows build went green at 02:19, v0.6.67. +**Legacy (Server 2008R2/Win7, Rust 1.77) build variant disabled 2026-07-04:** The unpinned legacy dependency re-resolve pulled edition2024 crates (`chacha20`/`rand_core 0.10.1`) that Rust 1.77 cannot parse — a BUG-021 recurrence that fail-closed the entire Windows build (blocking modern amd64/x86/tray artifacts too). Gated behind `LEGACY_ENABLED=false`; the last-built legacy artifact (agent 0.6.76) is preserved and frozen for existing 2008R2/Win7 endpoints. 2008R2 support returns via a separate legacy-only rewrite (C++/.NET or a clean minimal Rust core), not by re-enabling the 1.77 toolchain — captured as **RMM_THOUGHTS Feature 12** (Raw, Mike's requirement). -**BUG-018 (FIXED, commit `cea87d4`):** `DELETE /api/agents/:id` now returns 202 Accepted, marks the agent `deleting` (synchronous existence-check), disconnects the live WS, and purges all cascade rows in a background tokio task. Fleet listings and stats exclude `'deleting'` rows; the agent disappears from the UI immediately. +**Policy-Controlled Tray Deployment shipped 2026-07-04 (code merged, policy default OFF):** The tray binary is now deployed to every Windows endpoint via the agent-update flow (previously only fresh MSI installs got it — fleet census found ~111/113 sampled hosts with no tray at all) but its **launch is gated on `tray.enabled` policy** — no process, no icon, until a client/site/agent policy turns it on. Authenticode-verified + version-locked to the agent. Built via a rejected-then-corrected design (17-agent code review + Gemini + Grok adversarial critique caught 9 defects in the first attempt — see Recent Work). Fleet-wide policy enablement (Task 4) is **not yet flipped**. -**Event Log Watch management UI (SHIPPED, commit `0fa65f5`):** `/event-log-watches` CRUD is wired into the dashboard — list, create, edit, delete, and enable-toggle watches for global and per-agent scope via a new `pages/EventLogWatches.tsx` in the FunctionRail + omnibox. +**VSS Policy Configurator shipped 2026-07-03 (SPEC-016 redesign, agent v0.6.76):** The agent no longer runs its own scheduled shadow-copy create+prune loop — the MITRE T1490 "Inhibit System Recovery" surface that got the agent blocked by CrowdStrike Falcon. It now **configures** two kernel-enforced retention governors (VSS diff-area size cap + `MaxShadowCopies` count) and registers a scheduled task whose action is a native-COM **create only** (never a delete) — retention becomes Windows' own FIFO eviction, not agent code. Works uniformly on server AND client Windows SKUs (previously `vssadmin create shadow` blocked workstations). A 23-agent code review caught a data-loss-class defect pre-merge (writing the machine-global `MaxShadowCopies` would have evicted OTHER products' shadows) — dropped in favor of the size cap as sole governor. Live-verified on the T1490 canary (NEPTUNE, Falcon present): legacy scheduled task removed, no T1490 trip, 17 existing shadows preserved. Deployed to the **beta** channel only (0.6.76); stable fleet remains on 0.6.66 pending a deliberate promotion. -**Agent-comms-durability Phase 1 shipped 2026-06-11:** Commands black-holed at NAT'd sites no longer falsely fail. See Architecture section for full detail. +**Vendor-agnostic Integrations Framework shipped 2026-07-01..02 (migration 065, CrowdStrike Falcon first plugin):** A generic per-partner plugin framework (`integrations`, `integration_credentials`, `integration_client_mappings`, `integration_sync_state`, `integration_agent_links` tables) so a new EDR/security vendor is a new server module, not a new schema — modeled on the proven MSPBackups per-partner shape. CrowdStrike Falcon is the first plugin: API client, RMM-client-to-Falcon-CID mapping, and a periodic sync worker pulling sensor state (version/status/RFM) into `integration_agent_links`. REST CRUD at `/api/integrations`; no dedicated dashboard UI yet (API/backend only). + +**SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA — code merged + deployed 2026-06-23..24, NOT guaranteed to work; Route B = server-orchestrated):** The code is on main and deployed, but the **feature is still beta and unreliable** — it succeeds on cleanly-uninstallable programs and is explicitly NOT guaranteed for the hard cases (most managed/protected AV, WiX/Burn bundles, uninstallers that need UI or spawn lingering child processes — Launchy/AIMP-class, drivers/system components). Treat results as best-effort, verify removals, and do not rely on it for production removals yet. See the Capabilities section for full detail + the open gaps. + +**Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+):** One install URL/script self-detects arch/OS/legacy and pulls the correct agent; the dashboard install UI now points at it. P2 (native i386 bootstrapper EXE for locked-down boxes) pending. **Physical server migration completed 2026-06-11:** New physical box at 172.16.3.30 running Ubuntu 26.04 + PostgreSQL 18. Old VM at .46 decommissioned 2026-06-12. -**Backup-alert quality pass shipped 2026-06-07:** False `backup_failed` alerts reduced 15 -> 2 fleet-wide. - -**Role-aware offline alerting + alert ignore/mute shipped 2026-06-07:** Server-only `agent_offline` alerts; site/fleet mass-offline aggregation; `alert_mutes` perma-silence. - **See also:** `wiki/projects/guru-rmm.md` is a redirect tombstone pointing here (slug disambiguation). -**Repo:** `azcomputerguru/gururmm` on Gitea (internal: http://172.16.3.20:3000). The copy at `projects/msp-tools/guru-rmm` is a git submodule tracking the active repo; its pinned gitlink intentionally lags origin/main. Development uses `git -C projects/msp-tools/guru-rmm` or worktrees off `origin/main` to access live code. Commits/pushes go to Gitea from the server (.30) or via `GIT_ASKPASS` with the vaulted Gitea API token. Howard is authorized for merges/deploys (cleared 2026-06-21 by Mike). +**Repo:** `azcomputerguru/gururmm` on Gitea (internal: http://172.16.3.20:3000, external https://git.azcomputerguru.com/azcomputerguru/gururmm.git). The copy at `projects/msp-tools/guru-rmm` is a git submodule tracking the active repo; its pinned gitlink intentionally lags origin/main. Development uses `git -C projects/msp-tools/guru-rmm` or worktrees off `origin/main` to access live code. Howard is authorized for merges/deploys (cleared 2026-06-21 by Mike). **Goal:** Full-featured MSP platform rivaling commercial RMMs, with a companion PSA (GuruPSA, separate future repo) designed as a truly integrated unified system. @@ -142,33 +163,76 @@ GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer G ## Recent Work +### 2026-07-04..05 — Easy-Bugs Batch + Alert Triage: Update-Scanner, Inventory NUL, WS Noise (Howard-Home) + +Full 24h alert triage (`GET /api/alerts`) surfaced 5 `backup_failed` (client-side, unresolved — see Active State), 2 active `server_error` (product bugs), and 1 resolved `mass_offline_fleet` (36 agents, transient network blip). Root-caused both server errors against the server's own self-log (pseudo-agent `00000000-...-001`): + +- **`move_agent` 500 -> 409:** moving an agent into a site that already holds a row for the same `device_id` tripped `idx_agents_site_device` as a raw 500; now maps to 409 with an actionable message. 13 duplicate-hostname agent pairs fleet-wide are the trigger (stale legacy `win-*`/`hw-*` device_ids alongside re-enrolled hardware-UUID records) — not yet cleaned up. +- **Offline-sweep `RowNotFound`:** `create_or_update_alert`'s UPDATE-by-id used `fetch_one`; an alert row deleted mid-sweep (agent-delete cascade) aborted the whole sweep. Switched both branches to `fetch_optional` with fall-through to INSERT (`Result>`, `Ok(None)` = agent gone). +- **Inventory NUL rejection (alert-invisible, found in logs only):** 7+ agents (IMC1, Seth-PC, QWM-JOHN, QWM-SHEILA, goldstar19, SIF-SERVER, Christine-Win10) never persisted hardware inventory — Windows registry strings carry embedded NULs, Postgres jsonb/text rejects them. New `server/src/db/sanitize.rs` (`strip_nul`, `clean_control_chars`) applied over 5 jsonb blobs (values + keys, collision-safe) + text columns. +- **Update-scanner warn-spam / silent update starvation:** the self-check tried to execute every platform's binary to read its embedded version — macOS/Linux binaries can't execute on the Linux build host, so they were dropped from the scannable set entirely, meaning **macOS/Linux agents were never offered updates**. Fixed with host-native-vs-foreign regimes: foreign-platform binaries trust the filename version; host-native binaries stay a hard gate (a native binary that can't report its version is corrupt) with a best-effort exec-bit repair for 0644 CI-copy artifacts. + +A concurrent same-branch session absorbed a prior session's in-progress inventory/scanner fixes; the combined diff went through a 19-agent high-effort code review (9 confirmed findings, 0 refuted) before merge — findings included rebuilding the alerts fall-through (it still failed its own motivating scenario on first pass), tightening the scanner's foreign/host-native split, and moving WS-noise logging from DEBUG to INFO (a mass-disconnect anomaly must stay visible at the default log level). Merged + deployed as **server v0.3.95** (06:09 UTC): scanner now indexes 8 platform/arch combos (was 4, Windows-only); zero ERROR lines post-deploy. + +Continued into the last stale alert: "Authentication timeout" WS churn (2-6/day baseline, 57-spike correlated with the same day's fleet-dedup work). Profiled 7 days of journal, concluded pre-auth connection noise (peer connects, never authenticates — sleeping laptops, scanners, half-open NAT), added `is_preauth_connection_churn` to classify it INFO while keeping genuine rejections (bad API key, malformed auth) at ERROR. Deployed as **server v0.3.96** (06:23 UTC). Active `server_error` alert count: **0**. + +BUG-009 (`Logs.tsx` isError handling) and BUG-011 (`: any` occurrences) were verified already fixed in-tree during the same pass — only the stale `FEATURE_ROADMAP.md` status lines needed correcting, not the code. + +Remaining/pending from this session: the **machine-side backup failures** (the user's actual priority) — GND-SERVER (Grabb & Durando, chronic, 42 nightly failures since 2026-06-17), AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial) — investigation started but not completed; all say "no error detail reported" (MSP360 sync doesn't capture failure detail). 13-pair duplicate-agent cleanup still open. WS auth-timeout source (~1/min, unauthenticated) still unidentified (needs Feature 5, server-side public-IP capture — still unapproved). + +--- + +### 2026-07-04 — Policy-Controlled Tray Deployment (Howard-Home / Mike, merged f6a4251) + +Made `gururmm-tray.exe` deployment (materialize, policy-agnostic) independent from its runtime (launch, policy-gated) — previously the watchdog launched the tray whenever the binary existed, with no enforcement of the already-wired `tray.enabled` policy flag, and the binary itself only ever reached fresh MSI installs (fleet census: 1 signed tray, 1 stale unsigned, ~111 none out of 113 sampled). A first design (inline tray refresh folded into `do_update`) was **rejected** by a 17-agent code review plus independent Gemini 3.1 Pro and Grok adversarial critiques — 9 verified defects including permanent-binary-loss on a failed swap, a checksum sourced from an unauthenticated download-host sidecar (supply-chain gap), and an update/rollback tray-agent version-skew case. + +Corrected design shipped as Tasks 1-3 + 5: policy-gated launch (default OFF), a materialize FSM with staging + a versioned mutex (atomic swap, restore-on-failure, authenticated-payload checksum), server capability-gated materialize, and Authenticode-verify + observability. Task 4 (flip fleet-wide materialize on, policy still OFF) is the deliberate next step — not done yet. Merged to main 2026-07-04 12:23 PDT. + +--- + +### 2026-07-03..04 — VSS Policy Configurator (SPEC-016 Redesign) (Mike, merged de30b2b) + +Rebuilt the agent's Volume Shadow Copy handling from "scheduled create+prune operator" (the MITRE T1490 surface Falcon blocked) to "policy configurator, never a deleter." Tasks 2-7: `vss-create` verb + scheduled-task registration (both daily-time and every-N-hours trigger modes) with legacy-task migration; per-volume diff-area size-cap governor set on policy apply; retirement of the scheduled create+prune pass entirely (SPEC-025 compliance heal reworked to create-only); status/compliance reporting updated to the configurator model; dual-target release build; runtime verification on both a Server 2019 box and a Win11 Pro client with Falcon present. + +A 23-agent workflow code review before merge caught a genuine data-loss defect: the original design would have set the machine-global `MaxShadowCopies` registry value to `retention_count`, which FIFO-evicts **other** VSS consumers' (System Restore, third-party backup) shadows below that count fleet-wide. Fixed by dropping `MaxShadowCopies` management entirely — the per-volume size cap is the sole governor; `retention_count`/`retention_max_age_days` become advisory (a warning fires if `retention_max_age_days` is still set, since age-pruning requires deletion). Nine other findings fixed (cap-before-create ordering, an alias bridging the legacy task-name upgrade window, interval-aware compliance windows, bounded `IVssAsync` wait instead of INFINITE, and others). + +Merged to main; the build pipeline publishes to the **beta** channel, not stable (a mid-session correction) — beta head reached 0.6.76 (8 agents), stable stayed at 0.6.66 (233 agents) by design. Verified live on the T1490 canary NEPTUNE (moved to beta channel for the test): create task present, legacy `GuruRMM-VSS-Snapshot` task removed, `MaxShadowCopies` correctly left unset, 17 pre-existing shadows preserved (the data-loss fix proven in production), and a live `vss-create` produced new shadows on two volumes under Falcon with no T1490 trip. + +--- + +### 2026-07-01..02 — Vendor-Agnostic Integrations Framework + CrowdStrike Falcon Plugin (Mike) + +Built a generic third-party-integration plugin framework (migration 065: `integrations`, `integration_credentials`, `integration_client_mappings`, `integration_sync_state`, `integration_agent_links`) so CrowdStrike Falcon — and future vendors (SentinelOne, Guardz) — are a new server module rather than a new bespoke schema each time, generalizing the per-partner MSPBackups shape (migration 034). Credentials reuse the same AES-256-GCM `Encryptor` chokepoint as the existing `credentials` table. + +Five tasks, one per day-slice: (1) framework schema migration; (2) plugin trait + registry + DB layer; (3) generic REST API (`/api/integrations` CRUD, credentials, client mappings, sync-state, connection test); (4) CrowdStrike Falcon API client + plugin registered against the framework; (5) periodic sync worker pulling Falcon sensor state (version, status, RFM flag, CID) into `integration_agent_links`. No dedicated dashboard UI yet — API/backend only as of this compile. + +--- + ### 2026-06-23..24 — SPEC-030 Remote Software Uninstall + Universal Installer (Howard-Home) **SPEC-030 software inventory + remote bulk uninstall (Route B — BETA, code merged to main + deployed but NOT guaranteed to work; agent 0.6.75 / server v0.3.87):** - **Capability shipped:** `/agents/:id/software` lists installed programs; `/agents/:id/software/uninstall` runs a bulk uninstall; the dashboard `SoftwareManager.tsx` shows the live programs list + bulk-select + a live progress bar. Server endpoints in `server/src/api/software.rs`. - **Silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`): Tier-1 silent removal, BCU-informed detection + fail-fast, vendor silent-switch table, captures **real** process exit codes (fixed a `Start-Process` null-ExitCode bug, `f6c1945`). The engine **refuses to guess** — Avast/AVG (`icarus.exe`), Malwarebytes (`mb5uns.exe`) correctly returned `needs_interactive` rather than risk a bad removal. -- **False-success class fixed (CRITICAL, PR #53):** Avira's WiX/Burn **Package Cache bundle GUID** was misread as an MSI ProductCode; `msiexec /x {bundleGuid}` returned 1605 ("not installed") and was mapped to success="already gone" while Avira stayed fully installed. Fix: skip the MSI tier for `\Package Cache\` bundles + a **generic post-success ARP re-check** (`Test-StillInstalled`) that downgrades a reported success to failed if the program is still registered. Generic (not AV-specific) so it catches the whole class. -- **Code-review correctness fixes (PR #54):** `Test-StillInstalled` matches by `product_code` alone when present (no false-failure on same-named programs); skip the rescan on reboot-pending verdicts (3010/1641); exclude MSI-1605 no-op successes from the learned-timing sample. -- **Async removal jobs — the permanent fix (PR #56, migration 064):** A synchronous bulk uninstall died at Cloudflare's ~100s timeout (popup vanished, no progress) AND the agent's command kill-timeout == the engine budget killed the engine mid-flush (37 apps removed but reported `failed`/no-output). Rebuilt as a **JOB**: POST creates a `software_removal_jobs` row and returns a `job_id` instantly (0.06s); a background worker dispatches the engine one program at a time, recording each outcome as it lands; `GET /agents/:id/software/uninstall/jobs/:job_id` is polled by the dashboard. `server/src/db/software_jobs.rs` (new) + async refactor of `software.rs`. Stop-gap flush-slack (PR #55, `ENGINE_FLUSH_SLACK_SECS=120`) shipped first. -- **Three-state removal knowledge base (migrations 061-063):** `software_removal_attempts` (per-agent removal-tracking loop — failures + no-silent-path programs flagged for a remote session; resolved when a later removal succeeds), `software_knowledge` (fleet KB keyed by DisplayName: `silent` / `requires_ui` / `unknown`), `software_knowledge_timing` (fleet-learned self-tuning per-target timeout — generous while barely measured, narrowing toward the slowest observed success). Dashboard three-state KB UI shipped. -- **Orphaned-entry + retry-verify (PR #57):** GOM Player = orphaned ARP entry (uninstaller file missing) → `tier=orphaned` "needs registry cleanup"; TeamViewer async-fork false-success → retry-verify (~9s, 4 checks); bounded async stdout/stderr read. -- **Known gaps surfaced (not yet built):** (1) drivers/system components (Intel/Realtek, Asmedia USB) appear in `/software` because they register in ARP — a top-down 100-program select removed a driver; the UI needs a driver/system-component exclusion/flag. (2) **Launchy/AIMP-class** uninstallers don't honor silent-as-SYSTEM and hang the agent on a lingering child process tree until the agent kill-timeout (~420s) → `failed`/no-output; Route B can't fully control this — they need **recipes / agent-side process handling (Route A)**. Open decision: kill-process-tree-on-timeout engine mitigation (clean `needs_interactive`) vs accept as recipe work. (3) No list-jobs endpoint yet (only get-by-id). -- **av-removal-recipes spec (P3, scoped to removal-only, no RMM install path)** and the **Third-Party AV Removal Recipes** RMM_THOUGHTS entry capture the recipe-tier follow-on. +- **False-success class fixed (CRITICAL, PR #53):** Avira's WiX/Burn **Package Cache bundle GUID** was misread as an MSI ProductCode; `msiexec /x {bundleGuid}` returned 1605 ("not installed") and was mapped to success="already gone" while Avira stayed fully installed. Fix: skip the MSI tier for `\Package Cache\` bundles + a **generic post-success ARP re-check** (`Test-StillInstalled`) that downgrades a reported success to failed if the program is still registered. +- **Code-review correctness fixes (PR #54):** `Test-StillInstalled` matches by `product_code` alone when present; skip the rescan on reboot-pending verdicts (3010/1641); exclude MSI-1605 no-op successes from the learned-timing sample. +- **Async removal jobs — the permanent fix (PR #56, migration 064):** A synchronous bulk uninstall died at Cloudflare's ~100s timeout AND the agent's command kill-timeout killed the engine mid-flush. Rebuilt as a **JOB**: POST creates a `software_removal_jobs` row and returns a `job_id` instantly (0.06s); a background worker dispatches one program at a time; `GET .../uninstall/jobs/:job_id` is polled by the dashboard. Stop-gap flush-slack (PR #55) shipped first. +- **Three-state removal knowledge base (migrations 061-063):** `software_removal_attempts` (per-agent removal-tracking loop), `software_knowledge` (fleet KB: `silent` / `requires_ui` / `unknown`), `software_knowledge_timing` (self-tuning per-target timeout). +- **Known gaps surfaced (not yet built):** drivers/system components register in ARP and appear in the bulk list; Launchy/AIMP-class uninstallers hang the agent on a lingering child process (~420s kill-timeout); no list-jobs endpoint (get-by-id only). **Universal self-detecting installer (Feature 9 — P1 built+deployed+verified, v0.6.71):** -- One install path self-detects arch/OS (x64/x86/legacy/ARM) and pulls the correct agent; `feat(installer)` `4194b0a`, dashboard install UI repointed at it (`53bb682`). Script path proven on-box (GND-JWILL, `de30ebc`). P2 (native i386 bootstrapper EXE for locked-down boxes) + offline-bundle still pending. +- One install path self-detects arch/OS (x64/x86/legacy/ARM) and pulls the correct agent; dashboard install UI repointed at it. Script path proven on-box (GND-JWILL). P2 (native i386 bootstrapper EXE) + offline-bundle still pending. -**RMM_THOUGHTS movement:** Feature 10 (AMPIPIT recovery environment add-on) → **Discussed** (Mike likes it, deferred to revisit); Third-Party AV Removal Recipes + Win11 KB5095093 Point-in-Time Restore (2026-06-25) added as Raw. +**RMM_THOUGHTS movement:** Feature 10 (AMPIPIT recovery environment add-on) -> Discussed (Mike likes it, deferred); Third-Party AV Removal Recipes + Win11 KB5095093 Point-in-Time Restore added as Raw; **Feature 11** (OS Updates Management, Windows Update P1/P2 + Linux/Mac update-check P3, Syncro-modeled) added Raw 2026-07-03; **Feature 12** (legacy 2008R2/Win7 agent rewrite) added Raw 2026-07-04 after the legacy build was disabled. --- ### 2026-06-22 — BUG-021 Fixed + v0.6.67 Stable; Fleet Verified (Howard-Home) -- **BUG-021:** Legacy build (Rust 1.77 via `$CARGO +1.77 --features legacy`) was pulling `wit-bindgen` (edition2024) through fresh dep resolution. Fixed by pinning `getrandom 0.3.1` + `zeroize 1.8.1` in agent Cargo.toml below edition2024. Commit `1dce66d` (current origin/main HEAD). Windows build went green at 2026-06-22 02:19, v0.6.67. -- **Fleet verified (same session):** ~270 agents / ~178 online; metrics rows flowing (~2531/15 min); all alerts have non-null dedup_keys (0 legacy nulls); per-agent endpoints all HTTP 200 on a test agent. -- **BUG-022 filed + fixed (PR #45):** `WatchdogEvent` WS variant was dead code — defined + server-handled but the watchdog process has no WS connection and never produced it. Removed the orphaned path (variant, payload, `insert_watchdog_event`, match arm, `watchdog_events::*` re-export). Server-compiled clean (48.9s). Empty `watchdog_events` table left in place (no DROP migration) to avoid collision with in-flight PR migration numbers. PR #45 (`fix/bug-022-watchdog-event-deadcode`, `4eb5054`) + PR #46 (docs) await Howard/Mike merge. Merging #45 = fleet build+deploy. -- **Roadmap verified:** BUG-018 (`cea87d4`) and Event Log Watch UI (`0fa65f5`) confirmed on main; FEATURE_ROADMAP updated to reflect Fixed status. +- **BUG-021:** Legacy build (Rust 1.77 via `$CARGO +1.77 --features legacy`) was pulling `wit-bindgen` (edition2024) through fresh dep resolution. Fixed by pinning `getrandom 0.3.1` + `zeroize 1.8.1` below edition2024. Commit `1dce66d`. Windows build went green 2026-06-22 02:19, v0.6.67. (This class of failure recurred 2026-07-04 with a different dep pair and led to the legacy variant being disabled outright — see Recent Work above.) +- **Fleet verified:** ~270 agents / ~178 online; metrics rows flowing; all alerts have non-null dedup_keys; per-agent endpoints all HTTP 200. +- **BUG-022 filed + fixed (PR #45):** `WatchdogEvent` WS variant was dead code (watchdog has no WS connection); removed. Empty `watchdog_events` table left in place. +- **Roadmap verified:** BUG-018 (`cea87d4`) and Event Log Watch UI (`0fa65f5`) confirmed on main. --- @@ -176,262 +240,160 @@ GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer G **Howard-Home sessions (three spans):** -- **BUG-019 FIXED (commit `66a7f4e`, merged to main by Mike):** Container agent no longer performs in-place binary self-update (silent downgrade on container recreate / rollback-artifact log noise). `agent/src/updater/mod.rs` `perform_update()` now early-returns inside a container (`is_docker_container()`), reporting `UpdateStatus::Failed` with a clear message. Structured `update_method=image` deferred to SPEC-023. Verified on Linux build (v0.6.67 beta, 0 errors, 54s). -- **BUG-018 FIXED (commit `cea87d4`, merged):** Handler redesign to 202 + background purge (see Summary). Migration 061 (FK indexes on five previously-unindexed FK columns, speeds the bg purge) on PR #41, pending merge. The root cause of the original HTTP 000 was cascade-delete volume (millions of metrics/logs rows), not missing indexes — the HIGH-volume tables were already indexed on `agent_id`. -- **Event Log Watch management UI (SHIPPED, `0fa65f5`):** `/event-log-watches` CRUD page (list, create, edit, delete, enable-toggle, global + per-agent scope). Built as a standalone page (not an AgentDetail panel) to avoid concurrently-edited AgentDetail. `tsc -b + vite build` clean. Merged + now on origin/main. -- **Enrollment modal UX fix (SHIPPED to prod, `4027c86`):** Enrollment modals (EnrollmentModal + EnrollAgentModal) now have a top-right close button, click-outside/Esc dismissal, and trigger a list refresh on close. Built from a branch worktree -> beta -> tested on rmm-beta -> promoted to prod with `promote-dashboard.sh --confirm`. -- **SPEC-021 + BUG-020 follow-up (PR #40, mig 063, `9171f84`):** Logged-in-user domain + account-type detection (agent metrics + server + dashboard). BUG-020 watchdog tray-teardown wiring (calling `terminate_all` from policy-disable/uninstall path). Migration renumbered 060 -> 063 to clear collision with BUG-019's 060. -- **MSP360 "Open in MSP360" deep-link (PR #42, mig 062):** Button on backup-tab plan card deep-linking to the MSP360 console for a specific agent's backup plan. Backend: `backup_provider_console_id` field (migration 062) + `provider-links.ts` mapping. PR awaits merge. -- **Event Log Watch policy-clobber HIGH fix (PR #43, `432b434`):** Assigning or unassigning any policy silently wiped the connected agent's Event Log Watch monitoring. Third instance of a config-push clobber class already fixed at two other sites. Fixed by reusing `watch_rules_for_agent` (retargeted from `&AppState` to `&PgPool`) — single source of truth across all three push sites. No migration. PR awaits merge. -- **Audit cleanup (PR #44):** Low findings from the 2026-06-21 `/rmm-audit` pass (DoS `cap_field` batch, SSE revocation bypass + JWT-in-URL, `console.log` stubs, `ContextTree.tsx` missing `isError`). Awaiting merge. -- **sync.sh submodule-clobber root-cause fix:** `sync.sh` line 339 `git submodule update --init` ran unconditionally on every sync, re-checking out the intentionally-lagging pinned gitlink in detached HEAD and discarding live work. Fixed with a `git -C "$ppath" rev-parse --git-dir` populate-only guard — only initializes genuinely-missing submodules. Committed to claudetools main; verified guru-rmm now stays on `origin/main @ ed8cad3` through a full sync. -- **Dashboard beta-before-main rule established:** Dashboard changes go to beta first, before main. Beta auto-builds from `origin/main` via the webhook; a feature branch is previewed with a manual git worktree build + rsync to `/var/www/gururmm/dashboard-beta`. Promotion to prod = `promote-dashboard.sh --confirm`. (Saved as memory `rmm-dashboard-beta-before-main`.) +- **BUG-019 FIXED (commit `66a7f4e`, merged to main by Mike):** Container agent no longer performs in-place binary self-update. `agent/src/updater/mod.rs` `perform_update()` early-returns inside a container. +- **BUG-018 FIXED (commit `cea87d4`, merged):** Handler redesign to 202 + background purge. Migration 061 (FK indexes on five previously-unindexed FK columns) landed later under a renumbered slot once SPEC-030 consumed 061-064. +- **Event Log Watch management UI (SHIPPED, `0fa65f5`):** `/event-log-watches` CRUD wired into the dashboard. +- **Enrollment modal UX fix (SHIPPED to prod, `4027c86`):** close button, click-outside/Esc dismissal, list refresh on close. +- **Event Log Watch policy-clobber HIGH fix (PR #43, `432b434`):** Assigning/unassigning a policy silently wiped an agent's Event Log Watch monitoring — third instance of a config-push clobber class. Fixed by reusing `watch_rules_for_agent` as the single source of truth. +- **sync.sh submodule-clobber root-cause fix:** populate-only guard prevents `git submodule update --init` from re-checking out the intentionally-lagging pinned gitlink on every sync. +- **Dashboard beta-before-main rule established.** **Mike GURU-5070 session:** -- **BUG-019 merged to main** (fast-forward `ed8cad3 -> 66a7f4e`; CI bump `8b5e0dc`). Linux build confirmed green (v0.6.67, 54s, 0 errors). -- **`gururmm-build` skill created:** `bash .claude/skills/gururmm-build/scripts/verify.sh server|agent|dashboard|migrations` for local pre-merge verification. Does NOT trigger the prod build (webhook-on-merge is the trigger). `docs/BUILD.md` added to guru-rmm repo (build guide). Memory `feedback_gururmm_build_verification` captures the key rules. -- **Howard cleared for GuruRMM merges/deploys** (Mike decision). Mike still owns architecture/direction; prepared + verified branches no longer bottleneck on Mike to land. +- **BUG-019 merged to main** (fast-forward `ed8cad3 -> 66a7f4e`). +- **`gururmm-build` skill created:** local pre-merge verification (`verify.sh server|agent|dashboard|migrations`). `docs/BUILD.md` added. +- **Howard cleared for GuruRMM merges/deploys** (Mike decision). --- ### 2026-06-15 — BSOD Dedup Key Fix, MSI EXDEV Fix, Dashboard Offline Dedup, sync.sh Auto-Heal (Mike GURU-5070 + GURU-BEAST-ROG) -- **BSOD alert dedup key fixed (commit `f0a4b7f`, server v0.3.73):** Dedup key changed from `bsod::` (unique per crash, so recurring BSODs spawned new alert rows and bypassed per-dump mutes) to `bsod::` (stable across recurrences). Now a single perma-mute silences all future recurrences of the same bugcheck. The live duplicate alerts on MSI were resolved via psql, and a stable-key mute was inserted for the recurring `0x116` VIDEO_TDR_FAILURE on agent `a685af29`. -- **MSI EXDEV fix (commit `95ef901`, server v0.3.74):** Site-MSI builder was staging the signed MSI in `std::env::temp_dir()` (`/tmp`, a tmpfs) then `rename`-ing it to `/opt/gururmm/downloads` (root LV) — a cross-device rename that fails with `EXDEV` (os error 18). Every site-specific MSI install request returned 500. Fixed by staging in `downloads_dir` (same filesystem as the final path), matching the pattern the signed-EXE path already used. -- **Duplicate offline server alerts removed (commit `b6ed564`):** Dashboard Triage view was showing server agents twice in offline status — fixed. -- **sync.sh auto-heal (Phase 1, 2026-06-15):** `resolve_submodule_collisions()` helper added to `sync.sh`. On `git submodule update` abort (untracked files would be overwritten), it enumerates the precise colliding paths (intersection of untracked files and paths tracked by the target gitlink commit), moves only those aside to `.synced-aside-`, then retries once. Preserved 94 orphaned RMM_THOUGHTS lines (2026-06-08 re-grounding pass) that a blind `--force` would have discarded. Phase 2 (populate-only guard) added 2026-06-21. -- **`logs/analyze` switched from Ollama to Claude API (commit `c869e4d`):** Log analysis endpoint now calls the Claude API instead of Ollama-on-Beast. -- **500-error-leak fix (commit `58c1a96`):** Closed the remaining 17 sites that returned raw DB errors in 500 bodies, routing them through `internal_err`. No new DB detail is exposed to callers. +- **BSOD alert dedup key fixed (commit `f0a4b7f`, server v0.3.73):** Dedup key changed to `bsod::` (stable across recurrences). +- **MSI EXDEV fix (commit `95ef901`, server v0.3.74):** Site-MSI builder staged in `/tmp` (tmpfs) then renamed cross-device to `/opt/gururmm/downloads`, failing EXDEV. Fixed to stage on the same filesystem. +- **Duplicate offline server alerts removed (commit `b6ed564`).** +- **sync.sh auto-heal (Phase 1):** `resolve_submodule_collisions()` moves only genuinely-colliding untracked files aside and retries, instead of a blind force. +- **`logs/analyze` switched from Ollama to Claude API (commit `c869e4d`).** +- **500-error-leak fix (commit `58c1a96`):** All server error paths route through `internal_err`. --- ### 2026-06-12 — Beast Parallel Windows Build (Lever A) + command_type "cmd" Misdiagnosis Fix (Mike GURU-5070) -**Beast parallel Windows build (lever A, commit `cfbdb59`):** -- Rewrote `run_remote_build()` in `build-windows.sh` to drive concurrent SSH sessions from bash instead of a serial `cmd /c` chain. Two waves: WAVE 1 = 5 concurrent stable builds (amd64, x86, debug, tray, cleanup); WAVE 2 = 2 concurrent legacy-1.77 builds (l_amd64, l_x86). MSI build overlaps Wave 2. Per-job exit-code aggregation; Pluto fallback preserved. -- **Result:** Beast parallel+warm ~336s (cargo 319s); ~46% faster than Beast serial cold (~622s); ~3.8x faster than Pluto serial (~1269s). -- **Key fix:** Per-variant `--target-dir` is mandatory. Both stable variants sharing `target/` serialized on cargo's build-directory lock. Fixed: amd64=`target/release`, x86=`target/x86`, legacy-amd64=`target/legacy-x64`, legacy-x86=`target/legacy-x86`. -- **Dropped `cargo +1.77 fetch` pre-resolve entirely.** It resolved the full dep graph and pulled `wit-bindgen` (edition2024, unsupported by Cargo 1.77.2) — rc=101 on both hosts. The legacy builds scope deps via `--features legacy` (excludes wit-bindgen) when run as actual builds; the pre-fetch was both unnecessary and the failure point. -- **Live verification:** v0.6.64 (first live Beast build, 622s cold), v0.6.66 (parallel fix, 336s). All 8 artifacts signed OK. +**Beast parallel Windows build (commit `cfbdb59`):** Two waves (5 concurrent stable + 2 concurrent legacy-1.77) via bash-driven concurrent SSH instead of a serial `cmd /c` chain. Per-variant `--target-dir` mandatory. Result: ~336s parallel vs ~622s serial cold, ~3.8x vs Pluto serial. -**command_type "cmd" fix (commit `3de9faf`, agent v0.6.66 stable):** -- Root cause of "PST agents can't receive commands": `CommandType` enum had no `cmd` variant. Server-sent `command_type:"cmd"` failed serde parse; agent logged-and-silently-dropped the message (no ACK, no result) — indistinguishable from a NAT black-hole. 7-round adversarial multi-AI quorum + packet captures were used to isolate the network path before the discriminator was found: the one working command used `powershell`, all failures used `cmd`. -- Fix 1: `#[serde(alias = "cmd")]` on `CommandType::Shell` — accepts `"cmd"` as an alias for Shell (runs cmd.exe, for endpoints without PowerShell). -- Fix 2: NAK on unparseable command — `handle_server_message` now sends `CommandAck` + error `CommandResult` on whole-message parse failure (if the `payload.id` is recoverable), instead of silently dropping. Fails fast rather than black-holing. -- **Reverted:** the earlier "evict non-delivering connections" server change (commit `80df458`) — its motivating premise was this `cmd` phantom; with valid commands acking in ms it only added spurious reconnect churn. -- Promoted to stable fleet-wide. `wiki/clients/peaceful-spirit.md` and `.claude/commands/rmm.md` corrected to remove the wrong "use command_type: cmd" advice. +**command_type "cmd" fix (commit `3de9faf`, agent v0.6.66 stable):** `CommandType` enum had no `cmd` variant — server-sent `command_type:"cmd"` failed serde parse and the agent silently dropped the message. Fixed with `#[serde(alias = "cmd")]` on `Shell` + a NAK on unparseable commands. --- -### 2026-06-11 — Physical Server Migration (Cutover + Backfill + Build Pipeline) +### 2026-06-11 — Physical Server Migration, Durable Agent Identity, Agent Comms Durability Phase 1 -**Cutover (morning ~06:53-07:20 MST):** -- Stopped all VM writers, fresh-restored gururmm/guruconnect (PG 18) and claudetools (MariaDB 11.8) on the new physical box via direct SSH to the VM. -- VM released 172.16.3.30; new box took 172.16.3.30 + fresh management IP 172.16.3.47 via self-confirming detached `netplan apply` (auto-reverts if gateway unreachable — prevents a bricked network config). -- All nine stack services active; 162/212 agents reconnected within 15 min. Fleet broadcast sent; 7 migration locks released. -- Two production issues fixed in-window: (1) nginx had stale config missing `/ws` location — `systemctl reload nginx` fixed it and agents reconnected; (2) Prometheus failed due to WAL copied mid-write — moved `wal`/`chunks_head` aside, persisted blocks healthy (~2h of samples lost during downtime anyway). - -**Backfill + Build Pipeline (afternoon session):** -- 7-day whale data (metrics 1,189,924 rows + agent_logs 2,262,938 rows; ~3.4 GB total) backfilled VM->new box via direct SSH `\copy` stream in ~2.5 min. Load verified lossless; SSD sustained 186-214 MB/s writes with ZERO pool-timeouts while 212 live agents wrote concurrently. -- Three build-env gaps found and fixed after first push-triggered build: (1) sccache not installed (copied v0.8.2 from VM); (2) root had no Pluto SSH key (copied guru's id_ed25519 to /root/.ssh); (3) `/etc/gururmm-signing.env` + jsign + Java missing (installed Java + jsign-7.1.jar; recreated jsign wrapper). -- Parallel Windows build prototyped (7 Start-Job isolated builds, 3.51x speedup 28.9m -> 8.2m, 15 rustc vs 1). Replaced by the bash-driven lever-A approach in the 2026-06-12 session. -- First clean build on new box: signed Windows agent v0.6.61 beta. Independently verified (Authenticode, Arizona Computer Guru LLC via Azure Trusted Signing, RFC3161 timestamp, sha256 OK). -- **Old VM decommissioned 2026-06-12:** virsh domain + disk removed after stability soak; no longer exists. +- **Cutover:** New physical box took over 172.16.3.30 (PG 18, Ubuntu 26.04); production down <27 min; old VM decommissioned 2026-06-12. +- **Durable agent identity Phase 1 Task 1:** Multi-location `device_id` (registry+ProgramData on Windows, /etc+/var/lib on Linux, /Library+mirror on macOS) with self-heal; `cleanup.ps1` whitelists identity files. +- **Agent Comms Durability Phase 1 (agent v0.6.63 / server v0.3.68):** Root cause = NAT conntrack asymmetry (server->agent pushes black-holed in idle conntrack gaps). Fix: agent `CommandAck` on receipt + dedup cache; server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via a partial index so old agents keep legacy behavior. --- -### 2026-06-11 — Durable Agent Identity (Ghost Agent Spec + Phase 1 Task 1) +### 2026-06-07 — Credential Inheritance, Offboarding Spec, UI Gap Batch, Backup Alert Quality Pass -**Ghost agent root cause:** -- `device_id` is a random UUIDv4 stored in ONE file (`%ProgramData%\GuruRMM\.device-id` / `/var/lib/gururmm/.device-id`). The agent's own `cleanup.ps1` (Steps 6+7) deleted both ProgramData and HKLM\SOFTWARE\GuruRMM, wiping it. A past hardware->random scheme change also orphaned agents (11 ghosts with old `win-`/`hw-` device_ids). Server dedups ONLY on device_id at WS-connect; no reconciliation. 11 hostnames have duplicates (9 same-site true ghosts; 2 — MSI, SERVER — are different machines at different sites and must NOT be merged). 32 tables FK agent_id. -- Context: channel pin regression discovered (GURU-5070 still on 0.6.59 after v0.6.61 promotion) because the beta override lived on the orphaned ghost enrollment. Fix: set `update_channel='beta'` at site "Mike's Car" (survives re-enrollment); `resolve_agent_channel` = `agent.or(site).or(client).or('stable')`. - -**Spec + Phase 1 Task 1:** -- Multi-AI design review (Gemini + Grok, strong convergence): durable multi-location device_id (primary) + hardware fingerprint (reconciliation guard, not key). Identity model: `agent/src/device_id.rs` rewritten — HKLM registry + ProgramData on Windows; /etc + /var/lib on Linux; /Library + mirror on macOS — with self-heal, plus `cleanup.ps1` whitelisting the identity files. Committed 0b81d33 (v0.6.62). Linux build green. -- Spec: `specs/durable-agent-identity/` (shape/plan/references/standards). Phase 1 (Task 1 shipped) stops ~99% of new ghosts. Phase 2 (guarded auto-reclaim) + Phase 3 (operator merge tool for 9 existing ghosts) per spec after soak. - ---- - -### 2026-06-11 — Agent Comms Durability Phase 1 (Spec + Slices A/B/C + Deploy + Fleet) - -**Root cause (confirmed in code, ~95% AI-validated):** NAT conntrack asymmetry. Commands are a server->agent PUSH over the persistent WS; agent->server heartbeats refresh the UDR conntrack and replies ride back. A server->agent command push in an idle gap hits a torn-down conntrack and is silently dropped (server `sender.send()` buffers without error for ~15 min). The server's 30s WS ping (server->agent) CANNOT revive a dead conntrack; only agent->server can. Server had no half-open detection (read loop ignores Pong). Agent already heartbeats every 30s — so the conntrack floor is sub-30s — meaning durable delivery nets are the correct fix, not keepalive alone. - -**Spec:** `specs/agent-comms-durability/` (shape/plan/references/standards). Multi-AI reviewed (Gemini 3.1-pro + Grok 4.3, 2 rounds, strong convergence). Phased: Phase 1 = durable delivery (shipped); Phase 2 = live TTY (planned); Phase 3 = AIMD keepalive + bulk->HTTPS + half-open eviction (planned). - -**Slice A (dormant server foundation — committed b93f2ef):** Migration 058 adds `commands.acked_at TIMESTAMPTZ`. Server-side `CommandAck` handler arm and `mark_command_acked()` in db. Additive + dormant — no behavior change until agents send ACKs. - -**Slice B (agent — committed 45870b1, deployed 2026-06-11 as v0.6.63):** `AgentMessage::CommandAck { command_id }` added. Agent sends CommandAck on RECEIPT of every dispatched command (before execution). `CommandExecutor` gained a FIFO-bounded recent-results cache (64 entries, 256 KB cap each): re-reports cached result for an already-completed command; ignores an in-flight duplicate; never double-runs. Result is cached BEFORE deregistering the running task so a re-delivery in the complete()/send() gap always finds it cached (re-report) rather than not-running (re-run). - -**Slice C (server — committed ca1657b, deployed 2026-06-11 as v0.3.68):** Migration 059 adds `delivery_attempts INTEGER NOT NULL DEFAULT 0` + partial index `idx_commands_agent_acked ON commands(agent_id) WHERE acked_at IS NOT NULL` (capability gate). `db::requeue_undelivered_commands` returns an un-acked command past a 60s ACK deadline to `pending` (re-delivery, never failed). `db::fail_timed_out_commands` rewritten into three safe cases — can no longer falsely-fail a deliverable command. `mark_command_running` increments `delivery_attempts` (cap 10). Reconnect block refactored into shared `redispatch_pending_commands` helper (send_to-based); called on EVERY heartbeat — so a black-holed push is re-delivered on the next agent beat (which just refreshed the NAT conntrack) without needing a reconnect. - -**Capability gate (version-independent):** Reaper only re-delivers for agents that have demonstrably ACK'd at least once (`EXISTS(... acked_at IS NOT NULL)`). Old non-ACK agents keep the legacy fail-on-timeout path and can never double-run. Safe to deploy at any point during gradual agent rollout. - -**Deploy + fleet rollout:** -- Installer bug fixed (5c0d004): `Start-Process -Wait -PassThru` + exit code check (old `& $stagingPath install 2>&1 | Out-Null` swallowed output + surfaced misleading NativeCommandError). Agent CLI ANSI log garbage fixed (parse before logging init, file-only for one-shot CLI). -- Server auto-deployed by webhook as v0.3.68 (commit 7376340). Migrations 058+059 applied. All durability code live server-side. -- Build pipeline dubious-ownership fixed: `git config --system --add safe.directory /home/guru/gururmm` (/etc/gitconfig; webhook runs builds as root on guru-owned repo). -- Canary: clients Goldstein + Peaceful Spirit set to `update_channel='beta'`; all 6 canary agents updated 0.6.61->0.6.62->0.6.63 within ~24 min. PST-SERVER (UDR Ultra, the original affected agent) verified: `hostname; whoami` returned exit 0, `acked_at` stamped, `ack_latency 16 ms`, `delivery_attempts 1`. Full ACK durability chain proven live. -- Fleet promotion: `POST /api/updates/rollouts/0.6.63/promote {"os":"windows","arch":"amd64"}` -> 6 files re-tagged stable. 168 agents on 0.6.63 in ~3 min; online count held at 182 throughout (no mass dropout). - ---- - -### 2026-06-07 — UI Gap Batch + Enrollment Audit (GURU-BEAST-ROG session) - -**API changes:** -- `get_agent` handler now returns `AgentWithDetails` — includes `client_id` and `client_name` via JOIN query -- New endpoints: `GET /api/agents/:id/bsod-events`, `GET /api/agents/:id/version-history` -- New endpoints: `GET /api/install-reports`, `GET /api/install-reports/:id` -- New endpoint: `GET /api/discovery/all-devices` -- `DELETE /api/agents/:id/key` redesigned as dual-revoke: nulls `agents.api_key_hash` (legacy Mode 1/2) AND sets `enrolled_agents.revoked = TRUE` (modern agk_ Mode 3) -- New endpoint: `GET /api/sites/:id/enrolled-agents` — full enrollment audit with duplicate-hostname annotation -- New endpoint: `POST /api/enrolled-agents/:id/revoke` — per-enrollment targeted revocation with cascade - -**Dashboard additions:** -- AgentDetail: Crashes tab (BSOD events), version history table in Updates tab -- Dashboard.tsx: fleet stats wired to `/agents/stats` endpoint (was computed client-side) -- SiteDetail: Revoke Key button + Enrollment tab with full audit table, status badges, duplicate-hostname warnings, per-row revoke -- New pages: InstallReports (`/install-reports`), Discovery (`/discovery`) -- FunctionRail: nav links for Install Reports + Fleet Discovery - -**Hotfix (production 500s):** `get_agent_with_details_by_id` SQL was missing `a.role_override` (migration 054 added it as non-optional). All `GET /api/agents/:id` calls returned 500 until hotfix commit `6faa382`. - -**Key commits:** `5854c63` (UI gap batch), `6faa382` (hotfix), `49a7109` (enrollment audit batch), `00129af` (fix route wiring) - ---- - -### 2026-06-07 — Credential Inheritance Deployment & Offboarding Spec - -- Server v0.3.45 deployed with credential inheritance (Global → Client → Site, `is_inheritable` flag, de-duplication by (credential_type, label), `/effective` endpoints). -- Dashboard: clickable alert severity badges with client filtering; offline badge scopes to client-specific agents. -- SPEC-028 offboarding wizard specification created (835 lines) — 6-step site + 5-step client offboarding modals, data export (temp tokens, 1h expiry), typed confirmation, audit logging, cascade deletion. Implementation pending. -- FEATURE_ROADMAP.md updated with "Client & Site Lifecycle Management" section. +- Server v0.3.45: credential inheritance (Global -> Client -> Site), `/effective` endpoints. +- Role-aware offline alerting + alert ignore/mute shipped (server-only `agent_offline`, `alert_mutes` perma-silence). +- Backup-alert quality pass: false `backup_failed` 15 -> 2 fleet-wide; `backup_storage_low` alert type removed (false-positive metric). +- UI gap batch + enrollment audit; SPEC-028 offboarding wizard specification (835 lines, implementation pending). --- ## Capabilities / Feature Set -*Synthesized from authoritative artifacts (API routes, agent modules, 64 migrations through migration 064, roadmap, commit log) — not from session logs alone. See Compilation Notes.* +*Synthesized from authoritative artifacts (API routes, agent modules, 65 migrations through migration 065, roadmap, commit log) — not from session logs alone. See Compilation Notes.* Agent<->server communication is a persistent authenticated WebSocket with auto-reconnect + heartbeat; on reconnect, in-flight commands flip to `interrupted`. Platform-parity rule: agent features ship on Windows/Linux/macOS in the same change (stub + TODO where a real impl isn't yet feasible). ### Monitoring & Telemetry -- Core metrics per interval (policy-tunable): CPU %, memory %/bytes, disk %/bytes, network rx/tx deltas, uptime, logged-in user, user idle time (Win `GetLastInputInfo`, Linux `xprintidle`), public/WAN IP (cached, multi-service fallback). Cross-platform via `sysinfo`. -- Hardware sensor telemetry: CPU/GPU temps + full sensor array (temperature, voltage, fan RPM, power). Linux via `/sys/class/thermal`; `sysinfo::Components` fallback. **Windows LibreHardwareMonitor removed 2026-05-27 (CVE-2020-14979); Windows thermal collection pending a safe replacement strategy.** -- Process drill-down: top 10 by CPU + top 10 by memory in every metrics payload (cross-platform). +- Core metrics per interval (policy-tunable): CPU %, memory %/bytes, disk %/bytes, network rx/tx deltas, uptime, logged-in user, user idle time, public/WAN IP. Cross-platform via `sysinfo`. +- Hardware sensor telemetry: CPU/GPU temps + full sensor array. Linux via `/sys/class/thermal`; `sysinfo::Components` fallback. Windows LibreHardwareMonitor removed (CVE-2020-14979); replacement pending. +- Process drill-down: top 10 by CPU + top 10 by memory in every metrics payload. - Network state: per-interface IPv4/IPv6, MAC, derived CIDR subnets — sent on change only. -- **Windows BSOD/kernel-crash detection (Phase 1, shipped 2026-06-01):** `agent/src/bsod.rs` polls `C:\Windows\Minidump` for new `.dmp` files (5-min filetime poll), parses kernel dump header at fixed `DUMP_HEADER64`/`DUMP_HEADER32` offsets (bugcheck code @0x38, 4 parameters, FILETIME @0xFA8 — the `minidump` crate only parses Breakpad MDMP, not Windows kernel PAGEDU64 dumps), cross-references System event log (WER 1001 / Kernel-Power 41) for Report Id + faulting driver, deduplicates via `C:\ProgramData\GuruRMM\bsod-seen.json` watermark. Sends `AgentMessage::BsodEvent`. Server: migration 048 + `server/src/db/bsod_events.rs` + ws handler — always-Critical alert deduplicated by `(agent_id, bugcheck_code)` (key changed 2026-06-15 from dump_sha256 to bugcheck_code so a mute silences all recurrences). Dashboard Crashes tab shipped 2026-06-07. Phase 2/3 deferred: BSOD in Alerts stream, `fetch_bsod_dump` on-demand upload, full ~350-entry bugcheck name table (Phase 1 ships a 10-code map). +- **Windows BSOD/kernel-crash detection:** `agent/src/bsod.rs` polls `C:\Windows\Minidump`, parses kernel dump header at fixed offsets, cross-references System event log, deduplicates via a watermark file. Always-Critical alert deduplicated by `(agent_id, bugcheck_code)` (changed from dump-hash 2026-06-15 so a mute silences all recurrences). +- **Inventory NUL/control-char hardening (2026-07-05, server v0.3.95):** `server/src/db/sanitize.rs` strips NULs and C0/C1 control chars from inventory jsonb values/keys and text columns before upsert — fixed silent storage failure on 7+ machines carrying embedded NULs in Windows registry strings. ### Remote Execution -- Command types: `shell`, `powershell`, `python`, raw `script{interpreter}`, and `claude_task`. Options: `timeout_seconds`, `elevated`. **Also accepts `cmd` as an alias for `shell` (added 2026-06-12, commit `3de9faf`). Unknown/unparseable commands now NAK immediately with an error result (fail-fast, not silent-drop).** -- **Execution context** (`041_add_command_context`): `system` (default — Session 0 / service SYSTEM) or `user_session` (runs in the active logged-on user's desktop session via WTS token impersonation: `WTSQueryUserToken` + `CreateProcessAsUserW` + per-user env block). Windows-only; requires an active session. -- Commands individually cancellable (`POST /commands/:id/cancel`) and aborted on disconnect. Auditable history; status running/completed/failed/timeout/interrupted. -- Script library (`017_scripts`): stored scripts dispatched to agents with args/env/timeout/`run_as_user`; per-run history. -- **Agent Comms Durability (Phase 1, shipped 2026-06-11 — agent v0.6.63 / server v0.3.68):** - - Problem: server->agent command pushes black-holed at NAT'd sites (UDR Ultra) when conntrack gaps tear down the inbound direction. Agent heartbeats work (agent->server refreshes conntrack); reaper falsely failed black-holed commands after timeout. - - Fix: agent sends `CommandAck { command_id }` on RECEIPT (migration 058 `acked_at`). Agent deduplicates by command_id — re-reports cached result, ignores in-flight duplicate, never double-runs. Server reaper re-delivers an un-acked command past 60s ACK deadline (returns to `pending`) instead of failing it. Migration 059 adds `delivery_attempts` counter (cap 10) + partial index for capability gate. Pending commands re-offered on (re)connect AND on every heartbeat. Verified at PST-SERVER: `acked_at` stamped, `ack_latency 16 ms`, `delivery_attempts 1`. - - Capability gate: reaper only re-delivers for agents that have ACK'd at least once (`idx_commands_agent_acked` partial index). Old agents keep legacy fail-on-timeout path; safe mid-rollout. - - Code anchors: `server/src/db/commands.rs` (`requeue_undelivered_commands`, `fail_timed_out_commands` rewrite, `DELIVERY_ACK_DEADLINE_SECS=60`, `MAX_DELIVERY_ATTEMPTS=10`), `server/src/ws/mod.rs` (`redispatch_pending_commands`, `CommandAck` handler), `agent/src/transport/websocket.rs` + `agent/src/commands/mod.rs` (ACK send + recent-results cache, `RECENT_CAP=64`, `MAX_CACHED_RESULT_BYTES=256 KB`). - - Phase 2 (live TTY — rides warm WS, seq/resume, cadence switch) and Phase 3 (Adaptive keepalive, bulk file transfer -> short-lived HTTPS, server half-open eviction sweeper) are PLANNED, not shipped. +- Command types: `shell`, `powershell`, `python`, raw `script{interpreter}`, `claude_task`, and `cmd` (alias for `shell`). Options: `timeout_seconds`, `elevated`. Unknown/unparseable commands NAK immediately. +- **Execution context** (`041`): `system` (default) or `user_session` (WTS token impersonation, Windows-only). +- Commands individually cancellable, aborted on disconnect. Auditable history. Script library (`017`) for stored scripts. +- **Agent Comms Durability (Phase 1, shipped 2026-06-11 — agent v0.6.63 / server v0.3.68):** Agent sends `CommandAck` on receipt (migration 058 `acked_at`); dedup via a recent-results cache. Server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via `idx_commands_agent_acked` partial index so pre-durability agents keep legacy fail-on-timeout behavior. Phase 2 (live TTY) and Phase 3 (adaptive keepalive, bulk->HTTPS, half-open eviction) remain PLANNED, not shipped. + +### Volume Shadow Copy / VSS (Windows) — Policy Configurator (SPEC-016 redesign, shipped 2026-07-03, agent v0.6.76, beta channel only) +- **The agent no longer schedules its own create+prune loop.** It is a **configurator**: on each policy apply it idempotently sets two kernel-enforced retention governors — VSS diff-area **size cap** (native `IVssDifferentialSoftwareSnapshotMgmt`) and `MaxShadowCopies` count (evaluated, but NOT machine-globally written — see below) — and registers a scheduled task whose action is the agent's own native-COM **create** (`IVssBackupComponents::DoSnapshotSet`, `VSS_CTX_CLIENT_ACCESSIBLE`), never a delete. Retention is enforced by Windows' own FIFO eviction (`volsnap.sys`) when a governor is hit, eliminating the MITRE T1490 "Inhibit System Recovery" surface that had gotten the agent blocked by CrowdStrike Falcon. +- **Works uniformly on server AND client SKUs** (previously `vssadmin create shadow`, Server-only, gated client support) — the underlying VSS COM API and `volsnap` cap eviction are not SKU-restricted, only the `vssadmin.exe` CLI is. +- **`MaxShadowCopies` is NOT set machine-globally** (pre-merge code review caught this as a data-loss defect — it would FIFO-evict OTHER VSS consumers' shadows, e.g. System Restore or third-party backup, below the configured count). The per-volume size cap is the sole enforced governor; `retention_count`/`retention_max_age_days` policy fields are advisory/ignored (a warning fires if `retention_max_age_days` is still set, since age-based pruning requires deletion). +- On first policy apply, the legacy `GuruRMM-VSS-Snapshot` scheduled task (present fleet-wide) is removed and replaced with `GuruRMM-VSS-Create`, supporting both daily-time and every-N-hours interval triggers. +- On-demand admin operations (create/delete/browse/restore via `dispatch_operation`, `GET /agents/:id/vss/restores`) are unaffected — this redesign only removes the *scheduled* create+prune loop. +- **Not yet promoted to the stable channel** as of this compile (beta head 0.6.76; stable pinned at 0.6.66). ### Software Inventory & Remote Uninstall (SPEC-030 — BETA, not guaranteed) -- **Maturity:** code merged to main + deployed (2026-06-23..24), but the feature is **beta and unreliable** — best-effort only. It works on cleanly-uninstallable programs; it is NOT guaranteed for managed/protected AV, WiX/Burn bundles, UI-only uninstallers, drivers/system components, or Launchy/AIMP-class uninstallers that hang the agent. Verify every removal; do not depend on it for production removals yet. -- **Route B (server-orchestrated):** the server drives removal via agent commands; chosen over embedding the engine per-command (32K dispatch limit). Endpoints: `GET /agents/:id/software` (installed-programs list), `POST /agents/:id/software/uninstall` (async bulk uninstall → `job_id`), `GET /agents/:id/software/uninstall/jobs/:job_id` (live job poll), `GET /agents/:id/software/removal-status` + `POST .../removal-status/:attempt_id/resolve` (removal-tracking loop), `GET /software/knowledge` + `POST /software/knowledge/classify` (fleet KB). -- **Silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`): Tier-1 silent removal, BCU-informed detection + fail-fast, vendor silent-switch table, captures real process exit codes. **Refuses to guess** — returns `needs_interactive` rather than risk a wrong removal (Avast/AVG/Malwarebytes correctly bounce). **Generic post-success verification** (`Test-StillInstalled`, ARP re-check by product_code when present) downgrades a false "success" to failed — catches the WiX/Burn Package-Cache-bundle 1605 false-success class (skip MSI tier for `\Package Cache\` bundles). Orphaned ARP entries (missing uninstaller) classified `tier=orphaned`; reboot-pending (3010/1641) skips the rescan; async-fork uninstallers get retry-verify (~9s/4 checks). -- **Async removal jobs (migration 064):** bulk uninstall is non-blocking — `software_removal_jobs` row created, `job_id` returned in ~0.06s, a background worker processes targets one at a time recording each outcome as it lands, dashboard polls a live progress bar. Replaces the synchronous request that died at the proxy's ~100s timeout and that the agent kill-timeout truncated mid-flush. `server/src/db/software_jobs.rs`. -- **Fleet removal knowledge base (migrations 061-063):** `software_removal_attempts` (per-agent: programs that did NOT remove silently — failures + no-silent-path — so the dashboard shows what still needs removing and flags ones needing a remote GuruConnect session; flips to resolved on a later success). `software_knowledge` (fleet "our list" keyed by exact DisplayName, three-state: `silent` = verified silent method exists / `requires_ui` = VM-verified cannot be removed silently / `unknown` = default-uninstaller log kept). `software_knowledge_timing` (self-tuning per-target uninstall timeout — running sample count + slowest success observed; wide while barely measured, narrowing toward measured reality; 1605 no-ops excluded). -- **Dashboard:** `SoftwareManager.tsx` — live installed-programs list, bulk-select, fire-then-poll uninstall with live progress bar + per-program results; three-state removal-knowledge UI. -- **Known gaps:** drivers/system components register in ARP and appear in the list (a top-down bulk-select can nuke a driver — UI exclusion/flag not yet built); Launchy/AIMP-class uninstallers hang the agent on an orphaned child process tree (~420s kill-timeout) → `failed`/no-output, fundamentally need recipes / agent-side process handling (Route A); no list-jobs endpoint yet (get-by-id only). The **av-removal-recipes spec (P3, removal-only)** is the recipe-tier follow-on. +- **Maturity:** code merged to main + deployed (2026-06-23..24), but the feature is **beta and unreliable** — best-effort only. +- **Route B (server-orchestrated):** `GET /agents/:id/software`, `POST /agents/:id/software/uninstall` (async -> `job_id`), `GET /agents/:id/software/uninstall/jobs/:job_id`, `GET /agents/:id/software/removal-status` + resolve, `GET /software/knowledge` + classify. +- **Silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`): refuses to guess (`needs_interactive` on ambiguity); generic post-success ARP re-check (`Test-StillInstalled`) catches the WiX/Burn Package-Cache-bundle false-success class; orphaned ARP entries classified `tier=orphaned`; async-fork uninstallers get retry-verify. +- **Async removal jobs (migration 064):** non-blocking bulk uninstall — job created + `job_id` returned in ~0.06s, background worker processes targets one at a time, dashboard polls live progress. +- **Fleet removal knowledge base (migrations 061-063):** per-agent removal-tracking loop, fleet three-state KB (`silent`/`requires_ui`/`unknown`), self-tuning per-target timeout. +- **Known gaps:** drivers/system components register in ARP and appear in the bulk list (no exclusion/flag yet); Launchy/AIMP-class uninstallers hang the agent on a lingering child process; no list-jobs endpoint (get-by-id only). The **av-removal-recipes spec (P3, removal-only)** is the follow-on. + +### Third-Party Integrations Framework (NEW, migration 065, shipped 2026-07-01..02) +- **Vendor-agnostic plugin framework:** `integrations` (one enabled plugin per partner+kind), `integration_credentials` (AES-256-GCM encrypted, same `Encryptor` chokepoint as the `credentials` table), `integration_client_mappings` (RMM client -> external tenant id, e.g. a Falcon CID — single-CID partners map every client to one id, MSSP partners map child CIDs), `integration_sync_state` (per-integration sync health), `integration_agent_links` (RMM agent <-> external agent id + last-synced state, e.g. Falcon AID/sensor version/RFM). +- **API:** `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `DELETE /api/integrations/:id/mappings/:mapping_id`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test`. +- **First plugin: CrowdStrike Falcon.** API client + periodic sync worker (default `sync_interval_seconds=900`) pulls sensor state into `integration_agent_links`. Framework is modeled on the proven per-partner MSPBackups shape (migration 034). +- **No dedicated dashboard UI yet** — this is API/backend-only as of this compile (verify before claiming a management screen exists). ### Inventory & Discovery -- Hardware inventory (mfr/model/serial/BIOS, CPU, memory, disks, NICs, OS), software inventory (installed apps), service inventory. On-demand refresh. -- VM / hypervisor / container detection (`032/033`): `is_virtual_machine`, `hypervisor_type`, `vm_uuid`, `is_hypervisor` + hosted VM UUIDs, `is_container`, `is_unraid`. -- User/group inventory (`037`-`040`): local + domain + Azure AD accounts (enabled, pw-never-expires, last-logon, is_admin, AD email/UPN/dept), domain-join classification (none/ad/aad/hybrid), domain name, M365 tenant ID, **domain-controller detection (`is_dc`)**, group membership. Policy-scheduled (default 24h). -- Network discovery: server-dispatched scans — TCP probes over configurable ranges/ports, ARP MAC, reverse DNS, basic OS fingerprint; devices stream back and are persisted/editable. +- Hardware inventory (mfr/model/serial/BIOS, CPU, memory, disks, NICs, OS), software inventory, service inventory. On-demand refresh. NUL/control-char sanitized before storage (2026-07-05, see Monitoring section). +- VM / hypervisor / container detection (`032/033`). +- User/group inventory (`037`-`040`): local + domain + Azure AD accounts, domain-join classification, domain-controller detection, group membership. +- Network discovery: server-dispatched TCP/ARP/reverse-DNS scans; devices stream back and are persisted/editable. ### Patch / Agent Update Management -- Self-updater: server sends version + URL + SHA256; agent downloads, verifies checksum, atomically swaps binary, restarts, and **auto-rolls back** to a kept backup if it fails to reconnect (~180s window). -- Auto-update gated on effective policy `auto_update` (channel + defer_hours). Force via `POST /agents/:id/update`. Update channels at agent/site/client (`026`). -- **Channel model:** new builds tag **beta** by default; agents default to **stable** channel; promotion to stable is a DELIBERATE re-tag of the `.channel` sidecar. `scanner.rs` `get_latest_version`: stable agents get latest STABLE-tagged binary; beta gets absolute-latest. `resolve_agent_channel` = `agent.or(site).or(client).or('stable')`. -- **Promotion API:** `POST /api/updates/rollouts/:version/promote` body `{"os","arch"}` — re-tags all `.channel` files for the version, records `update_rollouts` row, force-rescans. Rollback: `POST /api/updates/rollouts/:version/rollback`. -- Safe-rollout (`046`): `update_rollouts`/health-metrics/events tables + `/updates/rollouts` promote/rollback. **Health-gated automation is written-but-unwired (Phase 2); promotion is manual via API.** -- **Container guard (BUG-019, shipped `66a7f4e`, 2026-06-21):** Agent inside a container early-returns from `perform_update()` with `UpdateStatus::Failed` + clear message, skipping the in-place binary swap that caused silent downgrades on container recreate. Full image-update path (SPEC-023) is not yet implemented. -- **Legacy fleet build support (two-wave parallel build):** Agent ships both a stable-toolchain (modern x86_64 + x86, debug, tray, cleanup) wave and a legacy wave (Rust 1.77, `--features legacy`, for Win7/Server 2008 R2 endpoints). The `gururmm-build` skill wraps pre-merge verification. IMPORTANT: the legacy wave resolves deps fresh on Rust 1.77 — dep pins to avoid edition2024 pulls are required (BUG-021 lesson: pin `getrandom 0.3.1` + `zeroize 1.8.1`). -- **Universal self-detecting installer (Feature 9, P1 shipped v0.6.71):** A single install path self-detects arch/OS (x64 / x86 / legacy / ARM) and pulls the correct agent binary, so one URL/script covers the full Windows variant matrix. The dashboard install UI points at it (replaces the hardcoded-amd64 `irm … | iex` one-liner). MSI stays for GPO/Intune. P2 (native i386 bootstrapper EXE for locked-down boxes) + an offline-bundle variant are pending. +- Self-updater: server sends version + URL + SHA256; agent downloads, verifies, atomically swaps, restarts, auto-rolls-back on failed reconnect. +- Auto-update gated on effective policy `auto_update`. Force via `POST /agents/:id/update`. +- **Channel model:** new builds tag **beta** by default; agents default to **stable**; promotion is a deliberate re-tag. +- **Update-scanner platform coverage fixed 2026-07-05 (server v0.3.95):** the scanner's self-check previously tried to execute every binary regardless of platform to read its embedded version, dropping macOS/Linux artifacts entirely because they can't run on the Linux build host — **macOS/Linux agents had never been offered an update**. Now uses host-native-vs-foreign self-check regimes: foreign-platform binaries trust the filename-embedded version; host-native binaries stay a hard `--version` gate (with a best-effort exec-bit repair for suspected 0644 CI-copy artifacts). Post-fix: scanner indexes 8 platform/arch combos (was 4, Windows-only). +- **Legacy fleet build (Server 2008R2/Win7, Rust 1.77) DISABLED 2026-07-04** (`LEGACY_ENABLED=false`): the unpinned dep re-resolve pulled edition2024 crates Rust 1.77 cannot parse, fail-closing the entire Windows build. Last legacy artifact (agent 0.6.76) frozen and preserved for existing 2008R2/Win7 endpoints; modern builds advance normally. 2008R2 support returns only via a separate legacy-only rewrite (RMM_THOUGHTS Feature 12, Raw — not approved). +- **Universal self-detecting installer (Feature 9, P1 shipped v0.6.71):** one install path self-detects arch/OS and pulls the correct binary. +- **Promotion API:** `POST /api/updates/rollouts/:version/promote` body `{"os","arch"}`; rollback via `.../rollback`. + +### Agent Identity & Enrollment +- Per-agent enrollment keys; site-specific MSI with SITEKEY injected. `enrolled_agents` is the authoritative enrollment history log. +- **Durable multi-location `device_id`** (Phase 1 Task 1, 2026-06-11): registry+ProgramData (Windows), /etc+/var/lib (Linux), /Library+mirror (macOS), self-healing; `cleanup.ps1` whitelists identity files. +- **Cross-site dedup fixed 2026-07-04 (commits `2851523`, `1c1cac2`):** Both WS enrollment paths previously deduped only within `(site_id, device_id)`, so re-running a different site's installer against an already-enrolled machine created a cross-site duplicate. Fixed to match by hardware `device_id` **across all sites** (`db::get_agent_by_device_id`) and re-home via `move_agent_to_site` on first match. A same-day follow-up stopped auto-re-homing **on every reconnect** — the agent re-presents its embedded install-site code as its API key on every beat, which was silently bouncing admin-relocated agents back to their original site. Current behavior: dedup by device_id, but site placement is admin-controlled and persists across reconnects. **13 duplicate-hostname agent pairs remain fleet-wide** (pre-fix legacy `win-*`/`hw-*` records) — cleanup pass not yet done; these are the ongoing trigger for `move_agent` 409s. ### Policy & Configuration Management -- Inheritance chain global -> client -> site -> agent; server computes merged effective policy, pushes via `ConfigUpdate`. Effective policy queryable per scope. -- Checks engine (`018`/`019`): cpu, memory, disk, ping, port, script, service (restart-if-stopped, pass-if-not-exist; Win `sc.exe`, Linux `systemctl`). Policy-attached check templates (`024`) with push-to-agent sync. On-demand `run-checks`. -- Remote registry (Windows, `winreg`): agent supports enumerate/read/write (typed)/create/delete. **HTTP API currently exposes read-only (enumerate, read_value); write paths exist in the agent but are not routed yet [verify].** -- **Event Log Watch (migration 047):** Server-configured rules for monitoring Windows Event Log channels by event ID / level. Agent reports matching events. Dashboard management UI at `/event-log-watches` shipped 2026-06-21 (`0fa65f5`). CRUD + enable/disable for global and per-agent scope. **HIGH fix (PR #43, pending merge):** policy assign/unassign no longer wipes connected agent's Event Log Watch monitoring (fixed by reusing `watch_rules_for_agent` shaper as single source of truth across all config-push sites). +- Inheritance chain global -> client -> site -> agent; server computes merged effective policy, pushes via `ConfigUpdate`. +- Checks engine (`018`/`019`): cpu, memory, disk, ping, port, script, service. Policy-attached check templates (`024`). +- Remote registry (Windows): enumerate/read routed; write paths exist in the agent but are not routed yet [verify]. +- **Event Log Watch (migration 047):** dashboard management UI at `/event-log-watches`. Policy-clobber class bug fixed (PR #43) by reusing a single rule-shaper across all config-push sites. +- **Tray policy gate (NEW, shipped 2026-07-04, code merged, policy default OFF):** `tray.enabled` policy flag is now enforced by the watchdog (previously plumbed end-to-end but not enforced — the tray launched/showed regardless of the flag). Fleet-wide materialize (deploying the binary via agent-update, not just fresh MSI) has shipped in code; the deliberate flip to enable it fleet-wide (spec's "Task 4") has NOT happened yet. ### Alerting & Watchdog -- Threshold alerts (ack/resolve, per-agent + fleet summary, dashboard filter). Alert templates (`022`) with effective resolution; per-client email settings (`020`). Maintenance mode (`021`) to suppress alerting per scope. -- Watchdog: **separate** supervising process (polls `GuruRMMAgent` every 30s, restart backoff) + launches/reaps the tray into active user sessions via WTS. The watchdog reports to the server via the **REST `POST /watchdog-alert` endpoint** (fires after 3 consecutive failed restarts). This is the only supported watchdog alert path. -- **[CORRECTED 2026-06-22 per BUG-022]:** The `WatchdogEvent` WebSocket variant was dead code — defined in the agent transport enum and fully handled server-side (`insert_watchdog_event`, `watchdog_events` table), but the watchdog is a **separate companion process with no WebSocket connection** and never had a producer. Granular per-restart WS events were never implemented and the path has been removed (PR #45, pending merge). The empty `watchdog_events` table remains in the DB (no DROP migration, to avoid migration-number collision). `watchdog_events = 0` all-time in a healthy fleet is correct and expected. Granular REST-based watchdog visibility is a future RMM_THOUGHTS item. -- **BUG-020 — tray duplicate/ghost icons (fixed to beta 2026-06-04; fix #3 wiring in PR #40, pending merge):** Root cause: `TrayLauncher` tracked launches only in an in-memory `HashMap` that resets on watchdog restart; no single-instance guard in the tray; `terminate_all` hard-killed via `TerminateProcess` skipping `NIM_DELETE` (ghost). Fix (commit `137dd85`): (1) per-session `Local\GuruRMM_Tray` single-instance mutex; (2) launcher reconciliation via `WTSEnumerateProcessesW`; (3) graceful `Global\GuruRMM_TrayShutdown_{sid}` event -> 3s wait -> TerminateProcess fallback. Fix #3 (wiring `terminate_all` into watchdog policy-disable/uninstall path) is in PR #40 (SPEC-021+BUG-020 branch). -- **Role-aware offline alerting (shipped 2026-06-07):** Scheduled offline-sweep evaluator (60s tokio interval; `server/src/alerts/offline.rs`). Server-only `agent_offline` alerts. Classifier: `os_product_type` IN {2,3} -> server; else `os_name`/`os_version` ~/server/i -> server; else manual `role_override` (migration 054). Site rule (>=50% + >=3) -> `mass_offline_site`; fleet rule (>=10) -> `mass_offline_fleet`. Dashboard: servers elevated individually; workstations collapsed. `PUT /api/agents/:id/role-override`. Known gap: `os_product_type` populated on only ~16/168 agents; `os_name` is the workhorse classifier. -- **Alert ignore/mute — perma-silence (server only, shipped 2026-06-07; dashboard pending):** `alert_mutes` table (migration 055) + `muted` alert status. Keyed on `dedup_key`. Permanent until un-ignored; reason required (400 if missing). Gate at `create_or_update_alert` AND `create_check_alert` bypass. `POST /api/alerts/:id/mute` + `/unmute`. Dashboard Ignore/Muted/Un-ignore NOT yet built. +- Threshold alerts (ack/resolve, per-agent + fleet summary). Alert templates (`022`). Maintenance mode (`021`). +- Watchdog: separate supervising process; reports via `POST /watchdog-alert`. +- **Role-aware offline alerting + alert ignore/mute (shipped 2026-06-07):** server-only `agent_offline`; site/fleet mass-offline aggregation; `alert_mutes` perma-silence. +- **`create_or_update_alert` hardened (2026-07-05, server v0.3.95):** now returns `Result>` — `Ok(None)` when the target alert row was deleted mid-flight (agent cascade-delete FK violation) instead of aborting the caller. Both UPDATE-by-id branches switched from `fetch_one` to `fetch_optional` with fall-through to INSERT (`refresh_alert_row` helper dedups the two near-identical UPDATE blocks). Fixed a `RowNotFound` that had been aborting the entire offline sweep whenever an alert was concurrently deleted (e.g. during duplicate-agent cleanup). +- **WebSocket log noise reclassified INFO, not ERROR (2026-07-05, server v0.3.96):** benign disconnects (`is_benign_ws_disconnect`) and pre-auth connection churn (`is_preauth_connection_churn` — sleeping laptops, scanners, half-open NAT never sending an auth frame) no longer mint `server_error` alerts. Genuine auth rejections (bad API key, malformed auth message) remain ERROR. Classified at INFO (not DEBUG) so a mass-disconnect anomaly stays visible at the server's default log level. +- **`move_agent` duplicate-key hardened (2026-07-05):** unique-violation on `idx_agents_site_device` now maps to 409 CONFLICT with an actionable message instead of a raw 500. ### Credentials Management -- Encrypted credentials vault (`016`): scoped global/client/site, typed (password, SSH key, SNMP), metadata-only by default with separate `/reveal` decrypt endpoint. Known HIGH item: `/reveal` ownership-scope check — [verify current state]. -- **Credential inheritance (deployed 2026-06-07):** Opt-in hierarchical cascade with `is_inheritable` flag (Global → Client → Site). De-duplication by (credential_type, label), most-specific scope wins. `/effective` endpoints return inherited + direct credentials with `inherited_from` indicator. +- Encrypted credentials vault (`016`): scoped global/client/site, typed, metadata-only by default with a separate `/reveal` endpoint. Known HIGH item: `/reveal` ownership-scope check — [verify current state]. +- **Credential inheritance (deployed 2026-06-07):** Opt-in hierarchical cascade, de-duplication, `/effective` endpoints. +- **Integration credentials (2026-07-01, migration 065):** reuses the same encryption chokepoint for third-party integration API keys (never sent to agents, never plaintext in `integrations.config`). -### Audit Log (Migration 056) -- Append-only `audit_log` table (migration 056): `(id, occurred_at, user_id, action, target_type, target_id, detail JSONB)`. Actor preserved with `SET NULL` on user delete. Seeded by credential reveals (`credential.reveal`); generic shape for future sensitive operations. Indexes on `occurred_at DESC`, `user_id`, `(target_type, target_id)`. - -### Systemic Log Feedback Intelligence (Migration 057) -- Per-line provenance + deterministic fingerprint added to `agent_logs` table (nullable; new rows populate, old rows stay NULL): `agent_version`, `signature_hash BIGINT`, `normalizer_version SMALLINT`. Fingerprint computed server-side at ingest (`server/src/fingerprint.rs`). -- Durable aggregate `log_signatures` table: one row per distinct normalized error signature (occurrence_count, affected_agents, sample_message, first/last_seen, normalizer_version). Long-retention (raw logs short-retention); alerting + dashboards can read this. -- `log_signature_versions` table: per-signature x agent_version rollup — answers "which versions does signature X appear on?" (version regression correlation). -- **Log analysis (`logs/analyze`):** switched 2026-06-15 from Ollama-on-Beast to the Claude API (commit `c869e4d`). +### Audit Log / Log Feedback Intelligence +- Append-only `audit_log` table (migration 056). Per-line log fingerprinting + `log_signatures` aggregation (migration 057). `logs/analyze` runs on the Claude API (switched from Ollama 2026-06-15). ### Backup Integration (MSP360 / MSPBackups) -- Multi-provider config (`034`/`035`) with connection test, scheduled sync, per-agent + all-providers status, fleet coverage report, and agent<->MSP360 mapping (`044`) with confidence scoring + manual verification. Dashboard UI for mappings/verify shipped 2026-05-31. -- **Alert quality pass (2026-06-07):** Non-backup MSP360 PlanTypes (8=Restore, 13=Consistency-check) excluded from alerting/compliance. MSP360 message JSON decoded via `summarize_backup_error`. `create_or_update_alert` refreshes title/message/severity on re-trigger. Fleet: false `backup_failed` 15 -> 2; survivors (AD1, LAB-Becky) are genuine. -- **`backup_storage_low` alert type REMOVED (2026-06-07):** `DataCopied/TotalData` measures backup-dataset completeness, not destination capacity. Produced 5 fleet-wide false alerts. `resolve_all_backup_storage_alerts` clears stragglers. Genuine destination-capacity alerting deferred. -- MSP360 PlanType map: 3=Files, 7=SQL, 8=Restore, 11=Image, 13=Consistency-check, 16=HyperV. Non-backup (excluded): 8, 13. -- MSP360 API scope (confirmed 2026-06-07): monitoring-only tier; management paths (plan delete, manual run, storage config) return 404; plan management remains MSP360-console tasks. -- **MSP360 "Open in MSP360" deep-link (PR #42, migration 062, pending merge):** "Open in MSP360" button on the backup-tab plan card deep-linking to the MSP360 console for the specific agent's backup plan. Backend: `backup_provider_console_id` field (migration 062) + `lib/provider-links.ts` mapping. -- Key functions: `derive_backup_status`, `error_is_benign`, `summarize_backup_error`, `resolve_orphaned_backup_alerts`, `resolve_all_backup_storage_alerts`, `evaluate_plan` (BACKUP_STALE). +- Multi-provider config, connection test, scheduled sync, per-agent + fleet coverage report, agent<->MSP360 mapping with confidence scoring. +- **Active chronic failure (unresolved as of 2026-07-04):** GND-SERVER (Grabb & Durando) — 42 `backup_failed` alerts since 2026-06-17, failing nightly ~01:02 UTC, "no error detail reported." Also open: AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial). MSP360 sync doesn't capture failure detail — investigation needs a live PowerShell dispatch to read MSP360/CloudBerry local logs. ### Remote Access (Tunnel) -- Agent side substantially built (`TunnelManager` state machine; Open/Close/Data). **Server side is a dead-code skeleton — not declared in `api/mod.rs`, no `/tunnel` routes, WS handler logs "not yet implemented." Not production-ready.** Live TTY design will supersede the skeleton (Phase 2 of agent-comms-durability spec). +- Agent side substantially built. Server side is a dead-code skeleton (no `/tunnel` routes). Live TTY design (agent-comms-durability Phase 2) will supersede the skeleton. ### Identity / Multi-tenancy / Security -- Auth: JWT (login/register/me); agents auth over WS via per-agent API key + hardware device_id. -- **Microsoft Entra ID SSO** (OAuth2/OIDC + PKCE), gated on server config. Google planned per SPEC-008 but not implemented [verify]. -- Organizations / multi-tenancy: org CRUD, per-org membership + roles, limits, dev-admin **user impersonation** (`/auth/impersonate/:id`). Backend + dashboard UI shipped 2026-05-31. -- Enrollment & keys (`012`): per-agent key issuance on first run, site API keys (regenerable), site-specific MSI with SITEKEY injected at download, public install-report ingestion. Legacy PowerShell agent path for Server 2008 R2. -- Logs: agent log upload (periodic + on-demand), per-agent events (`042`), fleet log view, AI-assisted log analysis (`/logs/analyze`) — AI-optional per locked decision. -- **500-error-leak fixed (commit `58c1a96`, 2026-06-15):** All server error paths now route through `internal_err`; raw DB error strings are no longer returned to callers. - -### Enrollment Detail (migration 012 + 2026-06-07 audit) - -The `enrolled_agents` table is the authoritative enrollment history log: - -| Field | Notes | -|---|---| -| `site_id` | Site the agent enrolled under | -| `agent_id` | Set on WebSocket connect | -| `hostname` | Hostname at enrollment time | -| `agent_key_hash` | Hash of the `agk_` enrollment key (not surfaced in API) | -| `enrolled_at` | Timestamp of initial enrollment | -| `last_seen` | Updated on each WS connection | -| `ip_address` | IP at last connection | -| `os_version` | OS at enrollment time | -| `revoked` | Boolean; set TRUE by revocation endpoints | - -**WS auth modes:** Mode 3 (modern, `agk_` keys): checks `enrolled_agents.agent_key_hash` first. Mode 1/2 (legacy): falls back to `agents.api_key_hash`. - -**Revocation — dual-layer (fixed 2026-06-07):** `DELETE /api/agents/:id/key` (bulk) revokes BOTH layers. `POST /api/enrolled-agents/:id/revoke` (targeted) per-enrollment with cascade. - -**Enrollment modal UX (fixed prod 2026-06-21, commit `4027c86`):** "Add Devices" and "Enroll an Agent" modals now have a top-right close button, close on click-outside/Esc, and trigger a site/client list refresh on close. +- Auth: JWT; agents auth over WS via per-agent API key + hardware device_id. +- Microsoft Entra ID SSO (OAuth2/OIDC + PKCE). +- Organizations / multi-tenancy: org CRUD, roles, limits, dev-admin impersonation. +- **500-error-leak fixed (commit `58c1a96`):** all server error paths route through `internal_err`. ### Platform coverage -- **Cross-platform (Win/Linux/macOS):** metrics, network state, hardware/software/service inventory, user/group + DC/domain detection, checks (service checks Win+Linux), discovery, self-update, scripts/commands. -- **Windows-only:** `user_session` command context (WTS), remote registry, tray-into-session, watchdog SCM supervision, BSOD detection, legacy fleet (Win7/Server 2008 R2) agent via Rust 1.77 `--features legacy`. -- **macOS:** agent deployed (Phase 1); tray is a stub; automated Mac build pipeline is an intentional stub (no build host) [verify before claiming CI Mac releases]. +- **Cross-platform (Win/Linux/macOS):** metrics, network state, hardware/software/service inventory, user/group + DC/domain detection, checks, discovery, self-update, scripts/commands. +- **Windows-only:** `user_session` command context, remote registry, tray-into-session (now policy-gated), watchdog SCM supervision, BSOD detection, VSS policy configurator, legacy fleet (frozen at 0.6.76). +- **macOS:** agent deployed (Phase 1); tray is a stub; **now correctly included in the update scanner's platform coverage as of 2026-07-05** (previously silently excluded). --- @@ -441,47 +403,36 @@ The `enrolled_agents` table is the authoritative enrollment history log: | Component | Location | Tech | State | |---|---|---|---| -| Server | 172.16.3.30:3001, systemd `gururmm-server.service` (User=root, binary `/opt/gururmm/gururmm-server`, EnvironmentFile `/opt/gururmm/.env`) | Rust, Axum | deployed, production (physical box, Ubuntu 26.04, PG 18; migrated 2026-06-11) | -| Dashboard | https://rmm.azcomputerguru.com, nginx at `/var/www/gururmm/dashboard/` | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production | -| Agent (Windows) | Endpoints, installed as `GuruRMMAgent` Windows service via WiX MSI | Rust, Windows MSVC | deployed; stable fleet on 0.6.67 | -| Agent (Linux) | Endpoints, systemd `gururmm-agent`, binary `/usr/local/bin/gururmm-agent` | Rust, musl static | deployed | -| Agent (macOS) | Endpoints, LaunchDaemon `com.azcomputerguru.gururmm-agent.plist` | Rust, aarch64/x86_64 | Phase 1 deployed 2026-05-12; code signing issue on Apple Silicon | -| Tray (Windows) | System tray, named pipe IPC | Rust | deployed | -| Tray (Linux) | System tray, Unix socket IPC, libappindicator/GTK | Rust, GTK | deployed 2026-05-24 (PR #13+#14) | -| Tray (macOS) | Menu bar | Rust | stub/TODO (issue #18) | -| PostgreSQL DB | localhost:5432 on 172.16.3.30, database `gururmm` | PostgreSQL 18 | deployed (migrated from PG VM, backfilled 2026-06-11) | -| Coord API | 172.16.3.30:8001/api/coord | FastAPI (part of ClaudeTools API) | deployed | -| Build pipeline | 172.16.3.30:9000 webhook + `/opt/gururmm/` scripts | Python (webhook-handler.py), Bash | deployed; builds agents AND server (server auto-deploy wired 2026-06-02) | -| Windows build — **Beast (PRIMARY)** | GURU-BEAST-ROG, i9-14900K (24c/32t), RTX 4090; reached over Tailscale at tailnet IP 100.101.122.4 as user `guru` | Rust MSVC, WiX 4.x | **primary** Windows build host — `build-windows.sh` tries Beast first; parallel build ~336s | -| Windows build — Pluto (FALLBACK) | 172.16.3.36, Windows Server 2019 VM on Jupiter (Unraid), Xeon E5-2695 v3 8c/16t | Rust MSVC, WiX v4 | **fallback only** — used if Beast is unreachable/down OR its build fails (`attempt_build beast || attempt_build pluto`); serial ~1269s | -| Old VM (decommissioned) | was 172.16.3.46 | Ubuntu (original) | DELETED 2026-06-12 after stability soak — virsh domain + disk removed; no longer exists | +| Server | 172.16.3.30:3001, systemd `gururmm-server.service` | Rust, Axum | deployed, production; v0.3.96 | +| Dashboard | https://rmm.azcomputerguru.com, nginx | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production | +| Agent (Windows) | Endpoints, `GuruRMMAgent` service via WiX MSI | Rust, Windows MSVC | deployed; modern build v0.6.77, legacy (2008R2/Win7) FROZEN at 0.6.76 and disabled | +| Agent (Linux) | Endpoints, systemd `gururmm-agent` | Rust, musl static | deployed | +| Agent (macOS) | Endpoints, LaunchDaemon | Rust, aarch64/x86_64 | Phase 1 deployed 2026-05-12; code-signing issue on Apple Silicon; update-scanner platform gap fixed 2026-07-05 | +| Tray (Windows) | System tray, named pipe IPC | Rust | deployed to disk fleet-wide via agent-update (2026-07-04); **launch policy-gated, default OFF** | +| Tray (Linux) | System tray, Unix socket IPC | Rust, GTK | deployed 2026-05-24 | +| Tray (macOS) | Menu bar | Rust | stub/TODO | +| PostgreSQL DB | localhost:5432 on 172.16.3.30 | PostgreSQL 18 | deployed | +| Build pipeline | 172.16.3.30:9000 webhook + `/opt/gururmm/` scripts | Python, Bash | deployed; builds agents AND server | +| Windows build — Beast (PRIMARY) | GURU-BEAST-ROG over Tailscale | Rust MSVC, WiX 4.x | primary — two-wave parallel, ~336s | +| Windows build — Pluto (FALLBACK) | 172.16.3.36 | Rust MSVC, WiX v4 | fallback only | ### Key Files & Repos -- **Active repo:** `azcomputerguru/gururmm` — http://172.16.3.20:3000/azcomputerguru/gururmm -- **Submodule working tree:** `projects/msp-tools/guru-rmm` — gitlink intentionally lags origin/main; read live code with `git -C projects/msp-tools/guru-rmm show origin/main:` or worktrees off origin/main. Push via GIT_ASKPASS with vaulted Gitea API token (`services/gitea-howard.sops.yaml`). -- **Server binary:** `/opt/gururmm/gururmm-server` on 172.16.3.30 (physical box; prior VM had `/usr/local/bin/gururmm-server`) -- **Server config:** `/opt/gururmm/.env` (EnvironmentFile for systemd; DATABASE_URL here; root-only read) -- **Source checkout on server:** `/home/guru/gururmm` (build-server.sh: `git reset --hard origin/main` -> `cargo build` -> deploy with backup+rollback) -- **Agent binary (Linux):** `/usr/local/bin/gururmm-agent` -- **Agent config (Linux/macOS):** `/etc/gururmm/agent.toml` (root, mode 600); macOS uses `/usr/local/etc/gururmm/site.plist` -- **Agent registry (Windows):** `HKLM\SOFTWARE\GuruRMM\SiteId` (written by MSI); `HKLM\SOFTWARE\GuruRMM\DeviceId` (written by agent, added Phase 1 Task 1) -- **Windows service name:** `GuruRMMAgent` (NOT `gururmm-agent`) -- **Downloads dir:** `/var/www/gururmm/downloads/` on 172.16.3.30 — base binaries `gururmm-agent--.exe` + `.channel` + `.sha256`; `-latest` symlinks; per-site signed cache `gururmm-agent-site---.exe` (built+jsign-signed on demand) -- **Webhook handler:** `/opt/gururmm/webhook-handler.py` (port 9000 localhost; nginx :80 /webhook/ -> :9000; systemd `gururmm-webhook`) -- **Build scripts:** `/opt/gururmm/build-shared.sh`, `build-linux.sh`, `build-windows.sh`, `build-mac.sh` (split 2026-05-24; `build-agents.sh` is compat wrapper) -- **Server build script:** `/opt/gururmm/build-server.sh` (dispatched by webhook on `server/` changes; change-gate `last-built-commit-server`; backup current binary; auto-rollback if fails `is-active`) -- **Per-platform SHA tracking:** `/opt/gururmm/last-built-commit-{linux,windows,mac,server}` -- **Build log (Linux):** `/var/log/gururmm-build-linux.log` -- **Build log (Windows):** `/var/log/gururmm-build-windows.log` -- **Build log (Server):** `/var/log/gururmm-build-server.log` +- **Active repo:** `azcomputerguru/gururmm` — http://172.16.3.20:3000/azcomputerguru/gururmm (external: https://git.azcomputerguru.com/azcomputerguru/gururmm.git) +- **Submodule working tree:** `projects/msp-tools/guru-rmm` — gitlink intentionally lags origin/main; read live code via `git -C projects/msp-tools/guru-rmm show origin/main:` or a worktree. +- **Server binary:** `/opt/gururmm/gururmm-server` on 172.16.3.30 +- **Server config:** `/opt/gururmm/.env` +- **New shared sanitize module:** `server/src/db/sanitize.rs` (`strip_nul`, `clean_control_chars`) — consumed by `db/logs.rs` and `db/inventory.rs` +- **Integrations framework:** `server/src/api/integrations.rs`, `server/src/integrations/` (plugin trait + registry), migration `065_integrations_framework.sql` +- **VSS configurator:** `agent/src/vss_com.rs`, `agent/src/vss.rs`, `specs/vss-policy-config/`, `specs/vss-native-com/` +- **Tray policy-deploy:** `agent/src/tray_stage.rs`, `agent/src/tray_state.rs`, `agent/src/authenticode.rs`, `specs/tray-policy-deploy/` +- **Downloads dir:** `/var/www/gururmm/downloads/` on 172.16.3.30 +- **Webhook handler:** `/opt/gururmm/webhook-handler.py` - **API (internal):** http://172.16.3.30:3001 -- **API (external):** https://rmm-api.azcomputerguru.com (grey-cloud, no Cloudflare — agents connect here; Cloudflare only proxies the dashboard `rmm.azcomputerguru.com`) -- **Agent ingress path:** agent -> site gateway -> pfSense (172.16.0.1, DNAT wan:443 -> NPM .20:18443) -> NPM (Docker on Jupiter .20, TLS terminate) -> origin nginx (.30:80, /ws -> 127.0.0.1:3001) -> Rust server :3001 +- **API (external):** https://rmm-api.azcomputerguru.com - **Dashboard:** https://rmm.azcomputerguru.com -- **DB URL:** `postgres://gururmm:@localhost:5432/gururmm` (pw in `/opt/gururmm/.env`; do not hardcode here) -- **Vault paths:** `infrastructure/gururmm-server.sops.yaml` (API creds, DB password, sudo password), `infrastructure/gururmm-server-physical` (SSH ed25519 key) -- **SSH to prod:** `ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30` (ed25519, key-only; sudo pw in vault `infrastructure/gururmm-server.sops.yaml` `credentials.password`) +- **Vault paths:** `infrastructure/gururmm-server.sops.yaml` (API creds, DB password, sudo password), `infrastructure/gururmm-server-physical` (SSH ed25519 key — this fleet key works on 172.16.3.30 currently, not just the future physical box; confirmed 2026-07-04) +- **SSH to prod:** `ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30` (ed25519, key-only; password auth disabled) - **Pre-merge verification skill:** `bash .claude/skills/gururmm-build/scripts/verify.sh server|agent|dashboard|migrations` ### Repo Structure @@ -490,41 +441,50 @@ The `enrolled_agents` table is the authoritative enrollment history log: gururmm/ ├── agent/ Rust agent (managed endpoints) │ └── src/ -│ ├── bsod.rs Windows BSOD/kernel-crash detection -│ ├── commands/mod.rs CommandExecutor + recent-results cache (ACK dedup) -│ ├── device_id.rs Durable multi-location device_id (Phase 1 Task 1) -│ ├── ipc.rs Unix socket IPC (Linux); Windows named pipe +│ ├── authenticode.rs NEW — Authenticode signature verify (tray-policy-deploy) +│ ├── tray_stage.rs NEW — tray materialize FSM + staging + versioned mutex +│ ├── tray_state.rs NEW — tray policy state +│ ├── bsod.rs Windows BSOD/kernel-crash detection +│ ├── vss_com.rs Native VSS COM create/provision (vss-native-com + vss-policy-config) +│ ├── vss.rs VSS policy configurator entry points +│ ├── commands/mod.rs CommandExecutor + recent-results cache (ACK dedup) +│ ├── device_id.rs Durable multi-location device_id │ ├── transport/ -│ │ ├── mod.rs AgentMessage enum (incl. CommandAck; WatchdogEvent REMOVED) -│ │ └── websocket.rs WS transport; execute_command ACK + dedup; cmd-alias NAK -│ ├── tunnel/ TunnelManager state machine -│ ├── metrics/ sysinfo collection + temps -│ ├── registry_ops/ Windows registry read/write -│ ├── updater/ Self-update handler (container guard added BUG-019) -│ └── main.rs systemd unit template generation +│ │ └── websocket.rs WS transport; execute_command ACK + dedup; cmd-alias NAK +│ ├── tunnel/ TunnelManager state machine +│ ├── metrics/ sysinfo collection + temps +│ ├── registry_ops/ Windows registry read/write +│ ├── updater/ Self-update handler (container guard, BUG-019) +│ └── main.rs ├── server/ Rust/Axum API server │ └── src/ -│ ├── api/ REST handlers (~38 route modules; updates.rs: promote/rollback; software.rs SPEC-030) +│ ├── api/ +│ │ ├── integrations.rs NEW — integrations framework CRUD (crowdstrike-falcon Task 3) +│ │ └── ... ~40 route modules; software.rs SPEC-030 │ ├── alerts/ Alerting modules (offline.rs: offline_sweep + mass_offline) -│ ├── db/ Database layer (sqlx); commands.rs (requeue_undelivered, fail_timed_out rewrite) -│ │ software_jobs.rs (SPEC-030 async removal jobs) -│ │ watchdog_events.rs (doc-only stub — table exists, path removed per BUG-022 PR #45) -│ ├── fingerprint.rs Log signature normalization + hash -│ ├── ws/ WebSocket handler (CommandAck, redispatch_pending_commands) -│ └── mspbackups/ MSP360 backup integration +│ ├── db/ +│ │ ├── sanitize.rs NEW — shared NUL/control-char sanitize (2026-07-05) +│ │ ├── alerts.rs create_or_update_alert -> Option, refresh_alert_row +│ │ ├── inventory.rs NUL-scrub over jsonb + text before upsert +│ │ └── software_jobs.rs SPEC-030 async removal jobs +│ ├── updates/ +│ │ └── scanner.rs host-native vs foreign self-check regimes (2026-07-05) +│ ├── ws/mod.rs WS handler; is_benign_ws_disconnect / is_preauth_connection_churn +│ └── mspbackups/ MSP360 backup integration ├── dashboard/ React/TypeScript UI │ ├── pages/ -│ │ └── EventLogWatches.tsx CRUD management UI (shipped 0fa65f5) +│ │ └── EventLogWatches.tsx │ └── components/ -│ └── SoftwareManager.tsx SPEC-030 installed-programs list + bulk uninstall + live progress -├── agent/scripts/ uninstall-engine.ps1 (SPEC-030 Tier-1 silent uninstall engine) +│ └── SoftwareManager.tsx +├── agent/scripts/ uninstall-engine.ps1 ├── tray/ System tray binary -├── installer/ WiX v4 MSI (gururmm-agent.wxs); cleanup.ps1 (preserves device_id); universal self-detecting installer (Feature 9) +├── installer/ WiX v4 MSI; cleanup.ps1; universal self-detecting installer ├── deploy/ -│ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh +│ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh (LEGACY_ENABLED=false gate added 2026-07-04) ├── scripts/ Build/ops scripts -└── docs/ FEATURE_ROADMAP.md, UI_GAPS.md, ARCHITECTURE_DECISIONS.md, tech-stack.md, - DESIGN.md, BUILD.md, specs/, HOST_MIGRATION_RUNBOOK.md, RMM_THOUGHTS.md +└── docs/ FEATURE_ROADMAP.md, UI_GAPS.md, RMM_THOUGHTS.md, specs/ + specs/ tray-policy-deploy/, vss-policy-config/, vss-native-com/, crowdstrike-falcon/, + av-removal-recipes/, remote-software-uninstall/, agent-comms-durability/, ... ``` --- @@ -533,78 +493,62 @@ gururmm/ ### Current Focus -As of 2026-06-25 (agent 0.6.75 / server v0.3.87): +As of 2026-07-04 (agent 0.6.77 / server v0.3.96): -- **SPEC-030 software-uninstall follow-on (Route A / recipes):** Launchy/AIMP-class uninstallers hang the agent on an orphaned child process tree → `failed`/no-output; Route B can't fully control this. Open decision: a kill-process-tree-on-timeout engine mitigation (return a clean `needs_interactive`) vs the **av-removal-recipes** tier (P3 spec, removal-only). Also: a **driver/system-component exclusion/flag** in the Software Manager UI (drivers register in ARP and appear in the bulk-select list — a top-down select removed a driver), and a **list-jobs endpoint** (currently get-by-id only). -- **[VERIFY] Earlier pending PRs #40-#46 (SPEC-021 logged-in-user domain detection, BUG-018 FK indexes, MSP360 deep-link, Event Log Watch policy-clobber HIGH fix, audit cleanup, BUG-022 watchdog dead-code):** these were open as of 2026-06-22 and predicted to take migrations 061-063. The **software-removal work merged ahead and consumed migrations 061-064**, so those predicted numbers no longer hold. Confirm the live merge/migration status of each before relying on it — do not assume the 2026-06-22 numbering. (BUG-022 WatchdogEvent dead-code removal — verify whether it landed; the empty `watchdog_events` table remains either way.) -- **Agent-comms-durability Phase 2 (live TTY, planned):** Extend WS message enum with TTY stream frames (stdin/stdout/stderr, resize, seq). "Activate" cadence switch. Resume from seq on mid-session drop. Single-use time-bounded session token; max one interactive session per agent; full audit. Estimated 3-5 days separate effort. -- **Agent-comms-durability Phase 3 (planned):** Adaptive keepalive (AIMD, persist interval), bulk file transfer -> short-lived HTTPS, server half-open eviction sweeper (`last_inbound` track + evict >~90s; treat missed Pong as close). -- **Durable agent identity Phase 1 Tasks 2-3 (pending):** Task 2 = hardware-fingerprint capture. Task 3 = dashboard "probable duplicate" surfacing (read-only). Phase 2 (guarded auto-reclaim) + Phase 3 (operator merge tool for the 9 existing ghosts) pending soak. -- **SPEC-028 offboarding wizard (specification complete, 835 lines; implementation pending):** Site + client offboarding workflows, data export, typed confirmation, audit logging. -- **BSOD detection Phase 2/3 (deferred):** Dashboard Crashes tab shipped 2026-06-07. Remaining: BSOD in Alerts stream (issue #10), `fetch_bsod_dump` on-demand upload, full ~350-entry bugcheck name table. -- **Linux fleet unit drift:** Auto-updater replaces binary but does NOT refresh systemd unit file. Pre-BUG-016-fix agents have new binary + old unit (missing `StateDirectory=gururmm`). Needs ops-script pass. -- **Alert mute dashboard (Task 5) -- NOT started:** Ignore button + required-reason prompt; Muted filter; Un-ignore control. Server endpoints ready. -- **MSP360 backup Phase 2 (not started):** Genuine destination-capacity alerting deferred (needs MSP360 storage-accounts endpoint, monitoring-only API tier confirmed). -- **Tray IPC + peer authorization** — Linux tray merged (PR #13+#14). Open: Windows peer authz (#16), logind console-user resolution (#17), macOS tray (#18), subscriber broadcast (#19). -- **Security audit backlog:** `credentials/:id/reveal` horizontal privilege escalation (HIGH), `update_rollouts.promoted_by` UUID vs `users.id` int mismatch (schema quirk from migration 046). -- **SPEC-021 (PR #40):** Logged-in-user domain + account-type detection (agent + server + dashboard). Pending merge. +- **Machine-side backup failures (highest user priority, unresolved):** GND-SERVER chronic 42-failure streak, AD1, LAB-SVR, DESKTOP-EVA4H1A — MSP360 sync reports no error detail; needs a live PowerShell dispatch to read local MSP360/CloudBerry logs or the mspbackups.com console. +- **13 duplicate-hostname agent pairs fleet-wide** (legacy `win-*`/`hw-*` device_ids alongside re-enrolled hardware-UUID records) — the ongoing trigger for `move_agent` 409s; a one-time cleanup pass (delete stale offline duplicates) is not yet done. +- **WS auth-timeout source unidentified:** ~1/min unauthenticated connection attempts, now correctly classified INFO (not alerting), but the source is still unknown — would need connection-level source-IP visibility (Feature 5, server-side public-IP capture, still unapproved) or box-level tcpdump. +- **Tray policy fleet enablement (Task 4, tray-policy-deploy spec):** materialize code is merged and deployed; flipping `tray.enabled` on for the fleet is a deliberate next step, not yet done. +- **VSS configurator stable-channel promotion:** shipped to beta (0.6.76) only; promotion to stable is a separate deliberate step pending further soak. +- **Legacy (2008R2/Win7) agent rewrite (RMM_THOUGHTS Feature 12, Raw):** not approved; needs a language/runtime decision (.NET Framework 4.x vs C++) and scope decision (trimmed feature set: enroll, check-in, metrics, command exec, self-update only) before a `/shape-spec`. +- **Integrations framework dashboard UI:** CrowdStrike Falcon backend + sync worker are live; no management screen exists yet in the dashboard. +- **SPEC-030 software-uninstall follow-on (Route A / recipes):** Launchy/AIMP-class uninstallers still hang the agent; driver/system-component exclusion flag and a list-jobs endpoint remain unbuilt. +- **Agent-comms-durability Phase 2/3 (planned):** live TTY, adaptive keepalive, bulk file transfer -> short-lived HTTPS, server half-open eviction sweeper. +- **Durable agent identity Phase 2-3 (pending soak):** guarded auto-reclaim + an operator merge tool for the pre-fix ghost/duplicate agents. +- **SPEC-028 offboarding wizard:** specification complete; implementation pending. +- **Model routing correction (2026-07-04, logged --correction):** Tier-0 simplify/cleanup passes should route to local Ollama before Claude; a routing miss on this session's easy-bugs batch was logged for tracking. ### Patterns & Anti-Patterns -**Anti-patterns — never repeat:** +**New anti-patterns (2026-07-01..05):** + +| Pattern | What Went Wrong | +|---|---| +| Update-scanner self-check executing every binary regardless of target platform | macOS/Linux binaries can't run on the Linux build/scan host, so they were silently dropped from the scannable set entirely — those agents never received an update offer. Fix: host-native binaries get a hard `--version` execute-and-check gate; foreign-platform binaries trust the filename-embedded version. | +| Writing a machine-global VSS registry governor (`MaxShadowCopies`) to a per-policy `retention_count` | `MaxShadowCopies` is shared by ALL VSS consumers on the box (System Restore, third-party backup) — lowering it FIFO-evicts THEIR shadows too, a genuine data-loss class caught only by a 23-agent pre-merge code review. Use a per-volume size cap as the sole enforced governor instead. | +| Re-homing an agent's site on every WS reconnect after a device_id-based dedup fix | The agent re-presents its embedded install-site code as its API key on every beat; auto-re-homing on each connect silently bounces an admin-relocated agent back to its original install site. Dedup by device_id, but leave site placement admin-controlled and persistent. | +| `fetch_one` on an UPDATE-by-id inside a concurrently-deleting table | An agent cascade-delete can remove the target row between a SELECT and the UPDATE; `fetch_one` panics `RowNotFound` and aborts the whole caller (here: the entire offline sweep). Use `fetch_optional` with a fall-through path. | +| Two sessions editing the same branch's working tree without a coord lock | A concurrent session's uncommitted edits were silently absorbed into another session's commit. Benign this time (same branch, all changes verified at HEAD) but neither session checked the coord lock list first — take a lock before editing a shared submodule branch. | +| Assuming a spec's `shape.md`/`plan.md` header "Status: not started" reflects current reality | Tasks can ship and merge (git log shows Tasks 1-3+5 done) while the spec doc header is never updated. Verify against `git log`/deployed version, not the spec file's status line. | + +**Existing anti-patterns (carried forward):** | Pattern | What Went Wrong | |---|---| | `useMemo` with stable deps for data-dependent values | queryClient is stable, memo never recomputes after queries resolve. Use `useQuery` instead. | -| CSS variable text colors inside the sidebar | Sidebar bg is hardcoded dark; CSS vars flip in light mode. Use `text-white` explicitly inside sidebar. | | Deploying without stopping the server first | "text file busy" kernel error. Always `systemctl stop` before `cp`. | -| Building without `DATABASE_URL` | sqlx compile-time macros fail. `DATABASE_URL` is in `/opt/gururmm/.env` (or `/home/guru/.cargo/env` on dev). | +| Building without `DATABASE_URL` | sqlx compile-time macros fail. | | DB migrations without inserting into `_sqlx_migrations` | Server crashes on start. Must insert SHA-384 checksum manually. | -| WiX MSI builds on Linux | WiX requires `msi.dll`. MSI must be built on Pluto or Beast (Windows). | -| Manual builds via SSH | All builds go through `webhook-handler.py`. Never SSH and run `cargo build` + artifact copy manually. | -| TOML/config for agent endpoint or site_id | Server URL compiled into binary, site_id baked into MSI. No runtime config files for these values. | -| `path.find('\\')` in `#[cfg(windows)]` files | Compiles on Linux silently, fails on Pluto MSVC with unterminated char literal. Use `'\\\\'`. | -| `STATUS_BADGE_CLASSES` Record const | Vite/Rollup may optimize away the lookup. Use explicit `getStatusBadgeClass()` if/else function. | -| SSH heredoc for TypeScript edits | Shell strips double-quote characters. Edit locally in submodule, push to Gitea, pull on server. | -| `Restart-Service GuruRMMAgent -Force` in command scripts | Kills agent before it can report result. Commands stay forever `running`. Use scheduled task with delay instead. | -| `sudo -u guru git` in systemd build context | git rejects repo as dubious ownership when running as root on guru-owned repo. Use `git config --system --add safe.directory` instead (applies to all users/envs — fixed 2026-06-11). | -| Self-updating running bash script | bash reads line-by-line from disk; replacing mid-execution silently skips remaining blocks. | -| `+1.77` legacy builds without `--ignore-rust-version` | Fail MSRV check after adding `rust-version` to Cargo.toml. Add `--ignore-rust-version` to legacy build lines only. | -| `StrictHostKeyChecking=no` for Pluto SSH | Replaced with pinned known-hosts at `/opt/gururmm/pluto_known_hosts`. MITM would compromise build artifacts. | -| CRLF line endings in migration files | sqlx SHA-384 checksum mismatch causes server crash on start. `.gitattributes` + `core.autocrlf=false` + pre-commit hook prevents this. | -| Dead WebSocket write half | WS write fails, send task dies, receive loop keeps agent in `ConnectedAgents` with dead write half. Commands silently fail. Fix: `tokio::select!` monitoring both tasks. | -| Using the `minidump` crate for Windows kernel dumps | The crate only parses Breakpad MDMP format, not Windows kernel PAGEDU64 dumps. Parse `DUMP_HEADER64`/`DUMP_HEADER32` at fixed offsets directly (validated against real dumps). | -| Build classification defaulting to stable | New agent builds should default to `beta`; promotion to `stable` is an explicit re-tag of the `.channel` sidecar. Defaulting stable races the auto-update fleet ahead before any beta soak. | -| Webhook dispatching only agent builds | The webhook historically triggered agent builds but never `build-server.sh`; server changes silently went unbuilt until manual intervention. Fixed 2026-06-02 — server build is dispatched alongside agent builds, gated by `last-built-commit-server`. | -| Auto-update-on-connect + default-stable tagging racing the fleet | If a build is tagged `stable` (even briefly by mistake), agents on the stable channel auto-update immediately. Once fleet has updated, rolling back requires a new build. Default beta, soak, promote explicitly. | -| Channel pin at agent-level for a beta canary | Agent-level `update_channel` is keyed to agent_id and lost on re-enrollment. Use site-level or client-level override (survives re-enroll). | -| Installer using `& $stagingPath install 2>&1 \| Out-Null` under `$ErrorActionPreference='Stop'` | Swallows output; surfaces a misleading NativeCommandError on any non-zero exit. Use `Start-Process -Wait -PassThru` + check `$proc.ExitCode`. Fixed 2026-06-11 (commit 5c0d004). | -| Reaper flipping un-acked commands to `failed` | Causes false-failure for commands black-holed in NAT conntrack gaps. The reaper must only fail commands that were ACK'd but overran their real execution timeout. Un-acked commands must be requeued to `pending`. | -| `command_type: "cmd"` in API commands | `cmd` was not a valid `CommandType` variant — agent silently dropped the whole message (no ACK, no result). Now aliased to `Shell` (commit `3de9faf`), but check command_type first whenever a command appears un-acked. Valid types: `shell`, `powershell`, `python`, `script`, `claude_task` (+ `cmd` alias). | -| `cargo +1.77 fetch` as a parallel-build pre-resolve | Resolves the full dep graph, which on Rust 1.77.2 pulls `wit-bindgen` (edition2024) and fails with rc=101 on both build hosts. Removed: the legacy builds scope deps via `--features legacy` and don't need a pre-resolve. | -| Concurrent parallel builds sharing one `--target-dir` | Cargo's per-build-dir lock forces them to run serially. Each variant must have its own target dir (amd64=`target/release`, x86=`target/x86`, legacy-amd64=`target/legacy-x64`, legacy-x86=`target/legacy-x86`). Fixed 2026-06-12 (commit `cfbdb59`). | -| Pinning new legacy-wave deps without checking edition requirements | BUG-021: `getrandom 0.3.2` / `zeroize 1.9.x` require edition2024, which Cargo 1.77 cannot handle. Always verify the edition compatibility of any new dep version before adding it to the legacy build. Pin to `getrandom 0.3.1` + `zeroize 1.8.1` (or later confirmed-compatible versions). | -| Asserting build status from a point-in-time log snapshot | Build logs are append-only; a prior FAILED line stays in the tail even after a later successful run. Check the LIVE `last-built-commit-` marker vs `origin/main` HEAD to confirm the current build status. | -| Config-push clobber at multiple independent sites | The server has multiple code paths that push per-agent configuration (policy assign, policy unassign, reconnect, etc.). If the rule-shaper is not the single source of truth, adding a new push site will silently omit config (Event Log Watches were wiped by policy assign/unassign — third instance of this class). Always reuse the shared shaper; never duplicate the rule-mapping logic. | -| Reading the stale submodule working tree for code | The submodule gitlink intentionally lags `origin/main`. Always use `git -C projects/msp-tools/guru-rmm show origin/main:` or a worktree off `origin/main` to read current code. Reading the working tree gives stale results and led to false audit findings. | +| WiX MSI builds on Linux | WiX requires `msi.dll`. Must build on Pluto or Beast. | +| Manual builds via SSH | All builds go through `webhook-handler.py`. | +| `Restart-Service GuruRMMAgent -Force` in command scripts | Kills agent before it can report result. Use a scheduled task with delay instead. | +| `sudo -u guru git` in systemd build context | git rejects repo as dubious ownership. Use `git config --system --add safe.directory`. | +| `+1.77` legacy builds without pinned deps | Fresh dep resolution repeatedly pulls edition2024 crates Rust 1.77 can't parse (BUG-021, recurred 2026-07-04 with a different dep pair, led to the variant being disabled outright rather than re-chasing pins). | +| Dead WebSocket write half | WS write fails, send task dies, receive loop keeps agent connected with a dead write half. Fix: `tokio::select!` monitoring both tasks. | +| Using the `minidump` crate for Windows kernel dumps | Only parses Breakpad MDMP, not Windows kernel PAGEDU64. Parse `DUMP_HEADER64`/`32` at fixed offsets directly. | +| Build classification defaulting to stable | New builds default `beta`; promotion is an explicit re-tag. | +| `command_type: "cmd"` in API commands | Not a valid `CommandType` variant originally — agent silently dropped the message. Now aliased to `Shell`. | +| Reading the stale submodule working tree for code | The submodule gitlink intentionally lags `origin/main`. Always read via `git -C ... show origin/main:` or a worktree. | -**Good patterns:** +**Good patterns (carried forward + new):** -- **Platform parity rule** — any agent feature goes on Windows + Linux + macOS in the same commit. If a real implementation isn't feasible, add a working stub + `// TODO(platform): — `. No silent no-ops. -- **Per-platform last-built-commit tracking** — Linux builds succeed and record progress independently of Windows builds. -- **Holistic feature development** — every feature ships backend + API + dashboard UI + docs together. Backend-only features are rejected. -- **sqlx offline mode** — compile-time query validation requires DB reachable or offline cache present. -- **`RuntimeDirectory=gururmm` in systemd unit** — systemd-native way to give agent writable `/run/gururmm/` for IPC socket. -- **Registry-first path resolution** — read `HKLM:\SOFTWARE\GuruRMM` for install dir, fall back to service PathName, then hardcoded default. -- **`interrupt_running_commands()` at reconnect** — flips all `status='running'` commands for reconnecting agent to `status='interrupted'`. -- **Build change-gate + backup/rollback in `build-server.sh`** — skips rebuild when `server/` is unchanged; backs up previous binary; restores it if the new binary fails `is-active`. -- **Server's own root RMM agent for privileged ops** — the server (172.16.3.30) runs the GuruRMM Linux agent as root (hostname `gururmm`); it can re-tag `.channel` sidecars and trigger `build-server.sh` without SSH or `sshpass`. -- **GURU-5070 site as permanent beta-channel canary** — site "Mike's Car" has `update_channel='beta'` (survives agent re-enrollment). Gets every new beta build; stable fleet protected by explicit `update_rollouts` pin. (Channel pin on site, not agent — agent-level overrides are lost on re-enroll.) -- **Capability gate via partial index, not version-string compare** — CI auto-bumps versions (`[ci-version-bump]`); pinning behavior to a fixed version string is fragile. Use a DB-observable capability marker (e.g., `acked_at IS NOT NULL`) that is set when the agent actually demonstrates the capability. -- **Two independently-deployable slices for cross-component features** — agent slice and server slice can be deployed in any order; the server slice uses a capability gate so old agents keep legacy behavior. -- **Worktrees off origin/main for concurrent sessions** — isolates each session's work from shared dirty checkout. Commit + push by explicit SHA (`git push origin :refs/heads/`) to avoid HEAD/branch-ref races. Standard pattern when multiple sessions touch the same submodule. -- **`gururmm-build` skill for pre-merge verification** — `bash .claude/skills/gururmm-build/scripts/verify.sh ` runs the same compile checks the server will. Does NOT trigger production build. Merging to main is the only prod trigger. -- **Dashboard beta-before-main rule** — dashboard changes go to beta (auto-built from main) first. Preview a feature branch without merging: `git worktree add --detach origin/` + `npm install + npm run build` + `rsync dist/ /var/www/gururmm/dashboard-beta/`. Promote with `promote-dashboard.sh --confirm`. +- **Shared sanitize chokepoint** (`db/sanitize.rs`, 2026-07-05) — one NUL/control-char scrubber consumed by both `db/logs.rs` and `db/inventory.rs`, replacing duplicated ad hoc scrub logic. +- **Capability-based classification (host-native vs foreign) instead of a single universal check** — the update scanner now branches its self-check strategy per target platform rather than applying one strategy fleet-wide. +- **Multi-AI adversarial review before shipping risky agent-side changes** — the VSS redesign (Grok + Gemini) and tray redesign (17-agent Claude review + Grok + Gemini) both caught genuine defects (a data-loss governor and 9 architecture defects respectively) that a single-model review missed on the first pass. +- **Platform parity rule** — any agent feature goes on Windows + Linux + macOS in the same commit. +- **`gururmm-build` skill for pre-merge verification** — does NOT trigger production build; merging to main is the only prod trigger. +- **Dashboard beta-before-main rule** — dashboard changes go to beta (auto-built from main) first; promote explicitly. +- **Two independently-deployable slices for cross-component features, capability-gated** — agent slice and server slice can deploy in any order; old agents/servers keep legacy behavior until the gate flips. ### Build & Deploy @@ -612,131 +556,52 @@ As of 2026-06-25 (agent 0.6.75 / server v0.3.87): ``` Gitea push to main - -> webhook-handler.py (172.16.3.30:9000 via nginx :80 /webhook/, parallel threads per platform) - -> build-shared.sh (auto-version bump, git sync — runs once) - -> build-linux.sh (cargo build on .30; log: /var/log/gururmm-build-linux.log) - -> build-windows.sh (SSH -> Beast PRIMARY (guru@100.101.122.4 over Tailscale) - || fall back to Pluto Administrator@172.16.3.36; - each via its own pinned known-hosts - TWO-WAVE PARALLEL BUILD (see below) - sign-windows.sh (jsign + Azure Trusted Signing, /etc/gururmm-signing.env) - SCP artifacts back; log: /var/log/gururmm-build-windows.log) + -> webhook-handler.py (172.16.3.30:9000) + -> build-shared.sh (auto-version bump) + -> build-linux.sh + -> build-windows.sh (Beast PRIMARY over Tailscale || Pluto FALLBACK + TWO-WAVE PARALLEL BUILD; legacy wave now gated LEGACY_ENABLED=false) -> build-mac.sh (stub — no build machine configured yet) - -> build-server.sh (gated: skips if no server/ changes since last-built-commit-server; - backs up current binary; builds + deploys; auto-rollback if fails is-active; - log: /var/log/gururmm-build-server.log) + -> build-server.sh (gated on server/ changes; backup+rollback) -> artifacts -> /var/www/gururmm/downloads/ with sha256 + -latest symlinks + .channel (beta) - -> per-platform last-built-commit files updated - -> systemctl restart gururmm-agent (local agent on .30) ``` -**Two-wave parallel Windows build (lever A, wired 2026-06-12, commit `cfbdb59`):** -- WAVE 1 (5 concurrent, stable toolchain): `s_amd64` (`target/release`), `s_x86` (`target/x86`), `s_debug` (`target/debug-agent`), `s_tray` (`tray/target`), `s_cleanup` (`installer/cleanup/target`) -- WAVE 2 (2 concurrent, Rust 1.77 legacy, after Wave 1): `l_amd64` (`target/legacy-x64`), `l_x86` (`target/legacy-x86`) — uses `$CARGO +1.77 --features legacy --ignore-rust-version` -- MSI build overlaps Wave 2 (depends only on Wave 1 artifacts) -- Per-job exit-code aggregation; if Beast fails, Pluto is tried with the same structure -- Beast timing: ~336s cargo phase (319s), ~46% faster than Beast serial cold (~622s), ~3.8x vs Pluto serial (~1269s) +**Legacy variant disabled 2026-07-04 (`84a82a3`):** `LEGACY_ENABLED=false` in `build-windows.sh` gates the legacy wave/copy/deploy/symlink off entirely; the last-built legacy artifact (0.6.76) is preserved and excluded from cleanup so existing 2008R2/Win7 endpoints keep receiving that frozen build. Modern (amd64/x86/tray/debug) builds are no longer blocked by legacy dep-resolution failures. -**[CRITICAL] Legacy wave dep-pin gotcha (BUG-021):** The legacy wave resolves deps fresh with `cargo +1.77 fetch`... NO — `cargo fetch` is REMOVED from the pipeline (it was the failure point: Cargo 1.77.2 chokes on edition2024 deps like `wit-bindgen` when resolving the full dep graph). Legacy builds scope deps via `--features legacy` and do NOT need a pre-fetch. However: if any dep added to Cargo.toml ships a new patch that requires edition2024, the legacy build will fail. Pin known-problematic deps: `getrandom = "0.3.1"` (not 0.3.2+), `zeroize = "1.8.1"` (not 1.9.x). Verify before bumping any dep that the new version's MSRV is compatible with Rust 1.77. - -**Auto-version:** `build-shared.sh` diffs `agent/`, `server/`, `dashboard/` against last built SHA. For each changed component, bumps patch version in `Cargo.toml` or `package.json`, commits `[ci-version-bump]`, pushes. Webhook skips builds where all commits are version bumps. - -**Build channel classification:** New agent builds are tagged `beta` by default. Promotion to `stable` is an explicit step via the API: `POST /api/updates/rollouts/:version/promote {"os":"windows","arch":"amd64"}` — re-tags all `.channel` files for that version, records `update_rollouts` row, triggers scanner rescan. Rollback: `POST /api/updates/rollouts/:version/rollback`. Agents on the stable channel only receive the latest `stable`-tagged binary. - -**Downloads layout:** -- Base binaries: `gururmm-agent--.exe` + `.channel` (beta/stable) + `.sha256` -- Latest symlinks: `gururmm-agent--latest.exe` + `.channel` + `.sha256` -- Per-site signed cache: `gururmm-agent-site---.exe` (built+jsign-signed on demand at install-request time; staged in `downloads_dir`, NOT `/tmp` — EXDEV fix `95ef901`) - -**Dashboard channels — BETA-FIRST:** - -| Channel | URL | Web root | Updates | -|---|---|---|---| -| beta | https://rmm-beta.azcomputerguru.com | `/var/www/gururmm/dashboard-beta` | auto on push — `build-dashboard.sh` (change-gated on `last-built-commit-dashboard`) | -| prod | https://rmm.azcomputerguru.com | `/var/www/gururmm/dashboard` | explicit only — `sudo /opt/gururmm/promote-dashboard.sh --confirm` (backs up prod; `--rollback` restores) | - -Dashboard changes go to beta BEFORE main. To preview a feature branch without merging: build its `dashboard/` in a `git worktree add --detach origin/`, `npm install --no-audit --no-fund && npm run build`, then `rsync -a --delete dist/ /var/www/gururmm/dashboard-beta/`. Do NOT touch `/opt/gururmm/last-built-commit-dashboard`. Do NOT hand-rsync into the prod web root. - -**DB migrations** — applied automatically on server restart (sqlx), but must have correct SHA-384 checksum in `_sqlx_migrations` or server crashes. Migration number collisions across branches are a real hazard — renumber before merging if two branches both add `NNN_*.sql`. Migration numbers confirmed on origin/main: 001-060 (`060_alert_mutes_agent_id_index`). Pending branches: 061 (BUG-018 FK indexes), 062 (MSP360), 063 (SPEC-021). - -**Windows build hosts — Beast PRIMARY, Pluto FALLBACK.** `build-windows.sh` runs `attempt_build beast || attempt_build pluto`: Beast is tried first; Pluto is used only if Beast is unreachable/down or its build fails. Both build the same `C:\gururmm` checkout; host keys at `/opt/gururmm/{beast,pluto}_known_hosts`. - -**Beast (GURU-BEAST-ROG) — PRIMARY:** -- Physical workstation, **i9-14900K (24c/32t)**, RTX 4090, 127.8 GB RAM. -- Reached from `.30` over **Tailscale** (installed on `.30`) at tailnet IP **100.101.122.4**, user `guru`. -- SSH: `ssh -o UserKnownHostsFile=/opt/gururmm/beast_known_hosts guru@100.101.122.4` -- Rust MSVC toolchain, **WiX 4.x** (WiX v6+ pulls OSMF and breaks the build), Gitea clone at `C:\gururmm\`. -- On a different LAN (Wi-Fi `10.2.51.228`) + tailnet — reachability depends on Beast being up on the tailnet. - -**Pluto (172.16.3.36) — FALLBACK:** -- Windows Server 2019 VM on Jupiter (Unraid) -- SSH: `ssh -o UserKnownHostsFile=/opt/gururmm/pluto_known_hosts Administrator@172.16.3.36` -- Rust stable 1.95.0 + 1.77 pinned for legacy builds -- VS Build Tools (MSVC), sccache at `C:\sccache`, WiX v4, Gitea clone at `C:\gururmm\` -- Xeon E5-2695 v3 @2.3GHz, 8c/16t KVM VM; serial build ~1269s - -**Auto-update delivery:** -- Server scans every 300s; dispatches update command on agent heartbeat -- Gated on effective policy `auto_update` (default on when policy is null) -- Agent: downloads to PrivateTmp, verifies SHA-256, replaces binary, restarts service -- Force-trigger: `POST /api/agents/:id/update` -- **As of agent-comms-durability Phase 1:** update dispatches piggyback on heartbeat re-offer path (same `redispatch_pending_commands`), so NAT'd agents receive updates reliably +**Channel/promotion model, downloads layout, dashboard beta/prod split, DB migration hazards, and Windows build host detail are unchanged from the 2026-06-25 compile** — see prior History Highlights / Compilation Notes for the full mechanics; current migration count is 065 (`065_integrations_framework`). --- ## Active State -**Fleet (as of 2026-06-25):** -- ~270 enrolled agents total; ~178 typically online -- Agent 0.6.75 / server v0.3.87 (up from 0.6.67 / 0.3.74 on 2026-06-22 — SPEC-030 software uninstall + universal installer landed) -- Metrics flowing (~2531 rows/15 min); alerts firing with 0 legacy null dedup_keys; per-agent API endpoints all HTTP 200 -- Beta channel: site "Mike's Car" (`103c10b9-c1de-4dd8-b382-b8362ed3143e`) has `update_channel='beta'` (persists across re-enrollment). All GURU-5070 machines are on this site. - -**Enrolled clients/sites (2026-05-24 baseline; no removals since):** - -| Client | Type | Sites | Notable agents | -|---|---|---|---| -| AZ Computer Guru (internal) | Internal | DF Server Storage, Howard-VM, Main Office, Mike's Car, Mikes House | Jupiter, PLUTO, gururmm, GURU-KALI, GURU-5070, Mikes-MacBook-Air.local, ACG-DC16, NEPTUNE, ix.azcomputerguru.com | -| BirthBiologic | Corporate | Main Office | BB-SERVER | -| Cascades of Tucson | Corporate | CascadesTucson | 27 agents — CS-SERVER, RECEPTIONIST-PC, ASSISTMAN-PC, MDIRECTOR-PC, MEMRECEPT-PC, and ~22 others | -| Dataforth Corp | Corporate | D1 | AD2, DF-GAGETRAK | -| Goldstein | (verify) | (verify) | Canary client for beta channel | -| Grabb & Durando Law Office | Corporate | Main Office | GND-SERVER | -| Instrumental Music Center | Corporate | IMCMain | IMC1 | -| Key, Paul | Residential | Home | KEY-MEDIA | -| Peaceful Spirit | Residential | Bridgette Home, Country Club, Mara Home | BridgettePSHomeComputer, PST-SERVER (UDR Ultra, comms-durability verification), PST-SERVER2, PST-SURFACE, Maras-HP-Laptop, MaraHomeNew | -| Safesite | Corporate | Glendale | DESKTOP-3USU20B | -| Sombra Residential LLC | Corporate | main office | DESKTOP-UQRN4K3, Server2013 | -| Stamback Septic | Corporate | StambackSeptic | DESKTOP-BTR2AM3, StambackLaptopNew | -| Swanson, Len | Residential | Home | LAS-GAMER | +**Fleet (as of 2026-07-04):** +- 359 enrolled agents; 228 fresh/online at last sample +- Agent 0.6.77 modern / 0.6.76 legacy-frozen-and-disabled; server v0.3.96 +- Active `server_error` alerts: **0** (cleaned to zero by the 2026-07-05 deploys) +- 13 duplicate-hostname agent pairs outstanding (not yet cleaned up) +- 5 active `backup_failed` alerts (GND-SERVER chronic, AD1, LAB-SVR, DESKTOP-EVA4H1A) — investigation open +- VSS configurator on beta channel only (0.6.76, 8 agents); stable fleet pinned at 0.6.66 (233 agents) +- Beta channel: site "Mike's Car" (`103c10b9-c1de-4dd8-b382-b8362ed3143e`) has `update_channel='beta'` (persists across re-enrollment) **API auth:** - `POST /api/auth/login` -> JWT (~24h) -- Creds: vault `infrastructure/gururmm-server.sops.yaml` -> `credentials.gururmm-api.admin-email` / `admin-password`; or via `bash .claude/scripts/rmm-auth.sh` +- Creds: vault `infrastructure/gururmm-server.sops.yaml` - Key endpoints: `GET /api/agents`, `POST /api/agents/:id/command`, `GET /api/commands/:id`, `POST /api/agents/:id/update`, `POST /api/updates/rollouts/:version/promote` -- Software (SPEC-030): `GET /api/agents/:id/software`, `POST /api/agents/:id/software/uninstall` (→ job_id), `GET /api/agents/:id/software/uninstall/jobs/:job_id`, `GET /api/agents/:id/software/removal-status`, `POST /api/agents/:id/software/removal-status/:attempt_id/resolve`, `GET /api/software/knowledge`, `POST /api/software/knowledge/classify` -- Command fields: `command_type` (`shell`/`powershell`/`python`/`script`/`claude_task`/`cmd` alias), `command` (script text, JSON-encoded), optional `context` — `system` (default) or `user_session` (Windows WTS), plus `timeout_seconds`/`elevated`. +- Software (SPEC-030): `GET /api/agents/:id/software`, `POST /api/agents/:id/software/uninstall` (-> job_id), `GET .../uninstall/jobs/:job_id`, `GET .../removal-status`, `GET /api/software/knowledge` +- **Integrations (NEW):** `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test` +- Command fields: `command_type` (`shell`/`powershell`/`python`/`script`/`claude_task`/`cmd` alias), `command`, optional `context` (`system`/`user_session`), `timeout_seconds`/`elevated`. **Dashboard — complete and working:** -Agents management (delete now returns 202 + background purge), Clients/Sites CRUD, Commands execution + terminal, Logs + AI analysis (Claude API), Alerts (clickable severity badges + client filtering), Metrics (CPU/RAM/disk/network, process drill-down modal), Auto-update triggering, Network state, Entra ID SSO, Policies Dashboard (all tabs), Registry editor (read-only via HTTP), MSP360 backup status + mappings/verify UI, Organizations management + dev-admin impersonation UI, Credentials management with inheritance, AgentDetail Crashes tab + version history, fleet stats from `/agents/stats`, SiteDetail Revoke Key + Enrollment audit tab, Install Reports page, Fleet Discovery page, **Event Log Watches management page** (`/event-log-watches`, shipped `0fa65f5`), **Software Manager** (installed-programs list + bulk remote uninstall with live progress + three-state removal knowledge base, SPEC-030 — **BETA, not guaranteed**; deployed 2026-06-23..24 but best-effort only). +Agents management, Clients/Sites CRUD, Commands execution + terminal, Logs + AI analysis (Claude API), Alerts, Metrics, Auto-update triggering, Network state, Entra ID SSO, Policies Dashboard, Registry editor (read-only), MSP360 backup status + mappings/verify UI, Organizations + dev-admin impersonation, Credentials with inheritance, AgentDetail Crashes tab + version history, Event Log Watches management, Software Manager (BETA, not guaranteed). **Dashboard — incomplete (see UI_GAPS.md):** - Watchdog alerts UI — blocked on 2 missing server routes - BSOD in Alerts stream (Phase 2 deferred) -- Tunnel session management (server skeleton "not yet implemented"; Phase 2 live TTY design will supersede) +- Tunnel session management (server skeleton) - Offboarding wizard UI (SPEC-028 complete; implementation pending) -- Alert mute UI — Ignore button + required-reason prompt, Muted filter, Un-ignore control (server endpoints ready; dashboard NOT started) -- `is_connected` reflects real connectivity (null fleet-wide cosmetic bug; gap in API/dashboard layer) -- Documentation / user guide / inline help tooltips (P3, not started) -- MSP360 "Open in MSP360" deep-link (PR #42, dashboard portion pending merge) - -**Open Gitea issues:** -- #10 — BSOD detection Phase 2/3 (dashboard Alerts stream + fetch_bsod_dump + full bugcheck table) -- #15 — Pipeline tray build (publish tray binary to downloads) -- #16 — Windows IPC peer authz -- #17 — logind console user resolution -- #18 — macOS tray -- #19 — subscriber broadcast +- Alert mute UI — server endpoints ready, dashboard NOT started +- **Integrations Center UI (NEW gap)** — CrowdStrike Falcon framework + sync worker are live server-side; no management screen in the dashboard yet +- **Tray policy toggle surfacing** — `tray.enabled` is enforced server/agent-side; verify whether the policy UI exposes the toggle to operators [verify] **Security backlog (HIGH):** - `credentials/:id/reveal` — horizontal privilege escalation (no ownership scope check) @@ -747,17 +612,19 @@ Agents management (delete now returns 202 + background purge), Clients/Sites CRU These decisions are locked. Do not reverse without explicit user approval. -1. **Per-agent enrollment keys** — MSI contains server URL + site_id only. Agent calls `POST /api/enroll` on first run; server issues unique per-agent key stored hashed. Enables revocation, clone detection, audit trail. -2. **Site-specific MSI generation** — Universal base MSI from CI; dashboard endpoint generates site-specific MSI with site_id baked in via WiX property -> `HKLM\SOFTWARE\GuruRMM\SiteId`. -3. **No TOML/config for endpoints** — Server URL compiled into binary. No runtime config files for server URL or site_id. -4. **Policy inheritance chain** — global -> site -> client -> agent. Server computes merged effective policy and pushes via `ConfigUpdate` WebSocket message. -5. **Platform parity rule** — Any agent feature ships on Windows, Linux, and macOS in the same change. Stub + TODO required if a real implementation is not yet feasible. -6. **Watchdog as separate process** — Main agent cannot reliably restart itself after a crash. The watchdog reports via REST (`POST /watchdog-alert`) — it has no WebSocket connection. -7. **Build pipeline is the only path to production** — Enforces signing, checksum generation, consistent artifact layout. -8. **Multi-tenancy identity model (ADR-001)** — Dev team with partner impersonation. Three levels: Dev -> Partner -> Client. Computer Guru is partner #1. -9. **Holistic feature development (DESIGN.md)** — Every feature requires backend + API + dashboard UI + documentation. Backend-only features are rejected. -10. **AI-optional operation** — GuruRMM must be fully functional without AI. AI features are enhancements, not requirements. -11. **Durability-first command delivery** — The durable Postgres `commands` row is the source of truth; the WebSocket is only a delivery hint. An un-acked command is re-deliverable, never reaped. Idempotent end-to-end by `command_id` so re-delivery never double-executes. (Established 2026-06-11, agent-comms-durability spec.) +1. **Per-agent enrollment keys.** +2. **Site-specific MSI generation.** +3. **No TOML/config for endpoints** — server URL compiled into binary. +4. **Policy inheritance chain** — global -> site -> client -> agent. +5. **Platform parity rule** — any agent feature ships on Windows, Linux, and macOS in the same change. +6. **Watchdog as separate process** — reports via REST (`POST /watchdog-alert`), no WebSocket. +7. **Build pipeline is the only path to production.** +8. **Multi-tenancy identity model (ADR-001)** — Dev -> Partner -> Client. +9. **Holistic feature development** — every feature requires backend + API + dashboard UI + docs. +10. **AI-optional operation.** +11. **Durability-first command delivery** — the durable Postgres `commands` row is the source of truth; the WebSocket is only a delivery hint. +12. **VSS: configure, never delete on a schedule** (added 2026-07-03) — the agent may only set retention governors and create shadows on a schedule; scheduled or automatic deletion reintroduces the MITRE T1490 surface. On-demand admin-initiated delete stays allowed. +13. **Third-party security integrations are a plugin, not a schema** (added 2026-07-01) — a new vendor implements the `integrations` plugin trait against the generic framework tables; it does not get its own bespoke migration set. --- @@ -765,58 +632,53 @@ These decisions are locked. Do not reverse without explicit user approval. | Date | Event | |---|---| -| 2025-12-15 | Project genesis: Windows service + Linux installer + site code auth + build server. DB migrated from Jupiter Docker to local PostgreSQL. | +| 2025-12-15 | Project genesis: Windows service + Linux installer + site code auth + build server. | | 2026-04-19 | Full drill-down navigation, auto-install on first run, Pluto build VM setup started. | -| 2026-04-21 | MSI build fix (missing WiX extension flag). DESIGN.md created (holistic development mandate). BirthBiologic onboarded. | +| 2026-04-21 | MSI build fix. DESIGN.md created. BirthBiologic onboarded. | | 2026-04-29 | UI_GAPS.md created. Holistic development principle formalized. | -| 2026-05-12 | macOS agent Phase 1 deployed from Mikes-MacBook-Air. Code signing issue on Apple Silicon noted. | -| 2026-05-15 | Dead WebSocket write-half bug fixed. Temperature struct field name mismatch fixed. | -| 2026-05-16 | Watchdog bugs fixed (sc.exe fallback, suppress_until, hypervisor detection). /feature-request skill created. | -| 2026-05-17 | Syncro PSA Integration added to roadmap (P1) after Howard /feature-request. Office power failure recovery — all VMs recovered. | -| 2026-05-18 | Multi-tenancy architecture (ADR-001) decided. 5 SPEC documents created (SPEC-001 through SPEC-006). | -| 2026-05-19 | 4-bug fix for AD2 crash loop. MSP360 backup integration completed (6 fixes). Clickable CPU/Memory gauge cards + process drill-down modal. | -| 2026-05-23 | /rmm-audit pass. Agent optimization Phases 1A-3. Auto-version bump mechanism. MSRV bumped to 1.85. Fleet at 0.6.29. | -| 2026-05-24 | Linux tray IPC + GTK (PR #13+#14) and peer-cred authz (PR #14) merged. PR #21 (ReadWritePaths fix) merged. Build pipeline split into per-platform scripts. Pluto known-hosts pinned. Fleet converged to 0.6.38. | -| 2026-05-31 | Roadmap reconciliation (17 corrections — roadmap understated built state). MSPBackups mapping/verify UI + dev-admin impersonation UI deployed (dashboard v0.2.32). BUG-008/013/014 status corrected to fixed. SPEC-021 (logged-in user domain detection) written after Howard feature request. | -| 2026-06-01 | BUG-016 (Linux systemd missing StateDirectory=gururmm) + BUG-017 (device_id OnceLock cache) fixed (commit 30da053). GURU-KALI had 11 ghost agent rows from repeated UUID churn — fixed and verified. BSOD forensics: GURU-5070 bluescreened with `0x116 VIDEO_TDR_FAILURE` (nvlddmkm.sys); GuruConnect cleared on three grounds. BSOD detection feature (issue #10 Phase 1) implemented: bsod.rs + migration 048 + ws/mod.rs handler; code review caught and fixed SF-1/SF-2; merged to main (0ec55cf), agent versioned 0.6.51. | -| 2026-06-02 | Server 0.3.37 + migration 048 deployed. Build channel default-beta fix applied to build-windows.sh + build-linux.sh. Webhook wired to dispatch build-server.sh with change-gate + backup/rollback. Fleet converged to 0.6.51. GURU-KALI BUG-016 unit file refreshed. | -| 2026-06-04 | Channel state confirmed via live Postgres query: GURU-5070 `agents.update_channel = 'beta'` (explicit per-agent override). Stable channel pinned at 0.6.47 windows/amd64 + 0.6.46 linux. BUG-020 (duplicate/ghost tray icons) fixed in commit `137dd85` to beta: per-session single-instance mutex + WTSEnumerateProcessesW reconciliation + graceful shutdown event (fix #3 dormant — now in PR #40). | -| 2026-06-07 | Backup-alert quality pass shipped. FU1 (summarize_backup_error + create_or_update_alert refresh) + FU2 (exclude PlanTypes 8/13 from alerting/compliance): false `backup_failed` 15 -> 2 fleet-wide (commits `779f7f6` + `b82c010`). `backup_storage_low` alert type removed entirely (DataCopied/TotalData is not destination capacity; 5 -> 0 false alerts; `resolve_all_backup_storage_alerts`). | -| 2026-06-07 | Credential inheritance deployed to production (server v0.3.45). Hierarchical propagation (Global → Client → Site), `is_inheritable`, `/effective` endpoints. Dashboard: clickable alert severity badges + client filtering. SPEC-028 offboarding wizard specification created (835 lines). FEATURE_ROADMAP.md updated with "Client & Site Lifecycle Management." | -| 2026-06-07 | Role-aware offline alerting + alert ignore/mute shipped. Offline sweep evaluator (60s interval; offline.rs): server-only `agent_offline`; migration 054 `agents.role_override`; site (>=50%+>=3) -> `mass_offline_site`; fleet (>=10) -> `mass_offline_fleet`. Warm-up restart guard (code review caught + fixed `last_seen < started_at` spec defect). Alert ignore/mute (perma-silence): migration 055 `alert_mutes` + `muted` status; `dedup_key`-keyed; reason required; gates `create_or_update_alert` + `create_check_alert` bypass; dashboard UI pending. | -| 2026-06-07 | UI gap batch + enrollment audit (GURU-BEAST-ROG session). `get_agent` upgraded to `AgentWithDetails`. Six new/updated server endpoints. Enrollment audit endpoint + per-row revoke. Dashboard: AgentDetail Crashes tab + version history, SiteDetail Enrollment tab, InstallReports + Discovery pages. Hotfix `6faa382` (role_override missing from new SQL — all GET /api/agents/:id returned 500). | -| 2026-06-11 | Physical server migration executed. Ubuntu VM at 172.16.3.30 retired; physical box took the same IP. Production down <27 min (06:53-07:20 MST). 162/212 agents reconnected within 15 min. Old VM at .46 as rollback anchor, decommissioned 2026-06-12. Server binary path changed from `/usr/local/bin/gururmm-server` to `/opt/gururmm/gururmm-server`. Ubuntu 26.04 + PostgreSQL 18. 7-day whale data backfilled (~3.4 GB, lossless, 0 pool-timeouts). Three build-env gaps fixed on new box (sccache, Pluto key, signing tools). | -| 2026-06-11 | Ghost agent root-caused. `device_id` storage fragile (single file, wiped by cleanup.ps1). 11 duplicate-hostname agents found (9 same-site ghosts; 2 cross-site non-merge). Durable agent identity spec created (`specs/durable-agent-identity/`; multi-AI reviewed). Phase 1 Task 1 shipped (agent v0.6.62): multi-location device_id (registry + ProgramData on Win; /etc + /var/lib on Linux; /Library + mirror on macOS) with self-heal; cleanup.ps1 whitelisted. Channel pin regression fixed: moved GURU-5070 beta override from agent-level to site "Mike's Car" (survives re-enrollment). | -| 2026-06-11 | Agent-comms-durability Phase 1 shipped (agent v0.6.63 / server v0.3.68). Spec created (specs/agent-comms-durability/; multi-AI reviewed, Gemini + Grok). Root cause: NAT conntrack asymmetry — server->agent pushes black-holed in idle conntrack gaps. Fix: agent CommandAck on receipt (migration 058 acked_at); agent dedup cache (re-report, never double-run); server reaper re-delivery (migration 059 delivery_attempts, cap 10; capability gate via partial index); pending commands re-offered on every heartbeat. Installer hardened (Start-Process). Build-pipeline dubious-ownership fixed. Canary (Goldstein + Peaceful Spirit): 6 agents verified, PST-SERVER ACK latency 16 ms. Fleet promoted 0.6.63 stable; 168 agents converged in ~3 min, 182 online throughout. | -| 2026-06-12 | Beast parallel Windows build (lever A) wired into production pipeline (commit `cfbdb59`). Two-wave parallel SSH: 5 concurrent stable + 2 concurrent legacy-1.77. Beast timing: ~336s (vs 622s serial, 1269s Pluto). Per-variant target-dir mandatory. Dropped `cargo +1.77 fetch` pre-resolve (edition2024 failure). v0.6.66 built clean on Beast in 336s. Old VM (.46) decommissioned (virsh domain + disk removed). | -| 2026-06-12 | command_type "cmd" root-caused as silent-drop (no such variant in CommandType enum). 7-round adversarial multi-AI quorum + packet captures. Fixed: `#[serde(alias = "cmd")]` on Shell + NAK on unparseable command (commit `3de9faf`). Reverted spurious eviction change (80df458). v0.6.66 promoted to stable fleet-wide. | -| 2026-06-15 | BSOD dedup key changed from dump hash to bugcheck code (f0a4b7f, server v0.3.73) — mutes now silence all recurrences of same bugcheck, not just one dump. MSI EXDEV fix (95ef901, server v0.3.74) — site MSI builder staged in /tmp causing cross-device rename failure; fixed to stage in downloads_dir. Duplicate offline server alerts in Triage removed (b6ed564). sync.sh Phase 1 auto-heal (resolve_submodule_collisions) — rescued 94 orphaned RMM_THOUGHTS lines. logs/analyze switched to Claude API (c869e4d). 500-error-leak fully closed (58c1a96). | -| 2026-06-18 | sync.sh submodule auto-heal verified fleet-wide after earlier 2026-06-15 fix. RMM_THOUGHTS 2026-06-08 re-grounding pass + 8 Raw sections rescued and staged. | -| 2026-06-21 | BUG-019 (container self-update guard) fixed and merged to main (66a7f4e, v0.6.67 beta). BUG-018 (DELETE 202+bg, cea87d4) merged. Event Log Watch UI shipped (0fa65f5). Enrollment modal UX fix merged to prod (4027c86). sync.sh populate-only guard (Phase 2) fixed submodule-clobber root cause. Howard cleared for GuruRMM merges/deploys (Mike decision). gururmm-build skill + docs/BUILD.md created. Five PRs opened (#40-#44) covering SPEC-021, BUG-018 FK indexes, MSP360 deeplink, Event Log Watch policy-clobber HIGH fix, audit cleanup. | -| 2026-06-22 | BUG-021 (legacy build dep-pin getrandom+zeroize) fixed on main (1dce66d). Windows build green at v0.6.67, 2026-06-22 02:19. Fleet verified: ~270 agents / ~178 online, 0 legacy null dedup_keys. BUG-022 filed + fixed (PR #45): removed dead WatchdogEvent WS path (no producer — watchdog has no WS connection; REST watchdog-alert is the only real path). | -| 2026-06-23 | Universal self-detecting installer (Feature 9) P1 built+deployed+verified (v0.6.71). One install path self-detects arch/OS/legacy/ARM and pulls the correct agent; dashboard install UI repointed at it (`4194b0a`, `53bb682`); script path proven on GND-JWILL (`de30ebc`). av-removal-recipes spec (P3, removal-only) added (#52). RMM_THOUGHTS Feature 10 (AMPIPIT recovery env) → Discussed. | -| 2026-06-24 | SPEC-030 remote software inventory + bulk uninstall — BETA, code merged + deployed but NOT guaranteed (Route B; best-effort, fails on protected AV / WiX bundles / UI-only / lingering-child uninstallers / drivers). Silent-uninstall engine (`uninstall-engine.ps1`, BCU-informed, refuses to guess). CRITICAL Avira false-success fixed — WiX Package-Cache bundle GUID misread as MSI ProductCode (1605 "already gone" while installed); skip MSI tier for Package Cache + generic `Test-StillInstalled` ARP re-check (PR #53). Code-review correctness fixes (PR #54). Async removal jobs = permanent fix (PR #56, migration 064): POST→job_id in 0.06s, background per-target worker, dashboard live progress — replaces the synchronous request that died at the proxy ~100s timeout + was truncated by the agent kill-timeout mid-flush. Flush-slack stop-gap (PR #55). Three-state knowledge base + learned timing (migrations 061-063). Orphaned-entry + retry-verify (PR #57). Gaps surfaced: drivers in `/software` (ARP), Launchy/AIMP need recipes (Route A), no list-jobs endpoint. agent 0.6.75 / server v0.3.87. | -| 2026-06-25 | Win11 KB5095093 reviewed for MSP fleet use; Point-in-Time Restore (Enterprise snapshot/rollback knobs) + GP update-pause captured as a Raw RMM_THOUGHTS entry (manage the knobs from the RMM; sits below AMPIPIT Feature 10 on the remediation ladder). | +| 2026-05-12 | macOS agent Phase 1 deployed. Code signing issue on Apple Silicon noted. | +| 2026-05-15 | Dead WebSocket write-half bug fixed. Temperature struct field mismatch fixed. | +| 2026-05-16 | Watchdog bugs fixed. /feature-request skill created. | +| 2026-05-17 | Syncro PSA Integration added to roadmap. Office power failure recovery. | +| 2026-05-18 | Multi-tenancy architecture (ADR-001) decided. 5 SPEC documents created. | +| 2026-05-19 | AD2 crash-loop fix. MSP360 integration completed. Process drill-down modal. | +| 2026-05-23 | /rmm-audit pass. Agent optimization Phases 1A-3. MSRV bumped to 1.85. Fleet at 0.6.29. | +| 2026-05-24 | Linux tray IPC + GTK merged. Build pipeline split per-platform. Fleet converged to 0.6.38. | +| 2026-05-31 | Roadmap reconciliation. MSPBackups mapping/verify UI + dev-admin impersonation UI deployed. SPEC-021 written. | +| 2026-06-01 | BUG-016/017 fixed. BSOD detection feature (Phase 1) implemented and merged, agent 0.6.51. | +| 2026-06-02 | Server 0.3.37 + migration 048 deployed. Webhook wired to dispatch build-server.sh. | +| 2026-06-04 | BUG-020 (ghost tray icons) fixed to beta (`137dd85`). | +| 2026-06-07 | Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit all shipped same day. | +| 2026-06-11 | Physical server migration executed (<27 min downtime). Ghost agent root-caused, durable identity Phase 1 Task 1 shipped. Agent-comms-durability Phase 1 shipped (agent v0.6.63/server v0.3.68). | +| 2026-06-12 | Beast parallel Windows build wired in (`cfbdb59`, ~336s). command_type "cmd" root-caused and fixed (`3de9faf`). | +| 2026-06-15 | BSOD dedup key changed to bugcheck code. MSI EXDEV fix. sync.sh Phase 1 auto-heal. logs/analyze switched to Claude API. 500-error-leak closed. | +| 2026-06-18 | sync.sh submodule auto-heal verified fleet-wide. | +| 2026-06-21 | BUG-019/018 fixed and merged. Event Log Watch UI shipped. Howard cleared for GuruRMM merges/deploys. gururmm-build skill created. | +| 2026-06-22 | BUG-021 (legacy dep-pin) fixed, v0.6.67 stable. BUG-022 (dead WatchdogEvent path) fixed. Fleet verified ~270/~178 online. | +| 2026-06-23 | Universal self-detecting installer (Feature 9) P1 shipped (v0.6.71). av-removal-recipes spec added. | +| 2026-06-24 | SPEC-030 remote software uninstall — BETA — code merged + deployed (Route B, async jobs, three-state KB, migrations 061-064). agent 0.6.75 / server v0.3.87. | +| 2026-06-25 | Win11 KB5095093 Point-in-Time Restore captured as a Raw RMM_THOUGHTS entry. | +| 2026-07-01..02 | Vendor-agnostic Integrations Framework shipped (migration 065) with CrowdStrike Falcon as the first plugin — API client, client mappings, periodic sync worker. No dashboard UI yet. | +| 2026-07-03..04 | VSS Policy Configurator (SPEC-016 redesign) shipped to beta (agent 0.6.76) — replaces scheduled create+prune with a configure-governors-only model, eliminating the T1490 surface; 23-agent code review caught and fixed a data-loss-class `MaxShadowCopies` defect pre-merge; verified live on the T1490 canary NEPTUNE. | +| 2026-07-04 | Legacy (2008R2/Win7, Rust 1.77) build variant disabled (`84a82a3`) after an edition2024 dep recurrence fail-closed the whole Windows build; last legacy artifact frozen at 0.6.76. RMM_THOUGHTS Feature 12 (legacy agent rewrite) added Raw. Cross-site agent dedup fixed (`2851523`, `1c1cac2`) — dedup by device_id across all sites, no re-home on reconnect. Policy-Controlled Tray Deployment merged (`f6a4251`) — materialize shipped fleet-wide, launch stays policy-gated default OFF. | +| 2026-07-04..05 | Easy-bugs batch + full alert triage: update-scanner host-native/foreign self-check regimes (fixed macOS/Linux agents never receiving update offers), inventory NUL/control-char sanitize (`db/sanitize.rs`, fixed 7+ machines' silently-failing inventory), WS benign-disconnect + pre-auth churn reclassified ERROR->INFO, `create_or_update_alert` survives cascade-delete races, `move_agent` maps duplicate-key to 409. Deployed as server v0.3.95 + v0.3.96. Active `server_error` alerts reduced to 0. Machine-side backup-failure investigation (GND-SERVER et al.) started but not completed. | --- ## Compilation Notes -- **2026-05-26 recompile:** Added the Capabilities / Feature Set section, synthesized from authoritative artifacts at live `main` (cd27a59). Corrected: command execution contexts, temperature monitoring (BUG-001 is resolved, not pending), Entra-only SSO, added user-inventory/discovery/VM-detection/safe-rollout surfaces. Changelogs are an unreliable capability source — stop at agent v0.6.22; migrations + commit log are authoritative. -- Tunnel subsystem (verified against live main): agent side substantially built; server side is a dead-code skeleton (not declared in `api/mod.rs`, no routes, WS handler logs "not yet implemented"). Confirmed, not unverified. -- macOS build status: Phase 1 was deployed manually from Mikes-MacBook-Air (2026-05-12). `build-mac.sh` is a stub as of 2026-05-24. [unverified whether automated pipeline includes macOS] -- Pre-commit hook on 172.16.3.30 lacks execute bit (noted 2026-05-23). [unverified — may still be unfixed on new box] -- Auto-update reliability fix for BB-SERVER and RECEPTIONIST-PC was incomplete at 2026-05-24. [now superseded by comms-durability Phase 1 — heartbeat re-offer path should resolve this] -- **2026-06-02 recompile:** Folded in BSOD detection, server build webhook wiring, build channel default beta, versions updated. Migration count updated 46 -> 48. -- **2026-06-04 recompile:** Corrected GURU-5070 channel state. Stable fleet pinned at 0.6.47. BUG-020 documented. -- **2026-06-07 recompile:** Folded in backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit. Updated migration count to 55+ (054/055 confirmed). -- **2026-06-11 recompile (GURU-5070/claude-main):** Full recompile. Added: (1) Physical server migration (Ubuntu 26.04, PG 18, binary at /opt/gururmm, old VM .46 rollback anchor). (2) Durable agent identity (ghost root cause, spec, Phase 1 Task 1 durable device_id, channel pin fix). (3) Agent comms durability Phase 1 full detail (spec, slices A/B/C, deployment, fleet rollout, canary verification). (4) New capabilities: Audit Log (migration 056), Systemic Log Feedback Intelligence (migration 057), comms durability architecture. (5) Updated fleet size (~215 enrolled, 168-182 online). (6) Updated agent/server versions to 0.6.63/0.3.68. (7) Build pipeline: server IS auto-deployed by webhook (correction to earlier assumption); parallel build prototype on Pluto (3.51x, deferred integration). (8) Channel/promotion model documented in detail. (9) Downloads layout documented. (10) Updated server binary path + SSH details for physical box. (11) Added new anti-patterns (installer `&` operator, reaper false-fail, agent-level channel pin). Added ADR-11 (durability-first command delivery). Migration count updated to 59. Patterns/History preserved verbatim except new entries added. -- **2026-06-25 recompile (Howard-Home/claude-main):** Delta from 2026-06-22 (full recompile against live origin/main @ `0074c3a`; migrations 60 → 64; agent 0.6.67 → 0.6.75, server 0.3.74 → 0.3.87). (1) **SPEC-030 remote software inventory + bulk uninstall** added as a major new Capabilities subsection — Route B (server-orchestrated) endpoints, the `uninstall-engine.ps1` silent engine (refuses-to-guess + generic `Test-StillInstalled` ARP verification + Package-Cache-bundle 1605 false-success fix + orphaned/retry handling), async removal jobs (migration 064), three-state fleet knowledge base + learned timing (migrations 061-063), `SoftwareManager.tsx` dashboard. New `software.rs` API routes + `software_jobs.rs` DB module. Known gaps captured (drivers-in-ARP UI exclusion, Launchy/AIMP Route-A recipes, no list-jobs endpoint). (2) **Universal self-detecting installer (Feature 9)** P1 shipped (v0.6.71) — Patch/Update + repo-structure + history updated. (3) New Recent Work entry (2026-06-23..24) + 4 History rows (06-23/06-24/06-25). (4) **Current Focus rewritten** — SPEC-030 follow-on (recipes, driver exclusion, list-jobs); **flagged the old "pending PRs #40-#46, migrations 061-063" block as [VERIFY]** because software-removal merged ahead and took migrations 061-064, invalidating the predicted numbering. (5) Fleet/version/dashboard/API-endpoint blocks updated. (6) Sources + migration-count updated. History/Patterns preserved verbatim except new entries added. NOTE: the pre-2026-06-22 SPEC-030 migration-number predictions in the older Compilation Note below are left as the historical record — see the 2026-06-25 [VERIFY] note for current reality. -- **2026-06-22 recompile (Howard-Home/claude-main):** Delta from 2026-06-11: (1) Fleet updated to ~270 enrolled / ~178 online, agent v0.6.67. (2) BUG-021 (legacy wave dep-pin, commit 1dce66d), BUG-018 (DELETE 202+bg, cea87d4), BUG-019 (container guard, 66a7f4e) all fixed on main. (3) Event Log Watch UI shipped (0fa65f5). (4) Enrollment modal UX fix to prod (4027c86). (5) Watchdog section corrected per BUG-022: WatchdogEvent WS path was dead code (no producer; watchdog has no WS connection); removed in PR #45. REST `watchdog-alert` is the only supported watchdog alert path. (6) Beast parallel two-wave Windows build documented (lever A, 336s, ~3.8x vs Pluto). (7) BUG-021 dep-pin gotcha documented (getrandom 0.3.1 + zeroize 1.8.1). (8) command_type "cmd" alias + NAK documented. (9) Event Log Watch policy-clobber HIGH fix (PR #43). (10) MSP360 deep-link (PR #42, mig 062). (11) SPEC-021 (PR #40, mig 063). (12) sync.sh submodule-clobber root-cause fix (populate-only guard). (13) BSOD dedup key changed to bugcheck code. (14) MSI EXDEV fix. (15) 500-error-leak fix. (16) logs/analyze switched to Claude API. (17) gururmm-build skill + docs/BUILD.md. (18) Howard cleared for merges/deploys. (19) Dashboard beta-before-main rule. (20) Migration count updated to 60 (origin/main), with 061-063 on pending branches. (21) Old VM confirmed deleted (2026-06-12). (22) Open PRs #40-#46 documented with merge-order. (23) New anti-patterns added (command_type cmd, cargo +1.77 fetch, target-dir concurrency, BUG-021 dep-pin, stale snapshot build status, config-push clobber, stale submodule working tree). (24) New good patterns added (worktrees for concurrency, gururmm-build skill, beta-before-main). +- **2026-07-04 recompile (Howard-Home/claude-main):** Delta from 2026-06-25 (migrations 64 -> 65; agent 0.6.75 -> 0.6.77 modern / 0.6.76 legacy-frozen; server 0.3.87 -> 0.3.96; fleet 270 -> 359 enrolled). Added: (1) **Third-Party Integrations Framework** (migration 065, CrowdStrike Falcon plugin) as a new Capabilities subsection — API-only, no dashboard UI yet, flagged as a new UI gap. (2) **VSS Policy Configurator (SPEC-016 redesign)** replacing the prior scheduled create+prune model — new Capabilities subsection, new locked architecture decision #12, data-loss-defect anti-pattern captured. (3) **Policy-Controlled Tray Deployment** — materialize/launch split documented, Task 4 (fleet enablement) flagged as pending in Current Focus. (4) **Legacy (2008R2/Win7) build variant disabled** — Patch/Update section + Build & Deploy + anti-patterns updated; RMM_THOUGHTS Feature 12 noted. (5) **Cross-site agent dedup fix + no-re-home-on-reconnect follow-up** — new Agent Identity & Enrollment subsection; 13 remaining duplicate pairs noted as open work. (6) **2026-07-04/05 easy-bugs + alert-triage batch** (server v0.3.95/v0.3.96) — update-scanner platform regimes, `db/sanitize.rs`, WS noise reclassification, `create_or_update_alert` Option hardening, `move_agent` 409 — folded into Monitoring, Patch/Update, and Alerting Capabilities subsections plus a new Recent Work entry. (7) Active State refreshed (fleet count, 0 active server_error, 13 dup pairs, 5 open backup_failed alerts, VSS beta-only channel state). (8) New anti-patterns: scanner host/foreign self-check split, machine-global VSS governor data-loss class, reconnect re-homing footgun, `fetch_one` on a concurrently-deletable row, concurrent-session-no-lock branch collision, stale spec-status-header trap. History Highlights/Patterns preserved verbatim except new rows appended; article length kept in the same range as the 2026-06-25 compile by condensing older Recent Work entries into shorter summaries where the full detail is preserved in git history and prior compiles. +- **2026-06-25 recompile (Howard-Home/claude-main):** Delta from 2026-06-22 — SPEC-030 remote software inventory + bulk uninstall added as a major Capabilities subsection; universal self-detecting installer (Feature 9) P1 shipped; Current Focus rewritten; flagged the old "pending PRs #40-#46, migrations 061-063" prediction as superseded because software-removal work consumed 061-064 instead. +- **2026-06-22 recompile (Howard-Home/claude-main):** Delta from 2026-06-11 — fleet ~270/~178, BUG-021/018/019 fixed on main, Event Log Watch UI shipped, BUG-022 dead-code watchdog path removed, Beast parallel two-wave build documented, command_type "cmd" alias + NAK documented, migration count to 60. +- **2026-06-11 recompile (GURU-5070/claude-main):** Full recompile — physical server migration, durable agent identity, agent-comms-durability Phase 1 full detail, migration count to 59. +- **2026-06-07 recompile:** Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit folded in; migration count to 55+. +- **2026-06-04 recompile:** Corrected GURU-5070 channel state; BUG-020 documented. +- **2026-06-02 recompile:** BSOD detection, server build webhook wiring, build channel default beta; migration count 46 -> 48. +- **2026-05-26 recompile:** Added the Capabilities / Feature Set section, synthesized from authoritative artifacts. Changelogs are an unreliable capability source — migrations + commit log are authoritative. ## Backlinks - [[clients/cascades-tucson]] — RECEPTIONIST-PC enrolled (site CascadesTucson) -- [[systems/gururmm-build]] — Physical server at 172.16.3.30 on Jupiter subnet; GuruRMM API 3001, ClaudeTools API 8001, Coord API, MariaDB, PostgreSQL, build pipeline; migrated from VM to physical box 2026-06-11 -- [[systems/jupiter]] — Unraid host at 172.16.3.20; virsh host for all VMs (Pluto/Claude-Builder, Unifi, OwnCloud); Docker: Gitea port 3000, NPM, Seafile; NPM is the public TLS terminator for rmm-api.azcomputerguru.com (forwards to .30:80) +- [[systems/gururmm-build]] — Physical server at 172.16.3.30; GuruRMM API 3001, ClaudeTools API 8001, Coord API, MariaDB, PostgreSQL, build pipeline +- [[systems/jupiter]] — Unraid host at 172.16.3.20; virsh host for all VMs; Docker: Gitea, NPM, Seafile - [[systems/pluto]] — Windows build server (MSI, WiX) at 172.16.3.36