--- type: project name: gururmm display_name: GuruRMM last_compiled: 2026-07-06 compiled_by: Howard-Home/claude-main aliases: - guru-rmm sources: - "gururmm@origin/main f0320a4: policy-ui Fleet-Policy Console (migration 068_visible_agents_folder_id, folder_id in API d028ca1, dashboard rebuild bd25fa9), PROD deploy; agent 0.6.79 / server 0.3.103" - "gururmm@origin/main 07307e3: WS-looper comms-durability (write-timeout + heartbeat-count diagnostic + AIMD keepalive, cd9e799), agent 0.6.79 beta / server 0.3.102" - "projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-05-howard-ws-looper-comms-durability.md" - "gururmm@origin/main 01d962a: migrations 066-067, policy-core merge e7e90c2, FEATURE_ROADMAP BUG-003/018/023/024" - "gururmm@origin/main: server/migrations/066_visible_agents_view.sql, 067_policy_folders.sql" - "gururmm@origin/main: server/src/api/mod.rs (route surface at 01d962a — policy-folders endpoints, 39 modules)" - "gururmm@origin/main: docs/specs/policy-core/{shape,plan,references,standards}.md, docs/specs/policy-ui/{shape,plan,references,standards}.md" - "gururmm@origin/main: commit e7e90c2 (policy-core merge), 6d2830d (066->067 renumber), 01d962a (policy-ui spec)" - "gururmm@origin/main: commit 76a6bd2 (repair batch 2 merge), 6ccd35e (bug-fix batch 3 merge), c34194a (batch 4 merge)" - "gururmm@origin/main: commit 519eef2/f2d02dd (easy-bugs+preauth), 238aad7 (batch 2 review hardening), 27aaab7 (batch 3 review hardening)" - "projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md" - "projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-easy-bugs-batch-deploy.md (incl. 00:45/01:00 PT Update sections — repair batch 2/v0.3.97/agent 0.6.78)" - "projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-05-howard-bugfix-batch3-channel-race.md (incl. 11:05 PT Update section — batch 4/v0.3.99/BUG-024)" - "projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md (BUG-003/BUG-018/BUG-023/BUG-024 sections)" - "gururmm@main: server/src/api/*.rs (REST API surface, ~40 route modules; integrations.rs new)" - "gururmm@main: server/src/api/mod.rs (route surface diff, 2026-06-25..2026-07-04 — integrations CRUD only new surface)" - "gururmm@main: server/migrations/*.sql (65 migrations — feature checkpoints through 065_integrations_framework)" - "gururmm@main: server/migrations/065_integrations_framework.sql" - "gururmm@main: server/src/db/sanitize.rs (NEW — shared NUL/control-char sanitize module)" - "gururmm@main: server/src/db/inventory.rs, server/src/db/alerts.rs, server/src/updates/scanner.rs, server/src/ws/mod.rs, server/src/api/agents.rs (2026-07-04 easy-bugs + alert-triage batch)" - "gururmm@main: agent/src/ (agent capabilities; tray_stage.rs, tray_state.rs, authenticode.rs new; device_id.rs, ohw.rs, watchdog/wts.rs, bsod.rs)" - "gururmm@main: specs/tray-policy-deploy/shape.md + plan.md (policy-controlled tray deployment)" - "gururmm@main: specs/vss-policy-config/shape.md (VSS policy configurator, SPEC-016 redesign)" - "gururmm@main: specs/vss-native-com/ (native VSS COM create/provision)" - "gururmm@main: specs/crowdstrike-falcon/shape.md (integrations framework + Falcon plugin)" - "gururmm@main: docs/RMM_THOUGHTS.md (header index Features 1-12; Refinement 4a #1 DONE; Feature 12 legacy rewrite Raw)" - "gururmm@main: docs/FEATURE_ROADMAP.md (BUG-009/BUG-011 verified fixed 2026-07-04; Known Bugs section)" - "gururmm@main: git log origin/main (2026-06-25..2026-07-04 — VSS, tray, integrations, easy-bugs batches)" - "gururmm@main: commit 8b75274..5f38ef3 (crowdstrike-falcon Tasks 1-5)" - "gururmm@main: commit de30b2b (VSS policy configurator merge)" - "gururmm@main: commit f6a4251 (policy-controlled tray deployment merge)" - "gururmm@main: commit 84a82a3 (legacy Rust 1.77 variant disabled)" - "gururmm@main: commit 2851523 + 1c1cac2 (cross-site agent dedup fix + no re-home on reconnect)" - "gururmm@main: commit 519eef2 + f2d02dd (easy-bugs batch + pre-auth churn noise merges, v0.3.95/v0.3.96)" - "gururmm@main: session-logs/2026-07/2026-07-04-howard-easy-bugs-batch-deploy.md" - "gururmm@main: session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md" - "session-logs/2026-07/2026-07-02-mike-crowdstrike-rollout-365-appsuite.md" - "session-logs/2026-07/2026-07-03-mike-gururmm-vss-configurator-and-pst-recovery.md" - "session-logs/2026-07/2026-07-04-mike-gururmm-vss-build-merge-deploy.md" - ".claude/memory/project_gururmm_legacy_disabled.md" - "gururmm@main: agent/src/ (agent capabilities; transport/CommandContext, ohw.rs, watchdog/wts.rs, bsod.rs, device_id.rs)" - "gururmm@main: agent/scripts/uninstall-engine.ps1 (SPEC-030 Tier-1 silent uninstall engine)" - "gururmm@main: server/src/db/software_jobs.rs (async removal jobs)" - "gururmm@main: dashboard/src/components/SoftwareManager.tsx (installed-programs list + bulk uninstall + live progress)" - "gururmm@main: server/migrations/061_software_removal_attempts.sql" - "gururmm@main: server/migrations/062_software_knowledge.sql" - "gururmm@main: server/migrations/063_software_knowledge_timing.sql" - "gururmm@main: server/migrations/064_software_removal_jobs.sql" - "gururmm@main: git log origin/main (2026-06-22..2026-06-25 — SPEC-030 + universal installer)" - "gururmm@main: session-logs/2026-06/2026-06-24-howard-gururmm-async-removal-jobs.md" - "gururmm@main: session-logs/2026-06/2026-06-25-howard-win11-pit-restore-rmm-thought.md" - "gururmm@main: server/migrations/048_bsod_events.sql" - "gururmm@main: server/migrations/056_audit_log.sql" - "gururmm@main: server/migrations/057_log_signatures.sql" - "gururmm@main: server/migrations/058_command_acked_at.sql" - "gururmm@main: server/migrations/059_command_delivery_attempts.sql" - "gururmm@main: server/migrations/060_alert_mutes_agent_id_index.sql" - "gururmm@main: agent/src/bsod.rs" - "gururmm@main: server/src/api/updates.rs (promote/rollback endpoints)" - "gururmm@main: deploy/build-pipeline/webhook-handler.py" - "gururmm@main: deploy/build-pipeline/build-server.sh" - "gururmm@main: docs/BUILD.md (build + pre-merge verification guide)" - "gururmm@main: commit 1dce66d (BUG-021 dep-pin)" - "gururmm@main: commit cea87d4 (BUG-018 202+bg)" - "gururmm@main: commit 0fa65f5 (Event Log Watch UI)" - "gururmm@main: commit 3de9faf (command_type cmd alias + NAK)" - "gururmm@main: commit 4027c86 (enrollment modal UX)" - "gururmm@main: commit f0a4b7f (BSOD dedup key)" - "gururmm@main: commit 95ef901 (MSI EXDEV fix)" - "gururmm@main: commit 137dd85 (BUG-020 tray fix)" - "gururmm@main: docs/specs/, docs/FEATURE_ROADMAP.md, docs/BUILD.md" - projects/msp-tools/guru-rmm/CONTEXT.md - projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md - projects/msp-tools/guru-rmm/docs/UI_GAPS.md - projects/msp-tools/guru-rmm/docs/ARCHITECTURE_DECISIONS.md - projects/msp-tools/guru-rmm/docs/tech-stack.md - projects/msp-tools/guru-rmm/docs/DESIGN.md - projects/msp-tools/guru-rmm/docs/BUILD.md - projects/msp-tools/guru-rmm/specs/agent-comms-durability/shape.md - projects/msp-tools/guru-rmm/specs/agent-comms-durability/plan.md - projects/msp-tools/guru-rmm/specs/agent-comms-durability/references.md - projects/msp-tools/guru-rmm/specs/agent-comms-durability/standards.md - .claude/memory/reference_gururmm_server.md - .claude/memory/reference_gururmm_api.md - .claude/memory/gururmm-development-principles.md - .claude/memory/feedback_gururmm_agent_parity.md - .claude/memory/reference_pluto_build_server.md - .claude/memory/project_mac_gururmm_setup_pending.md - .claude/memory/feedback_gururmm_build_channel_default.md - .claude/memory/reference_gururmm.md - .claude/memory/feedback_gururmm_build_verification.md - .claude/memory/rmm-dashboard-beta-before-main.md - credentials.md - session-logs/2025-12-15-session.md - session-logs/2025-12-20-session.md - session-logs/2026-04-19-session.md - session-logs/2026-04-21-session.md - session-logs/2026-04-29-session.md - session-logs/2026-05-12-guru-rmm-macos-agent-phase1.md - session-logs/2026-05-15-session.md - session-logs/2026-05-16-session.md - session-logs/2026-05-17-session.md - session-logs/2026-05-19-gururmm-backup-fixes.md - session-logs/2026-05-19-session.md - session-logs/2026-05-21-session.md - session-logs/2026-05-23-session.md - session-logs/2026-05-24-session.md - session-logs/2026-05-24-GURU-KALI-session.md - session-logs/2026-05-31-howard-gururmm-roadmap-and-features.md - session-logs/2026-06-02-mike-bsod-detection-and-pipeline.md - session-logs/2026-06-07-mike-gururmm-offboarding-spec.md - session-logs/2026-06-07-mike-gururmm-backup-alert-cleanup.md - session-logs/2026-06-07-mike-gururmm-offline-alerting-mute.md - session-logs/2026-06-07-mike-gururmm-ui-gaps-enrollment.md - projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-host-migration-cutover.md - projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-backfill-build-pipeline.md - projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-promotion-ghost-identity.md - projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-comms-durability.md - projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-comms-durability-phase1.md - projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-12-mike-beast-parallel-windows-build.md - projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-12-mike-rmm-command-type-cmd-misdiagnosis.md - session-logs/2026-06/2026-06-15-mike-unifi-wifi-skill-and-gururmm-fixes.md - session-logs/2026-06/2026-06-18-mike-sync-submodule-autoheal-and-thoughts-rescue.md - session-logs/2026-06/2026-06-21-howard-gururmm-bug-018-019.md - session-logs/2026-06/2026-06-21-howard-rmm-bug018-eventlogwatch-ui.md - session-logs/2026-06/2026-06-21-howard-gururmm-features-audit-submodule-fix.md - session-logs/2026-06/2026-06-21-mike-rmm-build-skill-errorlog-lint-coord-purge.md backlinks: - clients/cascades-tucson - systems/gururmm-build - systems/jupiter - systems/pluto --- # GuruRMM ## Summary GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 360 enrolled agents (~229 fresh/online at last sample) across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints. **Current version (2026-07-06):** agent 0.6.78 fleet-wide stable (Windows amd64, ~229+ agents) with **0.6.79 soaking on beta** (WS-looper comms-durability hardening) / server **v0.3.103** / dashboard rebuilt with the **Fleet-Policy Console** (policy-ui, PROD). Migrations end at `068_visible_agents_folder_id.sql`. Active `server_error` alerts: 0 (last verified 2026-07-06 03:22Z). **Policy folders + inheritance ("policy-core") deployed 2026-07-05 (Mike, migration 067, merge `e7e90c2`):** the biggest architectural change of this compile window. An org-rooted **policy folder tree** now sits on top of the existing policy engine — each client gets a folder tree (exactly one root), any folder optionally references a library policy, and a device sits in exactly one folder, inheriting the merged policies of its folder ancestry (root->leaf) plus an optional per-asset override. This replaces the fixed System->Client->Site->Agent stack with a per-client tree while preserving identical effective-policy output during migration (the backfill is a deliberate, gated, dry-run-verified step — landing the migration changes zero behavior, since `agents.folder_id` is NULL fleet-wide until backfilled). Ships folder-tree CRUD + placement + version history REST API, recompute-and-push on policy edit (scoped to affected agents only), and policy versioning (`policy_versions`, immutable snapshot per edit, for diff/rollback). Migration renumbered 066->067 after colliding with `066_visible_agents_view.sql`, which landed on main in the same window from an unrelated bug-fix arc. Explicitly out of scope for this phase: module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI, and any agent-side wire-format change (none — this is server+dashboard only). A follow-on **policy-ui** shape spec (folder management dashboard UI) was added the same day (`01d962a`) — implementation not started. **Four bug-fix batches shipped 2026-07-04..05 (server v0.3.95 -> v0.3.99, agent 0.6.77 -> 0.6.78):** a dense back-to-back sequence of alert-triage, easy-bugs, repair, and issue-hunt passes. Highlights (full detail in Recent Work / History Highlights): update-scanner platform-coverage + channel-default fixes (macOS/Linux agents were never offered updates; then a scanner/build-pipeline race briefly mass-offered unsoaked beta to the stable fleet — **BUG-023**, fixed); inventory NUL-byte storage failures fixed for 7+ machines; WS log noise reclassified off ERROR; **BUG-003** (unsafe deploy pipeline) closed with a build lock + gated git reset + pre-stop migration validation; **BUG-018** (agent delete race) got its permanent altitude fix via a `visible_agents` DB view + schema trigger; **BUG-001** (Windows ACPI thermal) shipped and was proven live on 112+ agents; comms-durability Phase 1 completed (half-open WS eviction sweep); a pending-command TTL, an agent-status reconciler, and update-supersede logic shipped in batch 4; and a **critical Linux glibc incident (BUG-024, still OPEN/mitigated)** crash-looped every older-distro Linux agent that took the newly-enabled Linux update offer. **Cross-site agent duplication fixed 2026-07-04:** Root cause of ~40 duplicate agent rows (24 at Dataforth D2->D1 alone): both WS enrollment paths deduped only within `(site_id, device_id)`, so re-running a different site's installer on an already-enrolled machine created a second row instead of finding the existing one. Fixed to match by hardware `device_id` **across all sites** and re-home via `move_agent_to_site` — but a same-day follow-up (`1c1cac2`) stopped the auto-re-home on every reconnect (the agent re-presents its install-site code as its API key on every beat, which was bouncing admin-moved agents back to their original site). Net effect: dedup by device_id, but site placement stays admin-controlled and persists. **Legacy (Server 2008R2/Win7, Rust 1.77) build variant disabled 2026-07-04:** The unpinned legacy dependency re-resolve pulled edition2024 crates (`chacha20`/`rand_core 0.10.1`) that Rust 1.77 cannot parse — a BUG-021 recurrence that fail-closed the entire Windows build (blocking modern amd64/x86/tray artifacts too). Gated behind `LEGACY_ENABLED=false`; the last-built legacy artifact (agent 0.6.76) is preserved and frozen for existing 2008R2/Win7 endpoints. 2008R2 support returns via a separate legacy-only rewrite (C++/.NET or a clean minimal Rust core), not by re-enabling the 1.77 toolchain — captured as **RMM_THOUGHTS Feature 12** (Raw, Mike's requirement). **Policy-Controlled Tray Deployment shipped 2026-07-04 (code merged, policy default OFF):** The tray binary is now deployed to every Windows endpoint via the agent-update flow (previously only fresh MSI installs got it — fleet census found ~111/113 sampled hosts with no tray at all) but its **launch is gated on `tray.enabled` policy** — no process, no icon, until a client/site/agent policy turns it on. Authenticode-verified + version-locked to the agent. Fleet-wide policy enablement (Task 4) is **not yet flipped**. **VSS Policy Configurator shipped 2026-07-03 (SPEC-016 redesign, agent v0.6.76):** The agent no longer runs its own scheduled shadow-copy create+prune loop — the MITRE T1490 "Inhibit System Recovery" surface that got the agent blocked by CrowdStrike Falcon. It now **configures** two kernel-enforced retention governors (VSS diff-area size cap + `MaxShadowCopies` count evaluated-but-not-machine-globally-written) and registers a scheduled task whose action is a native-COM **create only** (never a delete). A 23-agent code review caught a data-loss-class defect pre-merge (writing the machine-global `MaxShadowCopies` would have evicted OTHER products' shadows) — dropped in favor of the size cap as sole governor. Deployed to the **beta** channel only (0.6.76 at the time of that compile); stable-channel promotion of the VSS behavior riding on 0.6.78 has not been separately re-verified this compile. **Vendor-agnostic Integrations Framework shipped 2026-07-01..02 (migration 065, CrowdStrike Falcon first plugin):** A generic per-partner plugin framework so a new EDR/security vendor is a new server module, not a new schema — modeled on the proven MSPBackups per-partner shape. CrowdStrike Falcon is the first plugin. REST CRUD at `/api/integrations`; no dedicated dashboard UI yet. **SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA, not guaranteed to work; Route B = server-orchestrated):** unchanged status this compile — best-effort, not reliable for the hard cases. See Capabilities. **Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+):** unchanged this compile. **Physical server migration completed 2026-06-11:** New physical box at 172.16.3.30 running Ubuntu 26.04 + PostgreSQL 18. Old VM at .46 decommissioned 2026-06-12. **See also:** `wiki/projects/guru-rmm.md` is a redirect tombstone pointing here (slug disambiguation). **Repo:** `azcomputerguru/gururmm` on Gitea (internal: http://172.16.3.20:3000, external https://git.azcomputerguru.com/azcomputerguru/gururmm.git). The copy at `projects/msp-tools/guru-rmm` is a git submodule tracking the active repo; its pinned gitlink intentionally lags origin/main. Development uses `git -C projects/msp-tools/guru-rmm` or worktrees off `origin/main` to access live code. Howard is authorized for merges/deploys (cleared 2026-06-21 by Mike). **Goal:** Full-featured MSP platform rivaling commercial RMMs, with a companion PSA (GuruPSA, separate future repo) designed as a truly integrated unified system. --- ## Recent Work ### 2026-07-06 — Fleet-Policy Management Console + agent folder_id in API (policy-ui, merge f0320a4, migration 068, PROD deploy) The follow-on **policy-ui** work — scoped only as a thin component-assembly MVP the day before — was built out and merged to prod as a full **Fleet-Policy Management Console**. `PolicyFolders.tsx` was rebuilt (999-line rewrite; +2614/-526 across 12 dashboard files) into a console with a folder **tree table** (`FolderTreeTable`), folder CRUD/rename/move dialogs (`PolicyFolderDialogs`), a **place-devices** flow (`PlaceDevicesDialog` + `FolderPicker`), an **effective-policy viewer** with source attribution (`EffectivePolicyView`), a per-edit **version-history** diff view (`VersionHistoryDialog`), and an admin **backfill** trigger (`BackfillDialog`); `AgentDetail.tsx` was slimmed as folder assignment moved into the console. Server-side, `d028ca1` exposed `agents.folder_id` in the agent API and migration **068** (`068_visible_agents_folder_id.sql`) recreated the `visible_agents` view to add the column — the BUG-018 column-freeze convention in practice (any `ALTER TABLE agents ADD COLUMN` must recreate the `SELECT *` view or listings 500 fleet-wide). Direction was locked as "tri-model, full scope" (`0bd5f67`) before the build. Server advanced to v0.3.103, agent to 0.6.79. --- ### 2026-07-06 — WS-Looper Comms-Durability: Write-Timeout + Heartbeat Diagnostics + AIMD Keepalive (Howard-Home, merge 07307e3, agent 0.6.79 beta / server 0.3.102) Follow-up on the chronic WS loopers (BUG-023/T2 field data): the four boxes cycle auth -> total agent->server silence -> 90s evict -> reconnect, and updating them did not clear it — a per-site network blackhole no cadence change can cure. The core keepalive-inversion (30s agent heartbeat), server half-open eviction (T2), and CommandAck durable delivery (T3) were all already shipped, so this session delivered the code-fixable + observability layers. **(1) Bounded WS writes** (`send_bounded`, `WRITE_STALL_TIMEOUT=90s`): the outbound + Pong sends were unbounded `write.send().await` sharing the `select!` arm with the reconnect timer, so a backpressured half-open socket (send buffer fills, no error for ~15 min) froze the whole loop **including reconnect** — a latent hang, now time-bounded -> force-reconnect. **(2) Heartbeat-count diagnostic:** the agent counts heartbeats written per session and reports the previous session's count in the auth payload; the server logs it — **nonzero + a prior "no inbound" eviction proves an agent->server network blackhole; 0 proves a local send stall** — making the blackhole-vs-stall verdict server-visible with no on-site packet capture. **(3) Adaptive keepalive / AIMD** (new `transport::keepalive` module): the heartbeat interval self-tunes per site across reconnects (halve on a short authenticated session, +5s when stable), clamped **5-45s** (max kept under the server's 90s evict window), persisted to disk, and gated on an authenticated-session flag so a server outage cannot drag the fleet to the floor. A 15-agent high-effort code review drove 5 fixes (AIMD firing on unauthenticated sessions -> fleet-to-floor collapse; max>evict window; pre-auth counter reset losing the diagnostic on the very failure it targets; write-timeout dropping large slow-link frames; duplicated bounded-write). Deployed agent 0.6.79 to **beta** (soaking); server 0.3.102 live+clean; the diagnostic fired on first reconnect; DESKTOP-3USU20B beta-pinned for an early read. **BUG-024 recurrence caught during deploy:** the build re-published a fresh glibc-2.43 Linux 0.6.79 straight into downloads — re-quarantined to hold the "scanner offers nothing for Linux" invariant; this recurs on every merge until the glibc floor lands. --- ### 2026-07-05 — Policy Folders + Inheritance ("policy-core") Deployed (Mike, merge e7e90c2, migration 067) Replaced the fixed System->Client->Site->Agent policy stack with a per-client **folder tree**: each client gets exactly one root folder; any folder may optionally reference a library policy; a device sits in exactly one folder and inherits the merged policies of its folder ancestry (root->leaf) plus an optional per-asset override. Nine tasks: folder-tree schema + inheritance resolution + versioning (migration 067, renumbered from 066 after a same-window collision with `066_visible_agents_view.sql`); recompute+push on policy edit scoped to affected agents only (not a fleet-wide recompute); folder tree REST API (`/clients/:id/policy-folders`, `/policy-folders/:id[/rename|/policy|/parent]`, `/agents/:id/folder`); migration backfill job with a verification gate + dry-run mode (`/policy-folders/backfill`); integration tests. `policy_versions` gives every policy edit an immutable snapshot for diff/rollback and per-device "what changed." Landing migration 067 changes **zero behavior** by construction: every new column/table is nullable and `agents.folder_id` is NULL fleet-wide until the backfill runs, so effective-policy resolution stays on the legacy Client->Site->Agent path until a deliberate, gated flip. Explicitly out of scope for this phase: module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI, and any agent-side change (wire format unchanged — server+dashboard only). A follow-on **policy-ui** shape spec (folder management dashboard UI: tree view, agent placement, effective-policy viewer with source attribution, version history, admin backfill trigger) was added the same day (`01d962a`) as an MVP largely assembling existing dashboard components (ContextTree, PolicySectionEditor, EffectivePolicyTable) — not yet implemented. --- ### 2026-07-05 — Bug-Fix Batch 3: BUG-023 Channel Race, BUG-003 Closed, BUG-018 Altitude Fix (Howard-Home, merge 6ccd35e, server v0.3.98) Post-deploy verification of the prior night's repair batch 2 came back clean, but the new T2 half-open-eviction sweep surfaced an anomaly: 127 evictions, 79 of them one agent (DESKTOP-3USU20B, Safesite Glendale) cycling connect->auth->silent-90s->evict->reconnect every ~6 minutes, plus 3 other chronic loopers. Root-causing why the loopers never got 0.6.78 uncovered **BUG-023**: a publish/tag race in the build pipeline. Builds are tagged `beta` by a late "MARK AS BETA" phase minutes *after* binaries land in the downloads directory, and the scanner defaulted a missing `.channel` sidecar to `stable` — so the 07:38 scan classified the untagged 0.6.78 as stable and mass-offered it to 209 stable-channel agents before the 07:43 scan reclassified it beta, splitting the fleet (214 agents on unsoaked-beta 0.6.78, ~31 stragglers pinned with no path forward). Fixed in three layers: the scanner now **fails closed** (missing/malformed `.channel` => beta, with a regression test) so the race is harmless even if a future pipeline step crashes between publish and tag; `build-windows.sh`'s `deploy_and_sign` writes the beta `.channel` sidecar atomically with each published artifact, closing the timing window at the source; and `promote_version` was rewritten to key on agent **binaries** rather than pre-existing sidecar files (the old logic only rewrote files whose content was exactly `beta`, making fail-closed sidecar-less binaries permanently unpromotable) — it now creates/overwrites `stable` sidecars and a partial write fails the request (500) instead of recording a `stable` rollout row that disagrees with disk. With Howard's go, 0.6.78 (already soaked 8+ hours on 214 agents) was manually promoted to stable, healing the split fleet (229 fresh agents converged by session end). Also shipped in the same batch: the **BUG-018 altitude fix** — migration `066_visible_agents_view.sql` (a `visible_agents` DB view filtering `status<>'deleting'`, replacing nine scattered inline filters including a previously-missed agent-detail-page query) plus a `BEFORE UPDATE` trigger pinning `status='deleting'` at the schema level, and all three WS `authenticate_agent` paths now reject a mid-purge row instead of binding a live session to an agent about to be hard-deleted. A 15-agent code review (22 verified, 9 findings, 1 refuted) drove a hardening commit: `set_ignore_missing(true)` on the migrator (so a rollback binary tolerates an ahead DB instead of `VersionMissing`-crash-looping — the review's top finding), and a pre-existing `matches_pattern` panic on no-wildcard patterns was fixed (previously hidden among "19 known" test failures — really 18 DB-env + 1 real bug). **BUG-003 closed:** `deploy/build-pipeline/build-server.sh` (confirmed this session to auto-sync to `/opt/gururmm` on every push) gained a `flock` build lock, `PIPESTATUS`-gated `git fetch`/`reset`, and pre-stop migration validation via a new `--migrate-only` server flag (validates+applies migrations against the live DB while the OLD binary keeps serving, then exits without binding a port). The gate fired correctly on its **first real run**: the merge webhook collided with the CI version-bump job's `index.lock` and aborted cleanly with nothing deployed — where the old script would have silently built a stale tree. Deployed as server v0.3.98 (16:46 UTC), 0 ERROR lines, view/trigger verified live in a rolled-back test transaction. DESKTOP-3USU20B's one-way-WS loop was fingerprinted as per-site network behavior (agent->server frames eaten post-auth; server->agent and REST both fine; reproduces on 0.6.66, 0.6.77, and 0.6.78 alike) — 4 chronic loopers total, telemetry exhausted server-side, packet capture is next. RMM live temperature reading was also spot-verified against ACG-Tech03L's (Snapdragon WoA) local sensor panel: zone-for-zone match, values within tenths of a degree. --- ### 2026-07-05 — Bug-Fix Batch 4: Command TTL, Status Reconciler, Update Supersede; BUG-024 Linux glibc Incident (Howard-Home, merge c34194a, server v0.3.99) A live-system "what's not working as expected" sweep found and shipped three fixes, twice review-hardened (16 findings addressed across two workflow passes): 1. **Pending-command 7-day TTL:** commands could sit `pending` indefinitely and execute on reconnect regardless of age (May-dated commands would have fired on a stale agent's reconnect). The reaper now expires anything past 7 days with an honest distinct message for never-sent vs. unconfirmed-delivery rows; the first tick expired 10 stale rows. 2. **Agent-status reconciler:** `agents.status` was never reconciled against `last_seen`, leaving zombie `'online'` rows (11 found, one stranded since June 2). A new sweep flips status-only (never touching `last_seen`, after a review finding caught an earlier version fabricating freshness and auto-resolving offline alerts), respects the shared 2-hour update-hold window, skips agents with a live dispatch-map connection, and broadcasts the change via event + SSE. First post-warm-up sweep flipped 4 stranded rows. 3. **Stale pending-update supersede:** re-dispatching a pending update at auth time now supersedes it when a newer version exists or the target binary was pruned from disk (all 13 previously-pruned targets were being blindly re-offered); guarded to only fire on `pending` rows over an hour old with a non-empty platform index. **BUG-024 (NEW, OPEN, mitigated) — Linux glibc incident:** Linux agent binaries >=0.6.77 were built against the build host's glibc 2.43 (the box is now Ubuntu 26.04), and the 2026-07-04 scanner fix enabled Linux update offers for the **first time ever** — so every older-distro Linux agent that took the offer crash-looped on `GLIBC_2.39` not found: ix.azcomputerguru.com (CloudLinux 9, 3898 restarts), the Jupiter agent container (Debian 12, 623 restarts), a Lonestar container, and presumed SL-SERVER (Scileppi). The Unix rollback watchdog never fired on either affected box (a distinct sub-bug, uninvestigated), and the 0.6.59 container agent still performs the BUG-019 ephemeral in-place update, so the bad binary persisted in the writable layer across restarts. **Mitigated:** Linux binaries quarantined out of the downloads directory (scanner now offers nothing for Linux); ix and the Jupiter container were manually revived from their pre-update backups and reconnected healthy. SL-SERVER and the Lonestar container host remain down pending manual recovery (no vault SSH creds for either). **Real fix pending:** a glibc compatibility floor for Linux builds (old-glibc build container, e.g. rockylinux:8/debian:11, or a switch to musl static linking) plus an `objdump -T` max-GLIBC build gate — tracked as roadmap BUG-024. Also this pass: 4 orphaned duplicate agent rows deleted (zombies with fresher same-site siblings); `.30`'s `gururmm-agent.service` found `disabled` and re-enabled; RECEPTIONIST-PC's two rows at Cascades confirmed to be two real machines sharing a hostname (rename recommended, not a bug). Operational flags relayed to Howard/Mike: LAB-SVR offline 17.7 days with an active critical alert; KSTEENBB2025 peaking 83.8C cpu_temp; a stale repo-root `build-server.sh` lineage is a deletion candidate (the canonical script lives under `deploy/build-pipeline/`). --- ### 2026-07-04..05 — Repair Batch 2: Comms-Durability Phase 1 Complete, BUG-001 WMI Thermal, Uninstall Teardown (Howard-Home, merge 76a6bd2, server v0.3.97, agent 0.6.78) **Comms-durability Phase 1 completed:** a gap analysis found T1 (30s agent heartbeat) and T3 (ACK'd durable delivery, migrations 058/059) already live; this batch implemented the missing T2 — a half-open-connection eviction sweep (`evict_half_open_connections`, 90s no-inbound threshold, 30s sweeper cadence) — closing out the phase. `SO_KEEPALIVE` was deliberately skipped (the TCP peer is the NPM proxy, not the agent). **BUG-001 (Windows WMI ACPI thermal) SHIPPED and PROVEN:** `collect_temps_windows_acpi()` reads `MSAcpi_ThermalZoneTemperature` via WMI (`root\WMI`), converts tenths-Kelvin, applies a -20..125C plausibility filter, and reports the hottest zone as `cpu_temp` with a `sysinfo` fallback. Verified live on 112+ Windows agents reporting `cpu_temp` within 10 minutes of deploy (was 0 fleet-wide since LibreHardwareMonitor's CVE-2020-14979 removal on 2026-05-27); sensor-for-sensor matched a local panel on a Snapdragon WoA machine (ACG-Tech03L) including correctly filtering frozen 0-Kelvin zones. A review finding moved the WMI query to `spawn_blocking` with a 5-second deadline so a wedged `winmgmt` service cannot stall heartbeats. **BUG-020 uninstall teardown fixed:** the real gap (policy-disable/swap `terminate_all` wiring was already correct; the roadmap note calling it broken was stale) was the uninstall path leaving the watchdog service and `gururmm-tray.exe` behind. Fixed ordering: `remove_watchdog_service()` first, then `TrayLauncher::terminate_all().await`, then service delete, plus removal of the tray binary from disk. **BUG-018 delete UX:** `delete_agent` now marks `status='deleting'` synchronously before the 202 response; all listing/stat queries exclude 'deleting' rows; the dashboard does an optimistic cache removal. Three resurrection paths that could flip a mid-purge row back to `'online'`/`'offline'` (heartbeat, metrics, and offline-grace writes) were closed by making `update_agent_status` refuse to overwrite `'deleting'`. A zombie-connection check in the WS read loop now closes any connection no longer matching the agent's currently-registered one (fixes a permanent-orphan class: agent shown online in the DB, absent from the dispatch map, commands pending forever). Startup reconciliation re-runs the purge for any row stuck `'deleting'` after a mid-purge restart. (Full altitude fix via the `visible_agents` view followed the next day — see Bug-Fix Batch 3.) **MSP360 sync noise:** transient 5xx from MSP360 now logs WARN, escalating to ERROR only after 4 consecutive failed cycles (~1h), so real outages still alert without every cloud-maintenance blip minting a `server_error`. A 26-candidate, 10-finding code review preceded merge; all 8 findings attributed to this batch were fixed pre-merge (`238aad7`). All 4 build targets went green: server v0.3.97, Windows agent 0.6.78 (wmi crate compiled clean), Linux 0.6.78, dashboard v0.2.83 beta. --- ### 2026-07-04..05 — Easy-Bugs Batch + Alert Triage: Update-Scanner, Inventory NUL, WS Noise (Howard-Home, merges 519eef2/f2d02dd, server v0.3.95/v0.3.96) Full 24h alert triage (`GET /api/alerts`) surfaced 5 `backup_failed` (client-side, unresolved — see Active State), 2 active `server_error` (product bugs), and 1 resolved `mass_offline_fleet` (36 agents, transient network blip). Root-caused both server errors against the server's own self-log (pseudo-agent `00000000-...-001`): - **`move_agent` 500 -> 409:** moving an agent into a site that already holds a row for the same `device_id` tripped `idx_agents_site_device` as a raw 500; now maps to 409 with an actionable message. 13 duplicate-hostname agent pairs fleet-wide are the trigger (stale legacy `win-*`/`hw-*` device_ids alongside re-enrolled hardware-UUID records) — not yet cleaned up. - **Offline-sweep `RowNotFound`:** `create_or_update_alert`'s UPDATE-by-id used `fetch_one`; an alert row deleted mid-sweep (agent-delete cascade) aborted the whole sweep. Switched both branches to `fetch_optional` with fall-through to INSERT (`Result>`, `Ok(None)` = agent gone). - **Inventory NUL rejection (alert-invisible, found in logs only):** 7+ agents (IMC1, Seth-PC, QWM-JOHN, QWM-SHEILA, goldstar19, SIF-SERVER, Christine-Win10) never persisted hardware inventory — Windows registry strings carry embedded NULs, Postgres jsonb/text rejects them. New `server/src/db/sanitize.rs` (`strip_nul`, `clean_control_chars`) applied over 5 jsonb blobs (values + keys, collision-safe) + text columns. - **Update-scanner warn-spam / silent update starvation:** the self-check tried to execute every platform's binary to read its embedded version — macOS/Linux binaries can't execute on the Linux build host, so they were dropped from the scannable set entirely, meaning **macOS/Linux agents were never offered updates**. Fixed with host-native-vs-foreign regimes: foreign-platform binaries trust the filename version; host-native binaries stay a hard gate (a native binary that can't report its version is corrupt) with a best-effort exec-bit repair for 0644 CI-copy artifacts. A concurrent same-branch session absorbed a prior session's in-progress inventory/scanner fixes; the combined diff went through a 19-agent high-effort code review (9 confirmed findings, 0 refuted) before merge — findings included rebuilding the alerts fall-through (it still failed its own motivating scenario on first pass), tightening the scanner's foreign/host-native split, and moving WS-noise logging from DEBUG to INFO (a mass-disconnect anomaly must stay visible at the default log level). Merged + deployed as **server v0.3.95** (06:09 UTC): scanner now indexes 8 platform/arch combos (was 4, Windows-only); zero ERROR lines post-deploy. Continued into the last stale alert: "Authentication timeout" WS churn (2-6/day baseline, 57-spike correlated with the same day's fleet-dedup work). Profiled 7 days of journal, concluded pre-auth connection noise (peer connects, never authenticates — sleeping laptops, scanners, half-open NAT), added `is_preauth_connection_churn` to classify it INFO while keeping genuine rejections (bad API key, malformed auth) at ERROR. Deployed as **server v0.3.96** (06:23 UTC). Active `server_error` alert count: **0**. BUG-009 (`Logs.tsx` isError handling) and BUG-011 (`: any` occurrences) were verified already fixed in-tree during the same pass — only the stale `FEATURE_ROADMAP.md` status lines needed correcting, not the code. Remaining/pending from this session, superseded by later batches where noted: machine-side backup failures (GND-SERVER, AD1, LAB-SVR, DESKTOP-EVA4H1A) stayed open; 13-pair duplicate-agent cleanup (4 orphaned rows cleaned in batch 4, full pass still open); WS auth-timeout narrowed to pre-auth churn here, but 4 specific chronic loopers were only fingerprinted, not fixed, in batch 3. --- ### 2026-07-04 — Policy-Controlled Tray Deployment (Howard-Home / Mike, merged f6a4251) Made `gururmm-tray.exe` deployment (materialize, policy-agnostic) independent from its runtime (launch, policy-gated) — previously the watchdog launched the tray whenever the binary existed, with no enforcement of the already-wired `tray.enabled` policy flag, and the binary itself only ever reached fresh MSI installs (fleet census: 1 signed tray, 1 stale unsigned, ~111 none out of 113 sampled). A first design (inline tray refresh folded into `do_update`) was **rejected** by a 17-agent code review plus independent Gemini 3.1 Pro and Grok adversarial critiques — 9 verified defects including permanent-binary-loss on a failed swap, a checksum sourced from an unauthenticated download-host sidecar (supply-chain gap), and an update/rollback tray-agent version-skew case. Corrected design shipped as Tasks 1-3 + 5: policy-gated launch (default OFF), a materialize FSM with staging + a versioned mutex (atomic swap, restore-on-failure, authenticated-payload checksum), server capability-gated materialize, and Authenticode-verify + observability. Task 4 (flip fleet-wide materialize on, policy still OFF) is the deliberate next step — not done yet. Merged to main 2026-07-04 12:23 PDT. --- ### 2026-07-03..04 — VSS Policy Configurator (SPEC-016 Redesign) (Mike, merged de30b2b) Rebuilt the agent's Volume Shadow Copy handling from "scheduled create+prune operator" (the MITRE T1490 surface Falcon blocked) to "policy configurator, never a deleter." Tasks 2-7: `vss-create` verb + scheduled-task registration (both daily-time and every-N-hours trigger modes) with legacy-task migration; per-volume diff-area size-cap governor set on policy apply; retirement of the scheduled create+prune pass entirely (SPEC-025 compliance heal reworked to create-only); status/compliance reporting updated to the configurator model; dual-target release build; runtime verification on both a Server 2019 box and a Win11 Pro client with Falcon present. A 23-agent workflow code review before merge caught a genuine data-loss defect: the original design would have set the machine-global `MaxShadowCopies` registry value to `retention_count`, which FIFO-evicts **other** VSS consumers' (System Restore, third-party backup) shadows below that count fleet-wide. Fixed by dropping `MaxShadowCopies` management entirely — the per-volume size cap is the sole governor; `retention_count`/`retention_max_age_days` become advisory (a warning fires if `retention_max_age_days` is still set, since age-pruning requires deletion). Nine other findings fixed (cap-before-create ordering, an alias bridging the legacy task-name upgrade window, interval-aware compliance windows, bounded `IVssAsync` wait instead of INFINITE, and others). Merged to main; the build pipeline publishes to the **beta** channel, not stable (a mid-session correction) — beta head reached 0.6.76 (8 agents), stable stayed at 0.6.66 (233 agents) by design. Verified live on the T1490 canary NEPTUNE (moved to beta channel for the test): create task present, legacy `GuruRMM-VSS-Snapshot` task removed, `MaxShadowCopies` correctly left unset, 17 pre-existing shadows preserved (the data-loss fix proven in production), and a live `vss-create` produced new shadows on two volumes under Falcon with no T1490 trip. --- ### 2026-07-01..02 — Vendor-Agnostic Integrations Framework + CrowdStrike Falcon Plugin (Mike) Built a generic third-party-integration plugin framework (migration 065: `integrations`, `integration_credentials`, `integration_client_mappings`, `integration_sync_state`, `integration_agent_links`) so CrowdStrike Falcon — and future vendors (SentinelOne, Guardz) — are a new server module rather than a new bespoke schema each time, generalizing the per-partner MSPBackups shape (migration 034). Credentials reuse the same AES-256-GCM `Encryptor` chokepoint as the existing `credentials` table. Five tasks, one per day-slice: (1) framework schema migration; (2) plugin trait + registry + DB layer; (3) generic REST API (`/api/integrations` CRUD, credentials, client mappings, sync-state, connection test); (4) CrowdStrike Falcon API client + plugin registered against the framework; (5) periodic sync worker pulling Falcon sensor state (version, status, RFM flag, CID) into `integration_agent_links`. No dedicated dashboard UI yet — API/backend only as of this compile. --- ### 2026-06-23..24 — SPEC-030 Remote Software Uninstall + Universal Installer (Howard-Home) **SPEC-030 software inventory + remote bulk uninstall (Route B — BETA, code merged to main + deployed but NOT guaranteed to work; agent 0.6.75 / server v0.3.87):** - **Capability shipped:** `/agents/:id/software` lists installed programs; `/agents/:id/software/uninstall` runs a bulk uninstall; the dashboard `SoftwareManager.tsx` shows the live programs list + bulk-select + a live progress bar. Server endpoints in `server/src/api/software.rs`. - **Silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`): Tier-1 silent removal, BCU-informed detection + fail-fast, vendor silent-switch table, captures **real** process exit codes (fixed a `Start-Process` null-ExitCode bug, `f6c1945`). The engine **refuses to guess** — Avast/AVG (`icarus.exe`), Malwarebytes (`mb5uns.exe`) correctly returned `needs_interactive` rather than risk a bad removal. - **False-success class fixed (CRITICAL, PR #53):** Avira's WiX/Burn **Package Cache bundle GUID** was misread as an MSI ProductCode; `msiexec /x {bundleGuid}` returned 1605 ("not installed") and was mapped to success="already gone" while Avira stayed fully installed. Fix: skip the MSI tier for `\Package Cache\` bundles + a **generic post-success ARP re-check** (`Test-StillInstalled`) that downgrades a reported success to failed if the program is still registered. - **Code-review correctness fixes (PR #54):** `Test-StillInstalled` matches by `product_code` alone when present; skip the rescan on reboot-pending verdicts (3010/1641); exclude MSI-1605 no-op successes from the learned-timing sample. - **Async removal jobs — the permanent fix (PR #56, migration 064):** A synchronous bulk uninstall died at Cloudflare's ~100s timeout AND the agent's command kill-timeout killed the engine mid-flush. Rebuilt as a **JOB**: POST creates a `software_removal_jobs` row and returns a `job_id` instantly (0.06s); a background worker dispatches one program at a time; `GET .../uninstall/jobs/:job_id` is polled by the dashboard. Stop-gap flush-slack (PR #55) shipped first. - **Three-state removal knowledge base (migrations 061-063):** `software_removal_attempts` (per-agent removal-tracking loop), `software_knowledge` (fleet KB: `silent` / `requires_ui` / `unknown`), `software_knowledge_timing` (self-tuning per-target timeout). - **Known gaps surfaced (not yet built):** drivers/system components register in ARP and appear in the bulk list; Launchy/AIMP-class uninstallers hang the agent on a lingering child process (~420s kill-timeout); no list-jobs endpoint (get-by-id only). **Universal self-detecting installer (Feature 9 — P1 built+deployed+verified, v0.6.71):** - One install path self-detects arch/OS (x64/x86/legacy/ARM) and pulls the correct agent; dashboard install UI repointed at it. Script path proven on-box (GND-JWILL). P2 (native i386 bootstrapper EXE) + offline-bundle still pending. **RMM_THOUGHTS movement:** Feature 10 (AMPIPIT recovery environment add-on) -> Discussed (Mike likes it, deferred); Third-Party AV Removal Recipes + Win11 KB5095093 Point-in-Time Restore added as Raw; **Feature 11** (OS Updates Management, Windows Update P1/P2 + Linux/Mac update-check P3, Syncro-modeled) added Raw 2026-07-03; **Feature 12** (legacy 2008R2/Win7 agent rewrite) added Raw 2026-07-04 after the legacy build was disabled. --- ### 2026-06-22 — BUG-021 Fixed + v0.6.67 Stable; Fleet Verified (Howard-Home) - **BUG-021:** Legacy build (Rust 1.77 via `$CARGO +1.77 --features legacy`) was pulling `wit-bindgen` (edition2024) through fresh dep resolution. Fixed by pinning `getrandom 0.3.1` + `zeroize 1.8.1` below edition2024. Commit `1dce66d`. Windows build went green 2026-06-22 02:19, v0.6.67. (This class of failure recurred 2026-07-04 with a different dep pair and led to the legacy variant being disabled outright — see Recent Work above.) - **Fleet verified:** ~270 agents / ~178 online; metrics rows flowing; all alerts have non-null dedup_keys; per-agent endpoints all HTTP 200. - **BUG-022 filed + fixed (PR #45):** `WatchdogEvent` WS variant was dead code (watchdog has no WS connection); removed. Empty `watchdog_events` table left in place. - **Roadmap verified:** BUG-018 (`cea87d4`) and Event Log Watch UI (`0fa65f5`) confirmed on main. --- ### 2026-06-21 — Feature Batch, Audit, BUG-018 202+bg, Event Log Watch UI, Submodule Fix (Howard-Home + Mike GURU-5070) **Howard-Home sessions (three spans):** - **BUG-019 FIXED (commit `66a7f4e`, merged to main by Mike):** Container agent no longer performs in-place binary self-update. `agent/src/updater/mod.rs` `perform_update()` early-returns inside a container. - **BUG-018 FIXED (commit `cea87d4`, merged):** Handler redesign to 202 + background purge. Migration 061 (FK indexes on five previously-unindexed FK columns) landed later under a renumbered slot once SPEC-030 consumed 061-064. - **Event Log Watch management UI (SHIPPED, `0fa65f5`):** `/event-log-watches` CRUD wired into the dashboard. - **Enrollment modal UX fix (SHIPPED to prod, `4027c86`):** close button, click-outside/Esc dismissal, list refresh on close. - **Event Log Watch policy-clobber HIGH fix (PR #43, `432b434`):** Assigning/unassigning a policy silently wiped an agent's Event Log Watch monitoring — third instance of a config-push clobber class. Fixed by reusing `watch_rules_for_agent` as the single source of truth. - **sync.sh submodule-clobber root-cause fix:** populate-only guard prevents `git submodule update --init` from re-checking out the intentionally-lagging pinned gitlink on every sync. - **Dashboard beta-before-main rule established.** **Mike GURU-5070 session:** - **BUG-019 merged to main** (fast-forward `ed8cad3 -> 66a7f4e`). - **`gururmm-build` skill created:** local pre-merge verification (`verify.sh server|agent|dashboard|migrations`). `docs/BUILD.md` added. - **Howard cleared for GuruRMM merges/deploys** (Mike decision). --- ### 2026-06-15 — BSOD Dedup Key Fix, MSI EXDEV Fix, Dashboard Offline Dedup, sync.sh Auto-Heal (Mike GURU-5070 + GURU-BEAST-ROG) - **BSOD alert dedup key fixed (commit `f0a4b7f`, server v0.3.73):** Dedup key changed to `bsod::` (stable across recurrences). - **MSI EXDEV fix (commit `95ef901`, server v0.3.74):** Site-MSI builder staged in `/tmp` (tmpfs) then renamed cross-device to `/opt/gururmm/downloads`, failing EXDEV. Fixed to stage on the same filesystem. - **Duplicate offline server alerts removed (commit `b6ed564`).** - **sync.sh auto-heal (Phase 1):** `resolve_submodule_collisions()` moves only genuinely-colliding untracked files aside and retries, instead of a blind force. - **`logs/analyze` switched from Ollama to Claude API (commit `c869e4d`).** - **500-error-leak fix (commit `58c1a96`):** All server error paths route through `internal_err`. --- ### 2026-06-12 — Beast Parallel Windows Build (Lever A) + command_type "cmd" Misdiagnosis Fix (Mike GURU-5070) **Beast parallel Windows build (commit `cfbdb59`):** Two waves (5 concurrent stable + 2 concurrent legacy-1.77) via bash-driven concurrent SSH instead of a serial `cmd /c` chain. Per-variant `--target-dir` mandatory. Result: ~336s parallel vs ~622s serial cold, ~3.8x vs Pluto serial. **command_type "cmd" fix (commit `3de9faf`, agent v0.6.66 stable):** `CommandType` enum had no `cmd` variant — server-sent `command_type:"cmd"` failed serde parse and the agent silently dropped the message. Fixed with `#[serde(alias = "cmd")]` on `Shell` + a NAK on unparseable commands. --- ### 2026-06-11 — Physical Server Migration, Durable Agent Identity, Agent Comms Durability Phase 1 - **Cutover:** New physical box took over 172.16.3.30 (PG 18, Ubuntu 26.04); production down <27 min; old VM decommissioned 2026-06-12. - **Durable agent identity Phase 1 Task 1:** Multi-location `device_id` (registry+ProgramData on Windows, /etc+/var/lib on Linux, /Library+mirror on macOS) with self-heal; `cleanup.ps1` whitelists identity files. - **Agent Comms Durability Phase 1 (agent v0.6.63 / server v0.3.68):** Root cause = NAT conntrack asymmetry (server->agent pushes black-holed in idle conntrack gaps). Fix: agent `CommandAck` on receipt + dedup cache; server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via a partial index so old agents keep legacy behavior. (Phase 1 fully completed 2026-07-05, see Repair Batch 2 above — the T2 half-open eviction sweep was the missing piece.) --- ### 2026-06-07 — Credential Inheritance, Offboarding Spec, UI Gap Batch, Backup Alert Quality Pass - Server v0.3.45: credential inheritance (Global -> Client -> Site), `/effective` endpoints. - Role-aware offline alerting + alert ignore/mute shipped (server-only `agent_offline`, `alert_mutes` perma-silence). - Backup-alert quality pass: false `backup_failed` 15 -> 2 fleet-wide; `backup_storage_low` alert type removed (false-positive metric). - UI gap batch + enrollment audit; SPEC-028 offboarding wizard specification (835 lines, implementation pending). --- ## Capabilities / Feature Set *Synthesized from authoritative artifacts (API routes, agent modules, 67 migrations, roadmap, commit log) — not from session logs alone. See Compilation Notes.* Agent<->server communication is a persistent authenticated WebSocket with auto-reconnect + heartbeat; on reconnect, in-flight commands flip to `interrupted`. Platform-parity rule: agent features ship on Windows/Linux/macOS in the same change (stub + TODO where a real impl isn't yet feasible). ### Monitoring & Telemetry - Core metrics per interval (policy-tunable): CPU %, memory %/bytes, disk %/bytes, network rx/tx deltas, uptime, logged-in user, user idle time, public/WAN IP. Cross-platform via `sysinfo`. - **Windows CPU/thermal (BUG-001, shipped + PROVEN 2026-07-05, server v0.3.97):** `collect_temps_windows_acpi()` reads `MSAcpi_ThermalZoneTemperature` via WMI (`root\WMI`, `spawn_blocking` + 5s deadline so a wedged `winmgmt` can't stall heartbeats), tenths-Kelvin conversion, -20..125C plausibility filter, hottest zone reported as `cpu_temp`, `sysinfo::Components` fallback. Verified fleet-wide: 112+ agents reporting within 10 minutes of deploy (was 0 since LibreHardwareMonitor's CVE-2020-14979 removal 2026-05-27); sensor-for-sensor match against a local panel on Snapdragon WoA hardware (ACG-Tech03L, 20 valid zones, frozen 0-K zones correctly filtered). GPU/NVAPI thermal still open. - Process drill-down: top 10 by CPU + top 10 by memory in every metrics payload. - Network state: per-interface IPv4/IPv6, MAC, derived CIDR subnets — sent on change only. - **Windows BSOD/kernel-crash detection:** `agent/src/bsod.rs` polls `C:\Windows\Minidump`, parses kernel dump header at fixed offsets, cross-references System event log, deduplicates via a watermark file. Always-Critical alert deduplicated by `(agent_id, bugcheck_code)` (changed from dump-hash 2026-06-15 so a mute silences all recurrences). - **Inventory NUL/control-char hardening (2026-07-05, server v0.3.95):** `server/src/db/sanitize.rs` strips NULs and C0/C1 control chars from inventory jsonb values/keys and text columns before upsert — fixed silent storage failure on 7+ machines carrying embedded NULs in Windows registry strings. ### Remote Execution - Command types: `shell`, `powershell`, `python`, raw `script{interpreter}`, `claude_task`, and `cmd` (alias for `shell`). Options: `timeout_seconds`, `elevated`. Unknown/unparseable commands NAK immediately. - **Execution context** (`041`): `system` (default) or `user_session` (WTS token impersonation, Windows-only). - Commands individually cancellable, aborted on disconnect. Auditable history. Script library (`017`) for stored scripts. - **Agent Comms Durability — Phase 1 COMPLETE as of 2026-07-05 (agent v0.6.63.., server v0.3.68->v0.3.97):** T1 (30s agent heartbeat) and T3 (`CommandAck` on receipt, migration 058 `acked_at`, dedup via a recent-results cache, reaper re-delivers un-acked commands past a 60s deadline) shipped 2026-06-11; T2 (server-side half-open-connection eviction sweep, 90s no-inbound threshold, 30s cadence, `conn_id`-guarded) shipped 2026-07-05 closing out the phase. Capability-gated via `idx_commands_agent_acked` partial index so pre-durability agents keep legacy fail-on-timeout behavior. Phase 2 (live TTY) and Phase 3 (adaptive keepalive, bulk->HTTPS) remain PLANNED, not shipped. - **Pending-command TTL + status hygiene (batch 4, server v0.3.99):** commands stranded `pending` past 7 days are expired by the reaper (previously unbounded — a May-dated command could still fire on a stale agent's reconnect) with distinct never-sent vs. unconfirmed-delivery messaging. `agents.status` is now reconciled against `last_seen` by a periodic sweep (status-only conditional flip, preserves real `last_seen`, respects the 2h update-hold window, skips live dispatch-map connections, full event+SSE broadcast) — fixes zombie `'online'` rows left by unobserved disconnects. ### Volume Shadow Copy / VSS (Windows) — Policy Configurator (SPEC-016 redesign, shipped 2026-07-03, agent v0.6.76+) - **The agent no longer schedules its own create+prune loop.** It is a **configurator**: on each policy apply it idempotently sets two kernel-enforced retention governors — VSS diff-area **size cap** (native `IVssDifferentialSoftwareSnapshotMgmt`) and `MaxShadowCopies` count (evaluated, but NOT machine-globally written — see below) — and registers a scheduled task whose action is the agent's own native-COM **create** (`IVssBackupComponents::DoSnapshotSet`, `VSS_CTX_CLIENT_ACCESSIBLE`), never a delete. Retention is enforced by Windows' own FIFO eviction (`volsnap.sys`) when a governor is hit, eliminating the MITRE T1490 "Inhibit System Recovery" surface that had gotten the agent blocked by CrowdStrike Falcon. - **Works uniformly on server AND client SKUs** (previously `vssadmin create shadow`, Server-only, gated client support) — the underlying VSS COM API and `volsnap` cap eviction are not SKU-restricted, only the `vssadmin.exe` CLI is. - **`MaxShadowCopies` is NOT set machine-globally** (pre-merge code review caught this as a data-loss defect — it would FIFO-evict OTHER VSS consumers' shadows, e.g. System Restore or third-party backup, below the configured count). The per-volume size cap is the sole enforced governor; `retention_count`/`retention_max_age_days` policy fields are advisory/ignored (a warning fires if `retention_max_age_days` is still set, since age-based pruning requires deletion). - On first policy apply, the legacy `GuruRMM-VSS-Snapshot` scheduled task (present fleet-wide) is removed and replaced with `GuruRMM-VSS-Create`, supporting both daily-time and every-N-hours interval triggers. - On-demand admin operations (create/delete/browse/restore via `dispatch_operation`, `GET /agents/:id/vss/restores`) are unaffected — this redesign only removes the *scheduled* create+prune loop. - Deployed to beta (0.6.76) at ship time; agent has since advanced to 0.6.78 stable-wide via later bug-fix batches — stable-channel behavior on 0.6.78 not separately re-verified this compile [verify]. ### Software Inventory & Remote Uninstall (SPEC-030 — BETA, not guaranteed) - **Maturity:** code merged to main + deployed (2026-06-23..24), but the feature is **beta and unreliable** — best-effort only. - **Route B (server-orchestrated):** `GET /agents/:id/software`, `POST /agents/:id/software/uninstall` (async -> `job_id`), `GET /agents/:id/software/uninstall/jobs/:job_id`, `GET /agents/:id/software/removal-status` + resolve, `GET /software/knowledge` + classify. - **Silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`): refuses to guess (`needs_interactive` on ambiguity); generic post-success ARP re-check (`Test-StillInstalled`) catches the WiX/Burn Package-Cache-bundle false-success class; orphaned ARP entries classified `tier=orphaned`; async-fork uninstallers get retry-verify. - **Async removal jobs (migration 064):** non-blocking bulk uninstall — job created + `job_id` returned in ~0.06s, background worker processes targets one at a time, dashboard polls live progress. - **Fleet removal knowledge base (migrations 061-063):** per-agent removal-tracking loop, fleet three-state KB (`silent`/`requires_ui`/`unknown`), self-tuning per-target timeout. - **Known gaps:** drivers/system components register in ARP and appear in the bulk list (no exclusion/flag yet); Launchy/AIMP-class uninstallers hang the agent on a lingering child process; no list-jobs endpoint (get-by-id only). The **av-removal-recipes spec (P3, removal-only)** is the follow-on. ### Third-Party Integrations Framework (migration 065, shipped 2026-07-01..02) - **Vendor-agnostic plugin framework:** `integrations` (one enabled plugin per partner+kind), `integration_credentials` (AES-256-GCM encrypted, same `Encryptor` chokepoint as the `credentials` table), `integration_client_mappings` (RMM client -> external tenant id, e.g. a Falcon CID — single-CID partners map every client to one id, MSSP partners map child CIDs), `integration_sync_state` (per-integration sync health), `integration_agent_links` (RMM agent <-> external agent id + last-synced state, e.g. Falcon AID/sensor version/RFM). - **API:** `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `DELETE /api/integrations/:id/mappings/:mapping_id`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test`. - **First plugin: CrowdStrike Falcon.** API client + periodic sync worker (default `sync_interval_seconds=900`) pulls sensor state into `integration_agent_links`. Framework is modeled on the proven per-partner MSPBackups shape (migration 034). - **No dedicated dashboard UI yet** — this is API/backend-only as of this compile (verify before claiming a management screen exists). ### Policy & Configuration Management - **Policy folders + inheritance ("policy-core", NEW, deployed 2026-07-05, migration 067, merge e7e90c2):** per-client **folder tree** replacing the fixed global->client->site->agent stack for agents placed in a folder. Exactly one root folder per client; any folder optionally references a library policy (`policy_folders.policy_id`); `agents.folder_id` places a device in the tree; effective policy is the merged ancestry root->leaf plus an optional per-asset override. Recompute+push on policy edit is scoped to affected agents only. `policy_versions` gives every policy edit an immutable snapshot for diff/rollback and per-device "what changed." **Non-breaking migration by construction:** every new column/table is nullable, `folder_id` is NULL fleet-wide until an explicit, verification-gated, dry-run-capable backfill job runs (`POST /policy-folders/backfill`) — until then effective-policy resolution stays on the legacy path. **API:** `GET/POST /clients/:id/policy-folders`, `DELETE /policy-folders/:id`, `PUT /policy-folders/:id/rename`, `PUT /policy-folders/:id/policy`, `PUT /policy-folders/:id/parent` (move), `PUT /agents/:id/folder` (placement), `POST /policy-folders/backfill`. **Explicitly out of scope for policy-core:** module editor UI, drag-and-drop, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI — no agent-side change (wire format unchanged). - **Fleet-Policy Management Console (policy-ui, SHIPPED 2026-07-06, merge f0320a4, migration 068, PROD):** the dashboard `PolicyFolders.tsx` page is now a full console — folder **tree table**, folder CRUD/rename/move, a **place-devices** flow, an **effective-policy viewer** with source attribution, a per-edit **version-history** diff, and an admin **backfill** trigger. `agents.folder_id` is exposed in the agent API (`d028ca1`) and migration **068** (`068_visible_agents_folder_id.sql`) recreated `visible_agents` to add the column. Replaces the "no dashboard UI yet" state from the policy-core phase. - **Legacy inheritance chain (agents not yet placed in a folder):** global -> client -> site -> agent; server computes merged effective policy, pushes via `ConfigUpdate`. - Checks engine (`018`/`019`): cpu, memory, disk, ping, port, script, service. Policy-attached check templates (`024`). - Remote registry (Windows): enumerate/read routed; write paths exist in the agent but are not routed yet [verify]. - **Event Log Watch (migration 047):** dashboard management UI at `/event-log-watches`. Policy-clobber class bug fixed (PR #43) by reusing a single rule-shaper across all config-push sites. - **Tray policy gate (shipped 2026-07-04, code merged, policy default OFF):** `tray.enabled` policy flag is now enforced by the watchdog (previously plumbed end-to-end but not enforced — the tray launched/showed regardless of the flag). Fleet-wide materialize (deploying the binary via agent-update, not just fresh MSI) has shipped in code; the deliberate flip to enable it fleet-wide (spec's "Task 4") has NOT happened yet. ### Inventory & Discovery - Hardware inventory (mfr/model/serial/BIOS, CPU, memory, disks, NICs, OS), software inventory, service inventory. On-demand refresh. NUL/control-char sanitized before storage (2026-07-05, see Monitoring section). - VM / hypervisor / container detection (`032/033`). - User/group inventory (`037`-`040`): local + domain + Azure AD accounts, domain-join classification, domain-controller detection, group membership. - Network discovery: server-dispatched TCP/ARP/reverse-DNS scans; devices stream back and are persisted/editable. ### Patch / Agent Update Management - Self-updater: server sends version + URL + SHA256; agent downloads, verifies, atomically swaps, restarts, auto-rolls-back on failed reconnect. - Auto-update gated on effective policy `auto_update`. Force via `POST /agents/:id/update`. - **Channel model:** new builds tag **beta**; agents default to **stable**; promotion is a deliberate re-tag. **Fail-closed as of 2026-07-05 (BUG-023 fix, server v0.3.98):** a missing or malformed `.channel` sidecar now classifies **beta**, not stable — closing a publish/tag race where a late "MARK AS BETA" pipeline phase left a multi-minute window during which an untagged build could be mass-offered to the entire stable fleet (incident detail in Recent Work / History). `build-windows.sh` now writes the beta sidecar atomically at publish; `promote_version` keys on agent binaries (not pre-existing sidecar files) and creates/overwrites `stable` sidecars, failing the request (500) rather than recording a rollout row that disagrees with disk on a partial write. - **Agent WS transport durability (comms-durability, agent 0.6.79 beta, merge 07307e3):** update pushes and commands ride the persistent WS, so its liveness governs delivery. Three hardening layers shipped: **bounded WS writes** (`WRITE_STALL_TIMEOUT=90s`) so a half-open socket can no longer freeze the agent's send loop (incl. the reconnect timer); a **heartbeat-write counter** reported in the auth payload and logged server-side (nonzero count + a prior half-open eviction = agent->server network blackhole; 0 = local send stall — a server-visible verdict replacing on-site packet capture); and an **AIMD adaptive keepalive** (`transport::keepalive`) self-tuning the heartbeat interval per site 5-45s (kept under the 90s eviction window), persisted, gated on authenticated sessions. Targets the chronic per-site WS loopers (BUG-023/T2); a total agent->server blackhole stays network-layer (uncurable in code) — this hardens the freeze bug and makes the diagnosis server-visible. - **Update-scanner platform coverage fixed 2026-07-05 (server v0.3.95):** the scanner's self-check previously tried to execute every binary regardless of platform to read its embedded version, dropping macOS/Linux artifacts entirely because they can't run on the Linux build host — **macOS/Linux agents had never been offered an update**. Now uses host-native-vs-foreign self-check regimes: foreign-platform binaries trust the filename-embedded version; host-native binaries stay a hard `--version` gate (with a best-effort exec-bit repair for suspected 0644 CI-copy artifacts). Post-fix: scanner indexes 8 platform/arch combos (was 4, Windows-only). - **BUG-024 (OPEN, CRITICAL, mitigated 2026-07-05):** enabling Linux update offers for the first time (the fix above) exposed that Linux agent binaries >=0.6.77 are built against the build host's glibc 2.43 (Ubuntu 26.04) with no compatibility floor — every older-distro Linux agent that took the offer crash-looped on `GLIBC_2.39` not found (ix.azcomputerguru.com CloudLinux 9, Jupiter's Debian 12 container, a Lonestar container, presumed SL-SERVER). Mitigated by quarantining Linux binaries out of the downloads directory (scanner currently offers nothing for Linux). Real fix (glibc floor via an old-glibc build container or a musl static target, plus an `objdump -T` build gate) is pending — see History Highlights and FEATURE_ROADMAP BUG-024. **This recurs on every merge:** each build re-publishes a fresh glibc-2.43 Linux binary into downloads and must be re-quarantined by hand (done again for 0.6.79 on 2026-07-06) until the floor + gate are wired into the pipeline; the fix decision (musl vs old-glibc container) is with Mike. - **Pending-update supersede (batch 4, server v0.3.99):** a stale `pending` update row is now superseded at re-dispatch time when a newer version exists or its target binary was pruned from disk (guarded: only rows over 1h old with a non-empty platform index) — previously all 13 pruned targets were blindly re-offered. - **Legacy fleet build (Server 2008R2/Win7, Rust 1.77) DISABLED 2026-07-04** (`LEGACY_ENABLED=false`): the unpinned dep re-resolve pulled edition2024 crates Rust 1.77 cannot parse, fail-closing the entire Windows build. Last legacy artifact (agent 0.6.76) frozen and preserved for existing 2008R2/Win7 endpoints; modern builds advance normally. 2008R2 support returns only via a separate legacy-only rewrite (RMM_THOUGHTS Feature 12, Raw — not approved). - **Universal self-detecting installer (Feature 9, P1 shipped v0.6.71):** one install path self-detects arch/OS and pulls the correct binary. - **Promotion API:** `POST /api/updates/rollouts/:version/promote` body `{"os","arch"}`; rollback via `.../rollback`. ### Agent Identity & Enrollment - Per-agent enrollment keys; site-specific MSI with SITEKEY injected. `enrolled_agents` is the authoritative enrollment history log. - **Durable multi-location `device_id`** (Phase 1 Task 1, 2026-06-11): registry+ProgramData (Windows), /etc+/var/lib (Linux), /Library+mirror (macOS), self-healing; `cleanup.ps1` whitelists identity files. - **Cross-site dedup fixed 2026-07-04 (commits `2851523`, `1c1cac2`):** Both WS enrollment paths previously deduped only within `(site_id, device_id)`, so re-running a different site's installer against an already-enrolled machine created a cross-site duplicate. Fixed to match by hardware `device_id` **across all sites** (`db::get_agent_by_device_id`) and re-home via `move_agent_to_site` on first match. A same-day follow-up stopped auto-re-homing **on every reconnect** — the agent re-presents its embedded install-site code as its API key on every beat, which was silently bouncing admin-relocated agents back to their original site. Current behavior: dedup by device_id, but site placement is admin-controlled and persists across reconnects. 13 duplicate-hostname agent pairs were identified fleet-wide (pre-fix legacy `win-*`/`hw-*` records) — 4 orphaned zombies were cleaned up in batch 4 (2026-07-05); the remaining full cleanup pass is still open and is the ongoing trigger for `move_agent` 409s. - **Delete race closed to the schema level (BUG-018 altitude fix, migration 066, server v0.3.98):** a `visible_agents` DB view (`status<>'deleting'`) is now the single source of truth for all listing/stat queries (nine call sites, including a previously-missed agent-detail-page query), a `BEFORE UPDATE` trigger pins `status='deleting'` at the schema level, and all three WS `authenticate_agent` paths reject a mid-purge row. **Note for future schema changes:** the view freezes the `agents` column list at `CREATE VIEW` time (`SELECT *`) — any `ALTER TABLE agents ADD COLUMN` migration must also recreate `visible_agents` or listings 500 fleet-wide with `ColumnNotFound` (documented in the migration 066 header). - **Agent status reconciliation (batch 4, server v0.3.99):** a periodic sweep flips zombie `'online'` rows (agent status stranded by an unobserved disconnect) back to `'offline'` without disturbing the real `last_seen` timestamp, respecting the update-hold window and skipping agents with a live dispatch-map connection. ### Alerting & Watchdog - Threshold alerts (ack/resolve, per-agent + fleet summary). Alert templates (`022`). Maintenance mode (`021`). - Watchdog: separate supervising process; reports via `POST /watchdog-alert`. - **Role-aware offline alerting + alert ignore/mute (shipped 2026-06-07):** server-only `agent_offline`; site/fleet mass-offline aggregation; `alert_mutes` perma-silence. - **`create_or_update_alert` hardened (2026-07-05, server v0.3.95):** now returns `Result>` — `Ok(None)` when the target alert row was deleted mid-flight (agent cascade-delete FK violation) instead of aborting the caller. Both UPDATE-by-id branches switched from `fetch_one` to `fetch_optional` with fall-through to INSERT (`refresh_alert_row` helper dedups the two near-identical UPDATE blocks). Fixed a `RowNotFound` that had been aborting the entire offline sweep whenever an alert was concurrently deleted (e.g. during duplicate-agent cleanup). - **WebSocket log noise reclassified INFO, not ERROR (2026-07-05, server v0.3.96):** benign disconnects (`is_benign_ws_disconnect`) and pre-auth connection churn (`is_preauth_connection_churn` — sleeping laptops, scanners, half-open NAT never sending an auth frame) no longer mint `server_error` alerts. Genuine auth rejections (bad API key, malformed auth message) remain ERROR. Classified at INFO (not DEBUG) so a mass-disconnect anomaly stays visible at the server's default log level. - **`move_agent` duplicate-key hardened (2026-07-05):** unique-violation on `idx_agents_site_device` now maps to 409 CONFLICT with an actionable message instead of a raw 500. - **MSP360 transient-5xx noise reduction (batch 2, server v0.3.97):** sync 5xx logs WARN, escalating to ERROR only after 4 consecutive cycles (~1h) so persistent outages still alert without every cloud-maintenance blip minting a `server_error`. ### Credentials Management - Encrypted credentials vault (`016`): scoped global/client/site, typed, metadata-only by default with a separate `/reveal` endpoint. Known HIGH item: `/reveal` ownership-scope check — [verify current state]. - **Credential inheritance (deployed 2026-06-07):** Opt-in hierarchical cascade, de-duplication, `/effective` endpoints. - **Integration credentials (2026-07-01, migration 065):** reuses the same encryption chokepoint for third-party integration API keys (never sent to agents, never plaintext in `integrations.config`). ### Audit Log / Log Feedback Intelligence - Append-only `audit_log` table (migration 056). Per-line log fingerprinting + `log_signatures` aggregation (migration 057). `logs/analyze` runs on the Claude API (switched from Ollama 2026-06-15). ### Backup Integration (MSP360 / MSPBackups) - Multi-provider config, connection test, scheduled sync, per-agent + fleet coverage report, agent<->MSP360 mapping with confidence scoring. - **Active chronic failure (unresolved as of 2026-07-05):** GND-SERVER (Grabb & Durando) — 42+ `backup_failed` alerts since 2026-06-17, failing nightly ~01:02 UTC, "no error detail reported." Also open: AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage, offline 17.7 days with an active critical alert as of batch 4), DESKTOP-EVA4H1A (Gary A Hartman, partial), Blaster2 (x2). MSP360 sync doesn't capture failure detail — investigation needs a live PowerShell dispatch to read MSP360/CloudBerry local logs. ### Remote Access (Tunnel) - Agent side substantially built. Server side is a dead-code skeleton (no `/tunnel` routes). Live TTY design (agent-comms-durability Phase 2) will supersede the skeleton. ### Identity / Multi-tenancy / Security - Auth: JWT; agents auth over WS via per-agent API key + hardware device_id. - Microsoft Entra ID SSO (OAuth2/OIDC + PKCE). - Organizations / multi-tenancy: org CRUD, roles, limits, dev-admin impersonation. - **500-error-leak fixed (commit `58c1a96`):** all server error paths route through `internal_err`. ### Platform coverage - **Cross-platform (Win/Linux/macOS):** metrics, network state, hardware/software/service inventory, user/group + DC/domain detection, checks, discovery, self-update, scripts/commands. - **Windows-only:** `user_session` command context, remote registry, tray-into-session (now policy-gated), watchdog SCM supervision, BSOD detection, VSS policy configurator, WMI ACPI thermal, legacy fleet (frozen at 0.6.76). - **Linux:** update offers currently QUARANTINED fleet-wide pending the BUG-024 glibc-floor fix (see Patch/Update section) — all other cross-platform capabilities unaffected. - **macOS:** agent deployed (Phase 1); tray is a stub; included in the update scanner's platform coverage since 2026-07-05. --- ## Architecture ### Components | Component | Location | Tech | State | |---|---|---|---| | Server | 172.16.3.30:3001, systemd `gururmm-server.service` | Rust, Axum | deployed, production; v0.3.99 | | Dashboard | https://rmm.azcomputerguru.com, nginx | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production; v0.2.83 beta | | Agent (Windows) | Endpoints, `GuruRMMAgent` service via WiX MSI | Rust, Windows MSVC | deployed; modern build v0.6.78 fleet-wide (promoted stable 2026-07-05), legacy (2008R2/Win7) FROZEN at 0.6.76 and disabled | | Agent (Linux) | Endpoints, systemd `gururmm-agent` | Rust, musl static | deployed; **update offers QUARANTINED fleet-wide (BUG-024, glibc floor pending)** — existing installs unaffected, no new updates offered | | Agent (macOS) | Endpoints, LaunchDaemon | Rust, aarch64/x86_64 | Phase 1 deployed 2026-05-12; code-signing issue on Apple Silicon; update-scanner platform gap fixed 2026-07-05 | | Tray (Windows) | System tray, named pipe IPC | Rust | deployed to disk fleet-wide via agent-update (2026-07-04); **launch policy-gated, default OFF** | | Tray (Linux) | System tray, Unix socket IPC | Rust, GTK | deployed 2026-05-24 | | Tray (macOS) | Menu bar | Rust | stub/TODO | | PostgreSQL DB | localhost:5432 on 172.16.3.30 | PostgreSQL 18 | deployed; migrations through 067 | | Build pipeline | 172.16.3.30:9000 webhook + `/opt/gururmm/` scripts | Python, Bash | deployed; builds agents AND server; build-lock + gated git reset added 2026-07-05 (BUG-003) | | Windows build — Beast (PRIMARY) | GURU-BEAST-ROG over Tailscale | Rust MSVC, WiX 4.x | primary — two-wave parallel, ~336s | | Windows build — Pluto (FALLBACK) | 172.16.3.36 | Rust MSVC, WiX v4 | fallback only | ### Key Files & Repos - **Active repo:** `azcomputerguru/gururmm` — http://172.16.3.20:3000/azcomputerguru/gururmm (external: https://git.azcomputerguru.com/azcomputerguru/gururmm.git) - **Submodule working tree:** `projects/msp-tools/guru-rmm` — gitlink intentionally lags origin/main; read live code via `git -C projects/msp-tools/guru-rmm show origin/main:` or a worktree. - **Server binary:** `/opt/gururmm/gururmm-server` on 172.16.3.30 - **Server config:** `/opt/gururmm/.env` - **New shared sanitize module:** `server/src/db/sanitize.rs` (`strip_nul`, `clean_control_chars`) — consumed by `db/logs.rs` and `db/inventory.rs` - **Visible-agents view + delete-race schema fix:** `server/migrations/066_visible_agents_view.sql` (view + BEFORE UPDATE trigger; column-freeze warning in the header — any `ALTER TABLE agents ADD COLUMN` must recreate the view) - **Policy folders / policy-core:** `server/migrations/067_policy_folders.sql` (`policy_folders`, `agents.folder_id`, `policy_versions`), `server/src/api/policies.rs` (folder CRUD + placement + backfill), `docs/specs/policy-core/`, `docs/specs/policy-ui/` (dashboard UI spec, not implemented) - **Integrations framework:** `server/src/api/integrations.rs`, `server/src/integrations/` (plugin trait + registry), migration `065_integrations_framework.sql` - **VSS configurator:** `agent/src/vss_com.rs`, `agent/src/vss.rs`, `specs/vss-policy-config/`, `specs/vss-native-com/` - **Tray policy-deploy:** `agent/src/tray_stage.rs`, `agent/src/tray_state.rs`, `agent/src/authenticode.rs`, `specs/tray-policy-deploy/` - **Windows WMI thermal:** `agent/src/` (BUG-001 `collect_temps_windows_acpi`, `wmi` crate dep) - **Downloads dir:** `/var/www/gururmm/downloads/` on 172.16.3.30 (Linux binaries currently quarantined to a sibling `quarantine/` dir pending BUG-024) - **Webhook handler:** `/opt/gururmm/webhook-handler.py` - **Deploy pipeline (canonical, auto-synced to /opt/gururmm on push):** `deploy/build-pipeline/build-server.sh` (flock lock, PIPESTATUS-gated git reset, `--migrate-only` pre-stop validation — BUG-003), `deploy/build-pipeline/build-windows.sh` (atomic beta `.channel` sidecar write — BUG-023) - **API (internal):** http://172.16.3.30:3001 - **API (external):** https://rmm-api.azcomputerguru.com - **Dashboard:** https://rmm.azcomputerguru.com - **Vault paths:** `infrastructure/gururmm-server.sops.yaml` (API creds, DB password, sudo password), `infrastructure/gururmm-server-physical` (SSH ed25519 key — this fleet key works on 172.16.3.30 currently, not just the future physical box; confirmed 2026-07-04) - **SSH to prod:** `ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30` (ed25519, key-only; password auth disabled for SSH; sudo over SSH still requires the vault password via `-S`) - **Pre-merge verification skill:** `bash .claude/skills/gururmm-build/scripts/verify.sh server|agent|dashboard|migrations` ### Repo Structure ``` gururmm/ ├── agent/ Rust agent (managed endpoints) │ └── src/ │ ├── authenticode.rs Authenticode signature verify (tray-policy-deploy) │ ├── tray_stage.rs tray materialize FSM + staging + versioned mutex │ ├── tray_state.rs tray policy state │ ├── bsod.rs Windows BSOD/kernel-crash detection │ ├── vss_com.rs Native VSS COM create/provision │ ├── vss.rs VSS policy configurator entry points │ ├── commands/mod.rs CommandExecutor + recent-results cache (ACK dedup) │ ├── device_id.rs Durable multi-location device_id │ ├── transport/ │ │ └── websocket.rs WS transport; execute_command ACK + dedup; cmd-alias NAK │ ├── tunnel/ TunnelManager state machine │ ├── metrics/ sysinfo collection + temps; WMI ACPI thermal (BUG-001) │ ├── registry_ops/ Windows registry read/write │ ├── updater/ Self-update handler (container guard BUG-019, unix rollback watchdog) │ └── main.rs ├── server/ Rust/Axum API server │ └── src/ │ ├── api/ │ │ ├── integrations.rs integrations framework CRUD (crowdstrike-falcon) │ │ ├── policies.rs policy CRUD + folder tree + placement + backfill (policy-core, NEW) │ │ └── ... 39 route modules; software.rs SPEC-030 │ ├── alerts/ Alerting modules (offline.rs: offline_sweep + mass_offline + status reconciler) │ ├── db/ │ │ ├── sanitize.rs shared NUL/control-char sanitize │ │ ├── alerts.rs create_or_update_alert -> Option, refresh_alert_row │ │ ├── inventory.rs NUL-scrub over jsonb + text before upsert │ │ ├── agents.rs visible_agents-view reads, status write guards │ │ └── software_jobs.rs SPEC-030 async removal jobs │ ├── updates/ │ │ └── scanner.rs host-native vs foreign self-check regimes; fail-closed channel default (BUG-023) │ ├── ws/mod.rs WS handler; is_benign_ws_disconnect / is_preauth_connection_churn; deleting-row auth rejection; half-open eviction sweep (T2) │ └── mspbackups/ MSP360 backup integration; transient-5xx WARN escalation ├── dashboard/ React/TypeScript UI │ ├── pages/ │ │ └── EventLogWatches.tsx │ └── components/ │ └── SoftwareManager.tsx ├── agent/scripts/ uninstall-engine.ps1 ├── tray/ System tray binary ├── installer/ WiX v4 MSI; cleanup.ps1; universal self-detecting installer ├── deploy/ │ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh (flock+PIPESTATUS+migrate-only, LEGACY_ENABLED=false) ├── scripts/ Build/ops scripts └── docs/ FEATURE_ROADMAP.md, UI_GAPS.md, RMM_THOUGHTS.md, specs/ docs/specs/ policy-core/, policy-ui/, SPEC-025-policy-compliance-posture.md, ... specs/ tray-policy-deploy/, vss-policy-config/, vss-native-com/, crowdstrike-falcon/, av-removal-recipes/, remote-software-uninstall/, agent-comms-durability/, ... ``` --- ## Development ### Current Focus As of 2026-07-05 (agent 0.6.78 / server v0.3.99): - **BUG-024 real fix (glibc floor for Linux builds):** Linux binaries remain quarantined out of downloads — no Linux agent gets update offers until an old-glibc build container (or musl static target) plus an `objdump -T` build gate lands. SL-SERVER (Scileppi) and Lonestar's container host still need manual agent recovery (no vault SSH creds for either). - **Machine-side backup failures (highest user priority, unresolved):** GND-SERVER chronic 42+-failure streak, AD1, LAB-SVR (offline 17.7 days as of batch 4), DESKTOP-EVA4H1A, Blaster2 (x2) — MSP360 sync reports no error detail; needs a live PowerShell dispatch to read local MSP360/CloudBerry logs or the mspbackups.com console. - **Duplicate-hostname agent cleanup:** 4 orphaned zombies cleared in batch 4; the remaining pairs of the original 13 fleet-wide are the ongoing trigger for `move_agent` 409s — a full one-time cleanup pass is not yet done. - **4 chronic one-way-WS loopers (T2 field data):** DESKTOP-3USU20B (Safesite Glendale, 79/127 evictions), DESKTOP-BA45FRS (Safesite Bell), DDDOffice072023 (ACG discovery), DESKTOP-PL2RCGL (Robyn Pittman) — agent->server WS frames eaten post-auth, version-independent, server-side telemetry exhausted; packet capture on an affected box is the next diagnostic step. - **Policy-ui (folder management dashboard UI):** shape spec added 2026-07-05 (`docs/specs/policy-ui/`); implementation not started. Policy-core backend is deployed but has no dashboard surface yet. - **Tray policy fleet enablement (Task 4, tray-policy-deploy spec):** materialize code is merged and deployed; flipping `tray.enabled` on for the fleet is a deliberate next step, not yet done. - **Legacy (2008R2/Win7) agent rewrite (RMM_THOUGHTS Feature 12, Raw):** not approved; needs a language/runtime decision (.NET Framework 4.x vs C++) and scope decision before a `/shape-spec`. - **Integrations framework dashboard UI:** CrowdStrike Falcon backend + sync worker are live; no management screen exists yet in the dashboard. - **SPEC-030 software-uninstall follow-on (Route A / recipes):** Launchy/AIMP-class uninstallers still hang the agent; driver/system-component exclusion flag and a list-jobs endpoint remain unbuilt. - **Agent-comms-durability Phase 2/3 (planned):** live TTY, adaptive keepalive, bulk file transfer -> short-lived HTTPS. Phase 1 is now fully complete (T1/T2/T3 all shipped). - **Phase-1 acceptance test:** interactive command dispatch to PST-SERVER / Safesite boxes during an idle window, still pending across three consecutive session logs. - **Durable agent identity Phase 2-3 (pending soak):** guarded auto-reclaim + an operator merge tool for the pre-fix ghost/duplicate agents. - **SPEC-028 offboarding wizard:** specification complete; implementation pending. - **Repo-root `build-server.sh`:** a stale divergent lineage of the canonical `deploy/build-pipeline/build-server.sh` — flagged as a deletion candidate to end confusion about which script is authoritative. - **KSTEENBB2025:** fleet-max `cpu_temp` peaking 83.8C — worth a glance once thermal data has a wider baseline. ### Patterns & Anti-Patterns **New anti-patterns (2026-07-01..05):** | Pattern | What Went Wrong | |---|---| | Publish-then-tag-later build pipeline ordering (BUG-023) | Binaries landed in the downloads dir minutes before a late "MARK AS BETA" phase tagged them; the scanner's missing-sidecar default was `stable`, so a 5-minute scan window could mass-offer an unsoaked build to the entire stable fleet. Fix: fail-closed scanner default (missing/malformed sidecar => beta) AND atomic sidecar write at publish — defense in depth, not either alone. | | Promotion logic keyed on pre-existing sidecar file content rather than the binary itself | `promote_version` only rewrote files whose content was exactly `beta`, so a fail-closed (sidecar-less) binary became permanently unpromotable the moment the scanner default flipped. Key promotion on the binary set, create/overwrite the sidecar, and fail loudly (500) on a partial write rather than recording a DB rollout row that disagrees with disk. | | Building Linux agent binaries against the build host's live glibc with no floor (BUG-024) | The build host's OS upgrade (Ubuntu 26.04, glibc 2.43) silently raised the minimum glibc every Linux agent binary requires; combined with a same-week fix that enabled Linux update offers for the first time, every older-distro Linux agent that took the update crash-looped on a missing GLIBC symbol. Any cross-compiled/host-compiled binary needs an explicit compatibility floor (old-glibc container or static linking) verified by a build-time `objdump -T` gate, not an assumption that "it compiled" means "it runs everywhere." | | Hand-copied `WHERE status <> 'deleting'` guards scattered per query/UPDATE | The same forgotten-guard class recurred (BUG-018) even after a prior "fix" — nine call sites needed the filter and one (agent detail page) was missed. Fixed at the schema level: a view for reads, a `BEFORE UPDATE` trigger for writes, so no future query/update can forget the guard. Tradeoff accepted: a `SELECT *` view freezes the column list at creation time — documented as a loud warning for future migrations. | | Update-scanner self-check executing every binary regardless of target platform | macOS/Linux binaries can't run on the Linux build/scan host, so they were silently dropped from the scannable set entirely — those agents never received an update offer. Fix: host-native binaries get a hard `--version` execute-and-check gate; foreign-platform binaries trust the filename-embedded version. | | Writing a machine-global VSS registry governor (`MaxShadowCopies`) to a per-policy `retention_count` | `MaxShadowCopies` is shared by ALL VSS consumers on the box (System Restore, third-party backup) — lowering it FIFO-evicts THEIR shadows too, a genuine data-loss class caught only by a 23-agent pre-merge code review. Use a per-volume size cap as the sole enforced governor instead. | | Re-homing an agent's site on every WS reconnect after a device_id-based dedup fix | The agent re-presents its embedded install-site code as its API key on every beat; auto-re-homing on each connect silently bounces an admin-relocated agent back to its original install site. Dedup by device_id, but leave site placement admin-controlled and persistent. | | `fetch_one` on an UPDATE-by-id inside a concurrently-deleting table | An agent cascade-delete can remove the target row between a SELECT and the UPDATE; `fetch_one` panics `RowNotFound` and aborts the whole caller (here: the entire offline sweep). Use `fetch_optional` with a fall-through path. | | Two sessions editing the same branch's working tree without a coord lock | A concurrent session's uncommitted edits were silently absorbed into another session's commit more than once across this window. Benign each time (same branch, all changes verified at HEAD) but a coord lock was only reliably taken starting with bug-fix batch 3 — take a lock before editing a shared submodule branch. | | Assuming a spec's `shape.md`/`plan.md` header "Status: not started" reflects current reality | Tasks can ship and merge (git log shows work done) while the spec doc header is never updated. Verify against `git log`/deployed version, not the spec file's status line. | | Testing a pipe's exit code (`if ! cmd \| tee ...`) instead of the upstream command's | Shell tests the last command in the pipe (`tee`), not `git`; a failed `git fetch`/`reset` could report success. Use the shell's `PIPESTATUS` array explicitly. | | Mischaracterizing "19 known test failures" as a single homogeneous class | All 19 were assumed to be the same pre-existing `DATABASE_URL`-missing environment gap; one was a genuine `matches_pattern` panic on no-wildcard input hiding in the same failure count. Bucket-count assumptions on a recurring "known failures" number should be re-verified periodically, not re-quoted. | **Existing anti-patterns (carried forward):** | Pattern | What Went Wrong | |---|---| | `useMemo` with stable deps for data-dependent values | queryClient is stable, memo never recomputes after queries resolve. Use `useQuery` instead. | | Deploying without stopping the server first | "text file busy" kernel error. Always `systemctl stop` before `cp`. | | Building without `DATABASE_URL` | sqlx compile-time macros fail. | | DB migrations without inserting into `_sqlx_migrations` | Server crashes on start. Must insert SHA-384 checksum manually. | | WiX MSI builds on Linux | WiX requires `msi.dll`. Must build on Pluto or Beast. | | Manual builds via SSH | All builds go through `webhook-handler.py`. | | `Restart-Service GuruRMMAgent -Force` in command scripts | Kills agent before it can report result. Use a scheduled task with delay instead. | | `sudo -u guru git` in systemd build context | git rejects repo as dubious ownership. Use `git config --system --add safe.directory`. | | `+1.77` legacy builds without pinned deps | Fresh dep resolution repeatedly pulls edition2024 crates Rust 1.77 can't parse (BUG-021, recurred 2026-07-04 with a different dep pair, led to the variant being disabled outright rather than re-chasing pins). | | Dead WebSocket write half | WS write fails, send task dies, receive loop keeps agent connected with a dead write half. Fix: `tokio::select!` monitoring both tasks. | | Using the `minidump` crate for Windows kernel dumps | Only parses Breakpad MDMP, not Windows kernel PAGEDU64. Parse `DUMP_HEADER64`/`32` at fixed offsets directly. | | Build classification defaulting to stable | Superseded 2026-07-05 (BUG-023) — the scanner's missing-sidecar default is now beta, not stable; new builds are tagged beta by the pipeline and promotion is an explicit re-tag on top of that. | | `command_type: "cmd"` in API commands | Not a valid `CommandType` variant originally — agent silently dropped the message. Now aliased to `Shell`. | | Reading the stale submodule working tree for code | The submodule gitlink intentionally lags `origin/main`. Always read via `git -C ... show origin/main:` or a worktree. | **Good patterns (carried forward + new):** - **Push guards to the schema, not the call site** (2026-07-05) — the `visible_agents` view + `BEFORE UPDATE` trigger closed a class of forgotten-guard bugs that a second round of hand-copied `WHERE` clauses had already failed to fully close once. - **Fail-closed defaults on ambiguous/missing state** (BUG-023 scanner fix) — treating a missing `.channel` sidecar as the risky classification (beta) rather than the safe-looking one (stable) makes a future pipeline crash harmless instead of dangerous. - **Shared sanitize chokepoint** (`db/sanitize.rs`) — one NUL/control-char scrubber consumed by both `db/logs.rs` and `db/inventory.rs`, replacing duplicated ad hoc scrub logic. - **Capability-based classification (host-native vs foreign) instead of a single universal check** — the update scanner branches its self-check strategy per target platform rather than applying one strategy fleet-wide. - **Multi-AI adversarial review before shipping risky agent-side changes** — the VSS redesign (Grok + Gemini) and tray redesign (17-agent Claude review + Grok + Gemini) both caught genuine defects (a data-loss governor and 9 architecture defects respectively) that a single-model review missed on the first pass; the 2026-07-04..05 batches ran 15-26-agent high-effort code-review workflows before every fleet-deploy merge and consistently surfaced findings (data-loss guards, race conditions, unpromotable-binary edge cases) that first-pass implementations missed. - **Platform parity rule** — any agent feature goes on Windows + Linux + macOS in the same commit. - **`gururmm-build` skill for pre-merge verification** — does NOT trigger production build; merging to main is the only prod trigger. - **Dashboard beta-before-main rule** — dashboard changes go to beta (auto-built from main) first; promote explicitly. - **Non-breaking migrations by construction for structural changes** (policy-core, migration 067) — new nullable columns/tables plus a deliberate, verification-gated, dry-run-capable backfill job means landing a large schema change changes zero live behavior until an explicit flip. - **Two independently-deployable slices for cross-component features, capability-gated** — agent slice and server slice can deploy in any order; old agents/servers keep legacy behavior until the gate flips. ### Build & Deploy **CRITICAL: Never trigger builds manually via SSH. All builds go through the webhook pipeline.** ``` Gitea push to main -> webhook-handler.py (172.16.3.30:9000) -> build-shared.sh (auto-version bump) -> build-linux.sh (Linux artifact publish currently QUARANTINED post-build pending BUG-024 glibc floor) -> build-windows.sh (Beast PRIMARY over Tailscale || Pluto FALLBACK TWO-WAVE PARALLEL BUILD; legacy wave gated LEGACY_ENABLED=false; deploy_and_sign writes the beta .channel sidecar ATOMICALLY per artifact — BUG-023 fix) -> build-mac.sh (stub — no build machine configured yet) -> build-server.sh (gated on server/ changes; flock lock, PIPESTATUS-gated git fetch/reset, pre-stop --migrate-only migration validation, backup+rollback — BUG-003 fix) -> artifacts -> /var/www/gururmm/downloads/ with sha256 + -latest symlinks + .channel (missing/malformed .channel now scans as beta, not stable — BUG-023 fix) ``` **BUG-003 closed 2026-07-05 (`6ccd35e`):** the deploy script confirmed to auto-sync to `/opt/gururmm` on every push. Gained a `flock` build lock, `PIPESTATUS`-gated `git fetch`/`reset` (fired correctly on its first live run — a merge-webhook build collided with the CI version-bump job's `index.lock` and aborted cleanly with nothing deployed, instead of silently building a stale tree), and pre-stop migration validation via a new `server --migrate-only` flag (validates+applies migrations against the live DB while the OLD binary keeps serving; paired with `set_ignore_missing(true)` on the migrator so a rollback binary tolerates an ahead DB instead of re-creating the outage the gate was meant to prevent). **BUG-023 closed 2026-07-05:** `build-windows.sh`'s `deploy_and_sign` now writes the beta `.channel` sidecar atomically with each published artifact, closing the multi-minute publish-then-tag window that had let an untagged build get scanned as stable. **Legacy variant disabled 2026-07-04 (`84a82a3`):** `LEGACY_ENABLED=false` in `build-windows.sh` gates the legacy wave/copy/deploy/symlink off entirely; the last-built legacy artifact (0.6.76) is preserved and excluded from cleanup so existing 2008R2/Win7 endpoints keep receiving that frozen build. Modern (amd64/x86/tray/debug) builds are no longer blocked by legacy dep-resolution failures. **Channel/promotion model, downloads layout, dashboard beta/prod split, and Windows build host detail otherwise unchanged from the 2026-07-04 compile** — see History Highlights / Compilation Notes for full prior mechanics; current migration count is 068 (`068_visible_agents_folder_id`). --- ## Active State **Fleet (as of 2026-07-05):** - ~360 enrolled agents; ~229 fresh/online at last sample - Agent 0.6.78 modern fleet-wide (stable, promoted 2026-07-05) / 0.6.76 legacy-frozen-and-disabled; server v0.3.99; dashboard v0.2.83 beta - Active `server_error` alerts: **0** - Duplicate-hostname agent pairs: 4 orphaned zombies cleaned in batch 4; remaining pairs of the original 13 still outstanding - Active `backup_failed` alerts: GND-SERVER (chronic), AD1, LAB-SVR (offline 17.7 days), DESKTOP-EVA4H1A, Blaster2 (x2) — investigation open - Linux update offers QUARANTINED fleet-wide (BUG-024) — existing Linux installs unaffected, no new offers until the glibc-floor fix ships - 4 chronic one-way-WS loopers under active diagnosis (DESKTOP-3USU20B, DESKTOP-BA45FRS, DDDOffice072023, DESKTOP-PL2RCGL) — packet capture pending - Policy-core (folder tree) deployed but `agents.folder_id` is NULL fleet-wide — no agent is yet on the new inheritance path pending the backfill job - SL-SERVER (Scileppi) and Lonestar's container host down pending manual agent recovery (no vault SSH creds for either) **API auth:** - `POST /api/auth/login` -> JWT (~24h) - Creds: vault `infrastructure/gururmm-server.sops.yaml` - Key endpoints: `GET /api/agents`, `POST /api/agents/:id/command`, `GET /api/commands/:id`, `POST /api/agents/:id/update`, `POST /api/updates/rollouts/:version/promote` - Software (SPEC-030): `GET /api/agents/:id/software`, `POST /api/agents/:id/software/uninstall` (-> job_id), `GET .../uninstall/jobs/:job_id`, `GET .../removal-status`, `GET /api/software/knowledge` - Integrations: `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test` - **Policy folders (NEW):** `GET/POST /api/clients/:id/policy-folders`, `DELETE /api/policy-folders/:id`, `PUT /api/policy-folders/:id/rename`, `PUT /api/policy-folders/:id/policy`, `PUT /api/policy-folders/:id/parent`, `PUT /api/agents/:id/folder`, `POST /api/policy-folders/backfill` - Command fields: `command_type` (`shell`/`powershell`/`python`/`script`/`claude_task`/`cmd` alias), `command`, optional `context` (`system`/`user_session`), `timeout_seconds`/`elevated`. **Dashboard — complete and working:** Agents management, Clients/Sites CRUD, Commands execution + terminal, Logs + AI analysis (Claude API), Alerts, Metrics, Auto-update triggering, Network state, Entra ID SSO, Policies Dashboard, Registry editor (read-only), MSP360 backup status + mappings/verify UI, Organizations + dev-admin impersonation, Credentials with inheritance, AgentDetail Crashes tab + version history, Event Log Watches management, Software Manager (BETA, not guaranteed). **Dashboard — incomplete (see UI_GAPS.md):** - Watchdog alerts UI — blocked on 2 missing server routes - BSOD in Alerts stream (Phase 2 deferred) - Tunnel session management (server skeleton) - Offboarding wizard UI (SPEC-028 complete; implementation pending) - Alert mute UI — server endpoints ready, dashboard NOT started - Integrations Center UI — CrowdStrike Falcon framework + sync worker are live server-side; no management screen in the dashboard yet - **Policy folders / policy-ui (NEW gap)** — policy-core backend + folder-tree API deployed 2026-07-05; the `policy-ui` shape spec exists but no implementation has started, so folder placement/inheritance is API-only today - **Tray policy toggle surfacing** — `tray.enabled` is enforced server/agent-side; verify whether the policy UI exposes the toggle to operators [verify] **Security backlog (HIGH):** - `credentials/:id/reveal` — horizontal privilege escalation (no ownership scope check) --- ## Key Architecture Decisions (LOCKED) These decisions are locked. Do not reverse without explicit user approval. 1. **Per-agent enrollment keys.** 2. **Site-specific MSI generation.** 3. **No TOML/config for endpoints** — server URL compiled into binary. 4. **Policy inheritance chain** — global -> site -> client -> agent (legacy path); a device may instead be placed in a per-client **policy folder** (added 2026-07-05, decision 14) inheriting its folder-ancestry chain — the two models coexist during migration via `agents.folder_id`. 5. **Platform parity rule** — any agent feature ships on Windows, Linux, and macOS in the same change. 6. **Watchdog as separate process** — reports via REST (`POST /watchdog-alert`), no WebSocket. 7. **Build pipeline is the only path to production.** 8. **Multi-tenancy identity model (ADR-001)** — Dev -> Partner -> Client. 9. **Holistic feature development** — every feature requires backend + API + dashboard UI + docs. 10. **AI-optional operation.** 11. **Durability-first command delivery** — the durable Postgres `commands` row is the source of truth; the WebSocket is only a delivery hint. 12. **VSS: configure, never delete on a schedule** (added 2026-07-03) — the agent may only set retention governors and create shadows on a schedule; scheduled or automatic deletion reintroduces the MITRE T1490 surface. On-demand admin-initiated delete stays allowed. 13. **Third-party security integrations are a plugin, not a schema** (added 2026-07-01) — a new vendor implements the `integrations` plugin trait against the generic framework tables; it does not get its own bespoke migration set. 14. **Policy scoping is a folder tree, not a fixed level stack** (added 2026-07-05) — a client's policy structure is an arbitrary-depth folder tree per client (policy-core); the legacy fixed global->client->site->agent stack remains the resolution path for any agent not yet placed in a folder, and migration between the two models is backfill-gated, never automatic or silent. 15. **Update-channel classification fails closed** (added 2026-07-05, BUG-023) — a missing or malformed `.channel` sidecar classifies as beta, never stable; only an explicit `stable` sidecar (written only by the atomic-publish path or an operator promotion) offers a build to the default fleet. --- ## History Highlights | Date | Event | |---|---| | 2025-12-15 | Project genesis: Windows service + Linux installer + site code auth + build server. | | 2026-04-19 | Full drill-down navigation, auto-install on first run, Pluto build VM setup started. | | 2026-04-21 | MSI build fix. DESIGN.md created. BirthBiologic onboarded. | | 2026-04-29 | UI_GAPS.md created. Holistic development principle formalized. | | 2026-05-12 | macOS agent Phase 1 deployed. Code signing issue on Apple Silicon noted. | | 2026-05-15 | Dead WebSocket write-half bug fixed. Temperature struct field mismatch fixed. | | 2026-05-16 | Watchdog bugs fixed. /feature-request skill created. | | 2026-05-17 | Syncro PSA Integration added to roadmap. Office power failure recovery. | | 2026-05-18 | Multi-tenancy architecture (ADR-001) decided. 5 SPEC documents created. | | 2026-05-19 | AD2 crash-loop fix. MSP360 integration completed. Process drill-down modal. | | 2026-05-23 | /rmm-audit pass. Agent optimization Phases 1A-3. MSRV bumped to 1.85. Fleet at 0.6.29. | | 2026-05-24 | Linux tray IPC + GTK merged. Build pipeline split per-platform. Fleet converged to 0.6.38. | | 2026-05-31 | Roadmap reconciliation. MSPBackups mapping/verify UI + dev-admin impersonation UI deployed. SPEC-021 written. | | 2026-06-01 | BUG-016/017 fixed. BSOD detection feature (Phase 1) implemented and merged, agent 0.6.51. | | 2026-06-02 | Server 0.3.37 + migration 048 deployed. Webhook wired to dispatch build-server.sh. | | 2026-06-04 | BUG-020 (ghost tray icons) fixed to beta (`137dd85`). | | 2026-06-07 | Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit all shipped same day. | | 2026-06-11 | Physical server migration executed (<27 min downtime). Ghost agent root-caused, durable identity Phase 1 Task 1 shipped. Agent-comms-durability Phase 1 (T1/T3) shipped (agent v0.6.63/server v0.3.68). | | 2026-06-12 | Beast parallel Windows build wired in (`cfbdb59`, ~336s). command_type "cmd" root-caused and fixed (`3de9faf`). | | 2026-06-15 | BSOD dedup key changed to bugcheck code. MSI EXDEV fix. sync.sh Phase 1 auto-heal. logs/analyze switched to Claude API. 500-error-leak closed. | | 2026-06-18 | sync.sh submodule auto-heal verified fleet-wide. | | 2026-06-21 | BUG-019/018 fixed and merged. Event Log Watch UI shipped. Howard cleared for GuruRMM merges/deploys. gururmm-build skill created. | | 2026-06-22 | BUG-021 (legacy dep-pin) fixed, v0.6.67 stable. BUG-022 (dead WatchdogEvent path) fixed. Fleet verified ~270/~178 online. | | 2026-06-23 | Universal self-detecting installer (Feature 9) P1 shipped (v0.6.71). av-removal-recipes spec added. | | 2026-06-24 | SPEC-030 remote software uninstall — BETA — code merged + deployed (Route B, async jobs, three-state KB, migrations 061-064). agent 0.6.75 / server v0.3.87. | | 2026-06-25 | Win11 KB5095093 Point-in-Time Restore captured as a Raw RMM_THOUGHTS entry. | | 2026-07-01..02 | Vendor-agnostic Integrations Framework shipped (migration 065) with CrowdStrike Falcon as the first plugin — API client, client mappings, periodic sync worker. No dashboard UI yet. | | 2026-07-03..04 | VSS Policy Configurator (SPEC-016 redesign) shipped to beta (agent 0.6.76) — replaces scheduled create+prune with a configure-governors-only model, eliminating the T1490 surface; 23-agent code review caught and fixed a data-loss-class `MaxShadowCopies` defect pre-merge; verified live on the T1490 canary NEPTUNE. | | 2026-07-04 | Legacy (2008R2/Win7, Rust 1.77) build variant disabled (`84a82a3`) after an edition2024 dep recurrence fail-closed the whole Windows build; last legacy artifact frozen at 0.6.76. RMM_THOUGHTS Feature 12 (legacy agent rewrite) added Raw. Cross-site agent dedup fixed (`2851523`, `1c1cac2`) — dedup by device_id across all sites, no re-home on reconnect. Policy-Controlled Tray Deployment merged (`f6a4251`) — materialize shipped fleet-wide, launch stays policy-gated default OFF. | | 2026-07-04..05 | Easy-bugs batch + full alert triage: update-scanner host-native/foreign self-check regimes (fixed macOS/Linux agents never receiving update offers), inventory NUL/control-char sanitize (`db/sanitize.rs`), WS benign-disconnect + pre-auth churn reclassified ERROR->INFO, `create_or_update_alert` survives cascade-delete races, `move_agent` maps duplicate-key to 409. Deployed as server v0.3.95 + v0.3.96. Active `server_error` alerts reduced to 0. | | 2026-07-05 (00:45-01:00 PT) | Repair batch 2 merged + deployed (`76a6bd2`, server v0.3.97, agent 0.6.78): comms-durability Phase 1 completed (T2 half-open eviction sweep); BUG-001 Windows WMI ACPI thermal SHIPPED and PROVEN live on 112+ agents; BUG-020 uninstall teardown fixed; BUG-018 delete-race resurrection paths closed; MSP360 transient-5xx downgraded to WARN. | | 2026-07-05 | BUG-023 discovered and fixed (Howard): update-channel publish/tag race had mass-offered unsoaked beta 0.6.78 to 209 stable agents; fixed with a fail-closed scanner default, an atomic sidecar write at publish, and a binaries-keyed `promote_version`. 0.6.78 manually promoted to stable, healing the split fleet (229 agents converged). | | 2026-07-05 | BUG-003 closed (Howard, server v0.3.98): deploy pipeline hardened with a build lock, PIPESTATUS-gated git reset, and pre-stop `--migrate-only` migration validation — the gate fired correctly on its first live run (aborted cleanly on a CI index.lock collision instead of building a stale tree). | | 2026-07-05 | BUG-018 altitude fix shipped (Howard, migration 066, server v0.3.98): `visible_agents` DB view + `BEFORE UPDATE` schema trigger replace nine scattered inline `status<>'deleting'` filters; all three WS auth paths reject mid-purge rows. | | 2026-07-05 (11:05 PT) | Bug-fix batch 4 merged (Howard, server v0.3.99): pending-command 7-day TTL, agent-status reconciler for zombie 'online' rows, stale pending-update supersede logic. BUG-024 discovered (CRITICAL, OPEN): Linux agent binaries built against the build host's newer glibc crash-looped every older-distro Linux agent that took the newly-enabled update offer; mitigated via quarantine, real glibc-floor fix pending. | | 2026-07-05 | Policy-core (policy folders + inheritance) deployed to production (Mike, merge `e7e90c2`, migration 067 renumbered from 066): per-client folder tree with inheritance resolution, versioning, scoped recompute+push, and a gated/dry-run backfill — non-breaking by construction. Follow-on policy-ui (dashboard UI) shape spec added same day (`01d962a`), not yet implemented. | --- ## Compilation Notes - **2026-07-05 recompile (Howard-Home/claude-main):** Full rebuild. Delta from 2026-07-04 (migrations 65 -> 67; agent 0.6.77 -> 0.6.78 fleet-wide stable; server 0.3.96 -> 0.3.99; dashboard v0.2.83 beta). Major additions: **Policy-core** (migration 067, `e7e90c2`) — new Capabilities subsection, locked decision #14, follow-on policy-ui gap; **four sequential bug-fix batches** (repair batch 2/v0.3.97, batch 3/v0.3.98, batch 4/v0.3.99) folded into Recent Work + Capabilities; **BUG-003 CLOSED** (was open); **BUG-018** corrected from the morning's status-write-guard fix to the full v0.3.98 schema-level view+trigger fix; **BUG-023 (NEW)** — channel-race incident caused by the 07-04 scanner fix, own Recent Work entry + anti-pattern + locked decision #15; **BUG-024 (NEW, OPEN, CRITICAL)** — Linux glibc incident, documented across Recent Work/Capabilities/Active State/Architecture; **comms-durability Phase 1 marked COMPLETE** (was T2/T3-planned). All prior anti-patterns/good-patterns preserved verbatim except one ("build classification defaulting to stable") marked superseded rather than deleted. - **2026-07-04 recompile (Howard-Home/claude-main):** Delta from 2026-06-25 (migrations 64 -> 65; agent 0.6.75 -> 0.6.77 modern / 0.6.76 legacy-frozen; server 0.3.87 -> 0.3.96; fleet 270 -> 359 enrolled). Added: Third-Party Integrations Framework, VSS Policy Configurator, Policy-Controlled Tray Deployment, legacy build variant disabled, cross-site agent dedup fix, 2026-07-04/05 easy-bugs + alert-triage batch. - **2026-06-25 recompile (Howard-Home/claude-main):** Delta from 2026-06-22 — SPEC-030 remote software inventory + bulk uninstall added as a major Capabilities subsection; universal self-detecting installer (Feature 9) P1 shipped; Current Focus rewritten; flagged the old "pending PRs #40-#46, migrations 061-063" prediction as superseded because software-removal work consumed 061-064 instead. - **2026-06-22 recompile (Howard-Home/claude-main):** Delta from 2026-06-11 — fleet ~270/~178, BUG-021/018/019 fixed on main, Event Log Watch UI shipped, BUG-022 dead-code watchdog path removed, Beast parallel two-wave build documented, command_type "cmd" alias + NAK documented, migration count to 60. - **2026-06-11 recompile (GURU-5070/claude-main):** Full recompile — physical server migration, durable agent identity, agent-comms-durability Phase 1 full detail, migration count to 59. - **2026-06-07 recompile:** Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit folded in; migration count to 55+. - **2026-06-04 recompile:** Corrected GURU-5070 channel state; BUG-020 documented. - **2026-06-02 recompile:** BSOD detection, server build webhook wiring, build channel default beta; migration count 46 -> 48. - **2026-05-26 recompile:** Added the Capabilities / Feature Set section, synthesized from authoritative artifacts. Changelogs are an unreliable capability source — migrations + commit log are authoritative. ## Backlinks - [[clients/cascades-tucson]] — RECEPTIONIST-PC enrolled (site CascadesTucson) - [[systems/gururmm-build]] — Physical server at 172.16.3.30; GuruRMM API 3001, ClaudeTools API 8001, Coord API, MariaDB, PostgreSQL, build pipeline - [[systems/jupiter]] — Unraid host at 172.16.3.20; virsh host for all VMs; Docker: Gitea, NPM, Seafile - [[systems/pluto]] — Windows build server (MSI, WiX) at 172.16.3.36