108 KiB
type, name, display_name, last_compiled, compiled_by, aliases, sources, backlinks
| type | name | display_name | last_compiled | compiled_by | aliases | sources | backlinks | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| project | gururmm | GuruRMM | 2026-07-05 | Howard-Home/claude-main |
|
|
|
GuruRMM
Summary
GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 360 enrolled agents (~229 fresh/online at last sample) across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints.
Current version (2026-07-05): agent 0.6.78 fleet-wide (Windows amd64, ~229+ agents, promoted stable 2026-07-05 after a channel-race incident — see BUG-023) / server v0.3.99 / dashboard v0.2.83 beta. Migrations end at 067_policy_folders.sql. Active server_error alerts: 0.
Policy folders + inheritance ("policy-core") deployed 2026-07-05 (Mike, migration 067, merge e7e90c2): the biggest architectural change of this compile window. An org-rooted policy folder tree now sits on top of the existing policy engine — each client gets a folder tree (exactly one root), any folder optionally references a library policy, and a device sits in exactly one folder, inheriting the merged policies of its folder ancestry (root->leaf) plus an optional per-asset override. This replaces the fixed System->Client->Site->Agent stack with a per-client tree while preserving identical effective-policy output during migration (the backfill is a deliberate, gated, dry-run-verified step — landing the migration changes zero behavior, since agents.folder_id is NULL fleet-wide until backfilled). Ships folder-tree CRUD + placement + version history REST API, recompute-and-push on policy edit (scoped to affected agents only), and policy versioning (policy_versions, immutable snapshot per edit, for diff/rollback). Migration renumbered 066->067 after colliding with 066_visible_agents_view.sql, which landed on main in the same window from an unrelated bug-fix arc. Explicitly out of scope for this phase: module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI, and any agent-side wire-format change (none — this is server+dashboard only). A follow-on policy-ui shape spec (folder management dashboard UI) was added the same day (01d962a) — implementation not started.
Four bug-fix batches shipped 2026-07-04..05 (server v0.3.95 -> v0.3.99, agent 0.6.77 -> 0.6.78): a dense back-to-back sequence of alert-triage, easy-bugs, repair, and issue-hunt passes. Highlights (full detail in Recent Work / History Highlights): update-scanner platform-coverage + channel-default fixes (macOS/Linux agents were never offered updates; then a scanner/build-pipeline race briefly mass-offered unsoaked beta to the stable fleet — BUG-023, fixed); inventory NUL-byte storage failures fixed for 7+ machines; WS log noise reclassified off ERROR; BUG-003 (unsafe deploy pipeline) closed with a build lock + gated git reset + pre-stop migration validation; BUG-018 (agent delete race) got its permanent altitude fix via a visible_agents DB view + schema trigger; BUG-001 (Windows ACPI thermal) shipped and was proven live on 112+ agents; comms-durability Phase 1 completed (half-open WS eviction sweep); a pending-command TTL, an agent-status reconciler, and update-supersede logic shipped in batch 4; and a critical Linux glibc incident (BUG-024, still OPEN/mitigated) crash-looped every older-distro Linux agent that took the newly-enabled Linux update offer.
Cross-site agent duplication fixed 2026-07-04: Root cause of ~40 duplicate agent rows (24 at Dataforth D2->D1 alone): both WS enrollment paths deduped only within (site_id, device_id), so re-running a different site's installer on an already-enrolled machine created a second row instead of finding the existing one. Fixed to match by hardware device_id across all sites and re-home via move_agent_to_site — but a same-day follow-up (1c1cac2) stopped the auto-re-home on every reconnect (the agent re-presents its install-site code as its API key on every beat, which was bouncing admin-moved agents back to their original site). Net effect: dedup by device_id, but site placement stays admin-controlled and persists.
Legacy (Server 2008R2/Win7, Rust 1.77) build variant disabled 2026-07-04: The unpinned legacy dependency re-resolve pulled edition2024 crates (chacha20/rand_core 0.10.1) that Rust 1.77 cannot parse — a BUG-021 recurrence that fail-closed the entire Windows build (blocking modern amd64/x86/tray artifacts too). Gated behind LEGACY_ENABLED=false; the last-built legacy artifact (agent 0.6.76) is preserved and frozen for existing 2008R2/Win7 endpoints. 2008R2 support returns via a separate legacy-only rewrite (C++/.NET or a clean minimal Rust core), not by re-enabling the 1.77 toolchain — captured as RMM_THOUGHTS Feature 12 (Raw, Mike's requirement).
Policy-Controlled Tray Deployment shipped 2026-07-04 (code merged, policy default OFF): The tray binary is now deployed to every Windows endpoint via the agent-update flow (previously only fresh MSI installs got it — fleet census found ~111/113 sampled hosts with no tray at all) but its launch is gated on tray.enabled policy — no process, no icon, until a client/site/agent policy turns it on. Authenticode-verified + version-locked to the agent. Fleet-wide policy enablement (Task 4) is not yet flipped.
VSS Policy Configurator shipped 2026-07-03 (SPEC-016 redesign, agent v0.6.76): The agent no longer runs its own scheduled shadow-copy create+prune loop — the MITRE T1490 "Inhibit System Recovery" surface that got the agent blocked by CrowdStrike Falcon. It now configures two kernel-enforced retention governors (VSS diff-area size cap + MaxShadowCopies count evaluated-but-not-machine-globally-written) and registers a scheduled task whose action is a native-COM create only (never a delete). A 23-agent code review caught a data-loss-class defect pre-merge (writing the machine-global MaxShadowCopies would have evicted OTHER products' shadows) — dropped in favor of the size cap as sole governor. Deployed to the beta channel only (0.6.76 at the time of that compile); stable-channel promotion of the VSS behavior riding on 0.6.78 has not been separately re-verified this compile.
Vendor-agnostic Integrations Framework shipped 2026-07-01..02 (migration 065, CrowdStrike Falcon first plugin): A generic per-partner plugin framework so a new EDR/security vendor is a new server module, not a new schema — modeled on the proven MSPBackups per-partner shape. CrowdStrike Falcon is the first plugin. REST CRUD at /api/integrations; no dedicated dashboard UI yet.
SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA, not guaranteed to work; Route B = server-orchestrated): unchanged status this compile — best-effort, not reliable for the hard cases. See Capabilities.
Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+): unchanged this compile.
Physical server migration completed 2026-06-11: New physical box at 172.16.3.30 running Ubuntu 26.04 + PostgreSQL 18. Old VM at .46 decommissioned 2026-06-12.
See also: wiki/projects/guru-rmm.md is a redirect tombstone pointing here (slug disambiguation).
Repo: azcomputerguru/gururmm on Gitea (internal: http://172.16.3.20:3000, external https://git.azcomputerguru.com/azcomputerguru/gururmm.git). The copy at projects/msp-tools/guru-rmm is a git submodule tracking the active repo; its pinned gitlink intentionally lags origin/main. Development uses git -C projects/msp-tools/guru-rmm or worktrees off origin/main to access live code. Howard is authorized for merges/deploys (cleared 2026-06-21 by Mike).
Goal: Full-featured MSP platform rivaling commercial RMMs, with a companion PSA (GuruPSA, separate future repo) designed as a truly integrated unified system.
Recent Work
2026-07-05 — Policy Folders + Inheritance ("policy-core") Deployed (Mike, merge e7e90c2, migration 067)
Replaced the fixed System->Client->Site->Agent policy stack with a per-client folder tree: each client gets exactly one root folder; any folder may optionally reference a library policy; a device sits in exactly one folder and inherits the merged policies of its folder ancestry (root->leaf) plus an optional per-asset override. Nine tasks: folder-tree schema + inheritance resolution + versioning (migration 067, renumbered from 066 after a same-window collision with 066_visible_agents_view.sql); recompute+push on policy edit scoped to affected agents only (not a fleet-wide recompute); folder tree REST API (/clients/:id/policy-folders, /policy-folders/:id[/rename|/policy|/parent], /agents/:id/folder); migration backfill job with a verification gate + dry-run mode (/policy-folders/backfill); integration tests. policy_versions gives every policy edit an immutable snapshot for diff/rollback and per-device "what changed."
Landing migration 067 changes zero behavior by construction: every new column/table is nullable and agents.folder_id is NULL fleet-wide until the backfill runs, so effective-policy resolution stays on the legacy Client->Site->Agent path until a deliberate, gated flip. Explicitly out of scope for this phase: module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI, and any agent-side change (wire format unchanged — server+dashboard only). A follow-on policy-ui shape spec (folder management dashboard UI: tree view, agent placement, effective-policy viewer with source attribution, version history, admin backfill trigger) was added the same day (01d962a) as an MVP largely assembling existing dashboard components (ContextTree, PolicySectionEditor, EffectivePolicyTable) — not yet implemented.
2026-07-05 — Bug-Fix Batch 3: BUG-023 Channel Race, BUG-003 Closed, BUG-018 Altitude Fix (Howard-Home, merge 6ccd35e, server v0.3.98)
Post-deploy verification of the prior night's repair batch 2 came back clean, but the new T2 half-open-eviction sweep surfaced an anomaly: 127 evictions, 79 of them one agent (DESKTOP-3USU20B, Safesite Glendale) cycling connect->auth->silent-90s->evict->reconnect every ~6 minutes, plus 3 other chronic loopers. Root-causing why the loopers never got 0.6.78 uncovered BUG-023: a publish/tag race in the build pipeline. Builds are tagged beta by a late "MARK AS BETA" phase minutes after binaries land in the downloads directory, and the scanner defaulted a missing .channel sidecar to stable — so the 07:38 scan classified the untagged 0.6.78 as stable and mass-offered it to 209 stable-channel agents before the 07:43 scan reclassified it beta, splitting the fleet (214 agents on unsoaked-beta 0.6.78, ~31 stragglers pinned with no path forward). Fixed in three layers: the scanner now fails closed (missing/malformed .channel => beta, with a regression test) so the race is harmless even if a future pipeline step crashes between publish and tag; build-windows.sh's deploy_and_sign writes the beta .channel sidecar atomically with each published artifact, closing the timing window at the source; and promote_version was rewritten to key on agent binaries rather than pre-existing sidecar files (the old logic only rewrote files whose content was exactly beta, making fail-closed sidecar-less binaries permanently unpromotable) — it now creates/overwrites stable sidecars and a partial write fails the request (500) instead of recording a stable rollout row that disagrees with disk. With Howard's go, 0.6.78 (already soaked 8+ hours on 214 agents) was manually promoted to stable, healing the split fleet (229 fresh agents converged by session end).
Also shipped in the same batch: the BUG-018 altitude fix — migration 066_visible_agents_view.sql (a visible_agents DB view filtering status<>'deleting', replacing nine scattered inline filters including a previously-missed agent-detail-page query) plus a BEFORE UPDATE trigger pinning status='deleting' at the schema level, and all three WS authenticate_agent paths now reject a mid-purge row instead of binding a live session to an agent about to be hard-deleted. A 15-agent code review (22 verified, 9 findings, 1 refuted) drove a hardening commit: set_ignore_missing(true) on the migrator (so a rollback binary tolerates an ahead DB instead of VersionMissing-crash-looping — the review's top finding), and a pre-existing matches_pattern panic on no-wildcard patterns was fixed (previously hidden among "19 known" test failures — really 18 DB-env + 1 real bug).
BUG-003 closed: deploy/build-pipeline/build-server.sh (confirmed this session to auto-sync to /opt/gururmm on every push) gained a flock build lock, PIPESTATUS-gated git fetch/reset, and pre-stop migration validation via a new --migrate-only server flag (validates+applies migrations against the live DB while the OLD binary keeps serving, then exits without binding a port). The gate fired correctly on its first real run: the merge webhook collided with the CI version-bump job's index.lock and aborted cleanly with nothing deployed — where the old script would have silently built a stale tree. Deployed as server v0.3.98 (16:46 UTC), 0 ERROR lines, view/trigger verified live in a rolled-back test transaction. DESKTOP-3USU20B's one-way-WS loop was fingerprinted as per-site network behavior (agent->server frames eaten post-auth; server->agent and REST both fine; reproduces on 0.6.66, 0.6.77, and 0.6.78 alike) — 4 chronic loopers total, telemetry exhausted server-side, packet capture is next. RMM live temperature reading was also spot-verified against ACG-Tech03L's (Snapdragon WoA) local sensor panel: zone-for-zone match, values within tenths of a degree.
2026-07-05 — Bug-Fix Batch 4: Command TTL, Status Reconciler, Update Supersede; BUG-024 Linux glibc Incident (Howard-Home, merge c34194a, server v0.3.99)
A live-system "what's not working as expected" sweep found and shipped three fixes, twice review-hardened (16 findings addressed across two workflow passes):
- Pending-command 7-day TTL: commands could sit
pendingindefinitely and execute on reconnect regardless of age (May-dated commands would have fired on a stale agent's reconnect). The reaper now expires anything past 7 days with an honest distinct message for never-sent vs. unconfirmed-delivery rows; the first tick expired 10 stale rows. - Agent-status reconciler:
agents.statuswas never reconciled againstlast_seen, leaving zombie'online'rows (11 found, one stranded since June 2). A new sweep flips status-only (never touchinglast_seen, after a review finding caught an earlier version fabricating freshness and auto-resolving offline alerts), respects the shared 2-hour update-hold window, skips agents with a live dispatch-map connection, and broadcasts the change via event + SSE. First post-warm-up sweep flipped 4 stranded rows. - Stale pending-update supersede: re-dispatching a pending update at auth time now supersedes it when a newer version exists or the target binary was pruned from disk (all 13 previously-pruned targets were being blindly re-offered); guarded to only fire on
pendingrows over an hour old with a non-empty platform index.
BUG-024 (NEW, OPEN, mitigated) — Linux glibc incident: Linux agent binaries >=0.6.77 were built against the build host's glibc 2.43 (the box is now Ubuntu 26.04), and the 2026-07-04 scanner fix enabled Linux update offers for the first time ever — so every older-distro Linux agent that took the offer crash-looped on GLIBC_2.39 not found: ix.azcomputerguru.com (CloudLinux 9, 3898 restarts), the Jupiter agent container (Debian 12, 623 restarts), a Lonestar container, and presumed SL-SERVER (Scileppi). The Unix rollback watchdog never fired on either affected box (a distinct sub-bug, uninvestigated), and the 0.6.59 container agent still performs the BUG-019 ephemeral in-place update, so the bad binary persisted in the writable layer across restarts. Mitigated: Linux binaries quarantined out of the downloads directory (scanner now offers nothing for Linux); ix and the Jupiter container were manually revived from their pre-update backups and reconnected healthy. SL-SERVER and the Lonestar container host remain down pending manual recovery (no vault SSH creds for either). Real fix pending: a glibc compatibility floor for Linux builds (old-glibc build container, e.g. rockylinux:8/debian:11, or a switch to musl static linking) plus an objdump -T max-GLIBC build gate — tracked as roadmap BUG-024. Also this pass: 4 orphaned duplicate agent rows deleted (zombies with fresher same-site siblings); .30's gururmm-agent.service found disabled and re-enabled; RECEPTIONIST-PC's two rows at Cascades confirmed to be two real machines sharing a hostname (rename recommended, not a bug). Operational flags relayed to Howard/Mike: LAB-SVR offline 17.7 days with an active critical alert; KSTEENBB2025 peaking 83.8C cpu_temp; a stale repo-root build-server.sh lineage is a deletion candidate (the canonical script lives under deploy/build-pipeline/).
2026-07-04..05 — Repair Batch 2: Comms-Durability Phase 1 Complete, BUG-001 WMI Thermal, Uninstall Teardown (Howard-Home, merge 76a6bd2, server v0.3.97, agent 0.6.78)
Comms-durability Phase 1 completed: a gap analysis found T1 (30s agent heartbeat) and T3 (ACK'd durable delivery, migrations 058/059) already live; this batch implemented the missing T2 — a half-open-connection eviction sweep (evict_half_open_connections, 90s no-inbound threshold, 30s sweeper cadence) — closing out the phase. SO_KEEPALIVE was deliberately skipped (the TCP peer is the NPM proxy, not the agent).
BUG-001 (Windows WMI ACPI thermal) SHIPPED and PROVEN: collect_temps_windows_acpi() reads MSAcpi_ThermalZoneTemperature via WMI (root\WMI), converts tenths-Kelvin, applies a -20..125C plausibility filter, and reports the hottest zone as cpu_temp with a sysinfo fallback. Verified live on 112+ Windows agents reporting cpu_temp within 10 minutes of deploy (was 0 fleet-wide since LibreHardwareMonitor's CVE-2020-14979 removal on 2026-05-27); sensor-for-sensor matched a local panel on a Snapdragon WoA machine (ACG-Tech03L) including correctly filtering frozen 0-Kelvin zones. A review finding moved the WMI query to spawn_blocking with a 5-second deadline so a wedged winmgmt service cannot stall heartbeats.
BUG-020 uninstall teardown fixed: the real gap (policy-disable/swap terminate_all wiring was already correct; the roadmap note calling it broken was stale) was the uninstall path leaving the watchdog service and gururmm-tray.exe behind. Fixed ordering: remove_watchdog_service() first, then TrayLauncher::terminate_all().await, then service delete, plus removal of the tray binary from disk.
BUG-018 delete UX: delete_agent now marks status='deleting' synchronously before the 202 response; all listing/stat queries exclude 'deleting' rows; the dashboard does an optimistic cache removal. Three resurrection paths that could flip a mid-purge row back to 'online'/'offline' (heartbeat, metrics, and offline-grace writes) were closed by making update_agent_status refuse to overwrite 'deleting'. A zombie-connection check in the WS read loop now closes any connection no longer matching the agent's currently-registered one (fixes a permanent-orphan class: agent shown online in the DB, absent from the dispatch map, commands pending forever). Startup reconciliation re-runs the purge for any row stuck 'deleting' after a mid-purge restart. (Full altitude fix via the visible_agents view followed the next day — see Bug-Fix Batch 3.) MSP360 sync noise: transient 5xx from MSP360 now logs WARN, escalating to ERROR only after 4 consecutive failed cycles (~1h), so real outages still alert without every cloud-maintenance blip minting a server_error.
A 26-candidate, 10-finding code review preceded merge; all 8 findings attributed to this batch were fixed pre-merge (238aad7). All 4 build targets went green: server v0.3.97, Windows agent 0.6.78 (wmi crate compiled clean), Linux 0.6.78, dashboard v0.2.83 beta.
2026-07-04..05 — Easy-Bugs Batch + Alert Triage: Update-Scanner, Inventory NUL, WS Noise (Howard-Home, merges 519eef2/f2d02dd, server v0.3.95/v0.3.96)
Full 24h alert triage (GET /api/alerts) surfaced 5 backup_failed (client-side, unresolved — see Active State), 2 active server_error (product bugs), and 1 resolved mass_offline_fleet (36 agents, transient network blip). Root-caused both server errors against the server's own self-log (pseudo-agent 00000000-...-001):
move_agent500 -> 409: moving an agent into a site that already holds a row for the samedevice_idtrippedidx_agents_site_deviceas a raw 500; now maps to 409 with an actionable message. 13 duplicate-hostname agent pairs fleet-wide are the trigger (stale legacywin-*/hw-*device_ids alongside re-enrolled hardware-UUID records) — not yet cleaned up.- Offline-sweep
RowNotFound:create_or_update_alert's UPDATE-by-id usedfetch_one; an alert row deleted mid-sweep (agent-delete cascade) aborted the whole sweep. Switched both branches tofetch_optionalwith fall-through to INSERT (Result<Option<Alert>>,Ok(None)= agent gone). - Inventory NUL rejection (alert-invisible, found in logs only): 7+ agents (IMC1, Seth-PC, QWM-JOHN, QWM-SHEILA, goldstar19, SIF-SERVER, Christine-Win10) never persisted hardware inventory — Windows registry strings carry embedded NULs, Postgres jsonb/text rejects them. New
server/src/db/sanitize.rs(strip_nul,clean_control_chars) applied over 5 jsonb blobs (values + keys, collision-safe) + text columns. - Update-scanner warn-spam / silent update starvation: the self-check tried to execute every platform's binary to read its embedded version — macOS/Linux binaries can't execute on the Linux build host, so they were dropped from the scannable set entirely, meaning macOS/Linux agents were never offered updates. Fixed with host-native-vs-foreign regimes: foreign-platform binaries trust the filename version; host-native binaries stay a hard gate (a native binary that can't report its version is corrupt) with a best-effort exec-bit repair for 0644 CI-copy artifacts.
A concurrent same-branch session absorbed a prior session's in-progress inventory/scanner fixes; the combined diff went through a 19-agent high-effort code review (9 confirmed findings, 0 refuted) before merge — findings included rebuilding the alerts fall-through (it still failed its own motivating scenario on first pass), tightening the scanner's foreign/host-native split, and moving WS-noise logging from DEBUG to INFO (a mass-disconnect anomaly must stay visible at the default log level). Merged + deployed as server v0.3.95 (06:09 UTC): scanner now indexes 8 platform/arch combos (was 4, Windows-only); zero ERROR lines post-deploy.
Continued into the last stale alert: "Authentication timeout" WS churn (2-6/day baseline, 57-spike correlated with the same day's fleet-dedup work). Profiled 7 days of journal, concluded pre-auth connection noise (peer connects, never authenticates — sleeping laptops, scanners, half-open NAT), added is_preauth_connection_churn to classify it INFO while keeping genuine rejections (bad API key, malformed auth) at ERROR. Deployed as server v0.3.96 (06:23 UTC). Active server_error alert count: 0.
BUG-009 (Logs.tsx isError handling) and BUG-011 (: any occurrences) were verified already fixed in-tree during the same pass — only the stale FEATURE_ROADMAP.md status lines needed correcting, not the code. Remaining/pending from this session, superseded by later batches where noted: machine-side backup failures (GND-SERVER, AD1, LAB-SVR, DESKTOP-EVA4H1A) stayed open; 13-pair duplicate-agent cleanup (4 orphaned rows cleaned in batch 4, full pass still open); WS auth-timeout narrowed to pre-auth churn here, but 4 specific chronic loopers were only fingerprinted, not fixed, in batch 3.
2026-07-04 — Policy-Controlled Tray Deployment (Howard-Home / Mike, merged f6a4251)
Made gururmm-tray.exe deployment (materialize, policy-agnostic) independent from its runtime (launch, policy-gated) — previously the watchdog launched the tray whenever the binary existed, with no enforcement of the already-wired tray.enabled policy flag, and the binary itself only ever reached fresh MSI installs (fleet census: 1 signed tray, 1 stale unsigned, ~111 none out of 113 sampled). A first design (inline tray refresh folded into do_update) was rejected by a 17-agent code review plus independent Gemini 3.1 Pro and Grok adversarial critiques — 9 verified defects including permanent-binary-loss on a failed swap, a checksum sourced from an unauthenticated download-host sidecar (supply-chain gap), and an update/rollback tray-agent version-skew case.
Corrected design shipped as Tasks 1-3 + 5: policy-gated launch (default OFF), a materialize FSM with staging + a versioned mutex (atomic swap, restore-on-failure, authenticated-payload checksum), server capability-gated materialize, and Authenticode-verify + observability. Task 4 (flip fleet-wide materialize on, policy still OFF) is the deliberate next step — not done yet. Merged to main 2026-07-04 12:23 PDT.
2026-07-03..04 — VSS Policy Configurator (SPEC-016 Redesign) (Mike, merged de30b2b)
Rebuilt the agent's Volume Shadow Copy handling from "scheduled create+prune operator" (the MITRE T1490 surface Falcon blocked) to "policy configurator, never a deleter." Tasks 2-7: vss-create verb + scheduled-task registration (both daily-time and every-N-hours trigger modes) with legacy-task migration; per-volume diff-area size-cap governor set on policy apply; retirement of the scheduled create+prune pass entirely (SPEC-025 compliance heal reworked to create-only); status/compliance reporting updated to the configurator model; dual-target release build; runtime verification on both a Server 2019 box and a Win11 Pro client with Falcon present.
A 23-agent workflow code review before merge caught a genuine data-loss defect: the original design would have set the machine-global MaxShadowCopies registry value to retention_count, which FIFO-evicts other VSS consumers' (System Restore, third-party backup) shadows below that count fleet-wide. Fixed by dropping MaxShadowCopies management entirely — the per-volume size cap is the sole governor; retention_count/retention_max_age_days become advisory (a warning fires if retention_max_age_days is still set, since age-pruning requires deletion). Nine other findings fixed (cap-before-create ordering, an alias bridging the legacy task-name upgrade window, interval-aware compliance windows, bounded IVssAsync wait instead of INFINITE, and others).
Merged to main; the build pipeline publishes to the beta channel, not stable (a mid-session correction) — beta head reached 0.6.76 (8 agents), stable stayed at 0.6.66 (233 agents) by design. Verified live on the T1490 canary NEPTUNE (moved to beta channel for the test): create task present, legacy GuruRMM-VSS-Snapshot task removed, MaxShadowCopies correctly left unset, 17 pre-existing shadows preserved (the data-loss fix proven in production), and a live vss-create produced new shadows on two volumes under Falcon with no T1490 trip.
2026-07-01..02 — Vendor-Agnostic Integrations Framework + CrowdStrike Falcon Plugin (Mike)
Built a generic third-party-integration plugin framework (migration 065: integrations, integration_credentials, integration_client_mappings, integration_sync_state, integration_agent_links) so CrowdStrike Falcon — and future vendors (SentinelOne, Guardz) — are a new server module rather than a new bespoke schema each time, generalizing the per-partner MSPBackups shape (migration 034). Credentials reuse the same AES-256-GCM Encryptor chokepoint as the existing credentials table.
Five tasks, one per day-slice: (1) framework schema migration; (2) plugin trait + registry + DB layer; (3) generic REST API (/api/integrations CRUD, credentials, client mappings, sync-state, connection test); (4) CrowdStrike Falcon API client + plugin registered against the framework; (5) periodic sync worker pulling Falcon sensor state (version, status, RFM flag, CID) into integration_agent_links. No dedicated dashboard UI yet — API/backend only as of this compile.
2026-06-23..24 — SPEC-030 Remote Software Uninstall + Universal Installer (Howard-Home)
SPEC-030 software inventory + remote bulk uninstall (Route B — BETA, code merged to main + deployed but NOT guaranteed to work; agent 0.6.75 / server v0.3.87):
- Capability shipped:
/agents/:id/softwarelists installed programs;/agents/:id/software/uninstallruns a bulk uninstall; the dashboardSoftwareManager.tsxshows the live programs list + bulk-select + a live progress bar. Server endpoints inserver/src/api/software.rs. - Silent-uninstall engine (
agent/scripts/uninstall-engine.ps1): Tier-1 silent removal, BCU-informed detection + fail-fast, vendor silent-switch table, captures real process exit codes (fixed aStart-Processnull-ExitCode bug,f6c1945). The engine refuses to guess — Avast/AVG (icarus.exe), Malwarebytes (mb5uns.exe) correctly returnedneeds_interactiverather than risk a bad removal. - False-success class fixed (CRITICAL, PR #53): Avira's WiX/Burn Package Cache bundle GUID was misread as an MSI ProductCode;
msiexec /x {bundleGuid}returned 1605 ("not installed") and was mapped to success="already gone" while Avira stayed fully installed. Fix: skip the MSI tier for\Package Cache\bundles + a generic post-success ARP re-check (Test-StillInstalled) that downgrades a reported success to failed if the program is still registered. - Code-review correctness fixes (PR #54):
Test-StillInstalledmatches byproduct_codealone when present; skip the rescan on reboot-pending verdicts (3010/1641); exclude MSI-1605 no-op successes from the learned-timing sample. - Async removal jobs — the permanent fix (PR #56, migration 064): A synchronous bulk uninstall died at Cloudflare's ~100s timeout AND the agent's command kill-timeout killed the engine mid-flush. Rebuilt as a JOB: POST creates a
software_removal_jobsrow and returns ajob_idinstantly (0.06s); a background worker dispatches one program at a time;GET .../uninstall/jobs/:job_idis polled by the dashboard. Stop-gap flush-slack (PR #55) shipped first. - Three-state removal knowledge base (migrations 061-063):
software_removal_attempts(per-agent removal-tracking loop),software_knowledge(fleet KB:silent/requires_ui/unknown),software_knowledge_timing(self-tuning per-target timeout). - Known gaps surfaced (not yet built): drivers/system components register in ARP and appear in the bulk list; Launchy/AIMP-class uninstallers hang the agent on a lingering child process (~420s kill-timeout); no list-jobs endpoint (get-by-id only).
Universal self-detecting installer (Feature 9 — P1 built+deployed+verified, v0.6.71):
- One install path self-detects arch/OS (x64/x86/legacy/ARM) and pulls the correct agent; dashboard install UI repointed at it. Script path proven on-box (GND-JWILL). P2 (native i386 bootstrapper EXE) + offline-bundle still pending.
RMM_THOUGHTS movement: Feature 10 (AMPIPIT recovery environment add-on) -> Discussed (Mike likes it, deferred); Third-Party AV Removal Recipes + Win11 KB5095093 Point-in-Time Restore added as Raw; Feature 11 (OS Updates Management, Windows Update P1/P2 + Linux/Mac update-check P3, Syncro-modeled) added Raw 2026-07-03; Feature 12 (legacy 2008R2/Win7 agent rewrite) added Raw 2026-07-04 after the legacy build was disabled.
2026-06-22 — BUG-021 Fixed + v0.6.67 Stable; Fleet Verified (Howard-Home)
- BUG-021: Legacy build (Rust 1.77 via
$CARGO +1.77 --features legacy) was pullingwit-bindgen(edition2024) through fresh dep resolution. Fixed by pinninggetrandom 0.3.1+zeroize 1.8.1below edition2024. Commit1dce66d. Windows build went green 2026-06-22 02:19, v0.6.67. (This class of failure recurred 2026-07-04 with a different dep pair and led to the legacy variant being disabled outright — see Recent Work above.) - Fleet verified: ~270 agents / ~178 online; metrics rows flowing; all alerts have non-null dedup_keys; per-agent endpoints all HTTP 200.
- BUG-022 filed + fixed (PR #45):
WatchdogEventWS variant was dead code (watchdog has no WS connection); removed. Emptywatchdog_eventstable left in place. - Roadmap verified: BUG-018 (
cea87d4) and Event Log Watch UI (0fa65f5) confirmed on main.
2026-06-21 — Feature Batch, Audit, BUG-018 202+bg, Event Log Watch UI, Submodule Fix (Howard-Home + Mike GURU-5070)
Howard-Home sessions (three spans):
- BUG-019 FIXED (commit
66a7f4e, merged to main by Mike): Container agent no longer performs in-place binary self-update.agent/src/updater/mod.rsperform_update()early-returns inside a container. - BUG-018 FIXED (commit
cea87d4, merged): Handler redesign to 202 + background purge. Migration 061 (FK indexes on five previously-unindexed FK columns) landed later under a renumbered slot once SPEC-030 consumed 061-064. - Event Log Watch management UI (SHIPPED,
0fa65f5):/event-log-watchesCRUD wired into the dashboard. - Enrollment modal UX fix (SHIPPED to prod,
4027c86): close button, click-outside/Esc dismissal, list refresh on close. - Event Log Watch policy-clobber HIGH fix (PR #43,
432b434): Assigning/unassigning a policy silently wiped an agent's Event Log Watch monitoring — third instance of a config-push clobber class. Fixed by reusingwatch_rules_for_agentas the single source of truth. - sync.sh submodule-clobber root-cause fix: populate-only guard prevents
git submodule update --initfrom re-checking out the intentionally-lagging pinned gitlink on every sync. - Dashboard beta-before-main rule established.
Mike GURU-5070 session:
- BUG-019 merged to main (fast-forward
ed8cad3 -> 66a7f4e). gururmm-buildskill created: local pre-merge verification (verify.sh server|agent|dashboard|migrations).docs/BUILD.mdadded.- Howard cleared for GuruRMM merges/deploys (Mike decision).
2026-06-15 — BSOD Dedup Key Fix, MSI EXDEV Fix, Dashboard Offline Dedup, sync.sh Auto-Heal (Mike GURU-5070 + GURU-BEAST-ROG)
- BSOD alert dedup key fixed (commit
f0a4b7f, server v0.3.73): Dedup key changed tobsod:<agent>:<bugcheck_code>(stable across recurrences). - MSI EXDEV fix (commit
95ef901, server v0.3.74): Site-MSI builder staged in/tmp(tmpfs) then renamed cross-device to/opt/gururmm/downloads, failing EXDEV. Fixed to stage on the same filesystem. - Duplicate offline server alerts removed (commit
b6ed564). - sync.sh auto-heal (Phase 1):
resolve_submodule_collisions()moves only genuinely-colliding untracked files aside and retries, instead of a blind force. logs/analyzeswitched from Ollama to Claude API (commitc869e4d).- 500-error-leak fix (commit
58c1a96): All server error paths route throughinternal_err.
2026-06-12 — Beast Parallel Windows Build (Lever A) + command_type "cmd" Misdiagnosis Fix (Mike GURU-5070)
Beast parallel Windows build (commit cfbdb59): Two waves (5 concurrent stable + 2 concurrent legacy-1.77) via bash-driven concurrent SSH instead of a serial cmd /c chain. Per-variant --target-dir mandatory. Result: ~336s parallel vs ~622s serial cold, ~3.8x vs Pluto serial.
command_type "cmd" fix (commit 3de9faf, agent v0.6.66 stable): CommandType enum had no cmd variant — server-sent command_type:"cmd" failed serde parse and the agent silently dropped the message. Fixed with #[serde(alias = "cmd")] on Shell + a NAK on unparseable commands.
2026-06-11 — Physical Server Migration, Durable Agent Identity, Agent Comms Durability Phase 1
- Cutover: New physical box took over 172.16.3.30 (PG 18, Ubuntu 26.04); production down <27 min; old VM decommissioned 2026-06-12.
- Durable agent identity Phase 1 Task 1: Multi-location
device_id(registry+ProgramData on Windows, /etc+/var/lib on Linux, /Library+mirror on macOS) with self-heal;cleanup.ps1whitelists identity files. - Agent Comms Durability Phase 1 (agent v0.6.63 / server v0.3.68): Root cause = NAT conntrack asymmetry (server->agent pushes black-holed in idle conntrack gaps). Fix: agent
CommandAckon receipt + dedup cache; server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via a partial index so old agents keep legacy behavior. (Phase 1 fully completed 2026-07-05, see Repair Batch 2 above — the T2 half-open eviction sweep was the missing piece.)
2026-06-07 — Credential Inheritance, Offboarding Spec, UI Gap Batch, Backup Alert Quality Pass
- Server v0.3.45: credential inheritance (Global -> Client -> Site),
/effectiveendpoints. - Role-aware offline alerting + alert ignore/mute shipped (server-only
agent_offline,alert_mutesperma-silence). - Backup-alert quality pass: false
backup_failed15 -> 2 fleet-wide;backup_storage_lowalert type removed (false-positive metric). - UI gap batch + enrollment audit; SPEC-028 offboarding wizard specification (835 lines, implementation pending).
Capabilities / Feature Set
Synthesized from authoritative artifacts (API routes, agent modules, 67 migrations, roadmap, commit log) — not from session logs alone. See Compilation Notes.
Agent<->server communication is a persistent authenticated WebSocket with auto-reconnect + heartbeat; on reconnect, in-flight commands flip to interrupted. Platform-parity rule: agent features ship on Windows/Linux/macOS in the same change (stub + TODO where a real impl isn't yet feasible).
Monitoring & Telemetry
- Core metrics per interval (policy-tunable): CPU %, memory %/bytes, disk %/bytes, network rx/tx deltas, uptime, logged-in user, user idle time, public/WAN IP. Cross-platform via
sysinfo. - Windows CPU/thermal (BUG-001, shipped + PROVEN 2026-07-05, server v0.3.97):
collect_temps_windows_acpi()readsMSAcpi_ThermalZoneTemperaturevia WMI (root\WMI,spawn_blocking+ 5s deadline so a wedgedwinmgmtcan't stall heartbeats), tenths-Kelvin conversion, -20..125C plausibility filter, hottest zone reported ascpu_temp,sysinfo::Componentsfallback. Verified fleet-wide: 112+ agents reporting within 10 minutes of deploy (was 0 since LibreHardwareMonitor's CVE-2020-14979 removal 2026-05-27); sensor-for-sensor match against a local panel on Snapdragon WoA hardware (ACG-Tech03L, 20 valid zones, frozen 0-K zones correctly filtered). GPU/NVAPI thermal still open. - Process drill-down: top 10 by CPU + top 10 by memory in every metrics payload.
- Network state: per-interface IPv4/IPv6, MAC, derived CIDR subnets — sent on change only.
- Windows BSOD/kernel-crash detection:
agent/src/bsod.rspollsC:\Windows\Minidump, parses kernel dump header at fixed offsets, cross-references System event log, deduplicates via a watermark file. Always-Critical alert deduplicated by(agent_id, bugcheck_code)(changed from dump-hash 2026-06-15 so a mute silences all recurrences). - Inventory NUL/control-char hardening (2026-07-05, server v0.3.95):
server/src/db/sanitize.rsstrips NULs and C0/C1 control chars from inventory jsonb values/keys and text columns before upsert — fixed silent storage failure on 7+ machines carrying embedded NULs in Windows registry strings.
Remote Execution
- Command types:
shell,powershell,python, rawscript{interpreter},claude_task, andcmd(alias forshell). Options:timeout_seconds,elevated. Unknown/unparseable commands NAK immediately. - Execution context (
041):system(default) oruser_session(WTS token impersonation, Windows-only). - Commands individually cancellable, aborted on disconnect. Auditable history. Script library (
017) for stored scripts. - Agent Comms Durability — Phase 1 COMPLETE as of 2026-07-05 (agent v0.6.63.., server v0.3.68->v0.3.97): T1 (30s agent heartbeat) and T3 (
CommandAckon receipt, migration 058acked_at, dedup via a recent-results cache, reaper re-delivers un-acked commands past a 60s deadline) shipped 2026-06-11; T2 (server-side half-open-connection eviction sweep, 90s no-inbound threshold, 30s cadence,conn_id-guarded) shipped 2026-07-05 closing out the phase. Capability-gated viaidx_commands_agent_ackedpartial index so pre-durability agents keep legacy fail-on-timeout behavior. Phase 2 (live TTY) and Phase 3 (adaptive keepalive, bulk->HTTPS) remain PLANNED, not shipped. - Pending-command TTL + status hygiene (batch 4, server v0.3.99): commands stranded
pendingpast 7 days are expired by the reaper (previously unbounded — a May-dated command could still fire on a stale agent's reconnect) with distinct never-sent vs. unconfirmed-delivery messaging.agents.statusis now reconciled againstlast_seenby a periodic sweep (status-only conditional flip, preserves reallast_seen, respects the 2h update-hold window, skips live dispatch-map connections, full event+SSE broadcast) — fixes zombie'online'rows left by unobserved disconnects.
Volume Shadow Copy / VSS (Windows) — Policy Configurator (SPEC-016 redesign, shipped 2026-07-03, agent v0.6.76+)
- The agent no longer schedules its own create+prune loop. It is a configurator: on each policy apply it idempotently sets two kernel-enforced retention governors — VSS diff-area size cap (native
IVssDifferentialSoftwareSnapshotMgmt) andMaxShadowCopiescount (evaluated, but NOT machine-globally written — see below) — and registers a scheduled task whose action is the agent's own native-COM create (IVssBackupComponents::DoSnapshotSet,VSS_CTX_CLIENT_ACCESSIBLE), never a delete. Retention is enforced by Windows' own FIFO eviction (volsnap.sys) when a governor is hit, eliminating the MITRE T1490 "Inhibit System Recovery" surface that had gotten the agent blocked by CrowdStrike Falcon. - Works uniformly on server AND client SKUs (previously
vssadmin create shadow, Server-only, gated client support) — the underlying VSS COM API andvolsnapcap eviction are not SKU-restricted, only thevssadmin.exeCLI is. MaxShadowCopiesis NOT set machine-globally (pre-merge code review caught this as a data-loss defect — it would FIFO-evict OTHER VSS consumers' shadows, e.g. System Restore or third-party backup, below the configured count). The per-volume size cap is the sole enforced governor;retention_count/retention_max_age_dayspolicy fields are advisory/ignored (a warning fires ifretention_max_age_daysis still set, since age-based pruning requires deletion).- On first policy apply, the legacy
GuruRMM-VSS-Snapshotscheduled task (present fleet-wide) is removed and replaced withGuruRMM-VSS-Create, supporting both daily-time and every-N-hours interval triggers. - On-demand admin operations (create/delete/browse/restore via
dispatch_operation,GET /agents/:id/vss/restores) are unaffected — this redesign only removes the scheduled create+prune loop. - Deployed to beta (0.6.76) at ship time; agent has since advanced to 0.6.78 stable-wide via later bug-fix batches — stable-channel behavior on 0.6.78 not separately re-verified this compile [verify].
Software Inventory & Remote Uninstall (SPEC-030 — BETA, not guaranteed)
- Maturity: code merged to main + deployed (2026-06-23..24), but the feature is beta and unreliable — best-effort only.
- Route B (server-orchestrated):
GET /agents/:id/software,POST /agents/:id/software/uninstall(async ->job_id),GET /agents/:id/software/uninstall/jobs/:job_id,GET /agents/:id/software/removal-status+ resolve,GET /software/knowledge+ classify. - Silent-uninstall engine (
agent/scripts/uninstall-engine.ps1): refuses to guess (needs_interactiveon ambiguity); generic post-success ARP re-check (Test-StillInstalled) catches the WiX/Burn Package-Cache-bundle false-success class; orphaned ARP entries classifiedtier=orphaned; async-fork uninstallers get retry-verify. - Async removal jobs (migration 064): non-blocking bulk uninstall — job created +
job_idreturned in ~0.06s, background worker processes targets one at a time, dashboard polls live progress. - Fleet removal knowledge base (migrations 061-063): per-agent removal-tracking loop, fleet three-state KB (
silent/requires_ui/unknown), self-tuning per-target timeout. - Known gaps: drivers/system components register in ARP and appear in the bulk list (no exclusion/flag yet); Launchy/AIMP-class uninstallers hang the agent on a lingering child process; no list-jobs endpoint (get-by-id only). The av-removal-recipes spec (P3, removal-only) is the follow-on.
Third-Party Integrations Framework (migration 065, shipped 2026-07-01..02)
- Vendor-agnostic plugin framework:
integrations(one enabled plugin per partner+kind),integration_credentials(AES-256-GCM encrypted, sameEncryptorchokepoint as thecredentialstable),integration_client_mappings(RMM client -> external tenant id, e.g. a Falcon CID — single-CID partners map every client to one id, MSSP partners map child CIDs),integration_sync_state(per-integration sync health),integration_agent_links(RMM agent <-> external agent id + last-synced state, e.g. Falcon AID/sensor version/RFM). - API:
GET/POST /api/integrations,GET/DELETE /api/integrations/:id,GET/PUT /api/integrations/:id/credentials,GET/PUT /api/integrations/:id/mappings,DELETE /api/integrations/:id/mappings/:mapping_id,GET /api/integrations/:id/sync-state,POST /api/integrations/:id/test. - First plugin: CrowdStrike Falcon. API client + periodic sync worker (default
sync_interval_seconds=900) pulls sensor state intointegration_agent_links. Framework is modeled on the proven per-partner MSPBackups shape (migration 034). - No dedicated dashboard UI yet — this is API/backend-only as of this compile (verify before claiming a management screen exists).
Policy & Configuration Management
- Policy folders + inheritance ("policy-core", NEW, deployed 2026-07-05, migration 067, merge e7e90c2): per-client folder tree replacing the fixed global->client->site->agent stack for agents placed in a folder. Exactly one root folder per client; any folder optionally references a library policy (
policy_folders.policy_id);agents.folder_idplaces a device in the tree; effective policy is the merged ancestry root->leaf plus an optional per-asset override. Recompute+push on policy edit is scoped to affected agents only.policy_versionsgives every policy edit an immutable snapshot for diff/rollback and per-device "what changed." Non-breaking migration by construction: every new column/table is nullable,folder_idis NULL fleet-wide until an explicit, verification-gated, dry-run-capable backfill job runs (POST /policy-folders/backfill) — until then effective-policy resolution stays on the legacy path. API:GET/POST /clients/:id/policy-folders,DELETE /policy-folders/:id,PUT /policy-folders/:id/rename,PUT /policy-folders/:id/policy,PUT /policy-folders/:id/parent(move),PUT /agents/:id/folder(placement),POST /policy-folders/backfill. Explicitly out of scope this phase: module editor UI, drag-and-drop, simulate/dry-run preview UI, subtree RBAC, tags/multi-folder membership, conditional modules, compliance compare engine, App Store UI — no agent-side change (wire format unchanged). No dashboard UI yet — a follow-onpolicy-uishape spec (folder tree view, agent placement, effective-policy viewer with source attribution, version history, admin backfill trigger) was added 2026-07-05 (01d962a), implementation not started. - Legacy inheritance chain (agents not yet placed in a folder): global -> client -> site -> agent; server computes merged effective policy, pushes via
ConfigUpdate. - Checks engine (
018/019): cpu, memory, disk, ping, port, script, service. Policy-attached check templates (024). - Remote registry (Windows): enumerate/read routed; write paths exist in the agent but are not routed yet [verify].
- Event Log Watch (migration 047): dashboard management UI at
/event-log-watches. Policy-clobber class bug fixed (PR #43) by reusing a single rule-shaper across all config-push sites. - Tray policy gate (shipped 2026-07-04, code merged, policy default OFF):
tray.enabledpolicy flag is now enforced by the watchdog (previously plumbed end-to-end but not enforced — the tray launched/showed regardless of the flag). Fleet-wide materialize (deploying the binary via agent-update, not just fresh MSI) has shipped in code; the deliberate flip to enable it fleet-wide (spec's "Task 4") has NOT happened yet.
Inventory & Discovery
- Hardware inventory (mfr/model/serial/BIOS, CPU, memory, disks, NICs, OS), software inventory, service inventory. On-demand refresh. NUL/control-char sanitized before storage (2026-07-05, see Monitoring section).
- VM / hypervisor / container detection (
032/033). - User/group inventory (
037-040): local + domain + Azure AD accounts, domain-join classification, domain-controller detection, group membership. - Network discovery: server-dispatched TCP/ARP/reverse-DNS scans; devices stream back and are persisted/editable.
Patch / Agent Update Management
- Self-updater: server sends version + URL + SHA256; agent downloads, verifies, atomically swaps, restarts, auto-rolls-back on failed reconnect.
- Auto-update gated on effective policy
auto_update. Force viaPOST /agents/:id/update. - Channel model: new builds tag beta; agents default to stable; promotion is a deliberate re-tag. Fail-closed as of 2026-07-05 (BUG-023 fix, server v0.3.98): a missing or malformed
.channelsidecar now classifies beta, not stable — closing a publish/tag race where a late "MARK AS BETA" pipeline phase left a multi-minute window during which an untagged build could be mass-offered to the entire stable fleet (incident detail in Recent Work / History).build-windows.shnow writes the beta sidecar atomically at publish;promote_versionkeys on agent binaries (not pre-existing sidecar files) and creates/overwritesstablesidecars, failing the request (500) rather than recording a rollout row that disagrees with disk on a partial write. - Update-scanner platform coverage fixed 2026-07-05 (server v0.3.95): the scanner's self-check previously tried to execute every binary regardless of platform to read its embedded version, dropping macOS/Linux artifacts entirely because they can't run on the Linux build host — macOS/Linux agents had never been offered an update. Now uses host-native-vs-foreign self-check regimes: foreign-platform binaries trust the filename-embedded version; host-native binaries stay a hard
--versiongate (with a best-effort exec-bit repair for suspected 0644 CI-copy artifacts). Post-fix: scanner indexes 8 platform/arch combos (was 4, Windows-only). - BUG-024 (OPEN, CRITICAL, mitigated 2026-07-05): enabling Linux update offers for the first time (the fix above) exposed that Linux agent binaries >=0.6.77 are built against the build host's glibc 2.43 (Ubuntu 26.04) with no compatibility floor — every older-distro Linux agent that took the offer crash-looped on
GLIBC_2.39not found (ix.azcomputerguru.com CloudLinux 9, Jupiter's Debian 12 container, a Lonestar container, presumed SL-SERVER). Mitigated by quarantining Linux binaries out of the downloads directory (scanner currently offers nothing for Linux). Real fix (glibc floor via an old-glibc build container or a musl static target, plus anobjdump -Tbuild gate) is pending — see History Highlights and FEATURE_ROADMAP BUG-024. - Pending-update supersede (batch 4, server v0.3.99): a stale
pendingupdate row is now superseded at re-dispatch time when a newer version exists or its target binary was pruned from disk (guarded: only rows over 1h old with a non-empty platform index) — previously all 13 pruned targets were blindly re-offered. - Legacy fleet build (Server 2008R2/Win7, Rust 1.77) DISABLED 2026-07-04 (
LEGACY_ENABLED=false): the unpinned dep re-resolve pulled edition2024 crates Rust 1.77 cannot parse, fail-closing the entire Windows build. Last legacy artifact (agent 0.6.76) frozen and preserved for existing 2008R2/Win7 endpoints; modern builds advance normally. 2008R2 support returns only via a separate legacy-only rewrite (RMM_THOUGHTS Feature 12, Raw — not approved). - Universal self-detecting installer (Feature 9, P1 shipped v0.6.71): one install path self-detects arch/OS and pulls the correct binary.
- Promotion API:
POST /api/updates/rollouts/:version/promotebody{"os","arch"}; rollback via.../rollback.
Agent Identity & Enrollment
- Per-agent enrollment keys; site-specific MSI with SITEKEY injected.
enrolled_agentsis the authoritative enrollment history log. - Durable multi-location
device_id(Phase 1 Task 1, 2026-06-11): registry+ProgramData (Windows), /etc+/var/lib (Linux), /Library+mirror (macOS), self-healing;cleanup.ps1whitelists identity files. - Cross-site dedup fixed 2026-07-04 (commits
2851523,1c1cac2): Both WS enrollment paths previously deduped only within(site_id, device_id), so re-running a different site's installer against an already-enrolled machine created a cross-site duplicate. Fixed to match by hardwaredevice_idacross all sites (db::get_agent_by_device_id) and re-home viamove_agent_to_siteon first match. A same-day follow-up stopped auto-re-homing on every reconnect — the agent re-presents its embedded install-site code as its API key on every beat, which was silently bouncing admin-relocated agents back to their original site. Current behavior: dedup by device_id, but site placement is admin-controlled and persists across reconnects. 13 duplicate-hostname agent pairs were identified fleet-wide (pre-fix legacywin-*/hw-*records) — 4 orphaned zombies were cleaned up in batch 4 (2026-07-05); the remaining full cleanup pass is still open and is the ongoing trigger formove_agent409s. - Delete race closed to the schema level (BUG-018 altitude fix, migration 066, server v0.3.98): a
visible_agentsDB view (status<>'deleting') is now the single source of truth for all listing/stat queries (nine call sites, including a previously-missed agent-detail-page query), aBEFORE UPDATEtrigger pinsstatus='deleting'at the schema level, and all three WSauthenticate_agentpaths reject a mid-purge row. Note for future schema changes: the view freezes theagentscolumn list atCREATE VIEWtime (SELECT *) — anyALTER TABLE agents ADD COLUMNmigration must also recreatevisible_agentsor listings 500 fleet-wide withColumnNotFound(documented in the migration 066 header). - Agent status reconciliation (batch 4, server v0.3.99): a periodic sweep flips zombie
'online'rows (agent status stranded by an unobserved disconnect) back to'offline'without disturbing the reallast_seentimestamp, respecting the update-hold window and skipping agents with a live dispatch-map connection.
Alerting & Watchdog
- Threshold alerts (ack/resolve, per-agent + fleet summary). Alert templates (
022). Maintenance mode (021). - Watchdog: separate supervising process; reports via
POST /watchdog-alert. - Role-aware offline alerting + alert ignore/mute (shipped 2026-06-07): server-only
agent_offline; site/fleet mass-offline aggregation;alert_mutesperma-silence. create_or_update_alerthardened (2026-07-05, server v0.3.95): now returnsResult<Option<Alert>>—Ok(None)when the target alert row was deleted mid-flight (agent cascade-delete FK violation) instead of aborting the caller. Both UPDATE-by-id branches switched fromfetch_onetofetch_optionalwith fall-through to INSERT (refresh_alert_rowhelper dedups the two near-identical UPDATE blocks). Fixed aRowNotFoundthat had been aborting the entire offline sweep whenever an alert was concurrently deleted (e.g. during duplicate-agent cleanup).- WebSocket log noise reclassified INFO, not ERROR (2026-07-05, server v0.3.96): benign disconnects (
is_benign_ws_disconnect) and pre-auth connection churn (is_preauth_connection_churn— sleeping laptops, scanners, half-open NAT never sending an auth frame) no longer mintserver_erroralerts. Genuine auth rejections (bad API key, malformed auth message) remain ERROR. Classified at INFO (not DEBUG) so a mass-disconnect anomaly stays visible at the server's default log level. move_agentduplicate-key hardened (2026-07-05): unique-violation onidx_agents_site_devicenow maps to 409 CONFLICT with an actionable message instead of a raw 500.- MSP360 transient-5xx noise reduction (batch 2, server v0.3.97): sync 5xx logs WARN, escalating to ERROR only after 4 consecutive cycles (~1h) so persistent outages still alert without every cloud-maintenance blip minting a
server_error.
Credentials Management
- Encrypted credentials vault (
016): scoped global/client/site, typed, metadata-only by default with a separate/revealendpoint. Known HIGH item:/revealownership-scope check — [verify current state]. - Credential inheritance (deployed 2026-06-07): Opt-in hierarchical cascade, de-duplication,
/effectiveendpoints. - Integration credentials (2026-07-01, migration 065): reuses the same encryption chokepoint for third-party integration API keys (never sent to agents, never plaintext in
integrations.config).
Audit Log / Log Feedback Intelligence
- Append-only
audit_logtable (migration 056). Per-line log fingerprinting +log_signaturesaggregation (migration 057).logs/analyzeruns on the Claude API (switched from Ollama 2026-06-15).
Backup Integration (MSP360 / MSPBackups)
- Multi-provider config, connection test, scheduled sync, per-agent + fleet coverage report, agent<->MSP360 mapping with confidence scoring.
- Active chronic failure (unresolved as of 2026-07-05): GND-SERVER (Grabb & Durando) — 42+
backup_failedalerts since 2026-06-17, failing nightly ~01:02 UTC, "no error detail reported." Also open: AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage, offline 17.7 days with an active critical alert as of batch 4), DESKTOP-EVA4H1A (Gary A Hartman, partial), Blaster2 (x2). MSP360 sync doesn't capture failure detail — investigation needs a live PowerShell dispatch to read MSP360/CloudBerry local logs.
Remote Access (Tunnel)
- Agent side substantially built. Server side is a dead-code skeleton (no
/tunnelroutes). Live TTY design (agent-comms-durability Phase 2) will supersede the skeleton.
Identity / Multi-tenancy / Security
- Auth: JWT; agents auth over WS via per-agent API key + hardware device_id.
- Microsoft Entra ID SSO (OAuth2/OIDC + PKCE).
- Organizations / multi-tenancy: org CRUD, roles, limits, dev-admin impersonation.
- 500-error-leak fixed (commit
58c1a96): all server error paths route throughinternal_err.
Platform coverage
- Cross-platform (Win/Linux/macOS): metrics, network state, hardware/software/service inventory, user/group + DC/domain detection, checks, discovery, self-update, scripts/commands.
- Windows-only:
user_sessioncommand context, remote registry, tray-into-session (now policy-gated), watchdog SCM supervision, BSOD detection, VSS policy configurator, WMI ACPI thermal, legacy fleet (frozen at 0.6.76). - Linux: update offers currently QUARANTINED fleet-wide pending the BUG-024 glibc-floor fix (see Patch/Update section) — all other cross-platform capabilities unaffected.
- macOS: agent deployed (Phase 1); tray is a stub; included in the update scanner's platform coverage since 2026-07-05.
Architecture
Components
| Component | Location | Tech | State |
|---|---|---|---|
| Server | 172.16.3.30:3001, systemd gururmm-server.service |
Rust, Axum | deployed, production; v0.3.99 |
| Dashboard | https://rmm.azcomputerguru.com, nginx | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production; v0.2.83 beta |
| Agent (Windows) | Endpoints, GuruRMMAgent service via WiX MSI |
Rust, Windows MSVC | deployed; modern build v0.6.78 fleet-wide (promoted stable 2026-07-05), legacy (2008R2/Win7) FROZEN at 0.6.76 and disabled |
| Agent (Linux) | Endpoints, systemd gururmm-agent |
Rust, musl static | deployed; update offers QUARANTINED fleet-wide (BUG-024, glibc floor pending) — existing installs unaffected, no new updates offered |
| Agent (macOS) | Endpoints, LaunchDaemon | Rust, aarch64/x86_64 | Phase 1 deployed 2026-05-12; code-signing issue on Apple Silicon; update-scanner platform gap fixed 2026-07-05 |
| Tray (Windows) | System tray, named pipe IPC | Rust | deployed to disk fleet-wide via agent-update (2026-07-04); launch policy-gated, default OFF |
| Tray (Linux) | System tray, Unix socket IPC | Rust, GTK | deployed 2026-05-24 |
| Tray (macOS) | Menu bar | Rust | stub/TODO |
| PostgreSQL DB | localhost:5432 on 172.16.3.30 | PostgreSQL 18 | deployed; migrations through 067 |
| Build pipeline | 172.16.3.30:9000 webhook + /opt/gururmm/ scripts |
Python, Bash | deployed; builds agents AND server; build-lock + gated git reset added 2026-07-05 (BUG-003) |
| Windows build — Beast (PRIMARY) | GURU-BEAST-ROG over Tailscale | Rust MSVC, WiX 4.x | primary — two-wave parallel, ~336s |
| Windows build — Pluto (FALLBACK) | 172.16.3.36 | Rust MSVC, WiX v4 | fallback only |
Key Files & Repos
- Active repo:
azcomputerguru/gururmm— http://172.16.3.20:3000/azcomputerguru/gururmm (external: https://git.azcomputerguru.com/azcomputerguru/gururmm.git) - Submodule working tree:
projects/msp-tools/guru-rmm— gitlink intentionally lags origin/main; read live code viagit -C projects/msp-tools/guru-rmm show origin/main:<path>or a worktree. - Server binary:
/opt/gururmm/gururmm-serveron 172.16.3.30 - Server config:
/opt/gururmm/.env - New shared sanitize module:
server/src/db/sanitize.rs(strip_nul,clean_control_chars) — consumed bydb/logs.rsanddb/inventory.rs - Visible-agents view + delete-race schema fix:
server/migrations/066_visible_agents_view.sql(view + BEFORE UPDATE trigger; column-freeze warning in the header — anyALTER TABLE agents ADD COLUMNmust recreate the view) - Policy folders / policy-core:
server/migrations/067_policy_folders.sql(policy_folders,agents.folder_id,policy_versions),server/src/api/policies.rs(folder CRUD + placement + backfill),docs/specs/policy-core/,docs/specs/policy-ui/(dashboard UI spec, not implemented) - Integrations framework:
server/src/api/integrations.rs,server/src/integrations/(plugin trait + registry), migration065_integrations_framework.sql - VSS configurator:
agent/src/vss_com.rs,agent/src/vss.rs,specs/vss-policy-config/,specs/vss-native-com/ - Tray policy-deploy:
agent/src/tray_stage.rs,agent/src/tray_state.rs,agent/src/authenticode.rs,specs/tray-policy-deploy/ - Windows WMI thermal:
agent/src/(BUG-001collect_temps_windows_acpi,wmicrate dep) - Downloads dir:
/var/www/gururmm/downloads/on 172.16.3.30 (Linux binaries currently quarantined to a siblingquarantine/dir pending BUG-024) - Webhook handler:
/opt/gururmm/webhook-handler.py - Deploy pipeline (canonical, auto-synced to /opt/gururmm on push):
deploy/build-pipeline/build-server.sh(flock lock, PIPESTATUS-gated git reset,--migrate-onlypre-stop validation — BUG-003),deploy/build-pipeline/build-windows.sh(atomic beta.channelsidecar write — BUG-023) - API (internal): http://172.16.3.30:3001
- API (external): https://rmm-api.azcomputerguru.com
- Dashboard: https://rmm.azcomputerguru.com
- Vault paths:
infrastructure/gururmm-server.sops.yaml(API creds, DB password, sudo password),infrastructure/gururmm-server-physical(SSH ed25519 key — this fleet key works on 172.16.3.30 currently, not just the future physical box; confirmed 2026-07-04) - SSH to prod:
ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30(ed25519, key-only; password auth disabled for SSH; sudo over SSH still requires the vault password via-S) - Pre-merge verification skill:
bash .claude/skills/gururmm-build/scripts/verify.sh server|agent|dashboard|migrations
Repo Structure
gururmm/
├── agent/ Rust agent (managed endpoints)
│ └── src/
│ ├── authenticode.rs Authenticode signature verify (tray-policy-deploy)
│ ├── tray_stage.rs tray materialize FSM + staging + versioned mutex
│ ├── tray_state.rs tray policy state
│ ├── bsod.rs Windows BSOD/kernel-crash detection
│ ├── vss_com.rs Native VSS COM create/provision
│ ├── vss.rs VSS policy configurator entry points
│ ├── commands/mod.rs CommandExecutor + recent-results cache (ACK dedup)
│ ├── device_id.rs Durable multi-location device_id
│ ├── transport/
│ │ └── websocket.rs WS transport; execute_command ACK + dedup; cmd-alias NAK
│ ├── tunnel/ TunnelManager state machine
│ ├── metrics/ sysinfo collection + temps; WMI ACPI thermal (BUG-001)
│ ├── registry_ops/ Windows registry read/write
│ ├── updater/ Self-update handler (container guard BUG-019, unix rollback watchdog)
│ └── main.rs
├── server/ Rust/Axum API server
│ └── src/
│ ├── api/
│ │ ├── integrations.rs integrations framework CRUD (crowdstrike-falcon)
│ │ ├── policies.rs policy CRUD + folder tree + placement + backfill (policy-core, NEW)
│ │ └── ... 39 route modules; software.rs SPEC-030
│ ├── alerts/ Alerting modules (offline.rs: offline_sweep + mass_offline + status reconciler)
│ ├── db/
│ │ ├── sanitize.rs shared NUL/control-char sanitize
│ │ ├── alerts.rs create_or_update_alert -> Option<Alert>, refresh_alert_row
│ │ ├── inventory.rs NUL-scrub over jsonb + text before upsert
│ │ ├── agents.rs visible_agents-view reads, status write guards
│ │ └── software_jobs.rs SPEC-030 async removal jobs
│ ├── updates/
│ │ └── scanner.rs host-native vs foreign self-check regimes; fail-closed channel default (BUG-023)
│ ├── ws/mod.rs WS handler; is_benign_ws_disconnect / is_preauth_connection_churn; deleting-row auth rejection; half-open eviction sweep (T2)
│ └── mspbackups/ MSP360 backup integration; transient-5xx WARN escalation
├── dashboard/ React/TypeScript UI
│ ├── pages/
│ │ └── EventLogWatches.tsx
│ └── components/
│ └── SoftwareManager.tsx
├── agent/scripts/ uninstall-engine.ps1
├── tray/ System tray binary
├── installer/ WiX v4 MSI; cleanup.ps1; universal self-detecting installer
├── deploy/
│ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh (flock+PIPESTATUS+migrate-only, LEGACY_ENABLED=false)
├── scripts/ Build/ops scripts
└── docs/ FEATURE_ROADMAP.md, UI_GAPS.md, RMM_THOUGHTS.md, specs/
docs/specs/ policy-core/, policy-ui/, SPEC-025-policy-compliance-posture.md, ...
specs/ tray-policy-deploy/, vss-policy-config/, vss-native-com/, crowdstrike-falcon/,
av-removal-recipes/, remote-software-uninstall/, agent-comms-durability/, ...
Development
Current Focus
As of 2026-07-05 (agent 0.6.78 / server v0.3.99):
- BUG-024 real fix (glibc floor for Linux builds): Linux binaries remain quarantined out of downloads — no Linux agent gets update offers until an old-glibc build container (or musl static target) plus an
objdump -Tbuild gate lands. SL-SERVER (Scileppi) and Lonestar's container host still need manual agent recovery (no vault SSH creds for either). - Machine-side backup failures (highest user priority, unresolved): GND-SERVER chronic 42+-failure streak, AD1, LAB-SVR (offline 17.7 days as of batch 4), DESKTOP-EVA4H1A, Blaster2 (x2) — MSP360 sync reports no error detail; needs a live PowerShell dispatch to read local MSP360/CloudBerry logs or the mspbackups.com console.
- Duplicate-hostname agent cleanup: 4 orphaned zombies cleared in batch 4; the remaining pairs of the original 13 fleet-wide are the ongoing trigger for
move_agent409s — a full one-time cleanup pass is not yet done. - 4 chronic one-way-WS loopers (T2 field data): DESKTOP-3USU20B (Safesite Glendale, 79/127 evictions), DESKTOP-BA45FRS (Safesite Bell), DDDOffice072023 (ACG discovery), DESKTOP-PL2RCGL (Robyn Pittman) — agent->server WS frames eaten post-auth, version-independent, server-side telemetry exhausted; packet capture on an affected box is the next diagnostic step.
- Policy-ui (folder management dashboard UI): shape spec added 2026-07-05 (
docs/specs/policy-ui/); implementation not started. Policy-core backend is deployed but has no dashboard surface yet. - Tray policy fleet enablement (Task 4, tray-policy-deploy spec): materialize code is merged and deployed; flipping
tray.enabledon for the fleet is a deliberate next step, not yet done. - Legacy (2008R2/Win7) agent rewrite (RMM_THOUGHTS Feature 12, Raw): not approved; needs a language/runtime decision (.NET Framework 4.x vs C++) and scope decision before a
/shape-spec. - Integrations framework dashboard UI: CrowdStrike Falcon backend + sync worker are live; no management screen exists yet in the dashboard.
- SPEC-030 software-uninstall follow-on (Route A / recipes): Launchy/AIMP-class uninstallers still hang the agent; driver/system-component exclusion flag and a list-jobs endpoint remain unbuilt.
- Agent-comms-durability Phase 2/3 (planned): live TTY, adaptive keepalive, bulk file transfer -> short-lived HTTPS. Phase 1 is now fully complete (T1/T2/T3 all shipped).
- Phase-1 acceptance test: interactive command dispatch to PST-SERVER / Safesite boxes during an idle window, still pending across three consecutive session logs.
- Durable agent identity Phase 2-3 (pending soak): guarded auto-reclaim + an operator merge tool for the pre-fix ghost/duplicate agents.
- SPEC-028 offboarding wizard: specification complete; implementation pending.
- Repo-root
build-server.sh: a stale divergent lineage of the canonicaldeploy/build-pipeline/build-server.sh— flagged as a deletion candidate to end confusion about which script is authoritative. - KSTEENBB2025: fleet-max
cpu_temppeaking 83.8C — worth a glance once thermal data has a wider baseline.
Patterns & Anti-Patterns
New anti-patterns (2026-07-01..05):
| Pattern | What Went Wrong |
|---|---|
| Publish-then-tag-later build pipeline ordering (BUG-023) | Binaries landed in the downloads dir minutes before a late "MARK AS BETA" phase tagged them; the scanner's missing-sidecar default was stable, so a 5-minute scan window could mass-offer an unsoaked build to the entire stable fleet. Fix: fail-closed scanner default (missing/malformed sidecar => beta) AND atomic sidecar write at publish — defense in depth, not either alone. |
| Promotion logic keyed on pre-existing sidecar file content rather than the binary itself | promote_version only rewrote files whose content was exactly beta, so a fail-closed (sidecar-less) binary became permanently unpromotable the moment the scanner default flipped. Key promotion on the binary set, create/overwrite the sidecar, and fail loudly (500) on a partial write rather than recording a DB rollout row that disagrees with disk. |
| Building Linux agent binaries against the build host's live glibc with no floor (BUG-024) | The build host's OS upgrade (Ubuntu 26.04, glibc 2.43) silently raised the minimum glibc every Linux agent binary requires; combined with a same-week fix that enabled Linux update offers for the first time, every older-distro Linux agent that took the update crash-looped on a missing GLIBC symbol. Any cross-compiled/host-compiled binary needs an explicit compatibility floor (old-glibc container or static linking) verified by a build-time objdump -T gate, not an assumption that "it compiled" means "it runs everywhere." |
Hand-copied WHERE status <> 'deleting' guards scattered per query/UPDATE |
The same forgotten-guard class recurred (BUG-018) even after a prior "fix" — nine call sites needed the filter and one (agent detail page) was missed. Fixed at the schema level: a view for reads, a BEFORE UPDATE trigger for writes, so no future query/update can forget the guard. Tradeoff accepted: a SELECT * view freezes the column list at creation time — documented as a loud warning for future migrations. |
| Update-scanner self-check executing every binary regardless of target platform | macOS/Linux binaries can't run on the Linux build/scan host, so they were silently dropped from the scannable set entirely — those agents never received an update offer. Fix: host-native binaries get a hard --version execute-and-check gate; foreign-platform binaries trust the filename-embedded version. |
Writing a machine-global VSS registry governor (MaxShadowCopies) to a per-policy retention_count |
MaxShadowCopies is shared by ALL VSS consumers on the box (System Restore, third-party backup) — lowering it FIFO-evicts THEIR shadows too, a genuine data-loss class caught only by a 23-agent pre-merge code review. Use a per-volume size cap as the sole enforced governor instead. |
| Re-homing an agent's site on every WS reconnect after a device_id-based dedup fix | The agent re-presents its embedded install-site code as its API key on every beat; auto-re-homing on each connect silently bounces an admin-relocated agent back to its original install site. Dedup by device_id, but leave site placement admin-controlled and persistent. |
fetch_one on an UPDATE-by-id inside a concurrently-deleting table |
An agent cascade-delete can remove the target row between a SELECT and the UPDATE; fetch_one panics RowNotFound and aborts the whole caller (here: the entire offline sweep). Use fetch_optional with a fall-through path. |
| Two sessions editing the same branch's working tree without a coord lock | A concurrent session's uncommitted edits were silently absorbed into another session's commit more than once across this window. Benign each time (same branch, all changes verified at HEAD) but a coord lock was only reliably taken starting with bug-fix batch 3 — take a lock before editing a shared submodule branch. |
Assuming a spec's shape.md/plan.md header "Status: not started" reflects current reality |
Tasks can ship and merge (git log shows work done) while the spec doc header is never updated. Verify against git log/deployed version, not the spec file's status line. |
Testing a pipe's exit code (if ! cmd | tee ...) instead of the upstream command's |
Shell tests the last command in the pipe (tee), not git; a failed git fetch/reset could report success. Use the shell's PIPESTATUS array explicitly. |
| Mischaracterizing "19 known test failures" as a single homogeneous class | All 19 were assumed to be the same pre-existing DATABASE_URL-missing environment gap; one was a genuine matches_pattern panic on no-wildcard input hiding in the same failure count. Bucket-count assumptions on a recurring "known failures" number should be re-verified periodically, not re-quoted. |
Existing anti-patterns (carried forward):
| Pattern | What Went Wrong |
|---|---|
useMemo with stable deps for data-dependent values |
queryClient is stable, memo never recomputes after queries resolve. Use useQuery instead. |
| Deploying without stopping the server first | "text file busy" kernel error. Always systemctl stop before cp. |
Building without DATABASE_URL |
sqlx compile-time macros fail. |
DB migrations without inserting into _sqlx_migrations |
Server crashes on start. Must insert SHA-384 checksum manually. |
| WiX MSI builds on Linux | WiX requires msi.dll. Must build on Pluto or Beast. |
| Manual builds via SSH | All builds go through webhook-handler.py. |
Restart-Service GuruRMMAgent -Force in command scripts |
Kills agent before it can report result. Use a scheduled task with delay instead. |
sudo -u guru git in systemd build context |
git rejects repo as dubious ownership. Use git config --system --add safe.directory. |
+1.77 legacy builds without pinned deps |
Fresh dep resolution repeatedly pulls edition2024 crates Rust 1.77 can't parse (BUG-021, recurred 2026-07-04 with a different dep pair, led to the variant being disabled outright rather than re-chasing pins). |
| Dead WebSocket write half | WS write fails, send task dies, receive loop keeps agent connected with a dead write half. Fix: tokio::select! monitoring both tasks. |
Using the minidump crate for Windows kernel dumps |
Only parses Breakpad MDMP, not Windows kernel PAGEDU64. Parse DUMP_HEADER64/32 at fixed offsets directly. |
| Build classification defaulting to stable | Superseded 2026-07-05 (BUG-023) — the scanner's missing-sidecar default is now beta, not stable; new builds are tagged beta by the pipeline and promotion is an explicit re-tag on top of that. |
command_type: "cmd" in API commands |
Not a valid CommandType variant originally — agent silently dropped the message. Now aliased to Shell. |
| Reading the stale submodule working tree for code | The submodule gitlink intentionally lags origin/main. Always read via git -C ... show origin/main:<path> or a worktree. |
Good patterns (carried forward + new):
- Push guards to the schema, not the call site (2026-07-05) — the
visible_agentsview +BEFORE UPDATEtrigger closed a class of forgotten-guard bugs that a second round of hand-copiedWHEREclauses had already failed to fully close once. - Fail-closed defaults on ambiguous/missing state (BUG-023 scanner fix) — treating a missing
.channelsidecar as the risky classification (beta) rather than the safe-looking one (stable) makes a future pipeline crash harmless instead of dangerous. - Shared sanitize chokepoint (
db/sanitize.rs) — one NUL/control-char scrubber consumed by bothdb/logs.rsanddb/inventory.rs, replacing duplicated ad hoc scrub logic. - Capability-based classification (host-native vs foreign) instead of a single universal check — the update scanner branches its self-check strategy per target platform rather than applying one strategy fleet-wide.
- Multi-AI adversarial review before shipping risky agent-side changes — the VSS redesign (Grok + Gemini) and tray redesign (17-agent Claude review + Grok + Gemini) both caught genuine defects (a data-loss governor and 9 architecture defects respectively) that a single-model review missed on the first pass; the 2026-07-04..05 batches ran 15-26-agent high-effort code-review workflows before every fleet-deploy merge and consistently surfaced findings (data-loss guards, race conditions, unpromotable-binary edge cases) that first-pass implementations missed.
- Platform parity rule — any agent feature goes on Windows + Linux + macOS in the same commit.
gururmm-buildskill for pre-merge verification — does NOT trigger production build; merging to main is the only prod trigger.- Dashboard beta-before-main rule — dashboard changes go to beta (auto-built from main) first; promote explicitly.
- Non-breaking migrations by construction for structural changes (policy-core, migration 067) — new nullable columns/tables plus a deliberate, verification-gated, dry-run-capable backfill job means landing a large schema change changes zero live behavior until an explicit flip.
- Two independently-deployable slices for cross-component features, capability-gated — agent slice and server slice can deploy in any order; old agents/servers keep legacy behavior until the gate flips.
Build & Deploy
CRITICAL: Never trigger builds manually via SSH. All builds go through the webhook pipeline.
Gitea push to main
-> webhook-handler.py (172.16.3.30:9000)
-> build-shared.sh (auto-version bump)
-> build-linux.sh (Linux artifact publish currently QUARANTINED post-build pending BUG-024 glibc floor)
-> build-windows.sh (Beast PRIMARY over Tailscale || Pluto FALLBACK
TWO-WAVE PARALLEL BUILD; legacy wave gated LEGACY_ENABLED=false;
deploy_and_sign writes the beta .channel sidecar ATOMICALLY per artifact — BUG-023 fix)
-> build-mac.sh (stub — no build machine configured yet)
-> build-server.sh (gated on server/ changes; flock lock, PIPESTATUS-gated git fetch/reset,
pre-stop --migrate-only migration validation, backup+rollback — BUG-003 fix)
-> artifacts -> /var/www/gururmm/downloads/ with sha256 + -latest symlinks + .channel
(missing/malformed .channel now scans as beta, not stable — BUG-023 fix)
BUG-003 closed 2026-07-05 (6ccd35e): the deploy script confirmed to auto-sync to /opt/gururmm on every push. Gained a flock build lock, PIPESTATUS-gated git fetch/reset (fired correctly on its first live run — a merge-webhook build collided with the CI version-bump job's index.lock and aborted cleanly with nothing deployed, instead of silently building a stale tree), and pre-stop migration validation via a new server --migrate-only flag (validates+applies migrations against the live DB while the OLD binary keeps serving; paired with set_ignore_missing(true) on the migrator so a rollback binary tolerates an ahead DB instead of re-creating the outage the gate was meant to prevent).
BUG-023 closed 2026-07-05: build-windows.sh's deploy_and_sign now writes the beta .channel sidecar atomically with each published artifact, closing the multi-minute publish-then-tag window that had let an untagged build get scanned as stable.
Legacy variant disabled 2026-07-04 (84a82a3): LEGACY_ENABLED=false in build-windows.sh gates the legacy wave/copy/deploy/symlink off entirely; the last-built legacy artifact (0.6.76) is preserved and excluded from cleanup so existing 2008R2/Win7 endpoints keep receiving that frozen build. Modern (amd64/x86/tray/debug) builds are no longer blocked by legacy dep-resolution failures.
Channel/promotion model, downloads layout, dashboard beta/prod split, and Windows build host detail otherwise unchanged from the 2026-07-04 compile — see History Highlights / Compilation Notes for full prior mechanics; current migration count is 067 (067_policy_folders).
Active State
Fleet (as of 2026-07-05):
- ~360 enrolled agents; ~229 fresh/online at last sample
- Agent 0.6.78 modern fleet-wide (stable, promoted 2026-07-05) / 0.6.76 legacy-frozen-and-disabled; server v0.3.99; dashboard v0.2.83 beta
- Active
server_erroralerts: 0 - Duplicate-hostname agent pairs: 4 orphaned zombies cleaned in batch 4; remaining pairs of the original 13 still outstanding
- Active
backup_failedalerts: GND-SERVER (chronic), AD1, LAB-SVR (offline 17.7 days), DESKTOP-EVA4H1A, Blaster2 (x2) — investigation open - Linux update offers QUARANTINED fleet-wide (BUG-024) — existing Linux installs unaffected, no new offers until the glibc-floor fix ships
- 4 chronic one-way-WS loopers under active diagnosis (DESKTOP-3USU20B, DESKTOP-BA45FRS, DDDOffice072023, DESKTOP-PL2RCGL) — packet capture pending
- Policy-core (folder tree) deployed but
agents.folder_idis NULL fleet-wide — no agent is yet on the new inheritance path pending the backfill job - SL-SERVER (Scileppi) and Lonestar's container host down pending manual agent recovery (no vault SSH creds for either)
API auth:
POST /api/auth/login-> JWT (~24h)- Creds: vault
infrastructure/gururmm-server.sops.yaml - Key endpoints:
GET /api/agents,POST /api/agents/:id/command,GET /api/commands/:id,POST /api/agents/:id/update,POST /api/updates/rollouts/:version/promote - Software (SPEC-030):
GET /api/agents/:id/software,POST /api/agents/:id/software/uninstall(-> job_id),GET .../uninstall/jobs/:job_id,GET .../removal-status,GET /api/software/knowledge - Integrations:
GET/POST /api/integrations,GET/DELETE /api/integrations/:id,GET/PUT /api/integrations/:id/credentials,GET/PUT /api/integrations/:id/mappings,GET /api/integrations/:id/sync-state,POST /api/integrations/:id/test - Policy folders (NEW):
GET/POST /api/clients/:id/policy-folders,DELETE /api/policy-folders/:id,PUT /api/policy-folders/:id/rename,PUT /api/policy-folders/:id/policy,PUT /api/policy-folders/:id/parent,PUT /api/agents/:id/folder,POST /api/policy-folders/backfill - Command fields:
command_type(shell/powershell/python/script/claude_task/cmdalias),command, optionalcontext(system/user_session),timeout_seconds/elevated.
Dashboard — complete and working: Agents management, Clients/Sites CRUD, Commands execution + terminal, Logs + AI analysis (Claude API), Alerts, Metrics, Auto-update triggering, Network state, Entra ID SSO, Policies Dashboard, Registry editor (read-only), MSP360 backup status + mappings/verify UI, Organizations + dev-admin impersonation, Credentials with inheritance, AgentDetail Crashes tab + version history, Event Log Watches management, Software Manager (BETA, not guaranteed).
Dashboard — incomplete (see UI_GAPS.md):
- Watchdog alerts UI — blocked on 2 missing server routes
- BSOD in Alerts stream (Phase 2 deferred)
- Tunnel session management (server skeleton)
- Offboarding wizard UI (SPEC-028 complete; implementation pending)
- Alert mute UI — server endpoints ready, dashboard NOT started
- Integrations Center UI — CrowdStrike Falcon framework + sync worker are live server-side; no management screen in the dashboard yet
- Policy folders / policy-ui (NEW gap) — policy-core backend + folder-tree API deployed 2026-07-05; the
policy-uishape spec exists but no implementation has started, so folder placement/inheritance is API-only today - Tray policy toggle surfacing —
tray.enabledis enforced server/agent-side; verify whether the policy UI exposes the toggle to operators [verify]
Security backlog (HIGH):
credentials/:id/reveal— horizontal privilege escalation (no ownership scope check)
Key Architecture Decisions (LOCKED)
These decisions are locked. Do not reverse without explicit user approval.
- Per-agent enrollment keys.
- Site-specific MSI generation.
- No TOML/config for endpoints — server URL compiled into binary.
- Policy inheritance chain — global -> site -> client -> agent (legacy path); a device may instead be placed in a per-client policy folder (added 2026-07-05, decision 14) inheriting its folder-ancestry chain — the two models coexist during migration via
agents.folder_id. - Platform parity rule — any agent feature ships on Windows, Linux, and macOS in the same change.
- Watchdog as separate process — reports via REST (
POST /watchdog-alert), no WebSocket. - Build pipeline is the only path to production.
- Multi-tenancy identity model (ADR-001) — Dev -> Partner -> Client.
- Holistic feature development — every feature requires backend + API + dashboard UI + docs.
- AI-optional operation.
- Durability-first command delivery — the durable Postgres
commandsrow is the source of truth; the WebSocket is only a delivery hint. - VSS: configure, never delete on a schedule (added 2026-07-03) — the agent may only set retention governors and create shadows on a schedule; scheduled or automatic deletion reintroduces the MITRE T1490 surface. On-demand admin-initiated delete stays allowed.
- Third-party security integrations are a plugin, not a schema (added 2026-07-01) — a new vendor implements the
integrationsplugin trait against the generic framework tables; it does not get its own bespoke migration set. - Policy scoping is a folder tree, not a fixed level stack (added 2026-07-05) — a client's policy structure is an arbitrary-depth folder tree per client (policy-core); the legacy fixed global->client->site->agent stack remains the resolution path for any agent not yet placed in a folder, and migration between the two models is backfill-gated, never automatic or silent.
- Update-channel classification fails closed (added 2026-07-05, BUG-023) — a missing or malformed
.channelsidecar classifies as beta, never stable; only an explicitstablesidecar (written only by the atomic-publish path or an operator promotion) offers a build to the default fleet.
History Highlights
| Date | Event |
|---|---|
| 2025-12-15 | Project genesis: Windows service + Linux installer + site code auth + build server. |
| 2026-04-19 | Full drill-down navigation, auto-install on first run, Pluto build VM setup started. |
| 2026-04-21 | MSI build fix. DESIGN.md created. BirthBiologic onboarded. |
| 2026-04-29 | UI_GAPS.md created. Holistic development principle formalized. |
| 2026-05-12 | macOS agent Phase 1 deployed. Code signing issue on Apple Silicon noted. |
| 2026-05-15 | Dead WebSocket write-half bug fixed. Temperature struct field mismatch fixed. |
| 2026-05-16 | Watchdog bugs fixed. /feature-request skill created. |
| 2026-05-17 | Syncro PSA Integration added to roadmap. Office power failure recovery. |
| 2026-05-18 | Multi-tenancy architecture (ADR-001) decided. 5 SPEC documents created. |
| 2026-05-19 | AD2 crash-loop fix. MSP360 integration completed. Process drill-down modal. |
| 2026-05-23 | /rmm-audit pass. Agent optimization Phases 1A-3. MSRV bumped to 1.85. Fleet at 0.6.29. |
| 2026-05-24 | Linux tray IPC + GTK merged. Build pipeline split per-platform. Fleet converged to 0.6.38. |
| 2026-05-31 | Roadmap reconciliation. MSPBackups mapping/verify UI + dev-admin impersonation UI deployed. SPEC-021 written. |
| 2026-06-01 | BUG-016/017 fixed. BSOD detection feature (Phase 1) implemented and merged, agent 0.6.51. |
| 2026-06-02 | Server 0.3.37 + migration 048 deployed. Webhook wired to dispatch build-server.sh. |
| 2026-06-04 | BUG-020 (ghost tray icons) fixed to beta (137dd85). |
| 2026-06-07 | Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit all shipped same day. |
| 2026-06-11 | Physical server migration executed (<27 min downtime). Ghost agent root-caused, durable identity Phase 1 Task 1 shipped. Agent-comms-durability Phase 1 (T1/T3) shipped (agent v0.6.63/server v0.3.68). |
| 2026-06-12 | Beast parallel Windows build wired in (cfbdb59, ~336s). command_type "cmd" root-caused and fixed (3de9faf). |
| 2026-06-15 | BSOD dedup key changed to bugcheck code. MSI EXDEV fix. sync.sh Phase 1 auto-heal. logs/analyze switched to Claude API. 500-error-leak closed. |
| 2026-06-18 | sync.sh submodule auto-heal verified fleet-wide. |
| 2026-06-21 | BUG-019/018 fixed and merged. Event Log Watch UI shipped. Howard cleared for GuruRMM merges/deploys. gururmm-build skill created. |
| 2026-06-22 | BUG-021 (legacy dep-pin) fixed, v0.6.67 stable. BUG-022 (dead WatchdogEvent path) fixed. Fleet verified ~270/~178 online. |
| 2026-06-23 | Universal self-detecting installer (Feature 9) P1 shipped (v0.6.71). av-removal-recipes spec added. |
| 2026-06-24 | SPEC-030 remote software uninstall — BETA — code merged + deployed (Route B, async jobs, three-state KB, migrations 061-064). agent 0.6.75 / server v0.3.87. |
| 2026-06-25 | Win11 KB5095093 Point-in-Time Restore captured as a Raw RMM_THOUGHTS entry. |
| 2026-07-01..02 | Vendor-agnostic Integrations Framework shipped (migration 065) with CrowdStrike Falcon as the first plugin — API client, client mappings, periodic sync worker. No dashboard UI yet. |
| 2026-07-03..04 | VSS Policy Configurator (SPEC-016 redesign) shipped to beta (agent 0.6.76) — replaces scheduled create+prune with a configure-governors-only model, eliminating the T1490 surface; 23-agent code review caught and fixed a data-loss-class MaxShadowCopies defect pre-merge; verified live on the T1490 canary NEPTUNE. |
| 2026-07-04 | Legacy (2008R2/Win7, Rust 1.77) build variant disabled (84a82a3) after an edition2024 dep recurrence fail-closed the whole Windows build; last legacy artifact frozen at 0.6.76. RMM_THOUGHTS Feature 12 (legacy agent rewrite) added Raw. Cross-site agent dedup fixed (2851523, 1c1cac2) — dedup by device_id across all sites, no re-home on reconnect. Policy-Controlled Tray Deployment merged (f6a4251) — materialize shipped fleet-wide, launch stays policy-gated default OFF. |
| 2026-07-04..05 | Easy-bugs batch + full alert triage: update-scanner host-native/foreign self-check regimes (fixed macOS/Linux agents never receiving update offers), inventory NUL/control-char sanitize (db/sanitize.rs), WS benign-disconnect + pre-auth churn reclassified ERROR->INFO, create_or_update_alert survives cascade-delete races, move_agent maps duplicate-key to 409. Deployed as server v0.3.95 + v0.3.96. Active server_error alerts reduced to 0. |
| 2026-07-05 (00:45-01:00 PT) | Repair batch 2 merged + deployed (76a6bd2, server v0.3.97, agent 0.6.78): comms-durability Phase 1 completed (T2 half-open eviction sweep); BUG-001 Windows WMI ACPI thermal SHIPPED and PROVEN live on 112+ agents; BUG-020 uninstall teardown fixed; BUG-018 delete-race resurrection paths closed; MSP360 transient-5xx downgraded to WARN. |
| 2026-07-05 | BUG-023 discovered and fixed (Howard): update-channel publish/tag race had mass-offered unsoaked beta 0.6.78 to 209 stable agents; fixed with a fail-closed scanner default, an atomic sidecar write at publish, and a binaries-keyed promote_version. 0.6.78 manually promoted to stable, healing the split fleet (229 agents converged). |
| 2026-07-05 | BUG-003 closed (Howard, server v0.3.98): deploy pipeline hardened with a build lock, PIPESTATUS-gated git reset, and pre-stop --migrate-only migration validation — the gate fired correctly on its first live run (aborted cleanly on a CI index.lock collision instead of building a stale tree). |
| 2026-07-05 | BUG-018 altitude fix shipped (Howard, migration 066, server v0.3.98): visible_agents DB view + BEFORE UPDATE schema trigger replace nine scattered inline status<>'deleting' filters; all three WS auth paths reject mid-purge rows. |
| 2026-07-05 (11:05 PT) | Bug-fix batch 4 merged (Howard, server v0.3.99): pending-command 7-day TTL, agent-status reconciler for zombie 'online' rows, stale pending-update supersede logic. BUG-024 discovered (CRITICAL, OPEN): Linux agent binaries built against the build host's newer glibc crash-looped every older-distro Linux agent that took the newly-enabled update offer; mitigated via quarantine, real glibc-floor fix pending. |
| 2026-07-05 | Policy-core (policy folders + inheritance) deployed to production (Mike, merge e7e90c2, migration 067 renumbered from 066): per-client folder tree with inheritance resolution, versioning, scoped recompute+push, and a gated/dry-run backfill — non-breaking by construction. Follow-on policy-ui (dashboard UI) shape spec added same day (01d962a), not yet implemented. |
Compilation Notes
- 2026-07-05 recompile (Howard-Home/claude-main): Full rebuild. Delta from 2026-07-04 (migrations 65 -> 67; agent 0.6.77 -> 0.6.78 fleet-wide stable; server 0.3.96 -> 0.3.99; dashboard v0.2.83 beta). Major additions: Policy-core (migration 067,
e7e90c2) — new Capabilities subsection, locked decision #14, follow-on policy-ui gap; four sequential bug-fix batches (repair batch 2/v0.3.97, batch 3/v0.3.98, batch 4/v0.3.99) folded into Recent Work + Capabilities; BUG-003 CLOSED (was open); BUG-018 corrected from the morning's status-write-guard fix to the full v0.3.98 schema-level view+trigger fix; BUG-023 (NEW) — channel-race incident caused by the 07-04 scanner fix, own Recent Work entry + anti-pattern + locked decision #15; BUG-024 (NEW, OPEN, CRITICAL) — Linux glibc incident, documented across Recent Work/Capabilities/Active State/Architecture; comms-durability Phase 1 marked COMPLETE (was T2/T3-planned). All prior anti-patterns/good-patterns preserved verbatim except one ("build classification defaulting to stable") marked superseded rather than deleted. - 2026-07-04 recompile (Howard-Home/claude-main): Delta from 2026-06-25 (migrations 64 -> 65; agent 0.6.75 -> 0.6.77 modern / 0.6.76 legacy-frozen; server 0.3.87 -> 0.3.96; fleet 270 -> 359 enrolled). Added: Third-Party Integrations Framework, VSS Policy Configurator, Policy-Controlled Tray Deployment, legacy build variant disabled, cross-site agent dedup fix, 2026-07-04/05 easy-bugs + alert-triage batch.
- 2026-06-25 recompile (Howard-Home/claude-main): Delta from 2026-06-22 — SPEC-030 remote software inventory + bulk uninstall added as a major Capabilities subsection; universal self-detecting installer (Feature 9) P1 shipped; Current Focus rewritten; flagged the old "pending PRs #40-#46, migrations 061-063" prediction as superseded because software-removal work consumed 061-064 instead.
- 2026-06-22 recompile (Howard-Home/claude-main): Delta from 2026-06-11 — fleet ~270/~178, BUG-021/018/019 fixed on main, Event Log Watch UI shipped, BUG-022 dead-code watchdog path removed, Beast parallel two-wave build documented, command_type "cmd" alias + NAK documented, migration count to 60.
- 2026-06-11 recompile (GURU-5070/claude-main): Full recompile — physical server migration, durable agent identity, agent-comms-durability Phase 1 full detail, migration count to 59.
- 2026-06-07 recompile: Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit folded in; migration count to 55+.
- 2026-06-04 recompile: Corrected GURU-5070 channel state; BUG-020 documented.
- 2026-06-02 recompile: BSOD detection, server build webhook wiring, build channel default beta; migration count 46 -> 48.
- 2026-05-26 recompile: Added the Capabilities / Feature Set section, synthesized from authoritative artifacts. Changelogs are an unreliable capability source — migrations + commit log are authoritative.
Backlinks
- clients/cascades-tucson — RECEPTIONIST-PC enrolled (site CascadesTucson)
- systems/gururmm-build — Physical server at 172.16.3.30; GuruRMM API 3001, ClaudeTools API 8001, Coord API, MariaDB, PostgreSQL, build pipeline
- systems/jupiter — Unraid host at 172.16.3.20; virsh host for all VMs; Docker: Gitea, NPM, Seafile
- systems/pluto — Windows build server (MSI, WiX) at 172.16.3.36