77 KiB
type, name, display_name, last_compiled, compiled_by, aliases, sources, backlinks
| type | name | display_name | last_compiled | compiled_by | aliases | sources | backlinks | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| project | gururmm | GuruRMM | 2026-07-04 | Howard-Home/claude-main |
|
|
|
GuruRMM
Summary
GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 359 enrolled agents (228 typically fresh/online) across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints.
Current version: agent 0.6.77 / server v0.3.96 as of 2026-07-04 (was 0.6.75 / 0.3.87 on 2026-06-25). Active server_error alerts reduced to 0 fleet-wide after the 2026-07-04/05 fix batch.
Easy-bugs batch + alert-triage fixes deployed 2026-07-05 (server v0.3.95 + v0.3.96): Fixed three long-standing silent-failure classes surfaced by a full 24h alert triage: (1) the update scanner dropped macOS/Linux agent binaries during its self-check because they cannot execute on the Linux build host — every macOS/Linux agent had never been offered an update; fixed with host-native-vs-foreign self-check regimes (foreign platforms trust the filename version, host-native binaries stay a hard --version gate). (2) Hardware inventory silently failed to persist for 7+ fleet machines because Windows registry strings routinely carry embedded NUL bytes that Postgres jsonb/text rejects; fixed with a new shared db/sanitize.rs (NUL + C0/C1 control-char strip, collision-safe). (3) WebSocket log noise — benign disconnects and pre-auth connection churn (sleeping laptops, half-open NAT, scanners) were logged as ERROR and minted server_error alerts; reclassified to INFO, with genuine auth rejections (bad key, malformed auth) staying ERROR. Also fixed in the same batch: create_or_update_alert now returns Option<Alert> and survives an agent cascade-delete mid-flight (FK violation -> Ok(None) instead of aborting the offline sweep); move_agent maps a duplicate-site-device conflict to 409 instead of a raw 500. Went through a 19-agent high-effort code review before merge (9 confirmed findings, all fixed). See Recent Work for detail.
Cross-site agent duplication fixed 2026-07-04: Root cause of ~40 duplicate agent rows (24 at Dataforth D2->D1 alone): both WS enrollment paths deduped only within (site_id, device_id), so re-running a different site's installer on an already-enrolled machine created a second row instead of finding the existing one. Fixed to match by hardware device_id across all sites and re-home via move_agent_to_site — but a same-day follow-up (1c1cac2) stopped the auto-re-home on every reconnect (the agent re-presents its install-site code as its API key on every beat, which was bouncing admin-moved agents back to their original site). Net effect: dedup by device_id, but site placement stays admin-controlled and persists.
Legacy (Server 2008R2/Win7, Rust 1.77) build variant disabled 2026-07-04: The unpinned legacy dependency re-resolve pulled edition2024 crates (chacha20/rand_core 0.10.1) that Rust 1.77 cannot parse — a BUG-021 recurrence that fail-closed the entire Windows build (blocking modern amd64/x86/tray artifacts too). Gated behind LEGACY_ENABLED=false; the last-built legacy artifact (agent 0.6.76) is preserved and frozen for existing 2008R2/Win7 endpoints. 2008R2 support returns via a separate legacy-only rewrite (C++/.NET or a clean minimal Rust core), not by re-enabling the 1.77 toolchain — captured as RMM_THOUGHTS Feature 12 (Raw, Mike's requirement).
Policy-Controlled Tray Deployment shipped 2026-07-04 (code merged, policy default OFF): The tray binary is now deployed to every Windows endpoint via the agent-update flow (previously only fresh MSI installs got it — fleet census found ~111/113 sampled hosts with no tray at all) but its launch is gated on tray.enabled policy — no process, no icon, until a client/site/agent policy turns it on. Authenticode-verified + version-locked to the agent. Built via a rejected-then-corrected design (17-agent code review + Gemini + Grok adversarial critique caught 9 defects in the first attempt — see Recent Work). Fleet-wide policy enablement (Task 4) is not yet flipped.
VSS Policy Configurator shipped 2026-07-03 (SPEC-016 redesign, agent v0.6.76): The agent no longer runs its own scheduled shadow-copy create+prune loop — the MITRE T1490 "Inhibit System Recovery" surface that got the agent blocked by CrowdStrike Falcon. It now configures two kernel-enforced retention governors (VSS diff-area size cap + MaxShadowCopies count) and registers a scheduled task whose action is a native-COM create only (never a delete) — retention becomes Windows' own FIFO eviction, not agent code. Works uniformly on server AND client Windows SKUs (previously vssadmin create shadow blocked workstations). A 23-agent code review caught a data-loss-class defect pre-merge (writing the machine-global MaxShadowCopies would have evicted OTHER products' shadows) — dropped in favor of the size cap as sole governor. Live-verified on the T1490 canary (NEPTUNE, Falcon present): legacy scheduled task removed, no T1490 trip, 17 existing shadows preserved. Deployed to the beta channel only (0.6.76); stable fleet remains on 0.6.66 pending a deliberate promotion.
Vendor-agnostic Integrations Framework shipped 2026-07-01..02 (migration 065, CrowdStrike Falcon first plugin): A generic per-partner plugin framework (integrations, integration_credentials, integration_client_mappings, integration_sync_state, integration_agent_links tables) so a new EDR/security vendor is a new server module, not a new schema — modeled on the proven MSPBackups per-partner shape. CrowdStrike Falcon is the first plugin: API client, RMM-client-to-Falcon-CID mapping, and a periodic sync worker pulling sensor state (version/status/RFM) into integration_agent_links. REST CRUD at /api/integrations; no dedicated dashboard UI yet (API/backend only).
SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA — code merged + deployed 2026-06-23..24, NOT guaranteed to work; Route B = server-orchestrated): The code is on main and deployed, but the feature is still beta and unreliable — it succeeds on cleanly-uninstallable programs and is explicitly NOT guaranteed for the hard cases (most managed/protected AV, WiX/Burn bundles, uninstallers that need UI or spawn lingering child processes — Launchy/AIMP-class, drivers/system components). Treat results as best-effort, verify removals, and do not rely on it for production removals yet. See the Capabilities section for full detail + the open gaps.
Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+): One install URL/script self-detects arch/OS/legacy and pulls the correct agent; the dashboard install UI now points at it. P2 (native i386 bootstrapper EXE for locked-down boxes) pending.
Physical server migration completed 2026-06-11: New physical box at 172.16.3.30 running Ubuntu 26.04 + PostgreSQL 18. Old VM at .46 decommissioned 2026-06-12.
See also: wiki/projects/guru-rmm.md is a redirect tombstone pointing here (slug disambiguation).
Repo: azcomputerguru/gururmm on Gitea (internal: http://172.16.3.20:3000, external https://git.azcomputerguru.com/azcomputerguru/gururmm.git). The copy at projects/msp-tools/guru-rmm is a git submodule tracking the active repo; its pinned gitlink intentionally lags origin/main. Development uses git -C projects/msp-tools/guru-rmm or worktrees off origin/main to access live code. Howard is authorized for merges/deploys (cleared 2026-06-21 by Mike).
Goal: Full-featured MSP platform rivaling commercial RMMs, with a companion PSA (GuruPSA, separate future repo) designed as a truly integrated unified system.
Recent Work
2026-07-04..05 — Easy-Bugs Batch + Alert Triage: Update-Scanner, Inventory NUL, WS Noise (Howard-Home)
Full 24h alert triage (GET /api/alerts) surfaced 5 backup_failed (client-side, unresolved — see Active State), 2 active server_error (product bugs), and 1 resolved mass_offline_fleet (36 agents, transient network blip). Root-caused both server errors against the server's own self-log (pseudo-agent 00000000-...-001):
move_agent500 -> 409: moving an agent into a site that already holds a row for the samedevice_idtrippedidx_agents_site_deviceas a raw 500; now maps to 409 with an actionable message. 13 duplicate-hostname agent pairs fleet-wide are the trigger (stale legacywin-*/hw-*device_ids alongside re-enrolled hardware-UUID records) — not yet cleaned up.- Offline-sweep
RowNotFound:create_or_update_alert's UPDATE-by-id usedfetch_one; an alert row deleted mid-sweep (agent-delete cascade) aborted the whole sweep. Switched both branches tofetch_optionalwith fall-through to INSERT (Result<Option<Alert>>,Ok(None)= agent gone). - Inventory NUL rejection (alert-invisible, found in logs only): 7+ agents (IMC1, Seth-PC, QWM-JOHN, QWM-SHEILA, goldstar19, SIF-SERVER, Christine-Win10) never persisted hardware inventory — Windows registry strings carry embedded NULs, Postgres jsonb/text rejects them. New
server/src/db/sanitize.rs(strip_nul,clean_control_chars) applied over 5 jsonb blobs (values + keys, collision-safe) + text columns. - Update-scanner warn-spam / silent update starvation: the self-check tried to execute every platform's binary to read its embedded version — macOS/Linux binaries can't execute on the Linux build host, so they were dropped from the scannable set entirely, meaning macOS/Linux agents were never offered updates. Fixed with host-native-vs-foreign regimes: foreign-platform binaries trust the filename version; host-native binaries stay a hard gate (a native binary that can't report its version is corrupt) with a best-effort exec-bit repair for 0644 CI-copy artifacts.
A concurrent same-branch session absorbed a prior session's in-progress inventory/scanner fixes; the combined diff went through a 19-agent high-effort code review (9 confirmed findings, 0 refuted) before merge — findings included rebuilding the alerts fall-through (it still failed its own motivating scenario on first pass), tightening the scanner's foreign/host-native split, and moving WS-noise logging from DEBUG to INFO (a mass-disconnect anomaly must stay visible at the default log level). Merged + deployed as server v0.3.95 (06:09 UTC): scanner now indexes 8 platform/arch combos (was 4, Windows-only); zero ERROR lines post-deploy.
Continued into the last stale alert: "Authentication timeout" WS churn (2-6/day baseline, 57-spike correlated with the same day's fleet-dedup work). Profiled 7 days of journal, concluded pre-auth connection noise (peer connects, never authenticates — sleeping laptops, scanners, half-open NAT), added is_preauth_connection_churn to classify it INFO while keeping genuine rejections (bad API key, malformed auth) at ERROR. Deployed as server v0.3.96 (06:23 UTC). Active server_error alert count: 0.
BUG-009 (Logs.tsx isError handling) and BUG-011 (: any occurrences) were verified already fixed in-tree during the same pass — only the stale FEATURE_ROADMAP.md status lines needed correcting, not the code.
Remaining/pending from this session: the machine-side backup failures (the user's actual priority) — GND-SERVER (Grabb & Durando, chronic, 42 nightly failures since 2026-06-17), AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial) — investigation started but not completed; all say "no error detail reported" (MSP360 sync doesn't capture failure detail). 13-pair duplicate-agent cleanup still open. WS auth-timeout source (~1/min, unauthenticated) still unidentified (needs Feature 5, server-side public-IP capture — still unapproved).
2026-07-04 — Policy-Controlled Tray Deployment (Howard-Home / Mike, merged f6a4251)
Made gururmm-tray.exe deployment (materialize, policy-agnostic) independent from its runtime (launch, policy-gated) — previously the watchdog launched the tray whenever the binary existed, with no enforcement of the already-wired tray.enabled policy flag, and the binary itself only ever reached fresh MSI installs (fleet census: 1 signed tray, 1 stale unsigned, ~111 none out of 113 sampled). A first design (inline tray refresh folded into do_update) was rejected by a 17-agent code review plus independent Gemini 3.1 Pro and Grok adversarial critiques — 9 verified defects including permanent-binary-loss on a failed swap, a checksum sourced from an unauthenticated download-host sidecar (supply-chain gap), and an update/rollback tray-agent version-skew case.
Corrected design shipped as Tasks 1-3 + 5: policy-gated launch (default OFF), a materialize FSM with staging + a versioned mutex (atomic swap, restore-on-failure, authenticated-payload checksum), server capability-gated materialize, and Authenticode-verify + observability. Task 4 (flip fleet-wide materialize on, policy still OFF) is the deliberate next step — not done yet. Merged to main 2026-07-04 12:23 PDT.
2026-07-03..04 — VSS Policy Configurator (SPEC-016 Redesign) (Mike, merged de30b2b)
Rebuilt the agent's Volume Shadow Copy handling from "scheduled create+prune operator" (the MITRE T1490 surface Falcon blocked) to "policy configurator, never a deleter." Tasks 2-7: vss-create verb + scheduled-task registration (both daily-time and every-N-hours trigger modes) with legacy-task migration; per-volume diff-area size-cap governor set on policy apply; retirement of the scheduled create+prune pass entirely (SPEC-025 compliance heal reworked to create-only); status/compliance reporting updated to the configurator model; dual-target release build; runtime verification on both a Server 2019 box and a Win11 Pro client with Falcon present.
A 23-agent workflow code review before merge caught a genuine data-loss defect: the original design would have set the machine-global MaxShadowCopies registry value to retention_count, which FIFO-evicts other VSS consumers' (System Restore, third-party backup) shadows below that count fleet-wide. Fixed by dropping MaxShadowCopies management entirely — the per-volume size cap is the sole governor; retention_count/retention_max_age_days become advisory (a warning fires if retention_max_age_days is still set, since age-pruning requires deletion). Nine other findings fixed (cap-before-create ordering, an alias bridging the legacy task-name upgrade window, interval-aware compliance windows, bounded IVssAsync wait instead of INFINITE, and others).
Merged to main; the build pipeline publishes to the beta channel, not stable (a mid-session correction) — beta head reached 0.6.76 (8 agents), stable stayed at 0.6.66 (233 agents) by design. Verified live on the T1490 canary NEPTUNE (moved to beta channel for the test): create task present, legacy GuruRMM-VSS-Snapshot task removed, MaxShadowCopies correctly left unset, 17 pre-existing shadows preserved (the data-loss fix proven in production), and a live vss-create produced new shadows on two volumes under Falcon with no T1490 trip.
2026-07-01..02 — Vendor-Agnostic Integrations Framework + CrowdStrike Falcon Plugin (Mike)
Built a generic third-party-integration plugin framework (migration 065: integrations, integration_credentials, integration_client_mappings, integration_sync_state, integration_agent_links) so CrowdStrike Falcon — and future vendors (SentinelOne, Guardz) — are a new server module rather than a new bespoke schema each time, generalizing the per-partner MSPBackups shape (migration 034). Credentials reuse the same AES-256-GCM Encryptor chokepoint as the existing credentials table.
Five tasks, one per day-slice: (1) framework schema migration; (2) plugin trait + registry + DB layer; (3) generic REST API (/api/integrations CRUD, credentials, client mappings, sync-state, connection test); (4) CrowdStrike Falcon API client + plugin registered against the framework; (5) periodic sync worker pulling Falcon sensor state (version, status, RFM flag, CID) into integration_agent_links. No dedicated dashboard UI yet — API/backend only as of this compile.
2026-06-23..24 — SPEC-030 Remote Software Uninstall + Universal Installer (Howard-Home)
SPEC-030 software inventory + remote bulk uninstall (Route B — BETA, code merged to main + deployed but NOT guaranteed to work; agent 0.6.75 / server v0.3.87):
- Capability shipped:
/agents/:id/softwarelists installed programs;/agents/:id/software/uninstallruns a bulk uninstall; the dashboardSoftwareManager.tsxshows the live programs list + bulk-select + a live progress bar. Server endpoints inserver/src/api/software.rs. - Silent-uninstall engine (
agent/scripts/uninstall-engine.ps1): Tier-1 silent removal, BCU-informed detection + fail-fast, vendor silent-switch table, captures real process exit codes (fixed aStart-Processnull-ExitCode bug,f6c1945). The engine refuses to guess — Avast/AVG (icarus.exe), Malwarebytes (mb5uns.exe) correctly returnedneeds_interactiverather than risk a bad removal. - False-success class fixed (CRITICAL, PR #53): Avira's WiX/Burn Package Cache bundle GUID was misread as an MSI ProductCode;
msiexec /x {bundleGuid}returned 1605 ("not installed") and was mapped to success="already gone" while Avira stayed fully installed. Fix: skip the MSI tier for\Package Cache\bundles + a generic post-success ARP re-check (Test-StillInstalled) that downgrades a reported success to failed if the program is still registered. - Code-review correctness fixes (PR #54):
Test-StillInstalledmatches byproduct_codealone when present; skip the rescan on reboot-pending verdicts (3010/1641); exclude MSI-1605 no-op successes from the learned-timing sample. - Async removal jobs — the permanent fix (PR #56, migration 064): A synchronous bulk uninstall died at Cloudflare's ~100s timeout AND the agent's command kill-timeout killed the engine mid-flush. Rebuilt as a JOB: POST creates a
software_removal_jobsrow and returns ajob_idinstantly (0.06s); a background worker dispatches one program at a time;GET .../uninstall/jobs/:job_idis polled by the dashboard. Stop-gap flush-slack (PR #55) shipped first. - Three-state removal knowledge base (migrations 061-063):
software_removal_attempts(per-agent removal-tracking loop),software_knowledge(fleet KB:silent/requires_ui/unknown),software_knowledge_timing(self-tuning per-target timeout). - Known gaps surfaced (not yet built): drivers/system components register in ARP and appear in the bulk list; Launchy/AIMP-class uninstallers hang the agent on a lingering child process (~420s kill-timeout); no list-jobs endpoint (get-by-id only).
Universal self-detecting installer (Feature 9 — P1 built+deployed+verified, v0.6.71):
- One install path self-detects arch/OS (x64/x86/legacy/ARM) and pulls the correct agent; dashboard install UI repointed at it. Script path proven on-box (GND-JWILL). P2 (native i386 bootstrapper EXE) + offline-bundle still pending.
RMM_THOUGHTS movement: Feature 10 (AMPIPIT recovery environment add-on) -> Discussed (Mike likes it, deferred); Third-Party AV Removal Recipes + Win11 KB5095093 Point-in-Time Restore added as Raw; Feature 11 (OS Updates Management, Windows Update P1/P2 + Linux/Mac update-check P3, Syncro-modeled) added Raw 2026-07-03; Feature 12 (legacy 2008R2/Win7 agent rewrite) added Raw 2026-07-04 after the legacy build was disabled.
2026-06-22 — BUG-021 Fixed + v0.6.67 Stable; Fleet Verified (Howard-Home)
- BUG-021: Legacy build (Rust 1.77 via
$CARGO +1.77 --features legacy) was pullingwit-bindgen(edition2024) through fresh dep resolution. Fixed by pinninggetrandom 0.3.1+zeroize 1.8.1below edition2024. Commit1dce66d. Windows build went green 2026-06-22 02:19, v0.6.67. (This class of failure recurred 2026-07-04 with a different dep pair and led to the legacy variant being disabled outright — see Recent Work above.) - Fleet verified: ~270 agents / ~178 online; metrics rows flowing; all alerts have non-null dedup_keys; per-agent endpoints all HTTP 200.
- BUG-022 filed + fixed (PR #45):
WatchdogEventWS variant was dead code (watchdog has no WS connection); removed. Emptywatchdog_eventstable left in place. - Roadmap verified: BUG-018 (
cea87d4) and Event Log Watch UI (0fa65f5) confirmed on main.
2026-06-21 — Feature Batch, Audit, BUG-018 202+bg, Event Log Watch UI, Submodule Fix (Howard-Home + Mike GURU-5070)
Howard-Home sessions (three spans):
- BUG-019 FIXED (commit
66a7f4e, merged to main by Mike): Container agent no longer performs in-place binary self-update.agent/src/updater/mod.rsperform_update()early-returns inside a container. - BUG-018 FIXED (commit
cea87d4, merged): Handler redesign to 202 + background purge. Migration 061 (FK indexes on five previously-unindexed FK columns) landed later under a renumbered slot once SPEC-030 consumed 061-064. - Event Log Watch management UI (SHIPPED,
0fa65f5):/event-log-watchesCRUD wired into the dashboard. - Enrollment modal UX fix (SHIPPED to prod,
4027c86): close button, click-outside/Esc dismissal, list refresh on close. - Event Log Watch policy-clobber HIGH fix (PR #43,
432b434): Assigning/unassigning a policy silently wiped an agent's Event Log Watch monitoring — third instance of a config-push clobber class. Fixed by reusingwatch_rules_for_agentas the single source of truth. - sync.sh submodule-clobber root-cause fix: populate-only guard prevents
git submodule update --initfrom re-checking out the intentionally-lagging pinned gitlink on every sync. - Dashboard beta-before-main rule established.
Mike GURU-5070 session:
- BUG-019 merged to main (fast-forward
ed8cad3 -> 66a7f4e). gururmm-buildskill created: local pre-merge verification (verify.sh server|agent|dashboard|migrations).docs/BUILD.mdadded.- Howard cleared for GuruRMM merges/deploys (Mike decision).
2026-06-15 — BSOD Dedup Key Fix, MSI EXDEV Fix, Dashboard Offline Dedup, sync.sh Auto-Heal (Mike GURU-5070 + GURU-BEAST-ROG)
- BSOD alert dedup key fixed (commit
f0a4b7f, server v0.3.73): Dedup key changed tobsod:<agent>:<bugcheck_code>(stable across recurrences). - MSI EXDEV fix (commit
95ef901, server v0.3.74): Site-MSI builder staged in/tmp(tmpfs) then renamed cross-device to/opt/gururmm/downloads, failing EXDEV. Fixed to stage on the same filesystem. - Duplicate offline server alerts removed (commit
b6ed564). - sync.sh auto-heal (Phase 1):
resolve_submodule_collisions()moves only genuinely-colliding untracked files aside and retries, instead of a blind force. logs/analyzeswitched from Ollama to Claude API (commitc869e4d).- 500-error-leak fix (commit
58c1a96): All server error paths route throughinternal_err.
2026-06-12 — Beast Parallel Windows Build (Lever A) + command_type "cmd" Misdiagnosis Fix (Mike GURU-5070)
Beast parallel Windows build (commit cfbdb59): Two waves (5 concurrent stable + 2 concurrent legacy-1.77) via bash-driven concurrent SSH instead of a serial cmd /c chain. Per-variant --target-dir mandatory. Result: ~336s parallel vs ~622s serial cold, ~3.8x vs Pluto serial.
command_type "cmd" fix (commit 3de9faf, agent v0.6.66 stable): CommandType enum had no cmd variant — server-sent command_type:"cmd" failed serde parse and the agent silently dropped the message. Fixed with #[serde(alias = "cmd")] on Shell + a NAK on unparseable commands.
2026-06-11 — Physical Server Migration, Durable Agent Identity, Agent Comms Durability Phase 1
- Cutover: New physical box took over 172.16.3.30 (PG 18, Ubuntu 26.04); production down <27 min; old VM decommissioned 2026-06-12.
- Durable agent identity Phase 1 Task 1: Multi-location
device_id(registry+ProgramData on Windows, /etc+/var/lib on Linux, /Library+mirror on macOS) with self-heal;cleanup.ps1whitelists identity files. - Agent Comms Durability Phase 1 (agent v0.6.63 / server v0.3.68): Root cause = NAT conntrack asymmetry (server->agent pushes black-holed in idle conntrack gaps). Fix: agent
CommandAckon receipt + dedup cache; server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via a partial index so old agents keep legacy behavior.
2026-06-07 — Credential Inheritance, Offboarding Spec, UI Gap Batch, Backup Alert Quality Pass
- Server v0.3.45: credential inheritance (Global -> Client -> Site),
/effectiveendpoints. - Role-aware offline alerting + alert ignore/mute shipped (server-only
agent_offline,alert_mutesperma-silence). - Backup-alert quality pass: false
backup_failed15 -> 2 fleet-wide;backup_storage_lowalert type removed (false-positive metric). - UI gap batch + enrollment audit; SPEC-028 offboarding wizard specification (835 lines, implementation pending).
Capabilities / Feature Set
Synthesized from authoritative artifacts (API routes, agent modules, 65 migrations through migration 065, roadmap, commit log) — not from session logs alone. See Compilation Notes.
Agent<->server communication is a persistent authenticated WebSocket with auto-reconnect + heartbeat; on reconnect, in-flight commands flip to interrupted. Platform-parity rule: agent features ship on Windows/Linux/macOS in the same change (stub + TODO where a real impl isn't yet feasible).
Monitoring & Telemetry
- Core metrics per interval (policy-tunable): CPU %, memory %/bytes, disk %/bytes, network rx/tx deltas, uptime, logged-in user, user idle time, public/WAN IP. Cross-platform via
sysinfo. - Hardware sensor telemetry: CPU/GPU temps + full sensor array. Linux via
/sys/class/thermal;sysinfo::Componentsfallback. Windows LibreHardwareMonitor removed (CVE-2020-14979); replacement pending. - Process drill-down: top 10 by CPU + top 10 by memory in every metrics payload.
- Network state: per-interface IPv4/IPv6, MAC, derived CIDR subnets — sent on change only.
- Windows BSOD/kernel-crash detection:
agent/src/bsod.rspollsC:\Windows\Minidump, parses kernel dump header at fixed offsets, cross-references System event log, deduplicates via a watermark file. Always-Critical alert deduplicated by(agent_id, bugcheck_code)(changed from dump-hash 2026-06-15 so a mute silences all recurrences). - Inventory NUL/control-char hardening (2026-07-05, server v0.3.95):
server/src/db/sanitize.rsstrips NULs and C0/C1 control chars from inventory jsonb values/keys and text columns before upsert — fixed silent storage failure on 7+ machines carrying embedded NULs in Windows registry strings.
Remote Execution
- Command types:
shell,powershell,python, rawscript{interpreter},claude_task, andcmd(alias forshell). Options:timeout_seconds,elevated. Unknown/unparseable commands NAK immediately. - Execution context (
041):system(default) oruser_session(WTS token impersonation, Windows-only). - Commands individually cancellable, aborted on disconnect. Auditable history. Script library (
017) for stored scripts. - Agent Comms Durability (Phase 1, shipped 2026-06-11 — agent v0.6.63 / server v0.3.68): Agent sends
CommandAckon receipt (migration 058acked_at); dedup via a recent-results cache. Server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated viaidx_commands_agent_ackedpartial index so pre-durability agents keep legacy fail-on-timeout behavior. Phase 2 (live TTY) and Phase 3 (adaptive keepalive, bulk->HTTPS, half-open eviction) remain PLANNED, not shipped.
Volume Shadow Copy / VSS (Windows) — Policy Configurator (SPEC-016 redesign, shipped 2026-07-03, agent v0.6.76, beta channel only)
- The agent no longer schedules its own create+prune loop. It is a configurator: on each policy apply it idempotently sets two kernel-enforced retention governors — VSS diff-area size cap (native
IVssDifferentialSoftwareSnapshotMgmt) andMaxShadowCopiescount (evaluated, but NOT machine-globally written — see below) — and registers a scheduled task whose action is the agent's own native-COM create (IVssBackupComponents::DoSnapshotSet,VSS_CTX_CLIENT_ACCESSIBLE), never a delete. Retention is enforced by Windows' own FIFO eviction (volsnap.sys) when a governor is hit, eliminating the MITRE T1490 "Inhibit System Recovery" surface that had gotten the agent blocked by CrowdStrike Falcon. - Works uniformly on server AND client SKUs (previously
vssadmin create shadow, Server-only, gated client support) — the underlying VSS COM API andvolsnapcap eviction are not SKU-restricted, only thevssadmin.exeCLI is. MaxShadowCopiesis NOT set machine-globally (pre-merge code review caught this as a data-loss defect — it would FIFO-evict OTHER VSS consumers' shadows, e.g. System Restore or third-party backup, below the configured count). The per-volume size cap is the sole enforced governor;retention_count/retention_max_age_dayspolicy fields are advisory/ignored (a warning fires ifretention_max_age_daysis still set, since age-based pruning requires deletion).- On first policy apply, the legacy
GuruRMM-VSS-Snapshotscheduled task (present fleet-wide) is removed and replaced withGuruRMM-VSS-Create, supporting both daily-time and every-N-hours interval triggers. - On-demand admin operations (create/delete/browse/restore via
dispatch_operation,GET /agents/:id/vss/restores) are unaffected — this redesign only removes the scheduled create+prune loop. - Not yet promoted to the stable channel as of this compile (beta head 0.6.76; stable pinned at 0.6.66).
Software Inventory & Remote Uninstall (SPEC-030 — BETA, not guaranteed)
- Maturity: code merged to main + deployed (2026-06-23..24), but the feature is beta and unreliable — best-effort only.
- Route B (server-orchestrated):
GET /agents/:id/software,POST /agents/:id/software/uninstall(async ->job_id),GET /agents/:id/software/uninstall/jobs/:job_id,GET /agents/:id/software/removal-status+ resolve,GET /software/knowledge+ classify. - Silent-uninstall engine (
agent/scripts/uninstall-engine.ps1): refuses to guess (needs_interactiveon ambiguity); generic post-success ARP re-check (Test-StillInstalled) catches the WiX/Burn Package-Cache-bundle false-success class; orphaned ARP entries classifiedtier=orphaned; async-fork uninstallers get retry-verify. - Async removal jobs (migration 064): non-blocking bulk uninstall — job created +
job_idreturned in ~0.06s, background worker processes targets one at a time, dashboard polls live progress. - Fleet removal knowledge base (migrations 061-063): per-agent removal-tracking loop, fleet three-state KB (
silent/requires_ui/unknown), self-tuning per-target timeout. - Known gaps: drivers/system components register in ARP and appear in the bulk list (no exclusion/flag yet); Launchy/AIMP-class uninstallers hang the agent on a lingering child process; no list-jobs endpoint (get-by-id only). The av-removal-recipes spec (P3, removal-only) is the follow-on.
Third-Party Integrations Framework (NEW, migration 065, shipped 2026-07-01..02)
- Vendor-agnostic plugin framework:
integrations(one enabled plugin per partner+kind),integration_credentials(AES-256-GCM encrypted, sameEncryptorchokepoint as thecredentialstable),integration_client_mappings(RMM client -> external tenant id, e.g. a Falcon CID — single-CID partners map every client to one id, MSSP partners map child CIDs),integration_sync_state(per-integration sync health),integration_agent_links(RMM agent <-> external agent id + last-synced state, e.g. Falcon AID/sensor version/RFM). - API:
GET/POST /api/integrations,GET/DELETE /api/integrations/:id,GET/PUT /api/integrations/:id/credentials,GET/PUT /api/integrations/:id/mappings,DELETE /api/integrations/:id/mappings/:mapping_id,GET /api/integrations/:id/sync-state,POST /api/integrations/:id/test. - First plugin: CrowdStrike Falcon. API client + periodic sync worker (default
sync_interval_seconds=900) pulls sensor state intointegration_agent_links. Framework is modeled on the proven per-partner MSPBackups shape (migration 034). - No dedicated dashboard UI yet — this is API/backend-only as of this compile (verify before claiming a management screen exists).
Inventory & Discovery
- Hardware inventory (mfr/model/serial/BIOS, CPU, memory, disks, NICs, OS), software inventory, service inventory. On-demand refresh. NUL/control-char sanitized before storage (2026-07-05, see Monitoring section).
- VM / hypervisor / container detection (
032/033). - User/group inventory (
037-040): local + domain + Azure AD accounts, domain-join classification, domain-controller detection, group membership. - Network discovery: server-dispatched TCP/ARP/reverse-DNS scans; devices stream back and are persisted/editable.
Patch / Agent Update Management
- Self-updater: server sends version + URL + SHA256; agent downloads, verifies, atomically swaps, restarts, auto-rolls-back on failed reconnect.
- Auto-update gated on effective policy
auto_update. Force viaPOST /agents/:id/update. - Channel model: new builds tag beta by default; agents default to stable; promotion is a deliberate re-tag.
- Update-scanner platform coverage fixed 2026-07-05 (server v0.3.95): the scanner's self-check previously tried to execute every binary regardless of platform to read its embedded version, dropping macOS/Linux artifacts entirely because they can't run on the Linux build host — macOS/Linux agents had never been offered an update. Now uses host-native-vs-foreign self-check regimes: foreign-platform binaries trust the filename-embedded version; host-native binaries stay a hard
--versiongate (with a best-effort exec-bit repair for suspected 0644 CI-copy artifacts). Post-fix: scanner indexes 8 platform/arch combos (was 4, Windows-only). - Legacy fleet build (Server 2008R2/Win7, Rust 1.77) DISABLED 2026-07-04 (
LEGACY_ENABLED=false): the unpinned dep re-resolve pulled edition2024 crates Rust 1.77 cannot parse, fail-closing the entire Windows build. Last legacy artifact (agent 0.6.76) frozen and preserved for existing 2008R2/Win7 endpoints; modern builds advance normally. 2008R2 support returns only via a separate legacy-only rewrite (RMM_THOUGHTS Feature 12, Raw — not approved). - Universal self-detecting installer (Feature 9, P1 shipped v0.6.71): one install path self-detects arch/OS and pulls the correct binary.
- Promotion API:
POST /api/updates/rollouts/:version/promotebody{"os","arch"}; rollback via.../rollback.
Agent Identity & Enrollment
- Per-agent enrollment keys; site-specific MSI with SITEKEY injected.
enrolled_agentsis the authoritative enrollment history log. - Durable multi-location
device_id(Phase 1 Task 1, 2026-06-11): registry+ProgramData (Windows), /etc+/var/lib (Linux), /Library+mirror (macOS), self-healing;cleanup.ps1whitelists identity files. - Cross-site dedup fixed 2026-07-04 (commits
2851523,1c1cac2): Both WS enrollment paths previously deduped only within(site_id, device_id), so re-running a different site's installer against an already-enrolled machine created a cross-site duplicate. Fixed to match by hardwaredevice_idacross all sites (db::get_agent_by_device_id) and re-home viamove_agent_to_siteon first match. A same-day follow-up stopped auto-re-homing on every reconnect — the agent re-presents its embedded install-site code as its API key on every beat, which was silently bouncing admin-relocated agents back to their original site. Current behavior: dedup by device_id, but site placement is admin-controlled and persists across reconnects. 13 duplicate-hostname agent pairs remain fleet-wide (pre-fix legacywin-*/hw-*records) — cleanup pass not yet done; these are the ongoing trigger formove_agent409s.
Policy & Configuration Management
- Inheritance chain global -> client -> site -> agent; server computes merged effective policy, pushes via
ConfigUpdate. - Checks engine (
018/019): cpu, memory, disk, ping, port, script, service. Policy-attached check templates (024). - Remote registry (Windows): enumerate/read routed; write paths exist in the agent but are not routed yet [verify].
- Event Log Watch (migration 047): dashboard management UI at
/event-log-watches. Policy-clobber class bug fixed (PR #43) by reusing a single rule-shaper across all config-push sites. - Tray policy gate (NEW, shipped 2026-07-04, code merged, policy default OFF):
tray.enabledpolicy flag is now enforced by the watchdog (previously plumbed end-to-end but not enforced — the tray launched/showed regardless of the flag). Fleet-wide materialize (deploying the binary via agent-update, not just fresh MSI) has shipped in code; the deliberate flip to enable it fleet-wide (spec's "Task 4") has NOT happened yet.
Alerting & Watchdog
- Threshold alerts (ack/resolve, per-agent + fleet summary). Alert templates (
022). Maintenance mode (021). - Watchdog: separate supervising process; reports via
POST /watchdog-alert. - Role-aware offline alerting + alert ignore/mute (shipped 2026-06-07): server-only
agent_offline; site/fleet mass-offline aggregation;alert_mutesperma-silence. create_or_update_alerthardened (2026-07-05, server v0.3.95): now returnsResult<Option<Alert>>—Ok(None)when the target alert row was deleted mid-flight (agent cascade-delete FK violation) instead of aborting the caller. Both UPDATE-by-id branches switched fromfetch_onetofetch_optionalwith fall-through to INSERT (refresh_alert_rowhelper dedups the two near-identical UPDATE blocks). Fixed aRowNotFoundthat had been aborting the entire offline sweep whenever an alert was concurrently deleted (e.g. during duplicate-agent cleanup).- WebSocket log noise reclassified INFO, not ERROR (2026-07-05, server v0.3.96): benign disconnects (
is_benign_ws_disconnect) and pre-auth connection churn (is_preauth_connection_churn— sleeping laptops, scanners, half-open NAT never sending an auth frame) no longer mintserver_erroralerts. Genuine auth rejections (bad API key, malformed auth message) remain ERROR. Classified at INFO (not DEBUG) so a mass-disconnect anomaly stays visible at the server's default log level. move_agentduplicate-key hardened (2026-07-05): unique-violation onidx_agents_site_devicenow maps to 409 CONFLICT with an actionable message instead of a raw 500.
Credentials Management
- Encrypted credentials vault (
016): scoped global/client/site, typed, metadata-only by default with a separate/revealendpoint. Known HIGH item:/revealownership-scope check — [verify current state]. - Credential inheritance (deployed 2026-06-07): Opt-in hierarchical cascade, de-duplication,
/effectiveendpoints. - Integration credentials (2026-07-01, migration 065): reuses the same encryption chokepoint for third-party integration API keys (never sent to agents, never plaintext in
integrations.config).
Audit Log / Log Feedback Intelligence
- Append-only
audit_logtable (migration 056). Per-line log fingerprinting +log_signaturesaggregation (migration 057).logs/analyzeruns on the Claude API (switched from Ollama 2026-06-15).
Backup Integration (MSP360 / MSPBackups)
- Multi-provider config, connection test, scheduled sync, per-agent + fleet coverage report, agent<->MSP360 mapping with confidence scoring.
- Active chronic failure (unresolved as of 2026-07-04): GND-SERVER (Grabb & Durando) — 42
backup_failedalerts since 2026-06-17, failing nightly ~01:02 UTC, "no error detail reported." Also open: AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial). MSP360 sync doesn't capture failure detail — investigation needs a live PowerShell dispatch to read MSP360/CloudBerry local logs.
Remote Access (Tunnel)
- Agent side substantially built. Server side is a dead-code skeleton (no
/tunnelroutes). Live TTY design (agent-comms-durability Phase 2) will supersede the skeleton.
Identity / Multi-tenancy / Security
- Auth: JWT; agents auth over WS via per-agent API key + hardware device_id.
- Microsoft Entra ID SSO (OAuth2/OIDC + PKCE).
- Organizations / multi-tenancy: org CRUD, roles, limits, dev-admin impersonation.
- 500-error-leak fixed (commit
58c1a96): all server error paths route throughinternal_err.
Platform coverage
- Cross-platform (Win/Linux/macOS): metrics, network state, hardware/software/service inventory, user/group + DC/domain detection, checks, discovery, self-update, scripts/commands.
- Windows-only:
user_sessioncommand context, remote registry, tray-into-session (now policy-gated), watchdog SCM supervision, BSOD detection, VSS policy configurator, legacy fleet (frozen at 0.6.76). - macOS: agent deployed (Phase 1); tray is a stub; now correctly included in the update scanner's platform coverage as of 2026-07-05 (previously silently excluded).
Architecture
Components
| Component | Location | Tech | State |
|---|---|---|---|
| Server | 172.16.3.30:3001, systemd gururmm-server.service |
Rust, Axum | deployed, production; v0.3.96 |
| Dashboard | https://rmm.azcomputerguru.com, nginx | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production |
| Agent (Windows) | Endpoints, GuruRMMAgent service via WiX MSI |
Rust, Windows MSVC | deployed; modern build v0.6.77, legacy (2008R2/Win7) FROZEN at 0.6.76 and disabled |
| Agent (Linux) | Endpoints, systemd gururmm-agent |
Rust, musl static | deployed |
| Agent (macOS) | Endpoints, LaunchDaemon | Rust, aarch64/x86_64 | Phase 1 deployed 2026-05-12; code-signing issue on Apple Silicon; update-scanner platform gap fixed 2026-07-05 |
| Tray (Windows) | System tray, named pipe IPC | Rust | deployed to disk fleet-wide via agent-update (2026-07-04); launch policy-gated, default OFF |
| Tray (Linux) | System tray, Unix socket IPC | Rust, GTK | deployed 2026-05-24 |
| Tray (macOS) | Menu bar | Rust | stub/TODO |
| PostgreSQL DB | localhost:5432 on 172.16.3.30 | PostgreSQL 18 | deployed |
| Build pipeline | 172.16.3.30:9000 webhook + /opt/gururmm/ scripts |
Python, Bash | deployed; builds agents AND server |
| Windows build — Beast (PRIMARY) | GURU-BEAST-ROG over Tailscale | Rust MSVC, WiX 4.x | primary — two-wave parallel, ~336s |
| Windows build — Pluto (FALLBACK) | 172.16.3.36 | Rust MSVC, WiX v4 | fallback only |
Key Files & Repos
- Active repo:
azcomputerguru/gururmm— http://172.16.3.20:3000/azcomputerguru/gururmm (external: https://git.azcomputerguru.com/azcomputerguru/gururmm.git) - Submodule working tree:
projects/msp-tools/guru-rmm— gitlink intentionally lags origin/main; read live code viagit -C projects/msp-tools/guru-rmm show origin/main:<path>or a worktree. - Server binary:
/opt/gururmm/gururmm-serveron 172.16.3.30 - Server config:
/opt/gururmm/.env - New shared sanitize module:
server/src/db/sanitize.rs(strip_nul,clean_control_chars) — consumed bydb/logs.rsanddb/inventory.rs - Integrations framework:
server/src/api/integrations.rs,server/src/integrations/(plugin trait + registry), migration065_integrations_framework.sql - VSS configurator:
agent/src/vss_com.rs,agent/src/vss.rs,specs/vss-policy-config/,specs/vss-native-com/ - Tray policy-deploy:
agent/src/tray_stage.rs,agent/src/tray_state.rs,agent/src/authenticode.rs,specs/tray-policy-deploy/ - Downloads dir:
/var/www/gururmm/downloads/on 172.16.3.30 - Webhook handler:
/opt/gururmm/webhook-handler.py - API (internal): http://172.16.3.30:3001
- API (external): https://rmm-api.azcomputerguru.com
- Dashboard: https://rmm.azcomputerguru.com
- Vault paths:
infrastructure/gururmm-server.sops.yaml(API creds, DB password, sudo password),infrastructure/gururmm-server-physical(SSH ed25519 key — this fleet key works on 172.16.3.30 currently, not just the future physical box; confirmed 2026-07-04) - SSH to prod:
ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30(ed25519, key-only; password auth disabled) - Pre-merge verification skill:
bash .claude/skills/gururmm-build/scripts/verify.sh server|agent|dashboard|migrations
Repo Structure
gururmm/
├── agent/ Rust agent (managed endpoints)
│ └── src/
│ ├── authenticode.rs NEW — Authenticode signature verify (tray-policy-deploy)
│ ├── tray_stage.rs NEW — tray materialize FSM + staging + versioned mutex
│ ├── tray_state.rs NEW — tray policy state
│ ├── bsod.rs Windows BSOD/kernel-crash detection
│ ├── vss_com.rs Native VSS COM create/provision (vss-native-com + vss-policy-config)
│ ├── vss.rs VSS policy configurator entry points
│ ├── commands/mod.rs CommandExecutor + recent-results cache (ACK dedup)
│ ├── device_id.rs Durable multi-location device_id
│ ├── transport/
│ │ └── websocket.rs WS transport; execute_command ACK + dedup; cmd-alias NAK
│ ├── tunnel/ TunnelManager state machine
│ ├── metrics/ sysinfo collection + temps
│ ├── registry_ops/ Windows registry read/write
│ ├── updater/ Self-update handler (container guard, BUG-019)
│ └── main.rs
├── server/ Rust/Axum API server
│ └── src/
│ ├── api/
│ │ ├── integrations.rs NEW — integrations framework CRUD (crowdstrike-falcon Task 3)
│ │ └── ... ~40 route modules; software.rs SPEC-030
│ ├── alerts/ Alerting modules (offline.rs: offline_sweep + mass_offline)
│ ├── db/
│ │ ├── sanitize.rs NEW — shared NUL/control-char sanitize (2026-07-05)
│ │ ├── alerts.rs create_or_update_alert -> Option<Alert>, refresh_alert_row
│ │ ├── inventory.rs NUL-scrub over jsonb + text before upsert
│ │ └── software_jobs.rs SPEC-030 async removal jobs
│ ├── updates/
│ │ └── scanner.rs host-native vs foreign self-check regimes (2026-07-05)
│ ├── ws/mod.rs WS handler; is_benign_ws_disconnect / is_preauth_connection_churn
│ └── mspbackups/ MSP360 backup integration
├── dashboard/ React/TypeScript UI
│ ├── pages/
│ │ └── EventLogWatches.tsx
│ └── components/
│ └── SoftwareManager.tsx
├── agent/scripts/ uninstall-engine.ps1
├── tray/ System tray binary
├── installer/ WiX v4 MSI; cleanup.ps1; universal self-detecting installer
├── deploy/
│ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh (LEGACY_ENABLED=false gate added 2026-07-04)
├── scripts/ Build/ops scripts
└── docs/ FEATURE_ROADMAP.md, UI_GAPS.md, RMM_THOUGHTS.md, specs/
specs/ tray-policy-deploy/, vss-policy-config/, vss-native-com/, crowdstrike-falcon/,
av-removal-recipes/, remote-software-uninstall/, agent-comms-durability/, ...
Development
Current Focus
As of 2026-07-04 (agent 0.6.77 / server v0.3.96):
- Machine-side backup failures (highest user priority, unresolved): GND-SERVER chronic 42-failure streak, AD1, LAB-SVR, DESKTOP-EVA4H1A — MSP360 sync reports no error detail; needs a live PowerShell dispatch to read local MSP360/CloudBerry logs or the mspbackups.com console.
- 13 duplicate-hostname agent pairs fleet-wide (legacy
win-*/hw-*device_ids alongside re-enrolled hardware-UUID records) — the ongoing trigger formove_agent409s; a one-time cleanup pass (delete stale offline duplicates) is not yet done. - WS auth-timeout source unidentified: ~1/min unauthenticated connection attempts, now correctly classified INFO (not alerting), but the source is still unknown — would need connection-level source-IP visibility (Feature 5, server-side public-IP capture, still unapproved) or box-level tcpdump.
- Tray policy fleet enablement (Task 4, tray-policy-deploy spec): materialize code is merged and deployed; flipping
tray.enabledon for the fleet is a deliberate next step, not yet done. - VSS configurator stable-channel promotion: shipped to beta (0.6.76) only; promotion to stable is a separate deliberate step pending further soak.
- Legacy (2008R2/Win7) agent rewrite (RMM_THOUGHTS Feature 12, Raw): not approved; needs a language/runtime decision (.NET Framework 4.x vs C++) and scope decision (trimmed feature set: enroll, check-in, metrics, command exec, self-update only) before a
/shape-spec. - Integrations framework dashboard UI: CrowdStrike Falcon backend + sync worker are live; no management screen exists yet in the dashboard.
- SPEC-030 software-uninstall follow-on (Route A / recipes): Launchy/AIMP-class uninstallers still hang the agent; driver/system-component exclusion flag and a list-jobs endpoint remain unbuilt.
- Agent-comms-durability Phase 2/3 (planned): live TTY, adaptive keepalive, bulk file transfer -> short-lived HTTPS, server half-open eviction sweeper.
- Durable agent identity Phase 2-3 (pending soak): guarded auto-reclaim + an operator merge tool for the pre-fix ghost/duplicate agents.
- SPEC-028 offboarding wizard: specification complete; implementation pending.
- Model routing correction (2026-07-04, logged --correction): Tier-0 simplify/cleanup passes should route to local Ollama before Claude; a routing miss on this session's easy-bugs batch was logged for tracking.
Patterns & Anti-Patterns
New anti-patterns (2026-07-01..05):
| Pattern | What Went Wrong |
|---|---|
| Update-scanner self-check executing every binary regardless of target platform | macOS/Linux binaries can't run on the Linux build/scan host, so they were silently dropped from the scannable set entirely — those agents never received an update offer. Fix: host-native binaries get a hard --version execute-and-check gate; foreign-platform binaries trust the filename-embedded version. |
Writing a machine-global VSS registry governor (MaxShadowCopies) to a per-policy retention_count |
MaxShadowCopies is shared by ALL VSS consumers on the box (System Restore, third-party backup) — lowering it FIFO-evicts THEIR shadows too, a genuine data-loss class caught only by a 23-agent pre-merge code review. Use a per-volume size cap as the sole enforced governor instead. |
| Re-homing an agent's site on every WS reconnect after a device_id-based dedup fix | The agent re-presents its embedded install-site code as its API key on every beat; auto-re-homing on each connect silently bounces an admin-relocated agent back to its original install site. Dedup by device_id, but leave site placement admin-controlled and persistent. |
fetch_one on an UPDATE-by-id inside a concurrently-deleting table |
An agent cascade-delete can remove the target row between a SELECT and the UPDATE; fetch_one panics RowNotFound and aborts the whole caller (here: the entire offline sweep). Use fetch_optional with a fall-through path. |
| Two sessions editing the same branch's working tree without a coord lock | A concurrent session's uncommitted edits were silently absorbed into another session's commit. Benign this time (same branch, all changes verified at HEAD) but neither session checked the coord lock list first — take a lock before editing a shared submodule branch. |
Assuming a spec's shape.md/plan.md header "Status: not started" reflects current reality |
Tasks can ship and merge (git log shows Tasks 1-3+5 done) while the spec doc header is never updated. Verify against git log/deployed version, not the spec file's status line. |
Existing anti-patterns (carried forward):
| Pattern | What Went Wrong |
|---|---|
useMemo with stable deps for data-dependent values |
queryClient is stable, memo never recomputes after queries resolve. Use useQuery instead. |
| Deploying without stopping the server first | "text file busy" kernel error. Always systemctl stop before cp. |
Building without DATABASE_URL |
sqlx compile-time macros fail. |
DB migrations without inserting into _sqlx_migrations |
Server crashes on start. Must insert SHA-384 checksum manually. |
| WiX MSI builds on Linux | WiX requires msi.dll. Must build on Pluto or Beast. |
| Manual builds via SSH | All builds go through webhook-handler.py. |
Restart-Service GuruRMMAgent -Force in command scripts |
Kills agent before it can report result. Use a scheduled task with delay instead. |
sudo -u guru git in systemd build context |
git rejects repo as dubious ownership. Use git config --system --add safe.directory. |
+1.77 legacy builds without pinned deps |
Fresh dep resolution repeatedly pulls edition2024 crates Rust 1.77 can't parse (BUG-021, recurred 2026-07-04 with a different dep pair, led to the variant being disabled outright rather than re-chasing pins). |
| Dead WebSocket write half | WS write fails, send task dies, receive loop keeps agent connected with a dead write half. Fix: tokio::select! monitoring both tasks. |
Using the minidump crate for Windows kernel dumps |
Only parses Breakpad MDMP, not Windows kernel PAGEDU64. Parse DUMP_HEADER64/32 at fixed offsets directly. |
| Build classification defaulting to stable | New builds default beta; promotion is an explicit re-tag. |
command_type: "cmd" in API commands |
Not a valid CommandType variant originally — agent silently dropped the message. Now aliased to Shell. |
| Reading the stale submodule working tree for code | The submodule gitlink intentionally lags origin/main. Always read via git -C ... show origin/main:<path> or a worktree. |
Good patterns (carried forward + new):
- Shared sanitize chokepoint (
db/sanitize.rs, 2026-07-05) — one NUL/control-char scrubber consumed by bothdb/logs.rsanddb/inventory.rs, replacing duplicated ad hoc scrub logic. - Capability-based classification (host-native vs foreign) instead of a single universal check — the update scanner now branches its self-check strategy per target platform rather than applying one strategy fleet-wide.
- Multi-AI adversarial review before shipping risky agent-side changes — the VSS redesign (Grok + Gemini) and tray redesign (17-agent Claude review + Grok + Gemini) both caught genuine defects (a data-loss governor and 9 architecture defects respectively) that a single-model review missed on the first pass.
- Platform parity rule — any agent feature goes on Windows + Linux + macOS in the same commit.
gururmm-buildskill for pre-merge verification — does NOT trigger production build; merging to main is the only prod trigger.- Dashboard beta-before-main rule — dashboard changes go to beta (auto-built from main) first; promote explicitly.
- Two independently-deployable slices for cross-component features, capability-gated — agent slice and server slice can deploy in any order; old agents/servers keep legacy behavior until the gate flips.
Build & Deploy
CRITICAL: Never trigger builds manually via SSH. All builds go through the webhook pipeline.
Gitea push to main
-> webhook-handler.py (172.16.3.30:9000)
-> build-shared.sh (auto-version bump)
-> build-linux.sh
-> build-windows.sh (Beast PRIMARY over Tailscale || Pluto FALLBACK
TWO-WAVE PARALLEL BUILD; legacy wave now gated LEGACY_ENABLED=false)
-> build-mac.sh (stub — no build machine configured yet)
-> build-server.sh (gated on server/ changes; backup+rollback)
-> artifacts -> /var/www/gururmm/downloads/ with sha256 + -latest symlinks + .channel (beta)
Legacy variant disabled 2026-07-04 (84a82a3): LEGACY_ENABLED=false in build-windows.sh gates the legacy wave/copy/deploy/symlink off entirely; the last-built legacy artifact (0.6.76) is preserved and excluded from cleanup so existing 2008R2/Win7 endpoints keep receiving that frozen build. Modern (amd64/x86/tray/debug) builds are no longer blocked by legacy dep-resolution failures.
Channel/promotion model, downloads layout, dashboard beta/prod split, DB migration hazards, and Windows build host detail are unchanged from the 2026-06-25 compile — see prior History Highlights / Compilation Notes for the full mechanics; current migration count is 065 (065_integrations_framework).
Active State
Fleet (as of 2026-07-04):
- 359 enrolled agents; 228 fresh/online at last sample
- Agent 0.6.77 modern / 0.6.76 legacy-frozen-and-disabled; server v0.3.96
- Active
server_erroralerts: 0 (cleaned to zero by the 2026-07-05 deploys) - 13 duplicate-hostname agent pairs outstanding (not yet cleaned up)
- 5 active
backup_failedalerts (GND-SERVER chronic, AD1, LAB-SVR, DESKTOP-EVA4H1A) — investigation open - VSS configurator on beta channel only (0.6.76, 8 agents); stable fleet pinned at 0.6.66 (233 agents)
- Beta channel: site "Mike's Car" (
103c10b9-c1de-4dd8-b382-b8362ed3143e) hasupdate_channel='beta'(persists across re-enrollment)
API auth:
POST /api/auth/login-> JWT (~24h)- Creds: vault
infrastructure/gururmm-server.sops.yaml - Key endpoints:
GET /api/agents,POST /api/agents/:id/command,GET /api/commands/:id,POST /api/agents/:id/update,POST /api/updates/rollouts/:version/promote - Software (SPEC-030):
GET /api/agents/:id/software,POST /api/agents/:id/software/uninstall(-> job_id),GET .../uninstall/jobs/:job_id,GET .../removal-status,GET /api/software/knowledge - Integrations (NEW):
GET/POST /api/integrations,GET/DELETE /api/integrations/:id,GET/PUT /api/integrations/:id/credentials,GET/PUT /api/integrations/:id/mappings,GET /api/integrations/:id/sync-state,POST /api/integrations/:id/test - Command fields:
command_type(shell/powershell/python/script/claude_task/cmdalias),command, optionalcontext(system/user_session),timeout_seconds/elevated.
Dashboard — complete and working: Agents management, Clients/Sites CRUD, Commands execution + terminal, Logs + AI analysis (Claude API), Alerts, Metrics, Auto-update triggering, Network state, Entra ID SSO, Policies Dashboard, Registry editor (read-only), MSP360 backup status + mappings/verify UI, Organizations + dev-admin impersonation, Credentials with inheritance, AgentDetail Crashes tab + version history, Event Log Watches management, Software Manager (BETA, not guaranteed).
Dashboard — incomplete (see UI_GAPS.md):
- Watchdog alerts UI — blocked on 2 missing server routes
- BSOD in Alerts stream (Phase 2 deferred)
- Tunnel session management (server skeleton)
- Offboarding wizard UI (SPEC-028 complete; implementation pending)
- Alert mute UI — server endpoints ready, dashboard NOT started
- Integrations Center UI (NEW gap) — CrowdStrike Falcon framework + sync worker are live server-side; no management screen in the dashboard yet
- Tray policy toggle surfacing —
tray.enabledis enforced server/agent-side; verify whether the policy UI exposes the toggle to operators [verify]
Security backlog (HIGH):
credentials/:id/reveal— horizontal privilege escalation (no ownership scope check)
Key Architecture Decisions (LOCKED)
These decisions are locked. Do not reverse without explicit user approval.
- Per-agent enrollment keys.
- Site-specific MSI generation.
- No TOML/config for endpoints — server URL compiled into binary.
- Policy inheritance chain — global -> site -> client -> agent.
- Platform parity rule — any agent feature ships on Windows, Linux, and macOS in the same change.
- Watchdog as separate process — reports via REST (
POST /watchdog-alert), no WebSocket. - Build pipeline is the only path to production.
- Multi-tenancy identity model (ADR-001) — Dev -> Partner -> Client.
- Holistic feature development — every feature requires backend + API + dashboard UI + docs.
- AI-optional operation.
- Durability-first command delivery — the durable Postgres
commandsrow is the source of truth; the WebSocket is only a delivery hint. - VSS: configure, never delete on a schedule (added 2026-07-03) — the agent may only set retention governors and create shadows on a schedule; scheduled or automatic deletion reintroduces the MITRE T1490 surface. On-demand admin-initiated delete stays allowed.
- Third-party security integrations are a plugin, not a schema (added 2026-07-01) — a new vendor implements the
integrationsplugin trait against the generic framework tables; it does not get its own bespoke migration set.
History Highlights
| Date | Event |
|---|---|
| 2025-12-15 | Project genesis: Windows service + Linux installer + site code auth + build server. |
| 2026-04-19 | Full drill-down navigation, auto-install on first run, Pluto build VM setup started. |
| 2026-04-21 | MSI build fix. DESIGN.md created. BirthBiologic onboarded. |
| 2026-04-29 | UI_GAPS.md created. Holistic development principle formalized. |
| 2026-05-12 | macOS agent Phase 1 deployed. Code signing issue on Apple Silicon noted. |
| 2026-05-15 | Dead WebSocket write-half bug fixed. Temperature struct field mismatch fixed. |
| 2026-05-16 | Watchdog bugs fixed. /feature-request skill created. |
| 2026-05-17 | Syncro PSA Integration added to roadmap. Office power failure recovery. |
| 2026-05-18 | Multi-tenancy architecture (ADR-001) decided. 5 SPEC documents created. |
| 2026-05-19 | AD2 crash-loop fix. MSP360 integration completed. Process drill-down modal. |
| 2026-05-23 | /rmm-audit pass. Agent optimization Phases 1A-3. MSRV bumped to 1.85. Fleet at 0.6.29. |
| 2026-05-24 | Linux tray IPC + GTK merged. Build pipeline split per-platform. Fleet converged to 0.6.38. |
| 2026-05-31 | Roadmap reconciliation. MSPBackups mapping/verify UI + dev-admin impersonation UI deployed. SPEC-021 written. |
| 2026-06-01 | BUG-016/017 fixed. BSOD detection feature (Phase 1) implemented and merged, agent 0.6.51. |
| 2026-06-02 | Server 0.3.37 + migration 048 deployed. Webhook wired to dispatch build-server.sh. |
| 2026-06-04 | BUG-020 (ghost tray icons) fixed to beta (137dd85). |
| 2026-06-07 | Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit all shipped same day. |
| 2026-06-11 | Physical server migration executed (<27 min downtime). Ghost agent root-caused, durable identity Phase 1 Task 1 shipped. Agent-comms-durability Phase 1 shipped (agent v0.6.63/server v0.3.68). |
| 2026-06-12 | Beast parallel Windows build wired in (cfbdb59, ~336s). command_type "cmd" root-caused and fixed (3de9faf). |
| 2026-06-15 | BSOD dedup key changed to bugcheck code. MSI EXDEV fix. sync.sh Phase 1 auto-heal. logs/analyze switched to Claude API. 500-error-leak closed. |
| 2026-06-18 | sync.sh submodule auto-heal verified fleet-wide. |
| 2026-06-21 | BUG-019/018 fixed and merged. Event Log Watch UI shipped. Howard cleared for GuruRMM merges/deploys. gururmm-build skill created. |
| 2026-06-22 | BUG-021 (legacy dep-pin) fixed, v0.6.67 stable. BUG-022 (dead WatchdogEvent path) fixed. Fleet verified ~270/~178 online. |
| 2026-06-23 | Universal self-detecting installer (Feature 9) P1 shipped (v0.6.71). av-removal-recipes spec added. |
| 2026-06-24 | SPEC-030 remote software uninstall — BETA — code merged + deployed (Route B, async jobs, three-state KB, migrations 061-064). agent 0.6.75 / server v0.3.87. |
| 2026-06-25 | Win11 KB5095093 Point-in-Time Restore captured as a Raw RMM_THOUGHTS entry. |
| 2026-07-01..02 | Vendor-agnostic Integrations Framework shipped (migration 065) with CrowdStrike Falcon as the first plugin — API client, client mappings, periodic sync worker. No dashboard UI yet. |
| 2026-07-03..04 | VSS Policy Configurator (SPEC-016 redesign) shipped to beta (agent 0.6.76) — replaces scheduled create+prune with a configure-governors-only model, eliminating the T1490 surface; 23-agent code review caught and fixed a data-loss-class MaxShadowCopies defect pre-merge; verified live on the T1490 canary NEPTUNE. |
| 2026-07-04 | Legacy (2008R2/Win7, Rust 1.77) build variant disabled (84a82a3) after an edition2024 dep recurrence fail-closed the whole Windows build; last legacy artifact frozen at 0.6.76. RMM_THOUGHTS Feature 12 (legacy agent rewrite) added Raw. Cross-site agent dedup fixed (2851523, 1c1cac2) — dedup by device_id across all sites, no re-home on reconnect. Policy-Controlled Tray Deployment merged (f6a4251) — materialize shipped fleet-wide, launch stays policy-gated default OFF. |
| 2026-07-04..05 | Easy-bugs batch + full alert triage: update-scanner host-native/foreign self-check regimes (fixed macOS/Linux agents never receiving update offers), inventory NUL/control-char sanitize (db/sanitize.rs, fixed 7+ machines' silently-failing inventory), WS benign-disconnect + pre-auth churn reclassified ERROR->INFO, create_or_update_alert survives cascade-delete races, move_agent maps duplicate-key to 409. Deployed as server v0.3.95 + v0.3.96. Active server_error alerts reduced to 0. Machine-side backup-failure investigation (GND-SERVER et al.) started but not completed. |
Compilation Notes
- 2026-07-04 recompile (Howard-Home/claude-main): Delta from 2026-06-25 (migrations 64 -> 65; agent 0.6.75 -> 0.6.77 modern / 0.6.76 legacy-frozen; server 0.3.87 -> 0.3.96; fleet 270 -> 359 enrolled). Added: (1) Third-Party Integrations Framework (migration 065, CrowdStrike Falcon plugin) as a new Capabilities subsection — API-only, no dashboard UI yet, flagged as a new UI gap. (2) VSS Policy Configurator (SPEC-016 redesign) replacing the prior scheduled create+prune model — new Capabilities subsection, new locked architecture decision #12, data-loss-defect anti-pattern captured. (3) Policy-Controlled Tray Deployment — materialize/launch split documented, Task 4 (fleet enablement) flagged as pending in Current Focus. (4) Legacy (2008R2/Win7) build variant disabled — Patch/Update section + Build & Deploy + anti-patterns updated; RMM_THOUGHTS Feature 12 noted. (5) Cross-site agent dedup fix + no-re-home-on-reconnect follow-up — new Agent Identity & Enrollment subsection; 13 remaining duplicate pairs noted as open work. (6) 2026-07-04/05 easy-bugs + alert-triage batch (server v0.3.95/v0.3.96) — update-scanner platform regimes,
db/sanitize.rs, WS noise reclassification,create_or_update_alertOption hardening,move_agent409 — folded into Monitoring, Patch/Update, and Alerting Capabilities subsections plus a new Recent Work entry. (7) Active State refreshed (fleet count, 0 active server_error, 13 dup pairs, 5 open backup_failed alerts, VSS beta-only channel state). (8) New anti-patterns: scanner host/foreign self-check split, machine-global VSS governor data-loss class, reconnect re-homing footgun,fetch_oneon a concurrently-deletable row, concurrent-session-no-lock branch collision, stale spec-status-header trap. History Highlights/Patterns preserved verbatim except new rows appended; article length kept in the same range as the 2026-06-25 compile by condensing older Recent Work entries into shorter summaries where the full detail is preserved in git history and prior compiles. - 2026-06-25 recompile (Howard-Home/claude-main): Delta from 2026-06-22 — SPEC-030 remote software inventory + bulk uninstall added as a major Capabilities subsection; universal self-detecting installer (Feature 9) P1 shipped; Current Focus rewritten; flagged the old "pending PRs #40-#46, migrations 061-063" prediction as superseded because software-removal work consumed 061-064 instead.
- 2026-06-22 recompile (Howard-Home/claude-main): Delta from 2026-06-11 — fleet ~270/~178, BUG-021/018/019 fixed on main, Event Log Watch UI shipped, BUG-022 dead-code watchdog path removed, Beast parallel two-wave build documented, command_type "cmd" alias + NAK documented, migration count to 60.
- 2026-06-11 recompile (GURU-5070/claude-main): Full recompile — physical server migration, durable agent identity, agent-comms-durability Phase 1 full detail, migration count to 59.
- 2026-06-07 recompile: Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit folded in; migration count to 55+.
- 2026-06-04 recompile: Corrected GURU-5070 channel state; BUG-020 documented.
- 2026-06-02 recompile: BSOD detection, server build webhook wiring, build channel default beta; migration count 46 -> 48.
- 2026-05-26 recompile: Added the Capabilities / Feature Set section, synthesized from authoritative artifacts. Changelogs are an unreliable capability source — migrations + commit log are authoritative.
Backlinks
- clients/cascades-tucson — RECEPTIONIST-PC enrolled (site CascadesTucson)
- systems/gururmm-build — Physical server at 172.16.3.30; GuruRMM API 3001, ClaudeTools API 8001, Coord API, MariaDB, PostgreSQL, build pipeline
- systems/jupiter — Unraid host at 172.16.3.20; virsh host for all VMs; Docker: Gitea, NPM, Seafile
- systems/pluto — Windows build server (MSI, WiX) at 172.16.3.36