685 lines
77 KiB
Markdown
685 lines
77 KiB
Markdown
---
|
|
type: project
|
|
name: gururmm
|
|
display_name: GuruRMM
|
|
last_compiled: 2026-07-04
|
|
compiled_by: Howard-Home/claude-main
|
|
aliases:
|
|
- guru-rmm
|
|
sources:
|
|
- "gururmm@main: server/src/api/*.rs (REST API surface, ~40 route modules; integrations.rs new)"
|
|
- "gururmm@main: server/src/api/mod.rs (route surface diff, 2026-06-25..2026-07-04 — integrations CRUD only new surface)"
|
|
- "gururmm@main: server/migrations/*.sql (65 migrations — feature checkpoints through 065_integrations_framework)"
|
|
- "gururmm@main: server/migrations/065_integrations_framework.sql"
|
|
- "gururmm@main: server/src/db/sanitize.rs (NEW — shared NUL/control-char sanitize module)"
|
|
- "gururmm@main: server/src/db/inventory.rs, server/src/db/alerts.rs, server/src/updates/scanner.rs, server/src/ws/mod.rs, server/src/api/agents.rs (2026-07-04 easy-bugs + alert-triage batch)"
|
|
- "gururmm@main: agent/src/ (agent capabilities; tray_stage.rs, tray_state.rs, authenticode.rs new; device_id.rs, ohw.rs, watchdog/wts.rs, bsod.rs)"
|
|
- "gururmm@main: specs/tray-policy-deploy/shape.md + plan.md (policy-controlled tray deployment)"
|
|
- "gururmm@main: specs/vss-policy-config/shape.md (VSS policy configurator, SPEC-016 redesign)"
|
|
- "gururmm@main: specs/vss-native-com/ (native VSS COM create/provision)"
|
|
- "gururmm@main: specs/crowdstrike-falcon/shape.md (integrations framework + Falcon plugin)"
|
|
- "gururmm@main: docs/RMM_THOUGHTS.md (header index Features 1-12; Refinement 4a #1 DONE; Feature 12 legacy rewrite Raw)"
|
|
- "gururmm@main: docs/FEATURE_ROADMAP.md (BUG-009/BUG-011 verified fixed 2026-07-04; Known Bugs section)"
|
|
- "gururmm@main: git log origin/main (2026-06-25..2026-07-04 — VSS, tray, integrations, easy-bugs batches)"
|
|
- "gururmm@main: commit 8b75274..5f38ef3 (crowdstrike-falcon Tasks 1-5)"
|
|
- "gururmm@main: commit de30b2b (VSS policy configurator merge)"
|
|
- "gururmm@main: commit f6a4251 (policy-controlled tray deployment merge)"
|
|
- "gururmm@main: commit 84a82a3 (legacy Rust 1.77 variant disabled)"
|
|
- "gururmm@main: commit 2851523 + 1c1cac2 (cross-site agent dedup fix + no re-home on reconnect)"
|
|
- "gururmm@main: commit 519eef2 + f2d02dd (easy-bugs batch + pre-auth churn noise merges, v0.3.95/v0.3.96)"
|
|
- "gururmm@main: session-logs/2026-07/2026-07-04-howard-easy-bugs-batch-deploy.md"
|
|
- "gururmm@main: session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md"
|
|
- "session-logs/2026-07/2026-07-02-mike-crowdstrike-rollout-365-appsuite.md"
|
|
- "session-logs/2026-07/2026-07-03-mike-gururmm-vss-configurator-and-pst-recovery.md"
|
|
- "session-logs/2026-07/2026-07-04-mike-gururmm-vss-build-merge-deploy.md"
|
|
- ".claude/memory/project_gururmm_legacy_disabled.md"
|
|
- "gururmm@main: agent/src/ (agent capabilities; transport/CommandContext, ohw.rs, watchdog/wts.rs, bsod.rs, device_id.rs)"
|
|
- "gururmm@main: agent/scripts/uninstall-engine.ps1 (SPEC-030 Tier-1 silent uninstall engine)"
|
|
- "gururmm@main: server/src/db/software_jobs.rs (async removal jobs)"
|
|
- "gururmm@main: dashboard/src/components/SoftwareManager.tsx (installed-programs list + bulk uninstall + live progress)"
|
|
- "gururmm@main: server/migrations/061_software_removal_attempts.sql"
|
|
- "gururmm@main: server/migrations/062_software_knowledge.sql"
|
|
- "gururmm@main: server/migrations/063_software_knowledge_timing.sql"
|
|
- "gururmm@main: server/migrations/064_software_removal_jobs.sql"
|
|
- "gururmm@main: git log origin/main (2026-06-22..2026-06-25 — SPEC-030 + universal installer)"
|
|
- "gururmm@main: session-logs/2026-06/2026-06-24-howard-gururmm-async-removal-jobs.md"
|
|
- "gururmm@main: session-logs/2026-06/2026-06-25-howard-win11-pit-restore-rmm-thought.md"
|
|
- "gururmm@main: server/migrations/048_bsod_events.sql"
|
|
- "gururmm@main: server/migrations/056_audit_log.sql"
|
|
- "gururmm@main: server/migrations/057_log_signatures.sql"
|
|
- "gururmm@main: server/migrations/058_command_acked_at.sql"
|
|
- "gururmm@main: server/migrations/059_command_delivery_attempts.sql"
|
|
- "gururmm@main: server/migrations/060_alert_mutes_agent_id_index.sql"
|
|
- "gururmm@main: agent/src/bsod.rs"
|
|
- "gururmm@main: server/src/api/updates.rs (promote/rollback endpoints)"
|
|
- "gururmm@main: deploy/build-pipeline/webhook-handler.py"
|
|
- "gururmm@main: deploy/build-pipeline/build-server.sh"
|
|
- "gururmm@main: docs/BUILD.md (build + pre-merge verification guide)"
|
|
- "gururmm@main: commit 1dce66d (BUG-021 dep-pin)"
|
|
- "gururmm@main: commit cea87d4 (BUG-018 202+bg)"
|
|
- "gururmm@main: commit 0fa65f5 (Event Log Watch UI)"
|
|
- "gururmm@main: commit 3de9faf (command_type cmd alias + NAK)"
|
|
- "gururmm@main: commit 4027c86 (enrollment modal UX)"
|
|
- "gururmm@main: commit f0a4b7f (BSOD dedup key)"
|
|
- "gururmm@main: commit 95ef901 (MSI EXDEV fix)"
|
|
- "gururmm@main: commit 137dd85 (BUG-020 tray fix)"
|
|
- "gururmm@main: docs/specs/, docs/FEATURE_ROADMAP.md, docs/BUILD.md"
|
|
- projects/msp-tools/guru-rmm/CONTEXT.md
|
|
- projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md
|
|
- projects/msp-tools/guru-rmm/docs/UI_GAPS.md
|
|
- projects/msp-tools/guru-rmm/docs/ARCHITECTURE_DECISIONS.md
|
|
- projects/msp-tools/guru-rmm/docs/tech-stack.md
|
|
- projects/msp-tools/guru-rmm/docs/DESIGN.md
|
|
- projects/msp-tools/guru-rmm/docs/BUILD.md
|
|
- projects/msp-tools/guru-rmm/specs/agent-comms-durability/shape.md
|
|
- projects/msp-tools/guru-rmm/specs/agent-comms-durability/plan.md
|
|
- projects/msp-tools/guru-rmm/specs/agent-comms-durability/references.md
|
|
- projects/msp-tools/guru-rmm/specs/agent-comms-durability/standards.md
|
|
- .claude/memory/reference_gururmm_server.md
|
|
- .claude/memory/reference_gururmm_api.md
|
|
- .claude/memory/gururmm-development-principles.md
|
|
- .claude/memory/feedback_gururmm_agent_parity.md
|
|
- .claude/memory/reference_pluto_build_server.md
|
|
- .claude/memory/project_mac_gururmm_setup_pending.md
|
|
- .claude/memory/feedback_gururmm_build_channel_default.md
|
|
- .claude/memory/reference_gururmm.md
|
|
- .claude/memory/feedback_gururmm_build_verification.md
|
|
- .claude/memory/rmm-dashboard-beta-before-main.md
|
|
- credentials.md
|
|
- session-logs/2025-12-15-session.md
|
|
- session-logs/2025-12-20-session.md
|
|
- session-logs/2026-04-19-session.md
|
|
- session-logs/2026-04-21-session.md
|
|
- session-logs/2026-04-29-session.md
|
|
- session-logs/2026-05-12-guru-rmm-macos-agent-phase1.md
|
|
- session-logs/2026-05-15-session.md
|
|
- session-logs/2026-05-16-session.md
|
|
- session-logs/2026-05-17-session.md
|
|
- session-logs/2026-05-19-gururmm-backup-fixes.md
|
|
- session-logs/2026-05-19-session.md
|
|
- session-logs/2026-05-21-session.md
|
|
- session-logs/2026-05-23-session.md
|
|
- session-logs/2026-05-24-session.md
|
|
- session-logs/2026-05-24-GURU-KALI-session.md
|
|
- session-logs/2026-05-31-howard-gururmm-roadmap-and-features.md
|
|
- session-logs/2026-06-02-mike-bsod-detection-and-pipeline.md
|
|
- session-logs/2026-06-07-mike-gururmm-offboarding-spec.md
|
|
- session-logs/2026-06-07-mike-gururmm-backup-alert-cleanup.md
|
|
- session-logs/2026-06-07-mike-gururmm-offline-alerting-mute.md
|
|
- session-logs/2026-06-07-mike-gururmm-ui-gaps-enrollment.md
|
|
- projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-host-migration-cutover.md
|
|
- projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-backfill-build-pipeline.md
|
|
- projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-promotion-ghost-identity.md
|
|
- projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-comms-durability.md
|
|
- projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-11-mike-rmm-comms-durability-phase1.md
|
|
- projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-12-mike-beast-parallel-windows-build.md
|
|
- projects/msp-tools/guru-rmm/session-logs/2026-06/2026-06-12-mike-rmm-command-type-cmd-misdiagnosis.md
|
|
- session-logs/2026-06/2026-06-15-mike-unifi-wifi-skill-and-gururmm-fixes.md
|
|
- session-logs/2026-06/2026-06-18-mike-sync-submodule-autoheal-and-thoughts-rescue.md
|
|
- session-logs/2026-06/2026-06-21-howard-gururmm-bug-018-019.md
|
|
- session-logs/2026-06/2026-06-21-howard-rmm-bug018-eventlogwatch-ui.md
|
|
- session-logs/2026-06/2026-06-21-howard-gururmm-features-audit-submodule-fix.md
|
|
- session-logs/2026-06/2026-06-21-mike-rmm-build-skill-errorlog-lint-coord-purge.md
|
|
backlinks:
|
|
- clients/cascades-tucson
|
|
- systems/gururmm-build
|
|
- systems/jupiter
|
|
- systems/pluto
|
|
---
|
|
|
|
# GuruRMM
|
|
|
|
## Summary
|
|
|
|
GuruRMM is a Remote Monitoring & Management platform built by Arizona Computer Guru LLC for internal MSP operations and eventual productization. The server (Rust/Axum) and dashboard (React/TypeScript) are production-deployed at https://rmm.azcomputerguru.com with approximately 359 enrolled agents (228 typically fresh/online) across multiple client sites. The agent runs on managed Windows, Linux, and macOS endpoints.
|
|
|
|
**Current version:** agent 0.6.77 / server v0.3.96 as of 2026-07-04 (was 0.6.75 / 0.3.87 on 2026-06-25). Active `server_error` alerts reduced to 0 fleet-wide after the 2026-07-04/05 fix batch.
|
|
|
|
**Easy-bugs batch + alert-triage fixes deployed 2026-07-05 (server v0.3.95 + v0.3.96):** Fixed three long-standing silent-failure classes surfaced by a full 24h alert triage: (1) the update **scanner** dropped macOS/Linux agent binaries during its self-check because they cannot execute on the Linux build host — every macOS/Linux agent had **never** been offered an update; fixed with host-native-vs-foreign self-check regimes (foreign platforms trust the filename version, host-native binaries stay a hard `--version` gate). (2) **Hardware inventory** silently failed to persist for 7+ fleet machines because Windows registry strings routinely carry embedded NUL bytes that Postgres jsonb/text rejects; fixed with a new shared `db/sanitize.rs` (NUL + C0/C1 control-char strip, collision-safe). (3) **WebSocket log noise** — benign disconnects and pre-auth connection churn (sleeping laptops, half-open NAT, scanners) were logged as `ERROR` and minted `server_error` alerts; reclassified to `INFO`, with genuine auth rejections (bad key, malformed auth) staying `ERROR`. Also fixed in the same batch: `create_or_update_alert` now returns `Option<Alert>` and survives an agent cascade-delete mid-flight (FK violation -> `Ok(None)` instead of aborting the offline sweep); `move_agent` maps a duplicate-site-device conflict to 409 instead of a raw 500. Went through a 19-agent high-effort code review before merge (9 confirmed findings, all fixed). See Recent Work for detail.
|
|
|
|
**Cross-site agent duplication fixed 2026-07-04:** Root cause of ~40 duplicate agent rows (24 at Dataforth D2->D1 alone): both WS enrollment paths deduped only within `(site_id, device_id)`, so re-running a different site's installer on an already-enrolled machine created a second row instead of finding the existing one. Fixed to match by hardware `device_id` **across all sites** and re-home via `move_agent_to_site` — but a same-day follow-up (`1c1cac2`) stopped the auto-re-home on every reconnect (the agent re-presents its install-site code as its API key on every beat, which was bouncing admin-moved agents back to their original site). Net effect: dedup by device_id, but site placement stays admin-controlled and persists.
|
|
|
|
**Legacy (Server 2008R2/Win7, Rust 1.77) build variant disabled 2026-07-04:** The unpinned legacy dependency re-resolve pulled edition2024 crates (`chacha20`/`rand_core 0.10.1`) that Rust 1.77 cannot parse — a BUG-021 recurrence that fail-closed the entire Windows build (blocking modern amd64/x86/tray artifacts too). Gated behind `LEGACY_ENABLED=false`; the last-built legacy artifact (agent 0.6.76) is preserved and frozen for existing 2008R2/Win7 endpoints. 2008R2 support returns via a separate legacy-only rewrite (C++/.NET or a clean minimal Rust core), not by re-enabling the 1.77 toolchain — captured as **RMM_THOUGHTS Feature 12** (Raw, Mike's requirement).
|
|
|
|
**Policy-Controlled Tray Deployment shipped 2026-07-04 (code merged, policy default OFF):** The tray binary is now deployed to every Windows endpoint via the agent-update flow (previously only fresh MSI installs got it — fleet census found ~111/113 sampled hosts with no tray at all) but its **launch is gated on `tray.enabled` policy** — no process, no icon, until a client/site/agent policy turns it on. Authenticode-verified + version-locked to the agent. Built via a rejected-then-corrected design (17-agent code review + Gemini + Grok adversarial critique caught 9 defects in the first attempt — see Recent Work). Fleet-wide policy enablement (Task 4) is **not yet flipped**.
|
|
|
|
**VSS Policy Configurator shipped 2026-07-03 (SPEC-016 redesign, agent v0.6.76):** The agent no longer runs its own scheduled shadow-copy create+prune loop — the MITRE T1490 "Inhibit System Recovery" surface that got the agent blocked by CrowdStrike Falcon. It now **configures** two kernel-enforced retention governors (VSS diff-area size cap + `MaxShadowCopies` count) and registers a scheduled task whose action is a native-COM **create only** (never a delete) — retention becomes Windows' own FIFO eviction, not agent code. Works uniformly on server AND client Windows SKUs (previously `vssadmin create shadow` blocked workstations). A 23-agent code review caught a data-loss-class defect pre-merge (writing the machine-global `MaxShadowCopies` would have evicted OTHER products' shadows) — dropped in favor of the size cap as sole governor. Live-verified on the T1490 canary (NEPTUNE, Falcon present): legacy scheduled task removed, no T1490 trip, 17 existing shadows preserved. Deployed to the **beta** channel only (0.6.76); stable fleet remains on 0.6.66 pending a deliberate promotion.
|
|
|
|
**Vendor-agnostic Integrations Framework shipped 2026-07-01..02 (migration 065, CrowdStrike Falcon first plugin):** A generic per-partner plugin framework (`integrations`, `integration_credentials`, `integration_client_mappings`, `integration_sync_state`, `integration_agent_links` tables) so a new EDR/security vendor is a new server module, not a new schema — modeled on the proven MSPBackups per-partner shape. CrowdStrike Falcon is the first plugin: API client, RMM-client-to-Falcon-CID mapping, and a periodic sync worker pulling sensor state (version/status/RFM) into `integration_agent_links`. REST CRUD at `/api/integrations`; no dedicated dashboard UI yet (API/backend only).
|
|
|
|
**SPEC-030 Remote Software Inventory + Bulk Uninstall (BETA — code merged + deployed 2026-06-23..24, NOT guaranteed to work; Route B = server-orchestrated):** The code is on main and deployed, but the **feature is still beta and unreliable** — it succeeds on cleanly-uninstallable programs and is explicitly NOT guaranteed for the hard cases (most managed/protected AV, WiX/Burn bundles, uninstallers that need UI or spawn lingering child processes — Launchy/AIMP-class, drivers/system components). Treat results as best-effort, verify removals, and do not rely on it for production removals yet. See the Capabilities section for full detail + the open gaps.
|
|
|
|
**Universal self-detecting installer (Feature 9, SHIPPED P1, v0.6.71+):** One install URL/script self-detects arch/OS/legacy and pulls the correct agent; the dashboard install UI now points at it. P2 (native i386 bootstrapper EXE for locked-down boxes) pending.
|
|
|
|
**Physical server migration completed 2026-06-11:** New physical box at 172.16.3.30 running Ubuntu 26.04 + PostgreSQL 18. Old VM at .46 decommissioned 2026-06-12.
|
|
|
|
**See also:** `wiki/projects/guru-rmm.md` is a redirect tombstone pointing here (slug disambiguation).
|
|
|
|
**Repo:** `azcomputerguru/gururmm` on Gitea (internal: http://172.16.3.20:3000, external https://git.azcomputerguru.com/azcomputerguru/gururmm.git). The copy at `projects/msp-tools/guru-rmm` is a git submodule tracking the active repo; its pinned gitlink intentionally lags origin/main. Development uses `git -C projects/msp-tools/guru-rmm` or worktrees off `origin/main` to access live code. Howard is authorized for merges/deploys (cleared 2026-06-21 by Mike).
|
|
|
|
**Goal:** Full-featured MSP platform rivaling commercial RMMs, with a companion PSA (GuruPSA, separate future repo) designed as a truly integrated unified system.
|
|
|
|
---
|
|
|
|
## Recent Work
|
|
|
|
### 2026-07-04..05 — Easy-Bugs Batch + Alert Triage: Update-Scanner, Inventory NUL, WS Noise (Howard-Home)
|
|
|
|
Full 24h alert triage (`GET /api/alerts`) surfaced 5 `backup_failed` (client-side, unresolved — see Active State), 2 active `server_error` (product bugs), and 1 resolved `mass_offline_fleet` (36 agents, transient network blip). Root-caused both server errors against the server's own self-log (pseudo-agent `00000000-...-001`):
|
|
|
|
- **`move_agent` 500 -> 409:** moving an agent into a site that already holds a row for the same `device_id` tripped `idx_agents_site_device` as a raw 500; now maps to 409 with an actionable message. 13 duplicate-hostname agent pairs fleet-wide are the trigger (stale legacy `win-*`/`hw-*` device_ids alongside re-enrolled hardware-UUID records) — not yet cleaned up.
|
|
- **Offline-sweep `RowNotFound`:** `create_or_update_alert`'s UPDATE-by-id used `fetch_one`; an alert row deleted mid-sweep (agent-delete cascade) aborted the whole sweep. Switched both branches to `fetch_optional` with fall-through to INSERT (`Result<Option<Alert>>`, `Ok(None)` = agent gone).
|
|
- **Inventory NUL rejection (alert-invisible, found in logs only):** 7+ agents (IMC1, Seth-PC, QWM-JOHN, QWM-SHEILA, goldstar19, SIF-SERVER, Christine-Win10) never persisted hardware inventory — Windows registry strings carry embedded NULs, Postgres jsonb/text rejects them. New `server/src/db/sanitize.rs` (`strip_nul`, `clean_control_chars`) applied over 5 jsonb blobs (values + keys, collision-safe) + text columns.
|
|
- **Update-scanner warn-spam / silent update starvation:** the self-check tried to execute every platform's binary to read its embedded version — macOS/Linux binaries can't execute on the Linux build host, so they were dropped from the scannable set entirely, meaning **macOS/Linux agents were never offered updates**. Fixed with host-native-vs-foreign regimes: foreign-platform binaries trust the filename version; host-native binaries stay a hard gate (a native binary that can't report its version is corrupt) with a best-effort exec-bit repair for 0644 CI-copy artifacts.
|
|
|
|
A concurrent same-branch session absorbed a prior session's in-progress inventory/scanner fixes; the combined diff went through a 19-agent high-effort code review (9 confirmed findings, 0 refuted) before merge — findings included rebuilding the alerts fall-through (it still failed its own motivating scenario on first pass), tightening the scanner's foreign/host-native split, and moving WS-noise logging from DEBUG to INFO (a mass-disconnect anomaly must stay visible at the default log level). Merged + deployed as **server v0.3.95** (06:09 UTC): scanner now indexes 8 platform/arch combos (was 4, Windows-only); zero ERROR lines post-deploy.
|
|
|
|
Continued into the last stale alert: "Authentication timeout" WS churn (2-6/day baseline, 57-spike correlated with the same day's fleet-dedup work). Profiled 7 days of journal, concluded pre-auth connection noise (peer connects, never authenticates — sleeping laptops, scanners, half-open NAT), added `is_preauth_connection_churn` to classify it INFO while keeping genuine rejections (bad API key, malformed auth) at ERROR. Deployed as **server v0.3.96** (06:23 UTC). Active `server_error` alert count: **0**.
|
|
|
|
BUG-009 (`Logs.tsx` isError handling) and BUG-011 (`: any` occurrences) were verified already fixed in-tree during the same pass — only the stale `FEATURE_ROADMAP.md` status lines needed correcting, not the code.
|
|
|
|
Remaining/pending from this session: the **machine-side backup failures** (the user's actual priority) — GND-SERVER (Grabb & Durando, chronic, 42 nightly failures since 2026-06-17), AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial) — investigation started but not completed; all say "no error detail reported" (MSP360 sync doesn't capture failure detail). 13-pair duplicate-agent cleanup still open. WS auth-timeout source (~1/min, unauthenticated) still unidentified (needs Feature 5, server-side public-IP capture — still unapproved).
|
|
|
|
---
|
|
|
|
### 2026-07-04 — Policy-Controlled Tray Deployment (Howard-Home / Mike, merged f6a4251)
|
|
|
|
Made `gururmm-tray.exe` deployment (materialize, policy-agnostic) independent from its runtime (launch, policy-gated) — previously the watchdog launched the tray whenever the binary existed, with no enforcement of the already-wired `tray.enabled` policy flag, and the binary itself only ever reached fresh MSI installs (fleet census: 1 signed tray, 1 stale unsigned, ~111 none out of 113 sampled). A first design (inline tray refresh folded into `do_update`) was **rejected** by a 17-agent code review plus independent Gemini 3.1 Pro and Grok adversarial critiques — 9 verified defects including permanent-binary-loss on a failed swap, a checksum sourced from an unauthenticated download-host sidecar (supply-chain gap), and an update/rollback tray-agent version-skew case.
|
|
|
|
Corrected design shipped as Tasks 1-3 + 5: policy-gated launch (default OFF), a materialize FSM with staging + a versioned mutex (atomic swap, restore-on-failure, authenticated-payload checksum), server capability-gated materialize, and Authenticode-verify + observability. Task 4 (flip fleet-wide materialize on, policy still OFF) is the deliberate next step — not done yet. Merged to main 2026-07-04 12:23 PDT.
|
|
|
|
---
|
|
|
|
### 2026-07-03..04 — VSS Policy Configurator (SPEC-016 Redesign) (Mike, merged de30b2b)
|
|
|
|
Rebuilt the agent's Volume Shadow Copy handling from "scheduled create+prune operator" (the MITRE T1490 surface Falcon blocked) to "policy configurator, never a deleter." Tasks 2-7: `vss-create` verb + scheduled-task registration (both daily-time and every-N-hours trigger modes) with legacy-task migration; per-volume diff-area size-cap governor set on policy apply; retirement of the scheduled create+prune pass entirely (SPEC-025 compliance heal reworked to create-only); status/compliance reporting updated to the configurator model; dual-target release build; runtime verification on both a Server 2019 box and a Win11 Pro client with Falcon present.
|
|
|
|
A 23-agent workflow code review before merge caught a genuine data-loss defect: the original design would have set the machine-global `MaxShadowCopies` registry value to `retention_count`, which FIFO-evicts **other** VSS consumers' (System Restore, third-party backup) shadows below that count fleet-wide. Fixed by dropping `MaxShadowCopies` management entirely — the per-volume size cap is the sole governor; `retention_count`/`retention_max_age_days` become advisory (a warning fires if `retention_max_age_days` is still set, since age-pruning requires deletion). Nine other findings fixed (cap-before-create ordering, an alias bridging the legacy task-name upgrade window, interval-aware compliance windows, bounded `IVssAsync` wait instead of INFINITE, and others).
|
|
|
|
Merged to main; the build pipeline publishes to the **beta** channel, not stable (a mid-session correction) — beta head reached 0.6.76 (8 agents), stable stayed at 0.6.66 (233 agents) by design. Verified live on the T1490 canary NEPTUNE (moved to beta channel for the test): create task present, legacy `GuruRMM-VSS-Snapshot` task removed, `MaxShadowCopies` correctly left unset, 17 pre-existing shadows preserved (the data-loss fix proven in production), and a live `vss-create` produced new shadows on two volumes under Falcon with no T1490 trip.
|
|
|
|
---
|
|
|
|
### 2026-07-01..02 — Vendor-Agnostic Integrations Framework + CrowdStrike Falcon Plugin (Mike)
|
|
|
|
Built a generic third-party-integration plugin framework (migration 065: `integrations`, `integration_credentials`, `integration_client_mappings`, `integration_sync_state`, `integration_agent_links`) so CrowdStrike Falcon — and future vendors (SentinelOne, Guardz) — are a new server module rather than a new bespoke schema each time, generalizing the per-partner MSPBackups shape (migration 034). Credentials reuse the same AES-256-GCM `Encryptor` chokepoint as the existing `credentials` table.
|
|
|
|
Five tasks, one per day-slice: (1) framework schema migration; (2) plugin trait + registry + DB layer; (3) generic REST API (`/api/integrations` CRUD, credentials, client mappings, sync-state, connection test); (4) CrowdStrike Falcon API client + plugin registered against the framework; (5) periodic sync worker pulling Falcon sensor state (version, status, RFM flag, CID) into `integration_agent_links`. No dedicated dashboard UI yet — API/backend only as of this compile.
|
|
|
|
---
|
|
|
|
### 2026-06-23..24 — SPEC-030 Remote Software Uninstall + Universal Installer (Howard-Home)
|
|
|
|
**SPEC-030 software inventory + remote bulk uninstall (Route B — BETA, code merged to main + deployed but NOT guaranteed to work; agent 0.6.75 / server v0.3.87):**
|
|
|
|
- **Capability shipped:** `/agents/:id/software` lists installed programs; `/agents/:id/software/uninstall` runs a bulk uninstall; the dashboard `SoftwareManager.tsx` shows the live programs list + bulk-select + a live progress bar. Server endpoints in `server/src/api/software.rs`.
|
|
- **Silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`): Tier-1 silent removal, BCU-informed detection + fail-fast, vendor silent-switch table, captures **real** process exit codes (fixed a `Start-Process` null-ExitCode bug, `f6c1945`). The engine **refuses to guess** — Avast/AVG (`icarus.exe`), Malwarebytes (`mb5uns.exe`) correctly returned `needs_interactive` rather than risk a bad removal.
|
|
- **False-success class fixed (CRITICAL, PR #53):** Avira's WiX/Burn **Package Cache bundle GUID** was misread as an MSI ProductCode; `msiexec /x {bundleGuid}` returned 1605 ("not installed") and was mapped to success="already gone" while Avira stayed fully installed. Fix: skip the MSI tier for `\Package Cache\` bundles + a **generic post-success ARP re-check** (`Test-StillInstalled`) that downgrades a reported success to failed if the program is still registered.
|
|
- **Code-review correctness fixes (PR #54):** `Test-StillInstalled` matches by `product_code` alone when present; skip the rescan on reboot-pending verdicts (3010/1641); exclude MSI-1605 no-op successes from the learned-timing sample.
|
|
- **Async removal jobs — the permanent fix (PR #56, migration 064):** A synchronous bulk uninstall died at Cloudflare's ~100s timeout AND the agent's command kill-timeout killed the engine mid-flush. Rebuilt as a **JOB**: POST creates a `software_removal_jobs` row and returns a `job_id` instantly (0.06s); a background worker dispatches one program at a time; `GET .../uninstall/jobs/:job_id` is polled by the dashboard. Stop-gap flush-slack (PR #55) shipped first.
|
|
- **Three-state removal knowledge base (migrations 061-063):** `software_removal_attempts` (per-agent removal-tracking loop), `software_knowledge` (fleet KB: `silent` / `requires_ui` / `unknown`), `software_knowledge_timing` (self-tuning per-target timeout).
|
|
- **Known gaps surfaced (not yet built):** drivers/system components register in ARP and appear in the bulk list; Launchy/AIMP-class uninstallers hang the agent on a lingering child process (~420s kill-timeout); no list-jobs endpoint (get-by-id only).
|
|
|
|
**Universal self-detecting installer (Feature 9 — P1 built+deployed+verified, v0.6.71):**
|
|
- One install path self-detects arch/OS (x64/x86/legacy/ARM) and pulls the correct agent; dashboard install UI repointed at it. Script path proven on-box (GND-JWILL). P2 (native i386 bootstrapper EXE) + offline-bundle still pending.
|
|
|
|
**RMM_THOUGHTS movement:** Feature 10 (AMPIPIT recovery environment add-on) -> Discussed (Mike likes it, deferred); Third-Party AV Removal Recipes + Win11 KB5095093 Point-in-Time Restore added as Raw; **Feature 11** (OS Updates Management, Windows Update P1/P2 + Linux/Mac update-check P3, Syncro-modeled) added Raw 2026-07-03; **Feature 12** (legacy 2008R2/Win7 agent rewrite) added Raw 2026-07-04 after the legacy build was disabled.
|
|
|
|
---
|
|
|
|
### 2026-06-22 — BUG-021 Fixed + v0.6.67 Stable; Fleet Verified (Howard-Home)
|
|
|
|
- **BUG-021:** Legacy build (Rust 1.77 via `$CARGO +1.77 --features legacy`) was pulling `wit-bindgen` (edition2024) through fresh dep resolution. Fixed by pinning `getrandom 0.3.1` + `zeroize 1.8.1` below edition2024. Commit `1dce66d`. Windows build went green 2026-06-22 02:19, v0.6.67. (This class of failure recurred 2026-07-04 with a different dep pair and led to the legacy variant being disabled outright — see Recent Work above.)
|
|
- **Fleet verified:** ~270 agents / ~178 online; metrics rows flowing; all alerts have non-null dedup_keys; per-agent endpoints all HTTP 200.
|
|
- **BUG-022 filed + fixed (PR #45):** `WatchdogEvent` WS variant was dead code (watchdog has no WS connection); removed. Empty `watchdog_events` table left in place.
|
|
- **Roadmap verified:** BUG-018 (`cea87d4`) and Event Log Watch UI (`0fa65f5`) confirmed on main.
|
|
|
|
---
|
|
|
|
### 2026-06-21 — Feature Batch, Audit, BUG-018 202+bg, Event Log Watch UI, Submodule Fix (Howard-Home + Mike GURU-5070)
|
|
|
|
**Howard-Home sessions (three spans):**
|
|
|
|
- **BUG-019 FIXED (commit `66a7f4e`, merged to main by Mike):** Container agent no longer performs in-place binary self-update. `agent/src/updater/mod.rs` `perform_update()` early-returns inside a container.
|
|
- **BUG-018 FIXED (commit `cea87d4`, merged):** Handler redesign to 202 + background purge. Migration 061 (FK indexes on five previously-unindexed FK columns) landed later under a renumbered slot once SPEC-030 consumed 061-064.
|
|
- **Event Log Watch management UI (SHIPPED, `0fa65f5`):** `/event-log-watches` CRUD wired into the dashboard.
|
|
- **Enrollment modal UX fix (SHIPPED to prod, `4027c86`):** close button, click-outside/Esc dismissal, list refresh on close.
|
|
- **Event Log Watch policy-clobber HIGH fix (PR #43, `432b434`):** Assigning/unassigning a policy silently wiped an agent's Event Log Watch monitoring — third instance of a config-push clobber class. Fixed by reusing `watch_rules_for_agent` as the single source of truth.
|
|
- **sync.sh submodule-clobber root-cause fix:** populate-only guard prevents `git submodule update --init` from re-checking out the intentionally-lagging pinned gitlink on every sync.
|
|
- **Dashboard beta-before-main rule established.**
|
|
|
|
**Mike GURU-5070 session:**
|
|
|
|
- **BUG-019 merged to main** (fast-forward `ed8cad3 -> 66a7f4e`).
|
|
- **`gururmm-build` skill created:** local pre-merge verification (`verify.sh server|agent|dashboard|migrations`). `docs/BUILD.md` added.
|
|
- **Howard cleared for GuruRMM merges/deploys** (Mike decision).
|
|
|
|
---
|
|
|
|
### 2026-06-15 — BSOD Dedup Key Fix, MSI EXDEV Fix, Dashboard Offline Dedup, sync.sh Auto-Heal (Mike GURU-5070 + GURU-BEAST-ROG)
|
|
|
|
- **BSOD alert dedup key fixed (commit `f0a4b7f`, server v0.3.73):** Dedup key changed to `bsod:<agent>:<bugcheck_code>` (stable across recurrences).
|
|
- **MSI EXDEV fix (commit `95ef901`, server v0.3.74):** Site-MSI builder staged in `/tmp` (tmpfs) then renamed cross-device to `/opt/gururmm/downloads`, failing EXDEV. Fixed to stage on the same filesystem.
|
|
- **Duplicate offline server alerts removed (commit `b6ed564`).**
|
|
- **sync.sh auto-heal (Phase 1):** `resolve_submodule_collisions()` moves only genuinely-colliding untracked files aside and retries, instead of a blind force.
|
|
- **`logs/analyze` switched from Ollama to Claude API (commit `c869e4d`).**
|
|
- **500-error-leak fix (commit `58c1a96`):** All server error paths route through `internal_err`.
|
|
|
|
---
|
|
|
|
### 2026-06-12 — Beast Parallel Windows Build (Lever A) + command_type "cmd" Misdiagnosis Fix (Mike GURU-5070)
|
|
|
|
**Beast parallel Windows build (commit `cfbdb59`):** Two waves (5 concurrent stable + 2 concurrent legacy-1.77) via bash-driven concurrent SSH instead of a serial `cmd /c` chain. Per-variant `--target-dir` mandatory. Result: ~336s parallel vs ~622s serial cold, ~3.8x vs Pluto serial.
|
|
|
|
**command_type "cmd" fix (commit `3de9faf`, agent v0.6.66 stable):** `CommandType` enum had no `cmd` variant — server-sent `command_type:"cmd"` failed serde parse and the agent silently dropped the message. Fixed with `#[serde(alias = "cmd")]` on `Shell` + a NAK on unparseable commands.
|
|
|
|
---
|
|
|
|
### 2026-06-11 — Physical Server Migration, Durable Agent Identity, Agent Comms Durability Phase 1
|
|
|
|
- **Cutover:** New physical box took over 172.16.3.30 (PG 18, Ubuntu 26.04); production down <27 min; old VM decommissioned 2026-06-12.
|
|
- **Durable agent identity Phase 1 Task 1:** Multi-location `device_id` (registry+ProgramData on Windows, /etc+/var/lib on Linux, /Library+mirror on macOS) with self-heal; `cleanup.ps1` whitelists identity files.
|
|
- **Agent Comms Durability Phase 1 (agent v0.6.63 / server v0.3.68):** Root cause = NAT conntrack asymmetry (server->agent pushes black-holed in idle conntrack gaps). Fix: agent `CommandAck` on receipt + dedup cache; server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via a partial index so old agents keep legacy behavior.
|
|
|
|
---
|
|
|
|
### 2026-06-07 — Credential Inheritance, Offboarding Spec, UI Gap Batch, Backup Alert Quality Pass
|
|
|
|
- Server v0.3.45: credential inheritance (Global -> Client -> Site), `/effective` endpoints.
|
|
- Role-aware offline alerting + alert ignore/mute shipped (server-only `agent_offline`, `alert_mutes` perma-silence).
|
|
- Backup-alert quality pass: false `backup_failed` 15 -> 2 fleet-wide; `backup_storage_low` alert type removed (false-positive metric).
|
|
- UI gap batch + enrollment audit; SPEC-028 offboarding wizard specification (835 lines, implementation pending).
|
|
|
|
---
|
|
|
|
## Capabilities / Feature Set
|
|
|
|
*Synthesized from authoritative artifacts (API routes, agent modules, 65 migrations through migration 065, roadmap, commit log) — not from session logs alone. See Compilation Notes.*
|
|
|
|
Agent<->server communication is a persistent authenticated WebSocket with auto-reconnect + heartbeat; on reconnect, in-flight commands flip to `interrupted`. Platform-parity rule: agent features ship on Windows/Linux/macOS in the same change (stub + TODO where a real impl isn't yet feasible).
|
|
|
|
### Monitoring & Telemetry
|
|
- Core metrics per interval (policy-tunable): CPU %, memory %/bytes, disk %/bytes, network rx/tx deltas, uptime, logged-in user, user idle time, public/WAN IP. Cross-platform via `sysinfo`.
|
|
- Hardware sensor telemetry: CPU/GPU temps + full sensor array. Linux via `/sys/class/thermal`; `sysinfo::Components` fallback. Windows LibreHardwareMonitor removed (CVE-2020-14979); replacement pending.
|
|
- Process drill-down: top 10 by CPU + top 10 by memory in every metrics payload.
|
|
- Network state: per-interface IPv4/IPv6, MAC, derived CIDR subnets — sent on change only.
|
|
- **Windows BSOD/kernel-crash detection:** `agent/src/bsod.rs` polls `C:\Windows\Minidump`, parses kernel dump header at fixed offsets, cross-references System event log, deduplicates via a watermark file. Always-Critical alert deduplicated by `(agent_id, bugcheck_code)` (changed from dump-hash 2026-06-15 so a mute silences all recurrences).
|
|
- **Inventory NUL/control-char hardening (2026-07-05, server v0.3.95):** `server/src/db/sanitize.rs` strips NULs and C0/C1 control chars from inventory jsonb values/keys and text columns before upsert — fixed silent storage failure on 7+ machines carrying embedded NULs in Windows registry strings.
|
|
|
|
### Remote Execution
|
|
- Command types: `shell`, `powershell`, `python`, raw `script{interpreter}`, `claude_task`, and `cmd` (alias for `shell`). Options: `timeout_seconds`, `elevated`. Unknown/unparseable commands NAK immediately.
|
|
- **Execution context** (`041`): `system` (default) or `user_session` (WTS token impersonation, Windows-only).
|
|
- Commands individually cancellable, aborted on disconnect. Auditable history. Script library (`017`) for stored scripts.
|
|
- **Agent Comms Durability (Phase 1, shipped 2026-06-11 — agent v0.6.63 / server v0.3.68):** Agent sends `CommandAck` on receipt (migration 058 `acked_at`); dedup via a recent-results cache. Server reaper re-delivers un-acked commands past a 60s deadline instead of failing them; capability-gated via `idx_commands_agent_acked` partial index so pre-durability agents keep legacy fail-on-timeout behavior. Phase 2 (live TTY) and Phase 3 (adaptive keepalive, bulk->HTTPS, half-open eviction) remain PLANNED, not shipped.
|
|
|
|
### Volume Shadow Copy / VSS (Windows) — Policy Configurator (SPEC-016 redesign, shipped 2026-07-03, agent v0.6.76, beta channel only)
|
|
- **The agent no longer schedules its own create+prune loop.** It is a **configurator**: on each policy apply it idempotently sets two kernel-enforced retention governors — VSS diff-area **size cap** (native `IVssDifferentialSoftwareSnapshotMgmt`) and `MaxShadowCopies` count (evaluated, but NOT machine-globally written — see below) — and registers a scheduled task whose action is the agent's own native-COM **create** (`IVssBackupComponents::DoSnapshotSet`, `VSS_CTX_CLIENT_ACCESSIBLE`), never a delete. Retention is enforced by Windows' own FIFO eviction (`volsnap.sys`) when a governor is hit, eliminating the MITRE T1490 "Inhibit System Recovery" surface that had gotten the agent blocked by CrowdStrike Falcon.
|
|
- **Works uniformly on server AND client SKUs** (previously `vssadmin create shadow`, Server-only, gated client support) — the underlying VSS COM API and `volsnap` cap eviction are not SKU-restricted, only the `vssadmin.exe` CLI is.
|
|
- **`MaxShadowCopies` is NOT set machine-globally** (pre-merge code review caught this as a data-loss defect — it would FIFO-evict OTHER VSS consumers' shadows, e.g. System Restore or third-party backup, below the configured count). The per-volume size cap is the sole enforced governor; `retention_count`/`retention_max_age_days` policy fields are advisory/ignored (a warning fires if `retention_max_age_days` is still set, since age-based pruning requires deletion).
|
|
- On first policy apply, the legacy `GuruRMM-VSS-Snapshot` scheduled task (present fleet-wide) is removed and replaced with `GuruRMM-VSS-Create`, supporting both daily-time and every-N-hours interval triggers.
|
|
- On-demand admin operations (create/delete/browse/restore via `dispatch_operation`, `GET /agents/:id/vss/restores`) are unaffected — this redesign only removes the *scheduled* create+prune loop.
|
|
- **Not yet promoted to the stable channel** as of this compile (beta head 0.6.76; stable pinned at 0.6.66).
|
|
|
|
### Software Inventory & Remote Uninstall (SPEC-030 — BETA, not guaranteed)
|
|
- **Maturity:** code merged to main + deployed (2026-06-23..24), but the feature is **beta and unreliable** — best-effort only.
|
|
- **Route B (server-orchestrated):** `GET /agents/:id/software`, `POST /agents/:id/software/uninstall` (async -> `job_id`), `GET /agents/:id/software/uninstall/jobs/:job_id`, `GET /agents/:id/software/removal-status` + resolve, `GET /software/knowledge` + classify.
|
|
- **Silent-uninstall engine** (`agent/scripts/uninstall-engine.ps1`): refuses to guess (`needs_interactive` on ambiguity); generic post-success ARP re-check (`Test-StillInstalled`) catches the WiX/Burn Package-Cache-bundle false-success class; orphaned ARP entries classified `tier=orphaned`; async-fork uninstallers get retry-verify.
|
|
- **Async removal jobs (migration 064):** non-blocking bulk uninstall — job created + `job_id` returned in ~0.06s, background worker processes targets one at a time, dashboard polls live progress.
|
|
- **Fleet removal knowledge base (migrations 061-063):** per-agent removal-tracking loop, fleet three-state KB (`silent`/`requires_ui`/`unknown`), self-tuning per-target timeout.
|
|
- **Known gaps:** drivers/system components register in ARP and appear in the bulk list (no exclusion/flag yet); Launchy/AIMP-class uninstallers hang the agent on a lingering child process; no list-jobs endpoint (get-by-id only). The **av-removal-recipes spec (P3, removal-only)** is the follow-on.
|
|
|
|
### Third-Party Integrations Framework (NEW, migration 065, shipped 2026-07-01..02)
|
|
- **Vendor-agnostic plugin framework:** `integrations` (one enabled plugin per partner+kind), `integration_credentials` (AES-256-GCM encrypted, same `Encryptor` chokepoint as the `credentials` table), `integration_client_mappings` (RMM client -> external tenant id, e.g. a Falcon CID — single-CID partners map every client to one id, MSSP partners map child CIDs), `integration_sync_state` (per-integration sync health), `integration_agent_links` (RMM agent <-> external agent id + last-synced state, e.g. Falcon AID/sensor version/RFM).
|
|
- **API:** `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `DELETE /api/integrations/:id/mappings/:mapping_id`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test`.
|
|
- **First plugin: CrowdStrike Falcon.** API client + periodic sync worker (default `sync_interval_seconds=900`) pulls sensor state into `integration_agent_links`. Framework is modeled on the proven per-partner MSPBackups shape (migration 034).
|
|
- **No dedicated dashboard UI yet** — this is API/backend-only as of this compile (verify before claiming a management screen exists).
|
|
|
|
### Inventory & Discovery
|
|
- Hardware inventory (mfr/model/serial/BIOS, CPU, memory, disks, NICs, OS), software inventory, service inventory. On-demand refresh. NUL/control-char sanitized before storage (2026-07-05, see Monitoring section).
|
|
- VM / hypervisor / container detection (`032/033`).
|
|
- User/group inventory (`037`-`040`): local + domain + Azure AD accounts, domain-join classification, domain-controller detection, group membership.
|
|
- Network discovery: server-dispatched TCP/ARP/reverse-DNS scans; devices stream back and are persisted/editable.
|
|
|
|
### Patch / Agent Update Management
|
|
- Self-updater: server sends version + URL + SHA256; agent downloads, verifies, atomically swaps, restarts, auto-rolls-back on failed reconnect.
|
|
- Auto-update gated on effective policy `auto_update`. Force via `POST /agents/:id/update`.
|
|
- **Channel model:** new builds tag **beta** by default; agents default to **stable**; promotion is a deliberate re-tag.
|
|
- **Update-scanner platform coverage fixed 2026-07-05 (server v0.3.95):** the scanner's self-check previously tried to execute every binary regardless of platform to read its embedded version, dropping macOS/Linux artifacts entirely because they can't run on the Linux build host — **macOS/Linux agents had never been offered an update**. Now uses host-native-vs-foreign self-check regimes: foreign-platform binaries trust the filename-embedded version; host-native binaries stay a hard `--version` gate (with a best-effort exec-bit repair for suspected 0644 CI-copy artifacts). Post-fix: scanner indexes 8 platform/arch combos (was 4, Windows-only).
|
|
- **Legacy fleet build (Server 2008R2/Win7, Rust 1.77) DISABLED 2026-07-04** (`LEGACY_ENABLED=false`): the unpinned dep re-resolve pulled edition2024 crates Rust 1.77 cannot parse, fail-closing the entire Windows build. Last legacy artifact (agent 0.6.76) frozen and preserved for existing 2008R2/Win7 endpoints; modern builds advance normally. 2008R2 support returns only via a separate legacy-only rewrite (RMM_THOUGHTS Feature 12, Raw — not approved).
|
|
- **Universal self-detecting installer (Feature 9, P1 shipped v0.6.71):** one install path self-detects arch/OS and pulls the correct binary.
|
|
- **Promotion API:** `POST /api/updates/rollouts/:version/promote` body `{"os","arch"}`; rollback via `.../rollback`.
|
|
|
|
### Agent Identity & Enrollment
|
|
- Per-agent enrollment keys; site-specific MSI with SITEKEY injected. `enrolled_agents` is the authoritative enrollment history log.
|
|
- **Durable multi-location `device_id`** (Phase 1 Task 1, 2026-06-11): registry+ProgramData (Windows), /etc+/var/lib (Linux), /Library+mirror (macOS), self-healing; `cleanup.ps1` whitelists identity files.
|
|
- **Cross-site dedup fixed 2026-07-04 (commits `2851523`, `1c1cac2`):** Both WS enrollment paths previously deduped only within `(site_id, device_id)`, so re-running a different site's installer against an already-enrolled machine created a cross-site duplicate. Fixed to match by hardware `device_id` **across all sites** (`db::get_agent_by_device_id`) and re-home via `move_agent_to_site` on first match. A same-day follow-up stopped auto-re-homing **on every reconnect** — the agent re-presents its embedded install-site code as its API key on every beat, which was silently bouncing admin-relocated agents back to their original site. Current behavior: dedup by device_id, but site placement is admin-controlled and persists across reconnects. **13 duplicate-hostname agent pairs remain fleet-wide** (pre-fix legacy `win-*`/`hw-*` records) — cleanup pass not yet done; these are the ongoing trigger for `move_agent` 409s.
|
|
|
|
### Policy & Configuration Management
|
|
- Inheritance chain global -> client -> site -> agent; server computes merged effective policy, pushes via `ConfigUpdate`.
|
|
- Checks engine (`018`/`019`): cpu, memory, disk, ping, port, script, service. Policy-attached check templates (`024`).
|
|
- Remote registry (Windows): enumerate/read routed; write paths exist in the agent but are not routed yet [verify].
|
|
- **Event Log Watch (migration 047):** dashboard management UI at `/event-log-watches`. Policy-clobber class bug fixed (PR #43) by reusing a single rule-shaper across all config-push sites.
|
|
- **Tray policy gate (NEW, shipped 2026-07-04, code merged, policy default OFF):** `tray.enabled` policy flag is now enforced by the watchdog (previously plumbed end-to-end but not enforced — the tray launched/showed regardless of the flag). Fleet-wide materialize (deploying the binary via agent-update, not just fresh MSI) has shipped in code; the deliberate flip to enable it fleet-wide (spec's "Task 4") has NOT happened yet.
|
|
|
|
### Alerting & Watchdog
|
|
- Threshold alerts (ack/resolve, per-agent + fleet summary). Alert templates (`022`). Maintenance mode (`021`).
|
|
- Watchdog: separate supervising process; reports via `POST /watchdog-alert`.
|
|
- **Role-aware offline alerting + alert ignore/mute (shipped 2026-06-07):** server-only `agent_offline`; site/fleet mass-offline aggregation; `alert_mutes` perma-silence.
|
|
- **`create_or_update_alert` hardened (2026-07-05, server v0.3.95):** now returns `Result<Option<Alert>>` — `Ok(None)` when the target alert row was deleted mid-flight (agent cascade-delete FK violation) instead of aborting the caller. Both UPDATE-by-id branches switched from `fetch_one` to `fetch_optional` with fall-through to INSERT (`refresh_alert_row` helper dedups the two near-identical UPDATE blocks). Fixed a `RowNotFound` that had been aborting the entire offline sweep whenever an alert was concurrently deleted (e.g. during duplicate-agent cleanup).
|
|
- **WebSocket log noise reclassified INFO, not ERROR (2026-07-05, server v0.3.96):** benign disconnects (`is_benign_ws_disconnect`) and pre-auth connection churn (`is_preauth_connection_churn` — sleeping laptops, scanners, half-open NAT never sending an auth frame) no longer mint `server_error` alerts. Genuine auth rejections (bad API key, malformed auth message) remain ERROR. Classified at INFO (not DEBUG) so a mass-disconnect anomaly stays visible at the server's default log level.
|
|
- **`move_agent` duplicate-key hardened (2026-07-05):** unique-violation on `idx_agents_site_device` now maps to 409 CONFLICT with an actionable message instead of a raw 500.
|
|
|
|
### Credentials Management
|
|
- Encrypted credentials vault (`016`): scoped global/client/site, typed, metadata-only by default with a separate `/reveal` endpoint. Known HIGH item: `/reveal` ownership-scope check — [verify current state].
|
|
- **Credential inheritance (deployed 2026-06-07):** Opt-in hierarchical cascade, de-duplication, `/effective` endpoints.
|
|
- **Integration credentials (2026-07-01, migration 065):** reuses the same encryption chokepoint for third-party integration API keys (never sent to agents, never plaintext in `integrations.config`).
|
|
|
|
### Audit Log / Log Feedback Intelligence
|
|
- Append-only `audit_log` table (migration 056). Per-line log fingerprinting + `log_signatures` aggregation (migration 057). `logs/analyze` runs on the Claude API (switched from Ollama 2026-06-15).
|
|
|
|
### Backup Integration (MSP360 / MSPBackups)
|
|
- Multi-provider config, connection test, scheduled sync, per-agent + fleet coverage report, agent<->MSP360 mapping with confidence scoring.
|
|
- **Active chronic failure (unresolved as of 2026-07-04):** GND-SERVER (Grabb & Durando) — 42 `backup_failed` alerts since 2026-06-17, failing nightly ~01:02 UTC, "no error detail reported." Also open: AD1 (Dataforth), LAB-SVR (Len's Auto Brokerage), DESKTOP-EVA4H1A (Gary A Hartman, partial). MSP360 sync doesn't capture failure detail — investigation needs a live PowerShell dispatch to read MSP360/CloudBerry local logs.
|
|
|
|
### Remote Access (Tunnel)
|
|
- Agent side substantially built. Server side is a dead-code skeleton (no `/tunnel` routes). Live TTY design (agent-comms-durability Phase 2) will supersede the skeleton.
|
|
|
|
### Identity / Multi-tenancy / Security
|
|
- Auth: JWT; agents auth over WS via per-agent API key + hardware device_id.
|
|
- Microsoft Entra ID SSO (OAuth2/OIDC + PKCE).
|
|
- Organizations / multi-tenancy: org CRUD, roles, limits, dev-admin impersonation.
|
|
- **500-error-leak fixed (commit `58c1a96`):** all server error paths route through `internal_err`.
|
|
|
|
### Platform coverage
|
|
- **Cross-platform (Win/Linux/macOS):** metrics, network state, hardware/software/service inventory, user/group + DC/domain detection, checks, discovery, self-update, scripts/commands.
|
|
- **Windows-only:** `user_session` command context, remote registry, tray-into-session (now policy-gated), watchdog SCM supervision, BSOD detection, VSS policy configurator, legacy fleet (frozen at 0.6.76).
|
|
- **macOS:** agent deployed (Phase 1); tray is a stub; **now correctly included in the update scanner's platform coverage as of 2026-07-05** (previously silently excluded).
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
### Components
|
|
|
|
| Component | Location | Tech | State |
|
|
|---|---|---|---|
|
|
| Server | 172.16.3.30:3001, systemd `gururmm-server.service` | Rust, Axum | deployed, production; v0.3.96 |
|
|
| Dashboard | https://rmm.azcomputerguru.com, nginx | React + TypeScript + Vite, shadcn/ui, Tailwind CSS v4 | deployed, production |
|
|
| Agent (Windows) | Endpoints, `GuruRMMAgent` service via WiX MSI | Rust, Windows MSVC | deployed; modern build v0.6.77, legacy (2008R2/Win7) FROZEN at 0.6.76 and disabled |
|
|
| Agent (Linux) | Endpoints, systemd `gururmm-agent` | Rust, musl static | deployed |
|
|
| Agent (macOS) | Endpoints, LaunchDaemon | Rust, aarch64/x86_64 | Phase 1 deployed 2026-05-12; code-signing issue on Apple Silicon; update-scanner platform gap fixed 2026-07-05 |
|
|
| Tray (Windows) | System tray, named pipe IPC | Rust | deployed to disk fleet-wide via agent-update (2026-07-04); **launch policy-gated, default OFF** |
|
|
| Tray (Linux) | System tray, Unix socket IPC | Rust, GTK | deployed 2026-05-24 |
|
|
| Tray (macOS) | Menu bar | Rust | stub/TODO |
|
|
| PostgreSQL DB | localhost:5432 on 172.16.3.30 | PostgreSQL 18 | deployed |
|
|
| Build pipeline | 172.16.3.30:9000 webhook + `/opt/gururmm/` scripts | Python, Bash | deployed; builds agents AND server |
|
|
| Windows build — Beast (PRIMARY) | GURU-BEAST-ROG over Tailscale | Rust MSVC, WiX 4.x | primary — two-wave parallel, ~336s |
|
|
| Windows build — Pluto (FALLBACK) | 172.16.3.36 | Rust MSVC, WiX v4 | fallback only |
|
|
|
|
### Key Files & Repos
|
|
|
|
- **Active repo:** `azcomputerguru/gururmm` — http://172.16.3.20:3000/azcomputerguru/gururmm (external: https://git.azcomputerguru.com/azcomputerguru/gururmm.git)
|
|
- **Submodule working tree:** `projects/msp-tools/guru-rmm` — gitlink intentionally lags origin/main; read live code via `git -C projects/msp-tools/guru-rmm show origin/main:<path>` or a worktree.
|
|
- **Server binary:** `/opt/gururmm/gururmm-server` on 172.16.3.30
|
|
- **Server config:** `/opt/gururmm/.env`
|
|
- **New shared sanitize module:** `server/src/db/sanitize.rs` (`strip_nul`, `clean_control_chars`) — consumed by `db/logs.rs` and `db/inventory.rs`
|
|
- **Integrations framework:** `server/src/api/integrations.rs`, `server/src/integrations/` (plugin trait + registry), migration `065_integrations_framework.sql`
|
|
- **VSS configurator:** `agent/src/vss_com.rs`, `agent/src/vss.rs`, `specs/vss-policy-config/`, `specs/vss-native-com/`
|
|
- **Tray policy-deploy:** `agent/src/tray_stage.rs`, `agent/src/tray_state.rs`, `agent/src/authenticode.rs`, `specs/tray-policy-deploy/`
|
|
- **Downloads dir:** `/var/www/gururmm/downloads/` on 172.16.3.30
|
|
- **Webhook handler:** `/opt/gururmm/webhook-handler.py`
|
|
- **API (internal):** http://172.16.3.30:3001
|
|
- **API (external):** https://rmm-api.azcomputerguru.com
|
|
- **Dashboard:** https://rmm.azcomputerguru.com
|
|
- **Vault paths:** `infrastructure/gururmm-server.sops.yaml` (API creds, DB password, sudo password), `infrastructure/gururmm-server-physical` (SSH ed25519 key — this fleet key works on 172.16.3.30 currently, not just the future physical box; confirmed 2026-07-04)
|
|
- **SSH to prod:** `ssh -i ~/.ssh/gururmm-physical guru@172.16.3.30` (ed25519, key-only; password auth disabled)
|
|
- **Pre-merge verification skill:** `bash .claude/skills/gururmm-build/scripts/verify.sh server|agent|dashboard|migrations`
|
|
|
|
### Repo Structure
|
|
|
|
```
|
|
gururmm/
|
|
├── agent/ Rust agent (managed endpoints)
|
|
│ └── src/
|
|
│ ├── authenticode.rs NEW — Authenticode signature verify (tray-policy-deploy)
|
|
│ ├── tray_stage.rs NEW — tray materialize FSM + staging + versioned mutex
|
|
│ ├── tray_state.rs NEW — tray policy state
|
|
│ ├── bsod.rs Windows BSOD/kernel-crash detection
|
|
│ ├── vss_com.rs Native VSS COM create/provision (vss-native-com + vss-policy-config)
|
|
│ ├── vss.rs VSS policy configurator entry points
|
|
│ ├── commands/mod.rs CommandExecutor + recent-results cache (ACK dedup)
|
|
│ ├── device_id.rs Durable multi-location device_id
|
|
│ ├── transport/
|
|
│ │ └── websocket.rs WS transport; execute_command ACK + dedup; cmd-alias NAK
|
|
│ ├── tunnel/ TunnelManager state machine
|
|
│ ├── metrics/ sysinfo collection + temps
|
|
│ ├── registry_ops/ Windows registry read/write
|
|
│ ├── updater/ Self-update handler (container guard, BUG-019)
|
|
│ └── main.rs
|
|
├── server/ Rust/Axum API server
|
|
│ └── src/
|
|
│ ├── api/
|
|
│ │ ├── integrations.rs NEW — integrations framework CRUD (crowdstrike-falcon Task 3)
|
|
│ │ └── ... ~40 route modules; software.rs SPEC-030
|
|
│ ├── alerts/ Alerting modules (offline.rs: offline_sweep + mass_offline)
|
|
│ ├── db/
|
|
│ │ ├── sanitize.rs NEW — shared NUL/control-char sanitize (2026-07-05)
|
|
│ │ ├── alerts.rs create_or_update_alert -> Option<Alert>, refresh_alert_row
|
|
│ │ ├── inventory.rs NUL-scrub over jsonb + text before upsert
|
|
│ │ └── software_jobs.rs SPEC-030 async removal jobs
|
|
│ ├── updates/
|
|
│ │ └── scanner.rs host-native vs foreign self-check regimes (2026-07-05)
|
|
│ ├── ws/mod.rs WS handler; is_benign_ws_disconnect / is_preauth_connection_churn
|
|
│ └── mspbackups/ MSP360 backup integration
|
|
├── dashboard/ React/TypeScript UI
|
|
│ ├── pages/
|
|
│ │ └── EventLogWatches.tsx
|
|
│ └── components/
|
|
│ └── SoftwareManager.tsx
|
|
├── agent/scripts/ uninstall-engine.ps1
|
|
├── tray/ System tray binary
|
|
├── installer/ WiX v4 MSI; cleanup.ps1; universal self-detecting installer
|
|
├── deploy/
|
|
│ └── build-pipeline/ webhook-handler.py, build-*.sh, build-server.sh (LEGACY_ENABLED=false gate added 2026-07-04)
|
|
├── scripts/ Build/ops scripts
|
|
└── docs/ FEATURE_ROADMAP.md, UI_GAPS.md, RMM_THOUGHTS.md, specs/
|
|
specs/ tray-policy-deploy/, vss-policy-config/, vss-native-com/, crowdstrike-falcon/,
|
|
av-removal-recipes/, remote-software-uninstall/, agent-comms-durability/, ...
|
|
```
|
|
|
|
---
|
|
|
|
## Development
|
|
|
|
### Current Focus
|
|
|
|
As of 2026-07-04 (agent 0.6.77 / server v0.3.96):
|
|
|
|
- **Machine-side backup failures (highest user priority, unresolved):** GND-SERVER chronic 42-failure streak, AD1, LAB-SVR, DESKTOP-EVA4H1A — MSP360 sync reports no error detail; needs a live PowerShell dispatch to read local MSP360/CloudBerry logs or the mspbackups.com console.
|
|
- **13 duplicate-hostname agent pairs fleet-wide** (legacy `win-*`/`hw-*` device_ids alongside re-enrolled hardware-UUID records) — the ongoing trigger for `move_agent` 409s; a one-time cleanup pass (delete stale offline duplicates) is not yet done.
|
|
- **WS auth-timeout source unidentified:** ~1/min unauthenticated connection attempts, now correctly classified INFO (not alerting), but the source is still unknown — would need connection-level source-IP visibility (Feature 5, server-side public-IP capture, still unapproved) or box-level tcpdump.
|
|
- **Tray policy fleet enablement (Task 4, tray-policy-deploy spec):** materialize code is merged and deployed; flipping `tray.enabled` on for the fleet is a deliberate next step, not yet done.
|
|
- **VSS configurator stable-channel promotion:** shipped to beta (0.6.76) only; promotion to stable is a separate deliberate step pending further soak.
|
|
- **Legacy (2008R2/Win7) agent rewrite (RMM_THOUGHTS Feature 12, Raw):** not approved; needs a language/runtime decision (.NET Framework 4.x vs C++) and scope decision (trimmed feature set: enroll, check-in, metrics, command exec, self-update only) before a `/shape-spec`.
|
|
- **Integrations framework dashboard UI:** CrowdStrike Falcon backend + sync worker are live; no management screen exists yet in the dashboard.
|
|
- **SPEC-030 software-uninstall follow-on (Route A / recipes):** Launchy/AIMP-class uninstallers still hang the agent; driver/system-component exclusion flag and a list-jobs endpoint remain unbuilt.
|
|
- **Agent-comms-durability Phase 2/3 (planned):** live TTY, adaptive keepalive, bulk file transfer -> short-lived HTTPS, server half-open eviction sweeper.
|
|
- **Durable agent identity Phase 2-3 (pending soak):** guarded auto-reclaim + an operator merge tool for the pre-fix ghost/duplicate agents.
|
|
- **SPEC-028 offboarding wizard:** specification complete; implementation pending.
|
|
- **Model routing correction (2026-07-04, logged --correction):** Tier-0 simplify/cleanup passes should route to local Ollama before Claude; a routing miss on this session's easy-bugs batch was logged for tracking.
|
|
|
|
### Patterns & Anti-Patterns
|
|
|
|
**New anti-patterns (2026-07-01..05):**
|
|
|
|
| Pattern | What Went Wrong |
|
|
|---|---|
|
|
| Update-scanner self-check executing every binary regardless of target platform | macOS/Linux binaries can't run on the Linux build/scan host, so they were silently dropped from the scannable set entirely — those agents never received an update offer. Fix: host-native binaries get a hard `--version` execute-and-check gate; foreign-platform binaries trust the filename-embedded version. |
|
|
| Writing a machine-global VSS registry governor (`MaxShadowCopies`) to a per-policy `retention_count` | `MaxShadowCopies` is shared by ALL VSS consumers on the box (System Restore, third-party backup) — lowering it FIFO-evicts THEIR shadows too, a genuine data-loss class caught only by a 23-agent pre-merge code review. Use a per-volume size cap as the sole enforced governor instead. |
|
|
| Re-homing an agent's site on every WS reconnect after a device_id-based dedup fix | The agent re-presents its embedded install-site code as its API key on every beat; auto-re-homing on each connect silently bounces an admin-relocated agent back to its original install site. Dedup by device_id, but leave site placement admin-controlled and persistent. |
|
|
| `fetch_one` on an UPDATE-by-id inside a concurrently-deleting table | An agent cascade-delete can remove the target row between a SELECT and the UPDATE; `fetch_one` panics `RowNotFound` and aborts the whole caller (here: the entire offline sweep). Use `fetch_optional` with a fall-through path. |
|
|
| Two sessions editing the same branch's working tree without a coord lock | A concurrent session's uncommitted edits were silently absorbed into another session's commit. Benign this time (same branch, all changes verified at HEAD) but neither session checked the coord lock list first — take a lock before editing a shared submodule branch. |
|
|
| Assuming a spec's `shape.md`/`plan.md` header "Status: not started" reflects current reality | Tasks can ship and merge (git log shows Tasks 1-3+5 done) while the spec doc header is never updated. Verify against `git log`/deployed version, not the spec file's status line. |
|
|
|
|
**Existing anti-patterns (carried forward):**
|
|
|
|
| Pattern | What Went Wrong |
|
|
|---|---|
|
|
| `useMemo` with stable deps for data-dependent values | queryClient is stable, memo never recomputes after queries resolve. Use `useQuery` instead. |
|
|
| Deploying without stopping the server first | "text file busy" kernel error. Always `systemctl stop` before `cp`. |
|
|
| Building without `DATABASE_URL` | sqlx compile-time macros fail. |
|
|
| DB migrations without inserting into `_sqlx_migrations` | Server crashes on start. Must insert SHA-384 checksum manually. |
|
|
| WiX MSI builds on Linux | WiX requires `msi.dll`. Must build on Pluto or Beast. |
|
|
| Manual builds via SSH | All builds go through `webhook-handler.py`. |
|
|
| `Restart-Service GuruRMMAgent -Force` in command scripts | Kills agent before it can report result. Use a scheduled task with delay instead. |
|
|
| `sudo -u guru git` in systemd build context | git rejects repo as dubious ownership. Use `git config --system --add safe.directory`. |
|
|
| `+1.77` legacy builds without pinned deps | Fresh dep resolution repeatedly pulls edition2024 crates Rust 1.77 can't parse (BUG-021, recurred 2026-07-04 with a different dep pair, led to the variant being disabled outright rather than re-chasing pins). |
|
|
| Dead WebSocket write half | WS write fails, send task dies, receive loop keeps agent connected with a dead write half. Fix: `tokio::select!` monitoring both tasks. |
|
|
| Using the `minidump` crate for Windows kernel dumps | Only parses Breakpad MDMP, not Windows kernel PAGEDU64. Parse `DUMP_HEADER64`/`32` at fixed offsets directly. |
|
|
| Build classification defaulting to stable | New builds default `beta`; promotion is an explicit re-tag. |
|
|
| `command_type: "cmd"` in API commands | Not a valid `CommandType` variant originally — agent silently dropped the message. Now aliased to `Shell`. |
|
|
| Reading the stale submodule working tree for code | The submodule gitlink intentionally lags `origin/main`. Always read via `git -C ... show origin/main:<path>` or a worktree. |
|
|
|
|
**Good patterns (carried forward + new):**
|
|
|
|
- **Shared sanitize chokepoint** (`db/sanitize.rs`, 2026-07-05) — one NUL/control-char scrubber consumed by both `db/logs.rs` and `db/inventory.rs`, replacing duplicated ad hoc scrub logic.
|
|
- **Capability-based classification (host-native vs foreign) instead of a single universal check** — the update scanner now branches its self-check strategy per target platform rather than applying one strategy fleet-wide.
|
|
- **Multi-AI adversarial review before shipping risky agent-side changes** — the VSS redesign (Grok + Gemini) and tray redesign (17-agent Claude review + Grok + Gemini) both caught genuine defects (a data-loss governor and 9 architecture defects respectively) that a single-model review missed on the first pass.
|
|
- **Platform parity rule** — any agent feature goes on Windows + Linux + macOS in the same commit.
|
|
- **`gururmm-build` skill for pre-merge verification** — does NOT trigger production build; merging to main is the only prod trigger.
|
|
- **Dashboard beta-before-main rule** — dashboard changes go to beta (auto-built from main) first; promote explicitly.
|
|
- **Two independently-deployable slices for cross-component features, capability-gated** — agent slice and server slice can deploy in any order; old agents/servers keep legacy behavior until the gate flips.
|
|
|
|
### Build & Deploy
|
|
|
|
**CRITICAL: Never trigger builds manually via SSH. All builds go through the webhook pipeline.**
|
|
|
|
```
|
|
Gitea push to main
|
|
-> webhook-handler.py (172.16.3.30:9000)
|
|
-> build-shared.sh (auto-version bump)
|
|
-> build-linux.sh
|
|
-> build-windows.sh (Beast PRIMARY over Tailscale || Pluto FALLBACK
|
|
TWO-WAVE PARALLEL BUILD; legacy wave now gated LEGACY_ENABLED=false)
|
|
-> build-mac.sh (stub — no build machine configured yet)
|
|
-> build-server.sh (gated on server/ changes; backup+rollback)
|
|
-> artifacts -> /var/www/gururmm/downloads/ with sha256 + -latest symlinks + .channel (beta)
|
|
```
|
|
|
|
**Legacy variant disabled 2026-07-04 (`84a82a3`):** `LEGACY_ENABLED=false` in `build-windows.sh` gates the legacy wave/copy/deploy/symlink off entirely; the last-built legacy artifact (0.6.76) is preserved and excluded from cleanup so existing 2008R2/Win7 endpoints keep receiving that frozen build. Modern (amd64/x86/tray/debug) builds are no longer blocked by legacy dep-resolution failures.
|
|
|
|
**Channel/promotion model, downloads layout, dashboard beta/prod split, DB migration hazards, and Windows build host detail are unchanged from the 2026-06-25 compile** — see prior History Highlights / Compilation Notes for the full mechanics; current migration count is 065 (`065_integrations_framework`).
|
|
|
|
---
|
|
|
|
## Active State
|
|
|
|
**Fleet (as of 2026-07-04):**
|
|
- 359 enrolled agents; 228 fresh/online at last sample
|
|
- Agent 0.6.77 modern / 0.6.76 legacy-frozen-and-disabled; server v0.3.96
|
|
- Active `server_error` alerts: **0** (cleaned to zero by the 2026-07-05 deploys)
|
|
- 13 duplicate-hostname agent pairs outstanding (not yet cleaned up)
|
|
- 5 active `backup_failed` alerts (GND-SERVER chronic, AD1, LAB-SVR, DESKTOP-EVA4H1A) — investigation open
|
|
- VSS configurator on beta channel only (0.6.76, 8 agents); stable fleet pinned at 0.6.66 (233 agents)
|
|
- Beta channel: site "Mike's Car" (`103c10b9-c1de-4dd8-b382-b8362ed3143e`) has `update_channel='beta'` (persists across re-enrollment)
|
|
|
|
**API auth:**
|
|
- `POST /api/auth/login` -> JWT (~24h)
|
|
- Creds: vault `infrastructure/gururmm-server.sops.yaml`
|
|
- Key endpoints: `GET /api/agents`, `POST /api/agents/:id/command`, `GET /api/commands/:id`, `POST /api/agents/:id/update`, `POST /api/updates/rollouts/:version/promote`
|
|
- Software (SPEC-030): `GET /api/agents/:id/software`, `POST /api/agents/:id/software/uninstall` (-> job_id), `GET .../uninstall/jobs/:job_id`, `GET .../removal-status`, `GET /api/software/knowledge`
|
|
- **Integrations (NEW):** `GET/POST /api/integrations`, `GET/DELETE /api/integrations/:id`, `GET/PUT /api/integrations/:id/credentials`, `GET/PUT /api/integrations/:id/mappings`, `GET /api/integrations/:id/sync-state`, `POST /api/integrations/:id/test`
|
|
- Command fields: `command_type` (`shell`/`powershell`/`python`/`script`/`claude_task`/`cmd` alias), `command`, optional `context` (`system`/`user_session`), `timeout_seconds`/`elevated`.
|
|
|
|
**Dashboard — complete and working:**
|
|
Agents management, Clients/Sites CRUD, Commands execution + terminal, Logs + AI analysis (Claude API), Alerts, Metrics, Auto-update triggering, Network state, Entra ID SSO, Policies Dashboard, Registry editor (read-only), MSP360 backup status + mappings/verify UI, Organizations + dev-admin impersonation, Credentials with inheritance, AgentDetail Crashes tab + version history, Event Log Watches management, Software Manager (BETA, not guaranteed).
|
|
|
|
**Dashboard — incomplete (see UI_GAPS.md):**
|
|
- Watchdog alerts UI — blocked on 2 missing server routes
|
|
- BSOD in Alerts stream (Phase 2 deferred)
|
|
- Tunnel session management (server skeleton)
|
|
- Offboarding wizard UI (SPEC-028 complete; implementation pending)
|
|
- Alert mute UI — server endpoints ready, dashboard NOT started
|
|
- **Integrations Center UI (NEW gap)** — CrowdStrike Falcon framework + sync worker are live server-side; no management screen in the dashboard yet
|
|
- **Tray policy toggle surfacing** — `tray.enabled` is enforced server/agent-side; verify whether the policy UI exposes the toggle to operators [verify]
|
|
|
|
**Security backlog (HIGH):**
|
|
- `credentials/:id/reveal` — horizontal privilege escalation (no ownership scope check)
|
|
|
|
---
|
|
|
|
## Key Architecture Decisions (LOCKED)
|
|
|
|
These decisions are locked. Do not reverse without explicit user approval.
|
|
|
|
1. **Per-agent enrollment keys.**
|
|
2. **Site-specific MSI generation.**
|
|
3. **No TOML/config for endpoints** — server URL compiled into binary.
|
|
4. **Policy inheritance chain** — global -> site -> client -> agent.
|
|
5. **Platform parity rule** — any agent feature ships on Windows, Linux, and macOS in the same change.
|
|
6. **Watchdog as separate process** — reports via REST (`POST /watchdog-alert`), no WebSocket.
|
|
7. **Build pipeline is the only path to production.**
|
|
8. **Multi-tenancy identity model (ADR-001)** — Dev -> Partner -> Client.
|
|
9. **Holistic feature development** — every feature requires backend + API + dashboard UI + docs.
|
|
10. **AI-optional operation.**
|
|
11. **Durability-first command delivery** — the durable Postgres `commands` row is the source of truth; the WebSocket is only a delivery hint.
|
|
12. **VSS: configure, never delete on a schedule** (added 2026-07-03) — the agent may only set retention governors and create shadows on a schedule; scheduled or automatic deletion reintroduces the MITRE T1490 surface. On-demand admin-initiated delete stays allowed.
|
|
13. **Third-party security integrations are a plugin, not a schema** (added 2026-07-01) — a new vendor implements the `integrations` plugin trait against the generic framework tables; it does not get its own bespoke migration set.
|
|
|
|
---
|
|
|
|
## History Highlights
|
|
|
|
| Date | Event |
|
|
|---|---|
|
|
| 2025-12-15 | Project genesis: Windows service + Linux installer + site code auth + build server. |
|
|
| 2026-04-19 | Full drill-down navigation, auto-install on first run, Pluto build VM setup started. |
|
|
| 2026-04-21 | MSI build fix. DESIGN.md created. BirthBiologic onboarded. |
|
|
| 2026-04-29 | UI_GAPS.md created. Holistic development principle formalized. |
|
|
| 2026-05-12 | macOS agent Phase 1 deployed. Code signing issue on Apple Silicon noted. |
|
|
| 2026-05-15 | Dead WebSocket write-half bug fixed. Temperature struct field mismatch fixed. |
|
|
| 2026-05-16 | Watchdog bugs fixed. /feature-request skill created. |
|
|
| 2026-05-17 | Syncro PSA Integration added to roadmap. Office power failure recovery. |
|
|
| 2026-05-18 | Multi-tenancy architecture (ADR-001) decided. 5 SPEC documents created. |
|
|
| 2026-05-19 | AD2 crash-loop fix. MSP360 integration completed. Process drill-down modal. |
|
|
| 2026-05-23 | /rmm-audit pass. Agent optimization Phases 1A-3. MSRV bumped to 1.85. Fleet at 0.6.29. |
|
|
| 2026-05-24 | Linux tray IPC + GTK merged. Build pipeline split per-platform. Fleet converged to 0.6.38. |
|
|
| 2026-05-31 | Roadmap reconciliation. MSPBackups mapping/verify UI + dev-admin impersonation UI deployed. SPEC-021 written. |
|
|
| 2026-06-01 | BUG-016/017 fixed. BSOD detection feature (Phase 1) implemented and merged, agent 0.6.51. |
|
|
| 2026-06-02 | Server 0.3.37 + migration 048 deployed. Webhook wired to dispatch build-server.sh. |
|
|
| 2026-06-04 | BUG-020 (ghost tray icons) fixed to beta (`137dd85`). |
|
|
| 2026-06-07 | Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit all shipped same day. |
|
|
| 2026-06-11 | Physical server migration executed (<27 min downtime). Ghost agent root-caused, durable identity Phase 1 Task 1 shipped. Agent-comms-durability Phase 1 shipped (agent v0.6.63/server v0.3.68). |
|
|
| 2026-06-12 | Beast parallel Windows build wired in (`cfbdb59`, ~336s). command_type "cmd" root-caused and fixed (`3de9faf`). |
|
|
| 2026-06-15 | BSOD dedup key changed to bugcheck code. MSI EXDEV fix. sync.sh Phase 1 auto-heal. logs/analyze switched to Claude API. 500-error-leak closed. |
|
|
| 2026-06-18 | sync.sh submodule auto-heal verified fleet-wide. |
|
|
| 2026-06-21 | BUG-019/018 fixed and merged. Event Log Watch UI shipped. Howard cleared for GuruRMM merges/deploys. gururmm-build skill created. |
|
|
| 2026-06-22 | BUG-021 (legacy dep-pin) fixed, v0.6.67 stable. BUG-022 (dead WatchdogEvent path) fixed. Fleet verified ~270/~178 online. |
|
|
| 2026-06-23 | Universal self-detecting installer (Feature 9) P1 shipped (v0.6.71). av-removal-recipes spec added. |
|
|
| 2026-06-24 | SPEC-030 remote software uninstall — BETA — code merged + deployed (Route B, async jobs, three-state KB, migrations 061-064). agent 0.6.75 / server v0.3.87. |
|
|
| 2026-06-25 | Win11 KB5095093 Point-in-Time Restore captured as a Raw RMM_THOUGHTS entry. |
|
|
| 2026-07-01..02 | Vendor-agnostic Integrations Framework shipped (migration 065) with CrowdStrike Falcon as the first plugin — API client, client mappings, periodic sync worker. No dashboard UI yet. |
|
|
| 2026-07-03..04 | VSS Policy Configurator (SPEC-016 redesign) shipped to beta (agent 0.6.76) — replaces scheduled create+prune with a configure-governors-only model, eliminating the T1490 surface; 23-agent code review caught and fixed a data-loss-class `MaxShadowCopies` defect pre-merge; verified live on the T1490 canary NEPTUNE. |
|
|
| 2026-07-04 | Legacy (2008R2/Win7, Rust 1.77) build variant disabled (`84a82a3`) after an edition2024 dep recurrence fail-closed the whole Windows build; last legacy artifact frozen at 0.6.76. RMM_THOUGHTS Feature 12 (legacy agent rewrite) added Raw. Cross-site agent dedup fixed (`2851523`, `1c1cac2`) — dedup by device_id across all sites, no re-home on reconnect. Policy-Controlled Tray Deployment merged (`f6a4251`) — materialize shipped fleet-wide, launch stays policy-gated default OFF. |
|
|
| 2026-07-04..05 | Easy-bugs batch + full alert triage: update-scanner host-native/foreign self-check regimes (fixed macOS/Linux agents never receiving update offers), inventory NUL/control-char sanitize (`db/sanitize.rs`, fixed 7+ machines' silently-failing inventory), WS benign-disconnect + pre-auth churn reclassified ERROR->INFO, `create_or_update_alert` survives cascade-delete races, `move_agent` maps duplicate-key to 409. Deployed as server v0.3.95 + v0.3.96. Active `server_error` alerts reduced to 0. Machine-side backup-failure investigation (GND-SERVER et al.) started but not completed. |
|
|
|
|
---
|
|
|
|
## Compilation Notes
|
|
|
|
- **2026-07-04 recompile (Howard-Home/claude-main):** Delta from 2026-06-25 (migrations 64 -> 65; agent 0.6.75 -> 0.6.77 modern / 0.6.76 legacy-frozen; server 0.3.87 -> 0.3.96; fleet 270 -> 359 enrolled). Added: (1) **Third-Party Integrations Framework** (migration 065, CrowdStrike Falcon plugin) as a new Capabilities subsection — API-only, no dashboard UI yet, flagged as a new UI gap. (2) **VSS Policy Configurator (SPEC-016 redesign)** replacing the prior scheduled create+prune model — new Capabilities subsection, new locked architecture decision #12, data-loss-defect anti-pattern captured. (3) **Policy-Controlled Tray Deployment** — materialize/launch split documented, Task 4 (fleet enablement) flagged as pending in Current Focus. (4) **Legacy (2008R2/Win7) build variant disabled** — Patch/Update section + Build & Deploy + anti-patterns updated; RMM_THOUGHTS Feature 12 noted. (5) **Cross-site agent dedup fix + no-re-home-on-reconnect follow-up** — new Agent Identity & Enrollment subsection; 13 remaining duplicate pairs noted as open work. (6) **2026-07-04/05 easy-bugs + alert-triage batch** (server v0.3.95/v0.3.96) — update-scanner platform regimes, `db/sanitize.rs`, WS noise reclassification, `create_or_update_alert` Option<Alert> hardening, `move_agent` 409 — folded into Monitoring, Patch/Update, and Alerting Capabilities subsections plus a new Recent Work entry. (7) Active State refreshed (fleet count, 0 active server_error, 13 dup pairs, 5 open backup_failed alerts, VSS beta-only channel state). (8) New anti-patterns: scanner host/foreign self-check split, machine-global VSS governor data-loss class, reconnect re-homing footgun, `fetch_one` on a concurrently-deletable row, concurrent-session-no-lock branch collision, stale spec-status-header trap. History Highlights/Patterns preserved verbatim except new rows appended; article length kept in the same range as the 2026-06-25 compile by condensing older Recent Work entries into shorter summaries where the full detail is preserved in git history and prior compiles.
|
|
- **2026-06-25 recompile (Howard-Home/claude-main):** Delta from 2026-06-22 — SPEC-030 remote software inventory + bulk uninstall added as a major Capabilities subsection; universal self-detecting installer (Feature 9) P1 shipped; Current Focus rewritten; flagged the old "pending PRs #40-#46, migrations 061-063" prediction as superseded because software-removal work consumed 061-064 instead.
|
|
- **2026-06-22 recompile (Howard-Home/claude-main):** Delta from 2026-06-11 — fleet ~270/~178, BUG-021/018/019 fixed on main, Event Log Watch UI shipped, BUG-022 dead-code watchdog path removed, Beast parallel two-wave build documented, command_type "cmd" alias + NAK documented, migration count to 60.
|
|
- **2026-06-11 recompile (GURU-5070/claude-main):** Full recompile — physical server migration, durable agent identity, agent-comms-durability Phase 1 full detail, migration count to 59.
|
|
- **2026-06-07 recompile:** Backup-alert quality pass, credential inheritance, offline alerting + mute, UI gap batch + enrollment audit folded in; migration count to 55+.
|
|
- **2026-06-04 recompile:** Corrected GURU-5070 channel state; BUG-020 documented.
|
|
- **2026-06-02 recompile:** BSOD detection, server build webhook wiring, build channel default beta; migration count 46 -> 48.
|
|
- **2026-05-26 recompile:** Added the Capabilities / Feature Set section, synthesized from authoritative artifacts. Changelogs are an unreliable capability source — migrations + commit log are authoritative.
|
|
|
|
## Backlinks
|
|
|
|
- [[clients/cascades-tucson]] — RECEPTIONIST-PC enrolled (site CascadesTucson)
|
|
- [[systems/gururmm-build]] — Physical server at 172.16.3.30; GuruRMM API 3001, ClaudeTools API 8001, Coord API, MariaDB, PostgreSQL, build pipeline
|
|
- [[systems/jupiter]] — Unraid host at 172.16.3.20; virsh host for all VMs; Docker: Gitea, NPM, Seafile
|
|
- [[systems/pluto]] — Windows build server (MSI, WiX) at 172.16.3.36
|