# GuruRMM — Role-Aware Offline Alerting + Alert Ignore/Mute (+ MSP360 API probe, Jupiter VM diag)

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Shipped two GuruRMM alerting features end-to-end and investigated an MSP360 API
capability question, then diagnosed an unrelated Unraid VM network fault. After the
morning backup-alert cleanup (separate log), Mike asked whether the MSP360 API could
delete plans / trigger backups / configure storage for the three console follow-ups.
Probed `api.mspbackups.com` with the vaulted Provider token: it is monitoring-only
(Companies/Users/Monitoring, GET; every management path 404s; OPTIONS confirms
read-only). The agent-CLI fallback also failed -- the white-labeled MSP360 agent on
SERVER has no `cbb.exe` at any standard path. Concluded the three items stay
MSP360-console tasks and filed them as coord todos for Mike.

Built **role-aware offline alerting + correlated mass-offline detection** (spec
`role-aware-offline-alerting`). Servers offline are incidents; workstations offline
are routine. Implemented server-side (Tasks 1-5): migration 054 `role_override`
column + a shared `agent_role` classifier delegating to the canonical
`agent_is_server`; `OfflineAlertingConfig` in the policy JSONB; a new scheduled
offline-sweep evaluator generating server-only `agent_offline` alerts plus site
(>=50% + >=3) and fleet (>=10) `mass_offline_*` aggregates; a warm-up restart guard;
and `PUT /api/agents/:id/role-override`. Code Review REJECTED the first cut for a
CRITICAL defect in the spec's own restart guard (`last_seen < started_at`
permanently disabled detection in steady state because `last_seen` advances on every
heartbeat); fixed by making the warm-up window the sole guard. Re-review APPROVED.
Then the dashboard unit (Tasks 6-7): role-aware triage (offline servers individual +
elevated; offline workstations collapsed into a quiet "N workstations offline"
roll-up) and the role-override control on the agent detail page; plus a small server
DTO fix so `GET /api/agents/:id` returns `role_override`. Verified live: a real
offline server (WIN-TG2STMODJG8, "Windows Server 2019 Standard Evaluation",
auto-classified via `os_name`) fired `agent_offline`; zero false mass-offline despite
~55 chronically-offline agents; restart guard held across deploys.

Built **alert ignore/mute (perma-silence)** (spec `alert-mute`). Distinct from
ack/resolve, which only quiet the current cycle: a mute keyed on `dedup_key`
suppresses a recurring condition until un-ignored, with a required reason. Server
unit (Tasks 1-4): migration 055 `alert_mutes` table + `muted` status; an
`is_dedup_muted` gate inserted into the universal `create_or_update_alert` AND the
one `create_check_alert` bypass so muted conditions write `status='muted'` (never
active, never email); transactional `mute_condition`/`unmute_condition`;
`POST /api/alerts/:id/mute` (reason required -> 400) and `/unmute`. Code Review
APPROVED (the muted gate is a top-of-function early return, so the active path is
byte-for-byte unchanged for normal alerts). Verified live against WIN-TG2STMODJG8's
re-firing `agent_offline`: mute -> `muted`, survived a full offline sweep without
re-firing, unmute -> `active`. The dashboard unit (Task 5) is NOT yet built.

Finally, diagnosed an Unraid VM network fault on Jupiter (172.16.3.20): the libvirt
domain "Windows Server 2016" (guest hostname `ACG-DWP-X-BB`) had no LAN. Found the
host side healthy (vnet8 bridged to br0, forwarding, receiving LAN broadcast), and
the guest holding APIPA `169.254.157.152` with no gateway -- its single e1000 NIC is
DHCP-enabled but not getting a lease from pfSense (172.16.0.1). Paused for Mike's
call on DHCP-vs-static before changing the guest.

## Key Decisions

- Offline classification = auto (`os_product_type` 2/3, else `os_name`/`os_version`
  ~/server/i) + a manual `role_override` column. `os_product_type` is populated on
  only 16/168 agents, so `os_name` is the workhorse signal; inventory-less offline
  servers (SIF-SERVER, Server2013) auto-classify as workstations and need the manual
  override -- Mike held on tagging them.
- Mass/aggregate alerts pin to a representative offline agent with a site/fleet
  `dedup_key` rather than making `alerts.agent_id` nullable (avoids a rippling
  migration across every alert path).
- Restart guard = warm-up window only (max grace after boot). Dropped the broken
  `last_seen < started_at` clause. Individual server alerts fire independently of the
  site-outage alert (so a still-down server keeps paging after an outage clears).
- Alert mute keyed on `dedup_key` (universal recurring-condition id, always set);
  permanent until un-ignored; stays muted on severity escalation. Gate placed at the
  two creation choke points so it is universal across every alert type.
- MSP360 console items stay manual: the Provider API token is monitoring-tier only;
  no REST path for plan delete / run / storage config at our access level.

## Problems Encountered

- CRITICAL restart-guard defect (my spec error): `last_seen < started_at` silently
  disabled offline detection in steady state. Caught in Code Review; fixed to
  warm-up-only and re-reviewed. The bug-enshrining unit test was rewritten.
- Split-brain classifier: the new `agent_role` diverged from the existing
  `agent_is_server`. Unified by delegating to the canonical fn (threaded
  `os_version`).
- `GET /api/agents/:id` omitted `role_override` (returns `AgentResponse`, not
  `AgentWithDetails`), so the override card rendered blind. Fixed by adding the field
  to the base `Agent`/`AgentResponse` DTOs (all `Agent` SELECTs use `SELECT *`).
- Orphaned offline alerts + per-agent policy N+1 in the sweep: replaced scattered
  resolves with one authoritative resolve-except pass; read per-agent grace only for
  servers; mass membership uses global grace/window.
- MSP360 white-label: no `cbb.exe` on SERVER, so the agent-CLI automation route for
  the backup console tasks was not viable without per-box fingerprinting.
- Submodule gitlink: detached to the pinned commit before `/save` so the session-log
  commit does not fold a gitlink bump.

## Configuration Changes

GuruRMM submodule (`azcomputerguru/gururmm`), all merged to `main`:
- `server/migrations/054_agent_role_override.sql` (new) -- `role_override` column.
- `server/migrations/055_alert_mutes.sql` (new) -- `alert_mutes` table + partial
  unique index `ON dedup_key WHERE active`.
- `server/src/alerts/offline.rs` (new) -- offline sweep + mass detection.
- `server/src/db/alerts.rs` -- `AlertStatus::Muted`; `is_dedup_muted`/
  `mute_condition`/`unmute_condition`; muted gate in `create_or_update_alert`;
  `create_or_update_alert` title/message/severity refresh (from morning).
- `server/src/alerts/check_alerts.rs` -- mute gate + notify suppression.
- `server/src/db/agents.rs` -- `role_override` on `Agent`/`AgentResponse`/
  `AgentWithDetails`; `agent_role` + `set_agent_role_override`.
- `server/src/db/policies.rs`, `server/src/policy/{effective,merge}.rs` --
  `OfflineAlertingConfig` + merge.
- `server/src/main.rs` -- offline-sweep spawn (60s).
- `server/src/api/{alerts,agents,mod}.rs` -- mute/unmute + role-override endpoints.
- `dashboard/src/lib/agentRole.ts` (new), `dashboard/src/components/ExceptionStream.tsx`,
  `dashboard/src/pages/AgentDetail.tsx`, `dashboard/src/api/client.ts` -- triage +
  override control.
- specs/`role-aware-offline-alerting/` and specs/`alert-mute/` (new spec folders).

## Credentials & Secrets

No new secrets. MSP360 Provider API: vault `msp-tools/msp360-api.sops.yaml`
(`credentials.login` / `credentials.password`); base `https://api.mspbackups.com`,
`POST /api/Provider/Login` -> `access_token`; monitoring-only tier. GuruRMM API:
vault `infrastructure/gururmm-server.sops.yaml`. Jupiter: vault
`infrastructure/jupiter-unraid-primary.sops.yaml` (root@172.16.3.20:22, key auth
works from GURU-5070).

## Infrastructure & Servers

- GuruRMM API/server: 172.16.3.30:3001 (Linux VM on Jupiter). Dashboards
  rmm-beta / rmm.azcomputerguru.com (shared API serves both).
- Jupiter Unraid: 172.16.3.20, root:22, Unraid 6.12.85. VMs via virsh; bridge `br0`
  (uplink eth2), `172.16.3.20/22`. VM "Windows Server 2016" = guest `ACG-DWP-X-BB`,
  vnet8/br0/e1000, MAC 52:54:00:d4:8e:59, APIPA 169.254.157.152 (no DHCP lease).
- Office LAN 172.16.0.0/22; pfSense 172.16.0.1 (router + DNS + DHCP).

## Commands & Outputs

- Offline alerting commits: `f1cdf5d`, `30e4f23` (fix), `21d63bd` (DTO), `3eedf91`
  (dashboard). Alert-mute: `29c405e` (spec), `a120e71` (server).
- Live mute verify: empty-reason 400; mute -> status=muted; after a 60s sweep the
  muted alert did NOT re-fire active (count=0); unmute -> active.
- Jupiter diag: `virsh domiflist 9` (vnet8/br0/e1000); `virsh domifaddr 9 --source
  agent` -> 169.254.157.152; guest-exec `ipconfig /all` -> DHCP Enabled: Yes, APIPA,
  no gateway.

## Pending / Incomplete Tasks

- **alert-mute dashboard (Task 5)** -- NOT started. Ignore button + required-reason
  prompt + Muted filter + Un-ignore, on the alerts page AND the agent Alerts tab;
  muted rows show Un-ignore instead of Ack/Resolve. (Mike: "then continue" -> do this
  next.) Plus alert-mute Task 6 (roadmap) + Task 7 (final doc).
- **offline alerting** -- Task 8 (roadmap entry) outstanding; classification gap:
  tag SIF-SERVER / Server2013 (and WIN-TG2STMODJG8 if it should be a workstation) via
  role_override -- held for Mike.
- **Jupiter VM** -- decide DHCP vs static for ACG-DWP-X-BB; if DHCP, run `ipconfig
  /renew` via guest agent; if static, set the intended IP. Paused for Mike.
- **MSP360 console (Mike-side todos):** delete SERVER's Nov-2024 plan; AD1 full
  backup for retention; LAB-Becky storage-or-delete.

## Reference Information

- Commits (main): `f1cdf5d`,`30e4f23`,`21d63bd`,`3eedf91`,`29c405e`,`a120e71`.
- Specs: `projects/msp-tools/guru-rmm/specs/role-aware-offline-alerting/`,
  `.../specs/alert-mute/`.
- MSP360 PlanType map: 3=Files,7=SQL,8=Restore,11=Image,13=Consistency,16=HyperV.
- New alert types: `agent_offline`, `mass_offline_site`, `mass_offline_fleet`; new
  status `muted`; new tables `alert_mutes`; new columns `agents.role_override`.
- Test box: WIN-TG2STMODJG8 agent id b6c715df-09fe-4e97-b09a-82a1b535f041 (offline
  eval server, used for live verification).

---

## Update: 00:44 PT — alert-mute dashboard shipped, beta->prod promotion, MSI dedup cleanup, parked design

### Completed since the initial save
- **Alert ignore/mute dashboard (Task 5) — shipped + deployed** (commit `63d7c5d`). Reused the existing Radix Dialog for the required-reason prompt; shared `AlertRow` so both the alerts page and the agent Alerts tab get Ignore/Un-ignore + the Muted filter; muted rows show Un-ignore (not Ack/Resolve) + reason/who/when; status badge labels `muted` as "Ignored". Code Review APPROVED; `npm run build` clean. The alert-mute feature is now complete end-to-end (server `a120e71` + dashboard `63d7c5d`).
- **Roadmap entries** (commit `5f1d9f0`): both offline alerting and alert ignore/mute added to FEATURE_ROADMAP.md Alerting section; specs' roadmap tasks marked DONE.
- **Dashboard beta -> production promotion.** Prod (`/var/www/gururmm/dashboard`, rmm.azcomputerguru.com) was stuck at the 2026-06-04 build. Ran `sudo /opt/gururmm/promote-dashboard.sh --confirm` on 172.16.3.30 (the sanctioned promote: backs up prod first, keeps 10 backups, rsync beta->prod, `--rollback` to undo). Backup `dashboard-20260608-000926`; prod now byte-identical to beta. This shipped ALL accumulated beta work to prod: this session's offline triage + role-override + alert mute UI, plus other sessions' credential-inheritance mgmt + clickable alert badges. Coord component gururmm/dashboard -> deployed.
- **MSI BSOD duplicate cleanup.** MSI (Safesite/Glendale, agent a685af29...) had 4 BSOD VIDEO_TDR_FAILURE (0x116) alerts, all with `dedup_key = NULL` (legacy, from the early-June BSOD implementation before the dedup key was added). Mike had muted one; the other 3 active duplicates could not be spanned by the mute (NULL key falls back to single-row). Resolved the 3 duplicates (final: 0 active, 1 muted "Client to replace unit", 3 resolved). Two resolve calls hit transient HTTP 000; retried clean.

### Parked design conversation (NOT approved — captured for later)
A stream-of-thought design discussion produced two parked features, captured in
`projects/msp-tools/guru-rmm/docs/PARKED_alert-lifecycle-and-telemetry-cadence.md`
(commit `82a7b14`) + two coord todos (`eb14a4d2`, `bacc4f12`):
1. **Stateful alert lifecycle + occurrence history** (server): one alert per dedup_key spanning all statuses; re-fire reopens only when "substantively different" (severity escalation or re-fire after clear; not per-1%); occurrence count+first/last on the row, dated history via existing agent_events (no new table); ack reopens on substantive change, mute is absolute; server-side per-condition cooldown (~15min default, escalations bypass, recovery always prompt); subsumes BSOD per-bugcheck dedup + mute-across-refire + legacy null-key cleanup. Root cause it fixes: dedup currently matches only status='active', so ack/resolve breaks dedup and spawns duplicates.
2. **Tiered agent telemetry cadence** (agent): per-collector collection intervals (live/on-demand ~5-10s only when watched; standard ~30-60s; slow ~5-15min; daily). Disk consumption (slow) vs disk I/O (fast) are different collectors; fast diagnostics as on-demand/live mode; cadence config in policy, server-pushed; cross-platform scheduler. Larger, separate agent-side feature.

Saved a feedback memory (`.claude/memory/feedback_stream_of_thought_design.md`): Mike
prefers free-form design conversations; Claude sharpens as a design partner but does
not build until an explicit go, then decomposes into specs.

### Still open / parked
- Jupiter VM `ACG-DWP-X-BB` no-DHCP (diagnosed, paused for Mike's DHCP-vs-static call; see wiki/systems/jupiter.md).
- Offline classification overrides (SIF-SERVER/Server2013/WIN-TG2STMODJG8) — taggable from the agent detail role-override control; held.
- The two parked design features above; MSP360 console todos (3) from the morning.