# 2026-07-04 — Backup false-alarm root cause + remediation (MSP360 / GuruRMM machine alerts)

## User
- **User:** Howard Enos (howard)
- **Machine:** Howard-Home
- **Role:** tech

## Session Summary

Continuation of the GuruRMM alert-triage session (first half logged in `projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md` — server-side bug fixes on branch `fix/easy-bugs-batch-2026-07-04`, commit f6c163b). This half chased the MACHINE-side backup_failed alerts to root cause and remediated.

Pulled MSP360 logs via /rmm from the three reachable machines (AD1/Dataforth, GND-SERVER/Grabb & Durando, DESKTOP-EVA4H1A/Gary Hartman; LAB-SVR/Len's Auto unreachable — RMM agent offline since 2026-06-18). MSP360 is installed as "Online Backup" (branded "Arizona Computer Guru"), data dir `C:\ProgramData\Online Backup`, per-plan GUID logs in `Logs\`.

**Root cause of the backup_failed alert storm: none are real backup failures.** Plans that start exactly at a top-of-hour UTC time hit 503 Server Unavailable from `ws.mspbackups.com` (MSP360 cloud maintenance windows at ~01:00 and ~04:00 UTC, ~10 min). The client logs "MBS Maintenance: waiting ... will keep retrying", the status upload is recorded as a failed run in MSPBackups, GuruRMM's sync raises a critical backup_failed with "no error detail reported" — and the backup itself then completes Success. GND-SERVER's plan logged `Plan status: Success` on all 12 runs of 2026-07-04; only the 18:00 local (01:00 UTC) run hits the 503 — hence 42 nightly false criticals since 2026-06-17. AD1's Image2025 identical at 21:00 local (04:00 UTC).

**Remediation applied (user-approved):** shifted both schedules off the top of the hour. AD1 `Image2025`: `cbb.exe editBackupIBBPlan -n 'Image2025' -every day -at '21:30'` — verified Hour=21 Min=30. GND-SERVER (every-2h intra-daily recurrence, not expressible in cbb CLI): stopped "Online Backup Service", edited plan XML `e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb` `DailyFromMinutes` 0→30 (backup copy left at `<file>.bak-20260704`), restarted service — verified FromH=0 FromM=30 Period=120, ForceFullSchedule untouched, service Running. Runs now land at :30, clear of both windows. Tonight's runs confirm.

**DESKTOP-EVA4H1A:** the alert's "no access to 7 files" is benign (Office Click-to-Run phantom `AppvIsvStream*.dll` in the VSS snapshot). Real finding: System event log shows "IO operation failed due to a hardware error" for **Disk 1 at LBA 0 every ~1 min**, and Disk 1 no longer enumerates in Get-PhysicalDisk (only the healthy ST1000DM003 system disk shows; SMART clean). Dead/dying attached-external drive. **Syncro ticket #32502** created (Hartman, assigned Howard, Hardware, do-not-email initial issue) + **Reminder appointment Monday 2026-07-06 09:00-09:30** to call the customer. https://computerguru.syncromsp.com/tickets/113490650

**LAB-SVR:** RMM agent AND ScreenConnect offline (user checked SC). Possibly replaced/decommissioned — **Discord DM sent to Mike** asking status; if in service it needs RMM agent + ScreenConnect reinstalled. Its backup alerts (same 01:02 UTC timing) are almost certainly the same false alarm.

**Feature request filed** (/feature-request): RMM_THOUGHTS.md **Feature 13** — suppress/auto-resolve backup_failed alerts whose only failure is a maintenance-window status-upload error and the run later succeeds (delay-and-recheck), plus capture real failure detail in the MSPBackups sync. Committed docs-only to gururmm main (5ad4498) via a temporary worktree off origin/main — the fix branch working tree (another session actively working) was untouched. Coord todo fec70401 + fleet broadcast 38073821 sent.

## Key Decisions

- Schedule shift chosen over waiting for the GuruRMM alert-suppression feature — immediate stop to nightly false criticals; feature filed for the systemic fix.
- GND-SERVER edited via XML+service-restart because cbb CLI has no flags for intra-daily recurrence; AD1 via supported cbb CLI. Config backup left on the box before the XML edit.
- Docs commit for the feature request went through a temp `git worktree` off origin/main, NOT the checked-out fix branch — another session had new uncommitted work there (sanitize.rs etc.); switching/stashing would have disrupted it.
- Hartman ticket set do_not_email on the initial comment — internal diagnostic finding, tech calls the customer Monday instead.

## Problems Encountered

- `ps-encoded.sh` fails on Howard-Home (`iconv: command not found`) — logged to errorlog; fell back to jq-encoded inline dispatch (scripts kept single-quote-only, so quote-stripping not a risk).
- Env vars don't persist across Bash calls (known rule; logged --friction earlier in session) — re-auth or `./.rmm_token` per call.

## Configuration Changes

- **AD1 (Dataforth)**: MSP360 plan "Image2025" schedule 21:00 → **21:30 daily** (via cbb.exe editBackupIBBPlan).
- **GND-SERVER (Grabb & Durando)**: plan "Backup plan on 4/16/2020 1:40:24 PM" (`e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb`) `DailyFromMinutes` 0 → **30** (every-2h runs now at :30); backup at `C:\ProgramData\Online Backup\e25f4cc9-....cbb.bak-20260704`; "Online Backup Service" restarted.
- **gururmm repo main**: docs-only commit 5ad4498 (RMM_THOUGHTS.md Feature 13).
- No parent-repo config changes.

## Credentials & Secrets

None new. RMM auth via `rmm-auth.sh` (vault `infrastructure/gururmm-server.sops.yaml`); Syncro via Howard's baked-in skill key.

## Infrastructure & Servers

- MSP360 on endpoints: install `C:\Program Files\Arizona Computer Guru\Online Backup\` (cbb.exe), data/logs/plans `C:\ProgramData\Online Backup\`, services "Online Backup Service" (+ Remote Management). Version 8.6.0.338.
- MSP360 cloud maintenance windows observed: ~01:00 and ~04:00 UTC, ~10 min, `ws.mspbackups.com` returns 503.
- LAB-SVR (Len's Auto): RMM agent id 8afb1430-c8f2-4fa2-8335-35b13d8a732f, last_seen 2026-06-18; ScreenConnect also offline.

## Commands & Outputs

- Recon/dispatch pattern: `jq -n --arg cmd "$(cat script.ps1)" '{command_type:"powershell",command:$cmd,timeout_seconds:N}'` → POST `/api/agents/<id>/command` (ps-encoded.sh broken on this machine).
- Key log signature (both servers): `MBS Maintenance: waiting 00:0x:xx, will keep retrying 04:00:00` then `(503) Server Unavailable` then `Plan status: Success`.
- EVA disk: `Get-PhysicalDisk` shows only disk 0; System log `disk` source: hourly stream of `IO operation ... LBA 0x0 Disk 1 hardware error`.

## Pending / Incomplete Tasks

1. **Verify tonight/tomorrow**: no backup_failed alerts from AD1 (~04:30 UTC) or GND-SERVER (01:30 UTC window) — if any persist, dig deeper.
2. **Ticket #32502**: call Gary Hartman **Monday 2026-07-06 9:00 AM** (reminder appt 5631486605 on calendar); on-site to identify/replace dead Disk 1.
3. **LAB-SVR**: awaiting Mike's answer (Discord) — replaced/decommissioned vs needs RMM+SC reinstall. If decommissioned: remove agent record + MSP360 endpoint or alerts continue.
4. **Merge the fix branch** (`fix/easy-bugs-batch-2026-07-04`) — being handled by the OTHER active session (it has additional uncommitted work: sanitize.rs, logs.rs, ws/mod.rs beyond commit f6c163b). Merge-to-main deploys.
5. **RMM Thought Feature 13** — discuss → /shape-spec when ready.
6. Optional: exclusion for Office C2R phantom files on EVA's backup plan to silence the recurring "7 files" warning.

## Reference Information

- Syncro ticket #32502 (id 113490650): https://computerguru.syncromsp.com/tickets/113490650 — appt 5631486605 Mon 7/6 09:00.
- gururmm commits: f6c163b (fix branch, server fixes + first-half session log), 5ad4498 (main, docs Feature 13).
- Coord: todo fec70401-c122-482d-a173-ff236b9cf4a1, broadcast msg 38073821-61ce-49ee-91ed-727b967b5900.
- First-half session log: `projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md`.
- Discord: DM to Mike message_id 1523207214293323917; bot alerts in #dev-alerts (RMM dispatches) + #bot-alerts (Syncro #32502).
