Files

Howard Enos 9542de2b0a sync: auto-sync from HOWARD-HOME at 2026-07-04 23:15:20

Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-04 23:15:20

2026-07-04 23:15:49 -07:00

7.9 KiB

Raw Blame History

2026-07-04 — Backup false-alarm root cause + remediation (MSP360 / GuruRMM machine alerts)

User

User: Howard Enos (howard)
Machine: Howard-Home
Role: tech

Session Summary

Continuation of the GuruRMM alert-triage session (first half logged in projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md — server-side bug fixes on branch fix/easy-bugs-batch-2026-07-04, commit f6c163b). This half chased the MACHINE-side backup_failed alerts to root cause and remediated.

Pulled MSP360 logs via /rmm from the three reachable machines (AD1/Dataforth, GND-SERVER/Grabb & Durando, DESKTOP-EVA4H1A/Gary Hartman; LAB-SVR/Len's Auto unreachable — RMM agent offline since 2026-06-18). MSP360 is installed as "Online Backup" (branded "Arizona Computer Guru"), data dir C:\ProgramData\Online Backup, per-plan GUID logs in Logs\.

Root cause of the backup_failed alert storm: none are real backup failures. Plans that start exactly at a top-of-hour UTC time hit 503 Server Unavailable from ws.mspbackups.com (MSP360 cloud maintenance windows at ~01:00 and ~04:00 UTC, ~10 min). The client logs "MBS Maintenance: waiting ... will keep retrying", the status upload is recorded as a failed run in MSPBackups, GuruRMM's sync raises a critical backup_failed with "no error detail reported" — and the backup itself then completes Success. GND-SERVER's plan logged Plan status: Success on all 12 runs of 2026-07-04; only the 18:00 local (01:00 UTC) run hits the 503 — hence 42 nightly false criticals since 2026-06-17. AD1's Image2025 identical at 21:00 local (04:00 UTC).

Remediation applied (user-approved): shifted both schedules off the top of the hour. AD1 Image2025: cbb.exe editBackupIBBPlan -n 'Image2025' -every day -at '21:30' — verified Hour=21 Min=30. GND-SERVER (every-2h intra-daily recurrence, not expressible in cbb CLI): stopped "Online Backup Service", edited plan XML e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb DailyFromMinutes 0→30 (backup copy left at <file>.bak-20260704), restarted service — verified FromH=0 FromM=30 Period=120, ForceFullSchedule untouched, service Running. Runs now land at :30, clear of both windows. Tonight's runs confirm.

DESKTOP-EVA4H1A: the alert's "no access to 7 files" is benign (Office Click-to-Run phantom AppvIsvStream*.dll in the VSS snapshot). Real finding: System event log shows "IO operation failed due to a hardware error" for Disk 1 at LBA 0 every ~1 min, and Disk 1 no longer enumerates in Get-PhysicalDisk (only the healthy ST1000DM003 system disk shows; SMART clean). Dead/dying attached-external drive. Syncro ticket #32502 created (Hartman, assigned Howard, Hardware, do-not-email initial issue) + Reminder appointment Monday 2026-07-06 09:00-09:30 to call the customer. https://computerguru.syncromsp.com/tickets/113490650

LAB-SVR: RMM agent AND ScreenConnect offline (user checked SC). Possibly replaced/decommissioned — Discord DM sent to Mike asking status; if in service it needs RMM agent + ScreenConnect reinstalled. Its backup alerts (same 01:02 UTC timing) are almost certainly the same false alarm.

Feature request filed (/feature-request): RMM_THOUGHTS.md Feature 13 — suppress/auto-resolve backup_failed alerts whose only failure is a maintenance-window status-upload error and the run later succeeds (delay-and-recheck), plus capture real failure detail in the MSPBackups sync. Committed docs-only to gururmm main (5ad4498) via a temporary worktree off origin/main — the fix branch working tree (another session actively working) was untouched. Coord todo fec70401 + fleet broadcast 38073821 sent.

Key Decisions

Schedule shift chosen over waiting for the GuruRMM alert-suppression feature — immediate stop to nightly false criticals; feature filed for the systemic fix.
GND-SERVER edited via XML+service-restart because cbb CLI has no flags for intra-daily recurrence; AD1 via supported cbb CLI. Config backup left on the box before the XML edit.
Docs commit for the feature request went through a temp git worktree off origin/main, NOT the checked-out fix branch — another session had new uncommitted work there (sanitize.rs etc.); switching/stashing would have disrupted it.
Hartman ticket set do_not_email on the initial comment — internal diagnostic finding, tech calls the customer Monday instead.

Problems Encountered

ps-encoded.sh fails on Howard-Home (iconv: command not found) — logged to errorlog; fell back to jq-encoded inline dispatch (scripts kept single-quote-only, so quote-stripping not a risk).
Env vars don't persist across Bash calls (known rule; logged --friction earlier in session) — re-auth or ./.rmm_token per call.

Configuration Changes

AD1 (Dataforth): MSP360 plan "Image2025" schedule 21:00 → 21:30 daily (via cbb.exe editBackupIBBPlan).
GND-SERVER (Grabb & Durando): plan "Backup plan on 4/16/2020 1:40:24 PM" (e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb) DailyFromMinutes 0 → 30 (every-2h runs now at :30); backup at C:\ProgramData\Online Backup\e25f4cc9-....cbb.bak-20260704; "Online Backup Service" restarted.
gururmm repo main: docs-only commit 5ad4498 (RMM_THOUGHTS.md Feature 13).
No parent-repo config changes.

Credentials & Secrets

None new. RMM auth via rmm-auth.sh (vault infrastructure/gururmm-server.sops.yaml); Syncro via Howard's baked-in skill key.

Infrastructure & Servers

MSP360 on endpoints: install C:\Program Files\Arizona Computer Guru\Online Backup\ (cbb.exe), data/logs/plans C:\ProgramData\Online Backup\, services "Online Backup Service" (+ Remote Management). Version 8.6.0.338.
MSP360 cloud maintenance windows observed: ~01:00 and ~04:00 UTC, ~10 min, ws.mspbackups.com returns 503.
LAB-SVR (Len's Auto): RMM agent id 8afb1430-c8f2-4fa2-8335-35b13d8a732f, last_seen 2026-06-18; ScreenConnect also offline.

Commands & Outputs

Recon/dispatch pattern: jq -n --arg cmd "$(cat script.ps1)" '{command_type:"powershell",command:$cmd,timeout_seconds:N}' → POST /api/agents/<id>/command (ps-encoded.sh broken on this machine).
Key log signature (both servers): MBS Maintenance: waiting 00:0x:xx, will keep retrying 04:00:00 then (503) Server Unavailable then Plan status: Success.
EVA disk: Get-PhysicalDisk shows only disk 0; System log disk source: hourly stream of IO operation ... LBA 0x0 Disk 1 hardware error.

Pending / Incomplete Tasks

Verify tonight/tomorrow: no backup_failed alerts from AD1 (~04:30 UTC) or GND-SERVER (01:30 UTC window) — if any persist, dig deeper.
Ticket #32502: call Gary Hartman Monday 2026-07-06 9:00 AM (reminder appt 5631486605 on calendar); on-site to identify/replace dead Disk 1.
LAB-SVR: awaiting Mike's answer (Discord) — replaced/decommissioned vs needs RMM+SC reinstall. If decommissioned: remove agent record + MSP360 endpoint or alerts continue.
Merge the fix branch (fix/easy-bugs-batch-2026-07-04) — being handled by the OTHER active session (it has additional uncommitted work: sanitize.rs, logs.rs, ws/mod.rs beyond commit f6c163b). Merge-to-main deploys.
RMM Thought Feature 13 — discuss → /shape-spec when ready.
Optional: exclusion for Office C2R phantom files on EVA's backup plan to silence the recurring "7 files" warning.

Reference Information

Syncro ticket #32502 (id 113490650): https://computerguru.syncromsp.com/tickets/113490650 — appt 5631486605 Mon 7/6 09:00.
gururmm commits: f6c163b (fix branch, server fixes + first-half session log), 5ad4498 (main, docs Feature 13).
Coord: todo fec70401-c122-482d-a173-ff236b9cf4a1, broadcast msg 38073821-61ce-49ee-91ed-727b967b5900.
First-half session log: projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md.
Discord: DM to Mike message_id 1523207214293323917; bot alerts in #dev-alerts (RMM dispatches) + #bot-alerts (Syncro #32502).

7.9 KiB Raw Blame History