From 9542de2b0a850394a3e4f8dceb250bc4890af3b5 Mon Sep 17 00:00:00 2001 From: Howard Enos Date: Sat, 4 Jul 2026 23:15:49 -0700 Subject: [PATCH] sync: auto-sync from HOWARD-HOME at 2026-07-04 23:15:20 Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-07-04 23:15:20 --- .gps-imc-listener2.json | 1 + errorlog.md | 6 ++ ...6-07-04-howard-backup-alert-remediation.md | 74 +++++++++++++++++++ 3 files changed, 81 insertions(+) create mode 100644 .gps-imc-listener2.json create mode 100644 session-logs/2026-07/2026-07-04-howard-backup-alert-remediation.md diff --git a/.gps-imc-listener2.json b/.gps-imc-listener2.json new file mode 100644 index 00000000..b2427244 --- /dev/null +++ b/.gps-imc-listener2.json @@ -0,0 +1 @@ +{"command_type": "powershell", "command": "New-NetFirewallRule -DisplayName GPSDIAG-TMP -Direction Inbound -Protocol TCP -LocalPort 8765 -Action Allow | Out-Null\n$l = New-Object System.Net.HttpListener\n$l.Prefixes.Add('http://+:8765/')\n$l.Start()\n$sw = [Diagnostics.Stopwatch]::StartNew()\n$got = @()\nwhile ($sw.Elapsed.TotalSeconds -lt 150 -and $got.Count -lt 3) {\n $ar = $l.BeginGetContext($null, $null)\n if ($ar.AsyncWaitHandle.WaitOne(5000)) {\n $ctx = $l.EndGetContext($ar)\n $got += ('[' + [int]$sw.Elapsed.TotalSeconds + 's] ' + $ctx.Request.Url.Query)\n $b = [Text.Encoding]::UTF8.GetBytes('ok')\n $ctx.Response.OutputStream.Write($b, 0, $b.Length)\n $ctx.Response.Close()\n }\n}\n$l.Stop()\nRemove-NetFirewallRule -DisplayName GPSDIAG-TMP | Out-Null\nif ($got.Count -eq 0) { Write-Output 'NO-CONTACT' } else { $got | ForEach-Object { [uri]::UnescapeDataString($_) } }", "timeout_seconds": 200} diff --git a/errorlog.md b/errorlog.md index 7e0be113..7fbe70a1 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-05 | Howard-Home | model-routing | [correction] ran all verification passes on Claude; correct is Ollama Tier-0 for cheap classify/prose/pre-scan passes per CLAUDE.md model routing [ctx: ref=CLAUDE.md-model-routing session=easy-bugs-batch] + +2026-07-05 | Howard-Home | rmm/ps-encoded | ps-encoded.sh failed: iconv not found on Howard-Home Git-Bash; fell back to jq-inline dispatch [ctx: machine=Howard-Home] + +2026-07-05 | Howard-Home | ps-encoded | encode produced empty output [ctx: src=./.msp360-recon.ps1] (x3) + 2026-07-05 | Howard-Home | gururmm-build | server cargo check failed (rc=101) [ctx: os=windows] 2026-07-05 | Howard-Home | bash/env | [friction] reused $TOKEN/$RMM across Bash calls after fresh shell reset - curl HTTP:000; fix: re-auth or token file per call [ctx: ref=CLAUDE.md-windows-env-persistence] diff --git a/session-logs/2026-07/2026-07-04-howard-backup-alert-remediation.md b/session-logs/2026-07/2026-07-04-howard-backup-alert-remediation.md new file mode 100644 index 00000000..1833db69 --- /dev/null +++ b/session-logs/2026-07/2026-07-04-howard-backup-alert-remediation.md @@ -0,0 +1,74 @@ +# 2026-07-04 — Backup false-alarm root cause + remediation (MSP360 / GuruRMM machine alerts) + +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Continuation of the GuruRMM alert-triage session (first half logged in `projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md` — server-side bug fixes on branch `fix/easy-bugs-batch-2026-07-04`, commit f6c163b). This half chased the MACHINE-side backup_failed alerts to root cause and remediated. + +Pulled MSP360 logs via /rmm from the three reachable machines (AD1/Dataforth, GND-SERVER/Grabb & Durando, DESKTOP-EVA4H1A/Gary Hartman; LAB-SVR/Len's Auto unreachable — RMM agent offline since 2026-06-18). MSP360 is installed as "Online Backup" (branded "Arizona Computer Guru"), data dir `C:\ProgramData\Online Backup`, per-plan GUID logs in `Logs\`. + +**Root cause of the backup_failed alert storm: none are real backup failures.** Plans that start exactly at a top-of-hour UTC time hit 503 Server Unavailable from `ws.mspbackups.com` (MSP360 cloud maintenance windows at ~01:00 and ~04:00 UTC, ~10 min). The client logs "MBS Maintenance: waiting ... will keep retrying", the status upload is recorded as a failed run in MSPBackups, GuruRMM's sync raises a critical backup_failed with "no error detail reported" — and the backup itself then completes Success. GND-SERVER's plan logged `Plan status: Success` on all 12 runs of 2026-07-04; only the 18:00 local (01:00 UTC) run hits the 503 — hence 42 nightly false criticals since 2026-06-17. AD1's Image2025 identical at 21:00 local (04:00 UTC). + +**Remediation applied (user-approved):** shifted both schedules off the top of the hour. AD1 `Image2025`: `cbb.exe editBackupIBBPlan -n 'Image2025' -every day -at '21:30'` — verified Hour=21 Min=30. GND-SERVER (every-2h intra-daily recurrence, not expressible in cbb CLI): stopped "Online Backup Service", edited plan XML `e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb` `DailyFromMinutes` 0→30 (backup copy left at `.bak-20260704`), restarted service — verified FromH=0 FromM=30 Period=120, ForceFullSchedule untouched, service Running. Runs now land at :30, clear of both windows. Tonight's runs confirm. + +**DESKTOP-EVA4H1A:** the alert's "no access to 7 files" is benign (Office Click-to-Run phantom `AppvIsvStream*.dll` in the VSS snapshot). Real finding: System event log shows "IO operation failed due to a hardware error" for **Disk 1 at LBA 0 every ~1 min**, and Disk 1 no longer enumerates in Get-PhysicalDisk (only the healthy ST1000DM003 system disk shows; SMART clean). Dead/dying attached-external drive. **Syncro ticket #32502** created (Hartman, assigned Howard, Hardware, do-not-email initial issue) + **Reminder appointment Monday 2026-07-06 09:00-09:30** to call the customer. https://computerguru.syncromsp.com/tickets/113490650 + +**LAB-SVR:** RMM agent AND ScreenConnect offline (user checked SC). Possibly replaced/decommissioned — **Discord DM sent to Mike** asking status; if in service it needs RMM agent + ScreenConnect reinstalled. Its backup alerts (same 01:02 UTC timing) are almost certainly the same false alarm. + +**Feature request filed** (/feature-request): RMM_THOUGHTS.md **Feature 13** — suppress/auto-resolve backup_failed alerts whose only failure is a maintenance-window status-upload error and the run later succeeds (delay-and-recheck), plus capture real failure detail in the MSPBackups sync. Committed docs-only to gururmm main (5ad4498) via a temporary worktree off origin/main — the fix branch working tree (another session actively working) was untouched. Coord todo fec70401 + fleet broadcast 38073821 sent. + +## Key Decisions + +- Schedule shift chosen over waiting for the GuruRMM alert-suppression feature — immediate stop to nightly false criticals; feature filed for the systemic fix. +- GND-SERVER edited via XML+service-restart because cbb CLI has no flags for intra-daily recurrence; AD1 via supported cbb CLI. Config backup left on the box before the XML edit. +- Docs commit for the feature request went through a temp `git worktree` off origin/main, NOT the checked-out fix branch — another session had new uncommitted work there (sanitize.rs etc.); switching/stashing would have disrupted it. +- Hartman ticket set do_not_email on the initial comment — internal diagnostic finding, tech calls the customer Monday instead. + +## Problems Encountered + +- `ps-encoded.sh` fails on Howard-Home (`iconv: command not found`) — logged to errorlog; fell back to jq-encoded inline dispatch (scripts kept single-quote-only, so quote-stripping not a risk). +- Env vars don't persist across Bash calls (known rule; logged --friction earlier in session) — re-auth or `./.rmm_token` per call. + +## Configuration Changes + +- **AD1 (Dataforth)**: MSP360 plan "Image2025" schedule 21:00 → **21:30 daily** (via cbb.exe editBackupIBBPlan). +- **GND-SERVER (Grabb & Durando)**: plan "Backup plan on 4/16/2020 1:40:24 PM" (`e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb`) `DailyFromMinutes` 0 → **30** (every-2h runs now at :30); backup at `C:\ProgramData\Online Backup\e25f4cc9-....cbb.bak-20260704`; "Online Backup Service" restarted. +- **gururmm repo main**: docs-only commit 5ad4498 (RMM_THOUGHTS.md Feature 13). +- No parent-repo config changes. + +## Credentials & Secrets + +None new. RMM auth via `rmm-auth.sh` (vault `infrastructure/gururmm-server.sops.yaml`); Syncro via Howard's baked-in skill key. + +## Infrastructure & Servers + +- MSP360 on endpoints: install `C:\Program Files\Arizona Computer Guru\Online Backup\` (cbb.exe), data/logs/plans `C:\ProgramData\Online Backup\`, services "Online Backup Service" (+ Remote Management). Version 8.6.0.338. +- MSP360 cloud maintenance windows observed: ~01:00 and ~04:00 UTC, ~10 min, `ws.mspbackups.com` returns 503. +- LAB-SVR (Len's Auto): RMM agent id 8afb1430-c8f2-4fa2-8335-35b13d8a732f, last_seen 2026-06-18; ScreenConnect also offline. + +## Commands & Outputs + +- Recon/dispatch pattern: `jq -n --arg cmd "$(cat script.ps1)" '{command_type:"powershell",command:$cmd,timeout_seconds:N}'` → POST `/api/agents//command` (ps-encoded.sh broken on this machine). +- Key log signature (both servers): `MBS Maintenance: waiting 00:0x:xx, will keep retrying 04:00:00` then `(503) Server Unavailable` then `Plan status: Success`. +- EVA disk: `Get-PhysicalDisk` shows only disk 0; System log `disk` source: hourly stream of `IO operation ... LBA 0x0 Disk 1 hardware error`. + +## Pending / Incomplete Tasks + +1. **Verify tonight/tomorrow**: no backup_failed alerts from AD1 (~04:30 UTC) or GND-SERVER (01:30 UTC window) — if any persist, dig deeper. +2. **Ticket #32502**: call Gary Hartman **Monday 2026-07-06 9:00 AM** (reminder appt 5631486605 on calendar); on-site to identify/replace dead Disk 1. +3. **LAB-SVR**: awaiting Mike's answer (Discord) — replaced/decommissioned vs needs RMM+SC reinstall. If decommissioned: remove agent record + MSP360 endpoint or alerts continue. +4. **Merge the fix branch** (`fix/easy-bugs-batch-2026-07-04`) — being handled by the OTHER active session (it has additional uncommitted work: sanitize.rs, logs.rs, ws/mod.rs beyond commit f6c163b). Merge-to-main deploys. +5. **RMM Thought Feature 13** — discuss → /shape-spec when ready. +6. Optional: exclusion for Office C2R phantom files on EVA's backup plan to silence the recurring "7 files" warning. + +## Reference Information + +- Syncro ticket #32502 (id 113490650): https://computerguru.syncromsp.com/tickets/113490650 — appt 5631486605 Mon 7/6 09:00. +- gururmm commits: f6c163b (fix branch, server fixes + first-half session log), 5ad4498 (main, docs Feature 13). +- Coord: todo fec70401-c122-482d-a173-ff236b9cf4a1, broadcast msg 38073821-61ce-49ee-91ed-727b967b5900. +- First-half session log: `projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md`. +- Discord: DM to Mike message_id 1523207214293323917; bot alerts in #dev-alerts (RMM dispatches) + #bot-alerts (Syncro #32502).