sync: auto-sync from HOWARD-HOME at 2026-07-04 23:15:20

Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-04 23:15:20

.gps-imc-listener2.json

@@ -0,0 +1 @@
+{"command_type": "powershell", "command": "New-NetFirewallRule -DisplayName GPSDIAG-TMP -Direction Inbound -Protocol TCP -LocalPort 8765 -Action Allow | Out-Null\n$l = New-Object System.Net.HttpListener\n$l.Prefixes.Add('http://+:8765/')\n$l.Start()\n$sw = [Diagnostics.Stopwatch]::StartNew()\n$got = @()\nwhile ($sw.Elapsed.TotalSeconds -lt 150 -and $got.Count -lt 3) {\n  $ar = $l.BeginGetContext($null, $null)\n  if ($ar.AsyncWaitHandle.WaitOne(5000)) {\n    $ctx = $l.EndGetContext($ar)\n    $got += ('[' + [int]$sw.Elapsed.TotalSeconds + 's] ' + $ctx.Request.Url.Query)\n    $b = [Text.Encoding]::UTF8.GetBytes('ok')\n    $ctx.Response.OutputStream.Write($b, 0, $b.Length)\n    $ctx.Response.Close()\n  }\n}\n$l.Stop()\nRemove-NetFirewallRule -DisplayName GPSDIAG-TMP | Out-Null\nif ($got.Count -eq 0) { Write-Output 'NO-CONTACT' } else { $got | ForEach-Object { [uri]::UnescapeDataString($_) } }", "timeout_seconds": 200}

errorlog.md

@@ -19,6 +19,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
 
 <!-- Append entries below this line -->
 
+2026-07-05 | Howard-Home | model-routing | [correction] ran all verification passes on Claude; correct is Ollama Tier-0 for cheap classify/prose/pre-scan passes per CLAUDE.md model routing [ctx: ref=CLAUDE.md-model-routing session=easy-bugs-batch]
+
+2026-07-05 | Howard-Home | rmm/ps-encoded | ps-encoded.sh failed: iconv not found on Howard-Home Git-Bash; fell back to jq-inline dispatch [ctx: machine=Howard-Home]
+
+2026-07-05 | Howard-Home | ps-encoded | encode produced empty output [ctx: src=./.msp360-recon.ps1] (x3)
+
 2026-07-05 | Howard-Home | gururmm-build | server cargo check failed (rc=101) [ctx: os=windows]
 
 2026-07-05 | Howard-Home | bash/env | [friction] reused $TOKEN/$RMM across Bash calls after fresh shell reset - curl HTTP:000; fix: re-auth or token file per call [ctx: ref=CLAUDE.md-windows-env-persistence]

session-logs/2026-07/2026-07-04-howard-backup-alert-remediation.md

@@ -0,0 +1,74 @@
+# 2026-07-04 — Backup false-alarm root cause + remediation (MSP360 / GuruRMM machine alerts)
+
+## User
+- **User:** Howard Enos (howard)
+- **Machine:** Howard-Home
+- **Role:** tech
+
+## Session Summary
+
+Continuation of the GuruRMM alert-triage session (first half logged in `projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md` — server-side bug fixes on branch `fix/easy-bugs-batch-2026-07-04`, commit f6c163b). This half chased the MACHINE-side backup_failed alerts to root cause and remediated.
+
+Pulled MSP360 logs via /rmm from the three reachable machines (AD1/Dataforth, GND-SERVER/Grabb & Durando, DESKTOP-EVA4H1A/Gary Hartman; LAB-SVR/Len's Auto unreachable — RMM agent offline since 2026-06-18). MSP360 is installed as "Online Backup" (branded "Arizona Computer Guru"), data dir `C:\ProgramData\Online Backup`, per-plan GUID logs in `Logs\`.
+
+**Root cause of the backup_failed alert storm: none are real backup failures.** Plans that start exactly at a top-of-hour UTC time hit 503 Server Unavailable from `ws.mspbackups.com` (MSP360 cloud maintenance windows at ~01:00 and ~04:00 UTC, ~10 min). The client logs "MBS Maintenance: waiting ... will keep retrying", the status upload is recorded as a failed run in MSPBackups, GuruRMM's sync raises a critical backup_failed with "no error detail reported" — and the backup itself then completes Success. GND-SERVER's plan logged `Plan status: Success` on all 12 runs of 2026-07-04; only the 18:00 local (01:00 UTC) run hits the 503 — hence 42 nightly false criticals since 2026-06-17. AD1's Image2025 identical at 21:00 local (04:00 UTC).
+
+**Remediation applied (user-approved):** shifted both schedules off the top of the hour. AD1 `Image2025`: `cbb.exe editBackupIBBPlan -n 'Image2025' -every day -at '21:30'` — verified Hour=21 Min=30. GND-SERVER (every-2h intra-daily recurrence, not expressible in cbb CLI): stopped "Online Backup Service", edited plan XML `e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb` `DailyFromMinutes` 0→30 (backup copy left at `<file>.bak-20260704`), restarted service — verified FromH=0 FromM=30 Period=120, ForceFullSchedule untouched, service Running. Runs now land at :30, clear of both windows. Tonight's runs confirm.
+
+**DESKTOP-EVA4H1A:** the alert's "no access to 7 files" is benign (Office Click-to-Run phantom `AppvIsvStream*.dll` in the VSS snapshot). Real finding: System event log shows "IO operation failed due to a hardware error" for **Disk 1 at LBA 0 every ~1 min**, and Disk 1 no longer enumerates in Get-PhysicalDisk (only the healthy ST1000DM003 system disk shows; SMART clean). Dead/dying attached-external drive. **Syncro ticket #32502** created (Hartman, assigned Howard, Hardware, do-not-email initial issue) + **Reminder appointment Monday 2026-07-06 09:00-09:30** to call the customer. https://computerguru.syncromsp.com/tickets/113490650
+
+**LAB-SVR:** RMM agent AND ScreenConnect offline (user checked SC). Possibly replaced/decommissioned — **Discord DM sent to Mike** asking status; if in service it needs RMM agent + ScreenConnect reinstalled. Its backup alerts (same 01:02 UTC timing) are almost certainly the same false alarm.
+
+**Feature request filed** (/feature-request): RMM_THOUGHTS.md **Feature 13** — suppress/auto-resolve backup_failed alerts whose only failure is a maintenance-window status-upload error and the run later succeeds (delay-and-recheck), plus capture real failure detail in the MSPBackups sync. Committed docs-only to gururmm main (5ad4498) via a temporary worktree off origin/main — the fix branch working tree (another session actively working) was untouched. Coord todo fec70401 + fleet broadcast 38073821 sent.
+
+## Key Decisions
+
+- Schedule shift chosen over waiting for the GuruRMM alert-suppression feature — immediate stop to nightly false criticals; feature filed for the systemic fix.
+- GND-SERVER edited via XML+service-restart because cbb CLI has no flags for intra-daily recurrence; AD1 via supported cbb CLI. Config backup left on the box before the XML edit.
+- Docs commit for the feature request went through a temp `git worktree` off origin/main, NOT the checked-out fix branch — another session had new uncommitted work there (sanitize.rs etc.); switching/stashing would have disrupted it.
+- Hartman ticket set do_not_email on the initial comment — internal diagnostic finding, tech calls the customer Monday instead.
+
+## Problems Encountered
+
+- `ps-encoded.sh` fails on Howard-Home (`iconv: command not found`) — logged to errorlog; fell back to jq-encoded inline dispatch (scripts kept single-quote-only, so quote-stripping not a risk).
+- Env vars don't persist across Bash calls (known rule; logged --friction earlier in session) — re-auth or `./.rmm_token` per call.
+
+## Configuration Changes
+
+- **AD1 (Dataforth)**: MSP360 plan "Image2025" schedule 21:00 → **21:30 daily** (via cbb.exe editBackupIBBPlan).
+- **GND-SERVER (Grabb & Durando)**: plan "Backup plan on 4/16/2020 1:40:24 PM" (`e25f4cc9-7ed9-4010-b5e2-4684611fd794.cbb`) `DailyFromMinutes` 0 → **30** (every-2h runs now at :30); backup at `C:\ProgramData\Online Backup\e25f4cc9-....cbb.bak-20260704`; "Online Backup Service" restarted.
+- **gururmm repo main**: docs-only commit 5ad4498 (RMM_THOUGHTS.md Feature 13).
+- No parent-repo config changes.
+
+## Credentials & Secrets
+
+None new. RMM auth via `rmm-auth.sh` (vault `infrastructure/gururmm-server.sops.yaml`); Syncro via Howard's baked-in skill key.
+
+## Infrastructure & Servers
+
+- MSP360 on endpoints: install `C:\Program Files\Arizona Computer Guru\Online Backup\` (cbb.exe), data/logs/plans `C:\ProgramData\Online Backup\`, services "Online Backup Service" (+ Remote Management). Version 8.6.0.338.
+- MSP360 cloud maintenance windows observed: ~01:00 and ~04:00 UTC, ~10 min, `ws.mspbackups.com` returns 503.
+- LAB-SVR (Len's Auto): RMM agent id 8afb1430-c8f2-4fa2-8335-35b13d8a732f, last_seen 2026-06-18; ScreenConnect also offline.
+
+## Commands & Outputs
+
+- Recon/dispatch pattern: `jq -n --arg cmd "$(cat script.ps1)" '{command_type:"powershell",command:$cmd,timeout_seconds:N}'` → POST `/api/agents/<id>/command` (ps-encoded.sh broken on this machine).
+- Key log signature (both servers): `MBS Maintenance: waiting 00:0x:xx, will keep retrying 04:00:00` then `(503) Server Unavailable` then `Plan status: Success`.
+- EVA disk: `Get-PhysicalDisk` shows only disk 0; System log `disk` source: hourly stream of `IO operation ... LBA 0x0 Disk 1 hardware error`.
+
+## Pending / Incomplete Tasks
+
+1. **Verify tonight/tomorrow**: no backup_failed alerts from AD1 (~04:30 UTC) or GND-SERVER (01:30 UTC window) — if any persist, dig deeper.
+2. **Ticket #32502**: call Gary Hartman **Monday 2026-07-06 9:00 AM** (reminder appt 5631486605 on calendar); on-site to identify/replace dead Disk 1.
+3. **LAB-SVR**: awaiting Mike's answer (Discord) — replaced/decommissioned vs needs RMM+SC reinstall. If decommissioned: remove agent record + MSP360 endpoint or alerts continue.
+4. **Merge the fix branch** (`fix/easy-bugs-batch-2026-07-04`) — being handled by the OTHER active session (it has additional uncommitted work: sanitize.rs, logs.rs, ws/mod.rs beyond commit f6c163b). Merge-to-main deploys.
+5. **RMM Thought Feature 13** — discuss → /shape-spec when ready.
+6. Optional: exclusion for Office C2R phantom files on EVA's backup plan to silence the recurring "7 files" warning.
+
+## Reference Information
+
+- Syncro ticket #32502 (id 113490650): https://computerguru.syncromsp.com/tickets/113490650 — appt 5631486605 Mon 7/6 09:00.
+- gururmm commits: f6c163b (fix branch, server fixes + first-half session log), 5ad4498 (main, docs Feature 13).
+- Coord: todo fec70401-c122-482d-a173-ff236b9cf4a1, broadcast msg 38073821-61ce-49ee-91ed-727b967b5900.
+- First-half session log: `projects/msp-tools/guru-rmm/session-logs/2026-07/2026-07-04-howard-alert-triage-server-fixes.md`.
+- Discord: DM to Mike message_id 1523207214293323917; bot alerts in #dev-alerts (RMM dispatches) + #bot-alerts (Syncro #32502).