sync: auto-sync from HOWARD-HOME at 2026-07-14 19:13:30

Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-14 19:13:30

session-logs/2026-07/2026-07-14-howard-msp360-license-fix-triage.md

@@ -0,0 +1,79 @@
+# 2026-07-14 — MSP360 expired-license fix + backup triage worklist
+
+## User
+- **User:** Howard Enos (howard)
+- **Machine:** Howard-Home
+- **Role:** tech
+
+## Session Summary
+
+Started from a license-expired error on VWP-FILES ("The license (LicenseID: 0a120352-...) assigned to the computer has expired"). Diagnosis: the 2026-07-01 monthly renewal left 4 Server seats expired, stranding VWP-FILES + WINFileSvr (Valleywide), CS-SERVER (Cascades of Tucson), and DESKTOP-BMBTQLI (Len's Auto), all in automatic license mode. The Licensing API (Grant/Release/Revoke) is feature-gated off (HTTP 400), so the fix was console-side: Howard bought 4 Server licenses in the MSP360 console.
+
+Executed the swap on the 3 reachable machines: restart Online Backup service + force plan run via `cbb.exe plan -r` (service restart alone does NOT grab a new seat; the plan run does). All three verified backing up clean on new 8/1 seats (VWP-FILES 0.47GB, WINFileSvr 2.0GB, CS-SERVER 2.61GB). DESKTOP-BMBTQLI left to self-heal at next scheduled run (verify tomorrow via `msp360 licenses`).
+
+Merged the triage backlog from the parallel session (`2026-07-14-howard-msp360-backup-list-discord.md`) and worked it by priority: fixed the msp360 skill CLI (`--license-id` now required on release/revoke — API rejects without it), diagnosed Blaster2 disk-full (decision package for Mike), restarted the crashed backup service on Hartman DESKTOP-EVA4H1A and re-ran its plan, and Howard fixed FSG-SRV-02's schedule in console.
+
+Ran the stale-P2 sweep: root-caused IMC1's 628-day-stale Images plan (schedule `Enabled=false` in the plan XML — plan intact, Files/SQL plans fine), found LAB-SVR's RMM agent dead since 06-18 (SC service restarts did not reconnect it; Howard confirmed the box is offline — hands-on item), and confirmed GTI-INV-VMHOST, Tedards DESKTOP-SUFJR0J, TSAM-SERVER, and Desert RV DESKTOP-9T0UAON are unreachable by any remote path (not in RMM; SC absent or agent offline).
+
+## Key Decisions
+
+- License fix executed per Mike's "fix all"; Howard purchased 4 Server seats (console) because the Licensing API is feature-gated off.
+- Plan force-run (`cbb.exe plan -r "<name>"`), not just service restart, is the trigger that makes an auto-mode agent grab a fresh seat.
+- Monitoring feed lag pattern confirmed: a fresh LastStart with prior FAILED status + no error means a run is in flight — verify by re-poll, not by first read.
+- Blaster2 deletion/retention decision deferred to Mike at end of session (Howard: "we will pass that decision to mike when we are done").
+- No-plan clients ON HOLD per Howard: "we wont be deleting anyone just yet" — excluded from the Mike package.
+- IMC1 Images plan NOT silently re-enabled — 224GB job, deliberate-looking disable; flagged for decision (going over it next).
+- LAB-SVR marked needs-hands-on rather than chasing further blind SC commands.
+
+## Problems Encountered
+
+- `release-license --user-id` alone → HTTP 400 "LicenseID field is required" — fixed in CLI (`--license-id` now required for release/revoke); SKILL.md updated, errorlog'd.
+- Licensing API gate: POST /api/Licenses/* → HTTP 400 "not enabled for your account" — console-only; errorlog'd.
+- RMM dispatch: raw heredoc JSON with `\\` escapes rejected ("invalid escape") — build payloads with `jq -n --arg`.
+- RMM quote-stripping: bash `'...''...'` concatenation collapsed the PS single quotes → `C:\Program` not recognized; use bash double-quoted strings with embedded PS single quotes.
+- Service restart on stranded machines did not trigger license swap — force plan run did.
+- Foreground `sleep` blocked by harness — poll loops in background Bash instead.
+
+## Configuration Changes
+
+- `.claude/skills/msp360/scripts/msp360.py` (+ user copy `C:\Users\Howard\.claude\skills\msp360\scripts\msp360.py`): `--license-id` required for release-license/revoke-license.
+- `.claude/skills/msp360/SKILL.md` (+ user copy): usage line + gotchas (Licensing API gated, Release/Revoke need LicenseID).
+- `errorlog.md`: Licensing API gate entry.
+- Scratch (removed/ignorable): `./.lic.json`, `./.mon.json`, `./.bill.json`.
+
+## Credentials & Secrets
+
+- None new. MSP360 API creds from vault `msp-tools/msp360-api.sops.yaml`; RMM from `infrastructure/gururmm-server.sops.yaml`; SC from `msp-tools/screenconnect.sops.yaml`.
+
+## Infrastructure & Servers
+
+- New Server seats (bought 2026-07-14): VWP-FILES=fcc6c069..., WINFileSvr=5bf8d40c..., CS-SERVER=db511bb2... (all exp 2026-08-01); 4th reserved for DESKTOP-BMBTQLI self-heal.
+- Blaster2 (Jimmy Company) E: 7.45TB FULL — MSP360 repo `E:\MSPBackup\MBS-d94b286b-d22e-49c3-8bb1-415848f448f4` = 6,728.9 GB; reclaimable: `E:\VeeamBackup` 542.9 GB (newest file 3/29/2024), `E:\My backups` 98 GB, FileHistory 80.6 GB.
+- IMC1 (Instrumental Music, RMM id 88cbf7c0-abfa-4f12-846c-96274f718bff, online): plans Files dd1f1d8d / Images 2438483c (schedule Enabled=false, OnceDate 2020-03-18, last ran 2024-10-24) / SQL 2ede2a63 (Warning). Plan XMLs: `C:\ProgramData\Online Backup\<plan-guid>.cbb`.
+- LAB-SVR (Len's Auto): RMM agent 8afb1430-c8f2-4fa2-8335-35b13d8a732f last_seen 2026-06-18, v0.6.66; SC session 8d111067-7246-4c91-8027-d0fbb8c3a2de. Box offline per Howard.
+- Hartman DESKTOP-EVA4H1A (RMM 54ff7d66-8f3f-44a6-98a8-1419ffa78f6d): service crash mid-scan (net.pipe CBBPlanStatusMonitor gone); after restart the re-run was live at save time (log 2eff7410-70b5-468c-b17a-0440e3969283.log writing 20:55 local, deep repo sync).
+- FSG-SRV-02: Image plan OK (8.9GB); Files plan FAILED 2026-07-15T01:46 UTC 0.0B after Howard's console fix — re-check pending.
+- Unreachable stale machines: GTI-INV-VMHOST (no RMM/SC; SC has GTI-INV-DC/DC1/SQL only), DESKTOP-SUFJR0J (Tedards, none), TSAM-SERVER (SC a5845c99... agent offline since 2021 events), DESKTOP-9T0UAON (Desert RV, SC d2ecf021... offline).
+
+## Commands & Outputs
+
+- `py .claude/skills/msp360/scripts/msp360.py monitoring --company X` / `--stale 30` / `licenses --json` — primary reads.
+- RMM dispatch pattern: `PAYLOAD=$(jq -n --arg cmd "$SCRIPT" '{command_type:"powershell", command:$cmd, timeout_seconds:120}')` then POST `/api/agents/<id>/command`; PS paths in single quotes inside a bash double-quoted string.
+- cbb: `& 'C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe' plan -l` / `plan -r "<name>"`; logs `C:\ProgramData\Online Backup\Logs\<plan-guid>.log`.
+- Plan schedule check: `[xml]$x = Get-Content '<plan-guid>.cbb'; $x.DocumentElement.Schedule` → IMC1 Images `Enabled: false`.
+- SC blind restart: `sc.py send-command --session <id> --command "powershell ... Get-Service *guru*|Restart-Service" --confirm` → returns `{}`; verify by RMM last_seen poll (LAB-SVR: NOT reconnected).
+
+## Pending / Incomplete Tasks
+
+- Verify DESKTOP-BMBTQLI (Len's Auto) grabs the 4th new seat at next scheduled run (`msp360 licenses` tomorrow).
+- Re-check Hartman DESKTOP-EVA4H1A run completion (was in-flight at save).
+- Re-check FSG-SRV-02 Files plan (Howard fixed console-side; last poll FAILED 0.0B — may need log dig; box not in RMM, use SC).
+- IMC1 Images plan re-enable decision — being discussed next.
+- End-of-session Mike package: Blaster2 disk-full options (delete VeeamBackup 543GB / My backups 98GB / FileHistory 81GB + tighten MSP360 retention). No-plan clients EXCLUDED (on hold per Howard).
+- Hands-on list: LAB-SVR (offline), GTI-INV-VMHOST, Tedards DESKTOP-SUFJR0J, TSAM-SERVER, Desert RV DESKTOP-9T0UAON. Deferred per Howard: Air Pros, Tucson Coin.
+
+## Reference Information
+
+- Stranded-machine user/license IDs + status/error codes: see `session-logs/2026-07/2026-07-14-howard-msp360-backup-list-discord.md` (companion session).
+- MSP360 error codes: 1407 MBSLicense, 1506 dest-full, 1537 retention-without-full.
+- "Consistency check" plans stuck Running = UI artifact, ignore.