Files

Howard Enos 941de499c2 sync: auto-sync from HOWARD-HOME at 2026-07-14 19:13:30

Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-14 19:13:30

2026-07-14 19:14:01 -07:00

7.6 KiB

Raw Blame History

2026-07-14 — MSP360 expired-license fix + backup triage worklist

User

User: Howard Enos (howard)
Machine: Howard-Home
Role: tech

Session Summary

Started from a license-expired error on VWP-FILES ("The license (LicenseID: 0a120352-...) assigned to the computer has expired"). Diagnosis: the 2026-07-01 monthly renewal left 4 Server seats expired, stranding VWP-FILES + WINFileSvr (Valleywide), CS-SERVER (Cascades of Tucson), and DESKTOP-BMBTQLI (Len's Auto), all in automatic license mode. The Licensing API (Grant/Release/Revoke) is feature-gated off (HTTP 400), so the fix was console-side: Howard bought 4 Server licenses in the MSP360 console.

Executed the swap on the 3 reachable machines: restart Online Backup service + force plan run via cbb.exe plan -r (service restart alone does NOT grab a new seat; the plan run does). All three verified backing up clean on new 8/1 seats (VWP-FILES 0.47GB, WINFileSvr 2.0GB, CS-SERVER 2.61GB). DESKTOP-BMBTQLI left to self-heal at next scheduled run (verify tomorrow via msp360 licenses).

Merged the triage backlog from the parallel session (2026-07-14-howard-msp360-backup-list-discord.md) and worked it by priority: fixed the msp360 skill CLI (--license-id now required on release/revoke — API rejects without it), diagnosed Blaster2 disk-full (decision package for Mike), restarted the crashed backup service on Hartman DESKTOP-EVA4H1A and re-ran its plan, and Howard fixed FSG-SRV-02's schedule in console.

Ran the stale-P2 sweep: root-caused IMC1's 628-day-stale Images plan (schedule Enabled=false in the plan XML — plan intact, Files/SQL plans fine), found LAB-SVR's RMM agent dead since 06-18 (SC service restarts did not reconnect it; Howard confirmed the box is offline — hands-on item), and confirmed GTI-INV-VMHOST, Tedards DESKTOP-SUFJR0J, TSAM-SERVER, and Desert RV DESKTOP-9T0UAON are unreachable by any remote path (not in RMM; SC absent or agent offline).

Key Decisions

License fix executed per Mike's "fix all"; Howard purchased 4 Server seats (console) because the Licensing API is feature-gated off.
Plan force-run (cbb.exe plan -r "<name>"), not just service restart, is the trigger that makes an auto-mode agent grab a fresh seat.
Monitoring feed lag pattern confirmed: a fresh LastStart with prior FAILED status + no error means a run is in flight — verify by re-poll, not by first read.
Blaster2 deletion/retention decision deferred to Mike at end of session (Howard: "we will pass that decision to mike when we are done").
No-plan clients ON HOLD per Howard: "we wont be deleting anyone just yet" — excluded from the Mike package.
IMC1 Images plan NOT silently re-enabled — 224GB job, deliberate-looking disable; flagged for decision (going over it next).
LAB-SVR marked needs-hands-on rather than chasing further blind SC commands.

Problems Encountered

release-license --user-id alone → HTTP 400 "LicenseID field is required" — fixed in CLI (--license-id now required for release/revoke); SKILL.md updated, errorlog'd.
Licensing API gate: POST /api/Licenses/* → HTTP 400 "not enabled for your account" — console-only; errorlog'd.
RMM dispatch: raw heredoc JSON with \\ escapes rejected ("invalid escape") — build payloads with jq -n --arg.
RMM quote-stripping: bash '...''...' concatenation collapsed the PS single quotes → C:\Program not recognized; use bash double-quoted strings with embedded PS single quotes.
Service restart on stranded machines did not trigger license swap — force plan run did.
Foreground sleep blocked by harness — poll loops in background Bash instead.

Configuration Changes

.claude/skills/msp360/scripts/msp360.py (+ user copy C:\Users\Howard\.claude\skills\msp360\scripts\msp360.py): --license-id required for release-license/revoke-license.
.claude/skills/msp360/SKILL.md (+ user copy): usage line + gotchas (Licensing API gated, Release/Revoke need LicenseID).
errorlog.md: Licensing API gate entry.
Scratch (removed/ignorable): ./.lic.json, ./.mon.json, ./.bill.json.

Credentials & Secrets

None new. MSP360 API creds from vault msp-tools/msp360-api.sops.yaml; RMM from infrastructure/gururmm-server.sops.yaml; SC from msp-tools/screenconnect.sops.yaml.

Infrastructure & Servers

New Server seats (bought 2026-07-14): VWP-FILES=fcc6c069..., WINFileSvr=5bf8d40c..., CS-SERVER=db511bb2... (all exp 2026-08-01); 4th reserved for DESKTOP-BMBTQLI self-heal.
Blaster2 (Jimmy Company) E: 7.45TB FULL — MSP360 repo E:\MSPBackup\MBS-d94b286b-d22e-49c3-8bb1-415848f448f4 = 6,728.9 GB; reclaimable: E:\VeeamBackup 542.9 GB (newest file 3/29/2024), E:\My backups 98 GB, FileHistory 80.6 GB.
IMC1 (Instrumental Music, RMM id 88cbf7c0-abfa-4f12-846c-96274f718bff, online): plans Files dd1f1d8d / Images 2438483c (schedule Enabled=false, OnceDate 2020-03-18, last ran 2024-10-24) / SQL 2ede2a63 (Warning). Plan XMLs: C:\ProgramData\Online Backup\<plan-guid>.cbb.
LAB-SVR (Len's Auto): RMM agent 8afb1430-c8f2-4fa2-8335-35b13d8a732f last_seen 2026-06-18, v0.6.66; SC session 8d111067-7246-4c91-8027-d0fbb8c3a2de. Box offline per Howard.
Hartman DESKTOP-EVA4H1A (RMM 54ff7d66-8f3f-44a6-98a8-1419ffa78f6d): service crash mid-scan (net.pipe CBBPlanStatusMonitor gone); after restart the re-run was live at save time (log 2eff7410-70b5-468c-b17a-0440e3969283.log writing 20:55 local, deep repo sync).
FSG-SRV-02: Image plan OK (8.9GB); Files plan FAILED 2026-07-15T01:46 UTC 0.0B after Howard's console fix — re-check pending.
Unreachable stale machines: GTI-INV-VMHOST (no RMM/SC; SC has GTI-INV-DC/DC1/SQL only), DESKTOP-SUFJR0J (Tedards, none), TSAM-SERVER (SC a5845c99... agent offline since 2021 events), DESKTOP-9T0UAON (Desert RV, SC d2ecf021... offline).

Commands & Outputs

py .claude/skills/msp360/scripts/msp360.py monitoring --company X / --stale 30 / licenses --json — primary reads.
RMM dispatch pattern: PAYLOAD=$(jq -n --arg cmd "$SCRIPT" '{command_type:"powershell", command:$cmd, timeout_seconds:120}') then POST /api/agents/<id>/command; PS paths in single quotes inside a bash double-quoted string.
cbb: & 'C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe' plan -l / plan -r "<name>"; logs C:\ProgramData\Online Backup\Logs\<plan-guid>.log.
Plan schedule check: [xml]$x = Get-Content '<plan-guid>.cbb'; $x.DocumentElement.Schedule → IMC1 Images Enabled: false.
SC blind restart: sc.py send-command --session <id> --command "powershell ... Get-Service *guru*|Restart-Service" --confirm → returns {}; verify by RMM last_seen poll (LAB-SVR: NOT reconnected).

Pending / Incomplete Tasks

Verify DESKTOP-BMBTQLI (Len's Auto) grabs the 4th new seat at next scheduled run (msp360 licenses tomorrow).
Re-check Hartman DESKTOP-EVA4H1A run completion (was in-flight at save).
Re-check FSG-SRV-02 Files plan (Howard fixed console-side; last poll FAILED 0.0B — may need log dig; box not in RMM, use SC).
IMC1 Images plan re-enable decision — being discussed next.
End-of-session Mike package: Blaster2 disk-full options (delete VeeamBackup 543GB / My backups 98GB / FileHistory 81GB + tighten MSP360 retention). No-plan clients EXCLUDED (on hold per Howard).
Hands-on list: LAB-SVR (offline), GTI-INV-VMHOST, Tedards DESKTOP-SUFJR0J, TSAM-SERVER, Desert RV DESKTOP-9T0UAON. Deferred per Howard: Air Pros, Tucson Coin.

Reference Information

Stranded-machine user/license IDs + status/error codes: see session-logs/2026-07/2026-07-14-howard-msp360-backup-list-discord.md (companion session).
MSP360 error codes: 1407 MBSLicense, 1506 dest-full, 1537 retention-without-full.
"Consistency check" plans stuck Running = UI artifact, ignore.

7.6 KiB Raw Blame History