sync: auto-sync from GURU-5070 at 2026-06-19 15:52:19

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-19 15:52:19

clients/jimmy/session-logs/2026-06/2026-06-19-mike-blaster2-onboarding-cleanup.md

@@ -0,0 +1,76 @@
+# 2026-06-19 — Jimmy Company / BLASTER2 Onboarding, Cleanup, Billing
+
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Onboarded a new client, Jimmy Company, to GuruRMM and remediated their workstation BLASTER2 end to end. Created the RMM client + site "Main" (site code SILVER-LION-5647), captured and vaulted the one-time enrollment key, then found an agent (BLASTER2) had already enrolled. Ran the full onboarding health/security diagnostic against it — grade RED (3 critical / 4 warning).
+
+Remediated the two critical security findings via RMM PowerShell: enabled RDP Network Level Authentication (`UserAuthentication=1`; RDP was on with NLA off) and fully removed a leftover Kaseya RMM agent left by a prior provider (service `KaseyaConnectAPIService`, install dir, and both registry hives incl. `WOW6432Node\Kaseya\Agent`; verified clean sweep). A requested cleanup/bloatware scan came back clean — C: had 70 GB free, negligible temp junk, no real bloatware (only a dead Google IE Toolbar). The only disk problem was the external backup drive E:.
+
+Diagnosed and partially remediated the MSP360 ("mspbackups") backup failure. E: (7.45 TB external) was full (0.74 GB free); two local plans (image + file) were failing `NotEnoughSpaceOnLocalDestination` while the cloud (Backblaze B2) plan was healthy. Applied 90-day retention to both local plans via the agent CLI (`cbb editBackupPlan` / `editBackupIBBPlan -purge "3m"`). Attempted to reclaim space but hit a hard blocker: agent-side deletion (`cbb delete`) is refused by an MSP360 provider policy ("File deletion on backup storage is restricted"), and the MBS REST API is monitoring-only. Enumerated exactly what to purge (20 image generations ≤2026-03-01, ~65 file generations ≤2026-03-17, ~1 TB orphaned pre-2024 legacy bunches) and handed the worklist to Mike via Discord DM for the MSP360 management console — the only place the purge can run.
+
+Closed out the engagement administratively: wrote a client remediation note to `clients/jimmy/reports/`, posted a customer-visible update to Syncro ticket #32442, and billed 1 hr remote ($150.00, invoice 1650742128, ticket marked Invoiced).
+
+## Key Decisions
+
+- **Removed Kaseya leftover rather than leaving it disabled** — it was a foreign RMM agent from a prior provider (control/security risk); user authorized removal. Cleaned service + dir + registry, not just the service.
+- **Chose deleteIBB-by-date (90-day) for backup reclaim**, confirmed with the user, but it proved blocked by provider policy — pivoted to documenting a console worklist rather than forcing raw filesystem deletion (which would corrupt the MSP360 repository).
+- **Did NOT raw-delete the orphaned legacy data** — MSP360 still lists the legacy bunches in its repository, so filesystem deletion would desync it; flagged for console handling.
+- **Customer-visible ticket comment in plain language, no email** — softened the internal/technical detail (dropped provider-policy specifics), included the hardware-replacement recommendation; `do_not_email: true` to avoid an unexpected blast.
+- **Billed against #32442** (the active ticket, created 2026-06-18) since no ticket was dated today; user confirmed.
+
+## Problems Encountered
+
+- **`/self-check`-style RMM auth in a pipeline subshell** — `eval "$(rmm-auth.sh)" | tail` ran auth in a subshell, so `$TOKEN`/`$RMM` never set in the parent → empty curl response. Fixed by running `eval` standalone (logged as friction earlier-session pattern).
+- **MSP360 retention purge deadlock** — full disk blocks the backup pass that would trigger the purge; setting retention alone freed nothing.
+- **`cbb delete` silently no-op'd** — returned no error but deleted nothing; raw output revealed "File deletion on backup storage is restricted due to your service provider policy." Root cause = MSP360 provider policy; reclaim must be console-side.
+- **Vault file `client: null`** (earlier, onboarding) — sourcing an env file with `NAME=Jimmy Company` (space) ran the 2nd word as a command and left NAME unset; rebuilt the SOPS file with the value read via grep|cut. Logged as friction.
+- **PowerShell inline `if` expression** in a hashtable value returned empty tables (PS 5.1 doesn't allow it); rewrote with a pre-assigned `$nstr` variable.
+
+## Configuration Changes
+
+Created:
+- `clients/jimmy/onboarding-baselines/BLASTER2-20260619T191759.{json,md}` — onboarding diagnostic baseline (RED).
+- `clients/jimmy/reports/2026-06-19-blaster2-remediation.md` — remediation record.
+- Vault: `clients/jimmy/gururmm-site-main.sops.yaml` — RMM enrollment key.
+
+Endpoint changes on BLASTER2 (via RMM):
+- RDP `UserAuthentication` 0 → 1 (NLA required).
+- Removed Kaseya service + `C:\Program Files (x86)\Kaseya` + `HKLM\SOFTWARE\Kaseya` + `HKLM\SOFTWARE\WOW6432Node\Kaseya`.
+- MSP360 local plans (image aae0be51, file 5277ed3c): retention set to 90 days.
+
+## Credentials & Secrets
+
+- **GuruRMM site enrollment** — vaulted at `clients/jimmy/gururmm-site-main.sops.yaml`: client_id `0f831728-579d-4160-b18d-a2d0422f88d1`, site_id `42a3d2e7-d9c6-464d-bd76-fc8cec673098`, site_code `SILVER-LION-5647`, api_key `grmm_...` (vaulted, round-trip verified).
+- **MSP360 API** (existing) — `msp-tools/msp360-api.sops.yaml`, login `kY9PvDdWki`, base `https://api.mspbackups.com` (monitoring-only).
+- MSP360 end-user account for Blaster2: `jimmyco333@gmail.com` (matches Syncro contact Jimmy Hughes).
+- No new credentials created beyond the vaulted RMM key.
+
+## Infrastructure & Servers
+
+- **BLASTER2** — Windows 10 Pro 22H2 (build 19045, EOL 2025-10-14), Lenovo i5-3470 / 3.8 GB RAM / BIOS 2013, workgroup, IP 192.168.0.95. RMM agent `abddc0ce-a226-48f1-b913-263a81013389` (v0.6.66). LAN: E: = 7.45 TB external (Seagate Backup+ Hub).
+- GuruRMM API: `http://172.16.3.30:3001`. MSP360: `https://api.mspbackups.com`. Syncro: `https://computerguru.syncromsp.com/api/v1`.
+
+## Commands & Outputs
+
+- `cbb editBackupIBBPlan -n "..." -purge "3m"` → "Retention time is set to 90 days." (after dropping `-keepLastVersion`, which is NBF-incompatible).
+- `cbb delete -aid <local> -b <bunch> -g <genID>` → "WARNING: File deletion on backup storage is restricted due to your service provider policy" (the blocker).
+- MSP360 Local account: id `46deec2b-a3e9-4598-9a90-34bfd111ed6d`, repo-tracked 3.98 TB vs 6.73 TB on disk (~2.75 TB orphaned).
+- Syncro: comment id `419904430` (customer-visible), line item `42944377`, invoice `1650742128` ($150.00), ticket → Invoiced.
+
+## Pending / Incomplete Tasks
+
+- **MSP360 console purge (Mike)** — lift the restrict-deletion policy and purge: 20 image gens ≤2026-03-01 (keep 6/7, 5/4, 4/6/2026), ~65 file gens ≤2026-03-17 (keep ≥3/23/2026), 2 orphaned legacy bunches ("Image Based" 793 GB, "C:" 216 GB). Optional: 722 GB non-MSP360 on E: (Veeam 543 / My backups 98 / FileHistory 81). Then re-run both local plans to confirm Success. Worklist DM'd.
+- **BLASTER2 hardware** — Win10 EOL on 2012-era hardware; recommend replacement (told customer on #32442). Also: 5 pending Windows updates + pending reboot; 2 unexpected shutdowns/14d; verify BitLocker (likely unencrypted).
+- Optional: remove the dead Google IE Toolbar.
+
+## Reference Information
+
+- RMM: client_id `0f831728-...`, site_id `42a3d2e7-...`, site SILVER-LION-5647. Install: https://rmm.azcomputerguru.com/install/SILVER-LION-5647
+- Syncro: customer 18560272 (Jimmy Company / Jimmy Hughes), ticket #32442 (id 112819046), invoice 1650742128.
+- Baseline: `clients/jimmy/onboarding-baselines/BLASTER2-20260619T191759.md`. Remediation note: `clients/jimmy/reports/2026-06-19-blaster2-remediation.md`.
+- errorlog: provider-policy delete blocker + source/space friction logged.

errorlog.md

@@ -17,6 +17,10 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
 
 <!-- Append entries below this line -->
 
+2026-06-19 | GURU-5070 | rmm/mspbackups cbb delete | cbb delete -g (generation purge) on Blaster2 Local destination is blocked: 'File deletion on backup storage is restricted due to your service provider policy'. Agent-side deletion of MSP360 backup data is disabled by the provider policy; MBS REST API (api.mspbackups.com) is monitoring-only (no plan/storage delete endpoints, probed 404). Reclaiming local backup space must be done in the MSP360 management console (lift the restrict-deletion policy and let 90-day retention purge, or delete old generations/legacy bunches there). 90-day retention WAS set successfully via cbb editBackupPlan/editBackupIBBPlan. [ctx: machine=GURU-5070 client=jimmy host=Blaster2]
+
+2026-06-19 | GURU-5070 | rmm/onboard vault | [friction] stashed onboard vars in a scratch .env and sourced it; NAME=Jimmy Company (unquoted space) made 'source' exec the 2nd word as a command and left NAME unset -> vault file written with client: null. Fix: quote values when writing the env (printf '%s=%q'), or read back with grep|cut not source. [ctx: machine=GURU-5070 client=jimmy]
+
 2026-06-19 | GURU-5070 | coord/self-check publish | [friction] coord-queue.jsonl queued a census with an MSYS-mangled URL path (/api/coord/... -> C:/Program Files/Git/api/coord/...) AND was git-tracked (not gitignored), so a stale RED census propagated to the repo and could clobber a published GREEN if drained. Fix: gitignore .claude/coord-queue.jsonl; the queue writer must prefix the curl path with the full coord_api base or set MSYS2_ARG_CONV_EXCL/MSYS_NO_PATHCONV to stop path conversion. [ctx: machine=GURU-5070 ref=CLAUDE.md-softfail-queue]
 2026-06-19 | Howard-Home | unifi-wifi/gw-sitemanager | find subcommand crashed: GET /v1/hosts -> HTTP 500, then JSON decode traceback (no graceful handling of non-JSON error body) [ctx: client=khalsa cmd=find]