From 6149497ad156857d4c0b4a36313d9fddbb906c92 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Fri, 19 Jun 2026 15:53:30 -0700 Subject: [PATCH] sync: auto-sync from GURU-5070 at 2026-06-19 15:52:19 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-06-19 15:52:19 --- ...-06-19-mike-blaster2-onboarding-cleanup.md | 76 +++++++++++++++++++ errorlog.md | 4 + 2 files changed, 80 insertions(+) create mode 100644 clients/jimmy/session-logs/2026-06/2026-06-19-mike-blaster2-onboarding-cleanup.md diff --git a/clients/jimmy/session-logs/2026-06/2026-06-19-mike-blaster2-onboarding-cleanup.md b/clients/jimmy/session-logs/2026-06/2026-06-19-mike-blaster2-onboarding-cleanup.md new file mode 100644 index 00000000..8ed9e4fa --- /dev/null +++ b/clients/jimmy/session-logs/2026-06/2026-06-19-mike-blaster2-onboarding-cleanup.md @@ -0,0 +1,76 @@ +# 2026-06-19 — Jimmy Company / BLASTER2 Onboarding, Cleanup, Billing + +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Session Summary + +Onboarded a new client, Jimmy Company, to GuruRMM and remediated their workstation BLASTER2 end to end. Created the RMM client + site "Main" (site code SILVER-LION-5647), captured and vaulted the one-time enrollment key, then found an agent (BLASTER2) had already enrolled. Ran the full onboarding health/security diagnostic against it — grade RED (3 critical / 4 warning). + +Remediated the two critical security findings via RMM PowerShell: enabled RDP Network Level Authentication (`UserAuthentication=1`; RDP was on with NLA off) and fully removed a leftover Kaseya RMM agent left by a prior provider (service `KaseyaConnectAPIService`, install dir, and both registry hives incl. `WOW6432Node\Kaseya\Agent`; verified clean sweep). A requested cleanup/bloatware scan came back clean — C: had 70 GB free, negligible temp junk, no real bloatware (only a dead Google IE Toolbar). The only disk problem was the external backup drive E:. + +Diagnosed and partially remediated the MSP360 ("mspbackups") backup failure. E: (7.45 TB external) was full (0.74 GB free); two local plans (image + file) were failing `NotEnoughSpaceOnLocalDestination` while the cloud (Backblaze B2) plan was healthy. Applied 90-day retention to both local plans via the agent CLI (`cbb editBackupPlan` / `editBackupIBBPlan -purge "3m"`). Attempted to reclaim space but hit a hard blocker: agent-side deletion (`cbb delete`) is refused by an MSP360 provider policy ("File deletion on backup storage is restricted"), and the MBS REST API is monitoring-only. Enumerated exactly what to purge (20 image generations ≤2026-03-01, ~65 file generations ≤2026-03-17, ~1 TB orphaned pre-2024 legacy bunches) and handed the worklist to Mike via Discord DM for the MSP360 management console — the only place the purge can run. + +Closed out the engagement administratively: wrote a client remediation note to `clients/jimmy/reports/`, posted a customer-visible update to Syncro ticket #32442, and billed 1 hr remote ($150.00, invoice 1650742128, ticket marked Invoiced). + +## Key Decisions + +- **Removed Kaseya leftover rather than leaving it disabled** — it was a foreign RMM agent from a prior provider (control/security risk); user authorized removal. Cleaned service + dir + registry, not just the service. +- **Chose deleteIBB-by-date (90-day) for backup reclaim**, confirmed with the user, but it proved blocked by provider policy — pivoted to documenting a console worklist rather than forcing raw filesystem deletion (which would corrupt the MSP360 repository). +- **Did NOT raw-delete the orphaned legacy data** — MSP360 still lists the legacy bunches in its repository, so filesystem deletion would desync it; flagged for console handling. +- **Customer-visible ticket comment in plain language, no email** — softened the internal/technical detail (dropped provider-policy specifics), included the hardware-replacement recommendation; `do_not_email: true` to avoid an unexpected blast. +- **Billed against #32442** (the active ticket, created 2026-06-18) since no ticket was dated today; user confirmed. + +## Problems Encountered + +- **`/self-check`-style RMM auth in a pipeline subshell** — `eval "$(rmm-auth.sh)" | tail` ran auth in a subshell, so `$TOKEN`/`$RMM` never set in the parent → empty curl response. Fixed by running `eval` standalone (logged as friction earlier-session pattern). +- **MSP360 retention purge deadlock** — full disk blocks the backup pass that would trigger the purge; setting retention alone freed nothing. +- **`cbb delete` silently no-op'd** — returned no error but deleted nothing; raw output revealed "File deletion on backup storage is restricted due to your service provider policy." Root cause = MSP360 provider policy; reclaim must be console-side. +- **Vault file `client: null`** (earlier, onboarding) — sourcing an env file with `NAME=Jimmy Company` (space) ran the 2nd word as a command and left NAME unset; rebuilt the SOPS file with the value read via grep|cut. Logged as friction. +- **PowerShell inline `if` expression** in a hashtable value returned empty tables (PS 5.1 doesn't allow it); rewrote with a pre-assigned `$nstr` variable. + +## Configuration Changes + +Created: +- `clients/jimmy/onboarding-baselines/BLASTER2-20260619T191759.{json,md}` — onboarding diagnostic baseline (RED). +- `clients/jimmy/reports/2026-06-19-blaster2-remediation.md` — remediation record. +- Vault: `clients/jimmy/gururmm-site-main.sops.yaml` — RMM enrollment key. + +Endpoint changes on BLASTER2 (via RMM): +- RDP `UserAuthentication` 0 → 1 (NLA required). +- Removed Kaseya service + `C:\Program Files (x86)\Kaseya` + `HKLM\SOFTWARE\Kaseya` + `HKLM\SOFTWARE\WOW6432Node\Kaseya`. +- MSP360 local plans (image aae0be51, file 5277ed3c): retention set to 90 days. + +## Credentials & Secrets + +- **GuruRMM site enrollment** — vaulted at `clients/jimmy/gururmm-site-main.sops.yaml`: client_id `0f831728-579d-4160-b18d-a2d0422f88d1`, site_id `42a3d2e7-d9c6-464d-bd76-fc8cec673098`, site_code `SILVER-LION-5647`, api_key `grmm_...` (vaulted, round-trip verified). +- **MSP360 API** (existing) — `msp-tools/msp360-api.sops.yaml`, login `kY9PvDdWki`, base `https://api.mspbackups.com` (monitoring-only). +- MSP360 end-user account for Blaster2: `jimmyco333@gmail.com` (matches Syncro contact Jimmy Hughes). +- No new credentials created beyond the vaulted RMM key. + +## Infrastructure & Servers + +- **BLASTER2** — Windows 10 Pro 22H2 (build 19045, EOL 2025-10-14), Lenovo i5-3470 / 3.8 GB RAM / BIOS 2013, workgroup, IP 192.168.0.95. RMM agent `abddc0ce-a226-48f1-b913-263a81013389` (v0.6.66). LAN: E: = 7.45 TB external (Seagate Backup+ Hub). +- GuruRMM API: `http://172.16.3.30:3001`. MSP360: `https://api.mspbackups.com`. Syncro: `https://computerguru.syncromsp.com/api/v1`. + +## Commands & Outputs + +- `cbb editBackupIBBPlan -n "..." -purge "3m"` → "Retention time is set to 90 days." (after dropping `-keepLastVersion`, which is NBF-incompatible). +- `cbb delete -aid -b -g ` → "WARNING: File deletion on backup storage is restricted due to your service provider policy" (the blocker). +- MSP360 Local account: id `46deec2b-a3e9-4598-9a90-34bfd111ed6d`, repo-tracked 3.98 TB vs 6.73 TB on disk (~2.75 TB orphaned). +- Syncro: comment id `419904430` (customer-visible), line item `42944377`, invoice `1650742128` ($150.00), ticket → Invoiced. + +## Pending / Incomplete Tasks + +- **MSP360 console purge (Mike)** — lift the restrict-deletion policy and purge: 20 image gens ≤2026-03-01 (keep 6/7, 5/4, 4/6/2026), ~65 file gens ≤2026-03-17 (keep ≥3/23/2026), 2 orphaned legacy bunches ("Image Based" 793 GB, "C:" 216 GB). Optional: 722 GB non-MSP360 on E: (Veeam 543 / My backups 98 / FileHistory 81). Then re-run both local plans to confirm Success. Worklist DM'd. +- **BLASTER2 hardware** — Win10 EOL on 2012-era hardware; recommend replacement (told customer on #32442). Also: 5 pending Windows updates + pending reboot; 2 unexpected shutdowns/14d; verify BitLocker (likely unencrypted). +- Optional: remove the dead Google IE Toolbar. + +## Reference Information + +- RMM: client_id `0f831728-...`, site_id `42a3d2e7-...`, site SILVER-LION-5647. Install: https://rmm.azcomputerguru.com/install/SILVER-LION-5647 +- Syncro: customer 18560272 (Jimmy Company / Jimmy Hughes), ticket #32442 (id 112819046), invoice 1650742128. +- Baseline: `clients/jimmy/onboarding-baselines/BLASTER2-20260619T191759.md`. Remediation note: `clients/jimmy/reports/2026-06-19-blaster2-remediation.md`. +- errorlog: provider-policy delete blocker + source/space friction logged. diff --git a/errorlog.md b/errorlog.md index 42ea0924..a214bfc3 100644 --- a/errorlog.md +++ b/errorlog.md @@ -17,6 +17,10 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-06-19 | GURU-5070 | rmm/mspbackups cbb delete | cbb delete -g (generation purge) on Blaster2 Local destination is blocked: 'File deletion on backup storage is restricted due to your service provider policy'. Agent-side deletion of MSP360 backup data is disabled by the provider policy; MBS REST API (api.mspbackups.com) is monitoring-only (no plan/storage delete endpoints, probed 404). Reclaiming local backup space must be done in the MSP360 management console (lift the restrict-deletion policy and let 90-day retention purge, or delete old generations/legacy bunches there). 90-day retention WAS set successfully via cbb editBackupPlan/editBackupIBBPlan. [ctx: machine=GURU-5070 client=jimmy host=Blaster2] + +2026-06-19 | GURU-5070 | rmm/onboard vault | [friction] stashed onboard vars in a scratch .env and sourced it; NAME=Jimmy Company (unquoted space) made 'source' exec the 2nd word as a command and left NAME unset -> vault file written with client: null. Fix: quote values when writing the env (printf '%s=%q'), or read back with grep|cut not source. [ctx: machine=GURU-5070 client=jimmy] + 2026-06-19 | GURU-5070 | coord/self-check publish | [friction] coord-queue.jsonl queued a census with an MSYS-mangled URL path (/api/coord/... -> C:/Program Files/Git/api/coord/...) AND was git-tracked (not gitignored), so a stale RED census propagated to the repo and could clobber a published GREEN if drained. Fix: gitignore .claude/coord-queue.jsonl; the queue writer must prefix the curl path with the full coord_api base or set MSYS2_ARG_CONV_EXCL/MSYS_NO_PATHCONV to stop path conversion. [ctx: machine=GURU-5070 ref=CLAUDE.md-softfail-queue] 2026-06-19 | Howard-Home | unifi-wifi/gw-sitemanager | find subcommand crashed: GET /v1/hosts -> HTTP 500, then JSON decode traceback (no graceful handling of non-JSON error body) [ctx: client=khalsa cmd=find]