claudetools/session-logs/2026-06-01-session.md

# Session Log — 2026-06-01

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Three distinct work streams ran this session: Jupiter Unraid container templating, a GURU-KALI ghost-agent investigation that turned into a GuruRMM bug-tracking cleanup, and an EZ Fast Auto Glass email-deliverability investigation.

**Jupiter Docker → Unraid templating.** Inspected all 21 containers on Jupiter (172.16.3.20). 14 were created via raw `docker run`/`docker-compose` and lack the `net.unraid.docker.*` labels that drive the Unraid UI (WebUI button, icon, update-check, Edit form). Because those labels are immutable, the fix requires recreating each container. Executed target #1 (lowest risk): wrote an Unraid template for `gururmm-agent`, recreated it with `net.unraid.docker.managed=dockerman` and the faithful config; verified it re-authenticated and resumed reporting. Then added a device-id persistence mapping (`/var/lib/gururmm` → appdata) after noticing the agent's identity was ephemeral. Produced a full plan doc and held the higher-risk recreates (seafile, gitea, npm — compose stacks/public proxy) for a maintenance window.

**GURU-KALI ghost agents.** A ghost check after the Jupiter work surfaced ~11 duplicate GuruRMM agent rows for hostname GURU-KALI. Traced it to a daily device_id churn: the agent could not persist `/var/lib/gururmm/.device-id`. My initial diagnosis ("host root filesystem is read-only") was wrong — commands dispatched through the RMM agent execute inside the agent's `ProtectSystem=strict` systemd sandbox, so they showed the agent's namespace view, not the host. Sent the diagnosis to GURU-KALI's own Claude session via the coord API as a self-heal experiment; it correctly identified the real cause (the systemd unit's `ReadWritePaths` omitted `/var/lib/gururmm`), applied a drop-in override, and reported back. Purged the ghosts (keeper `9bca5090`, stable device-id `ec975630`), discovering the `DELETE /api/agents/:id` endpoint is buggy (resets the connection / commits asynchronously). Filed three bugs and reconciled the bug-tracking system: GuruRMM bugs live as `BUG-NNN` sections in `docs/FEATURE_ROADMAP.md` (not Gitea issues), so BUG-018 was added there and the duplicate coord todos were closed.

**EZ Fast Auto Glass Amazon email.** Jon Shailer reported Amazon emails possibly blocked/filtered. ezfastautoglass.com email is hosted on the IX cPanel server (172.16.3.10 / 72.194.62.5). Exim logs over ~3 weeks showed every Amazon SES message accepted, scored NOT spam, delivered to the local mailbox AND forwarded to jshailer1@gmail.com (Gmail returns 250 OK). Nothing is blocked server-side. The likely cause is Gmail filing the forwarded mail into Spam/Promotions, aggravated by SRS being disabled (srs=0) so forwards fail SPF at Gmail. Drafted client instructions (Gmail filter + POP3 pull) and opened Syncro tracking ticket #32359.

## Key Decisions

- **Templated gururmm-agent via CLI `docker run` with the `net.unraid.docker.managed=dockerman` label** rather than the Unraid web UI — replicates exactly what Unraid's docker manager does, and works over SSH. Verified the label + faithful config post-recreate.
- **Mapped `/var/lib/gururmm` to appdata** for gururmm-agent so the agent's device-id survives recreates; copied the existing device-id out first so identity stayed stable (no new enrollment).
- **Held seafile/gitea/npm recreates** for a maintenance window — compose stacks should be adopted into the Compose Manager plugin (already installed), and npm/gitea are high-blast-radius (public proxy / repos + build pipeline). Discourse (`app`) left alone — self-managed by its own `./launcher`.
- **Used the coord API to hand the GURU-KALI diagnosis to its own Claude session** as a self-heal test instead of remediating it remotely. It succeeded and corrected my misdiagnosis.
- **Finished the ghost purge via the Database Agent (direct SQL)** rather than the flaky `DELETE /api/agents/:id` API. The API deletes turned out to commit asynchronously, so the ghosts were already gone by the time the DB Agent checked.
- **Kept GuruRMM bug tracking in `FEATURE_ROADMAP.md` (no Gitea issues)** per Mike, and added BUG-018 there for consistency; closed the interim coord todos as duplicates.
- **For ezfastautoglass, concluded the server is not at fault** and pointed remediation at Gmail (filter) + an optional POP3-pull / SRS change, rather than touching the mail server reactively.
- **Did not include the mailbox password in the client email draft** — instructed that ACG provides it securely.

## Problems Encountered

- **Misdiagnosed GURU-KALI as "host fs read-only."** RMM-agent-dispatched shell commands run inside the agent's `ProtectSystem=strict` mount namespace (`/` is ro there), so `findmnt`/`touch` reflected the sandbox, not the host. Resolved by GURU-KALI's own session (the real cause was the unit's `ReadWritePaths` missing `/var/lib/gururmm`). Saved as memory `reference_rmm_agent_runs_in_systemd_sandbox.md`.
- **`DELETE /api/agents/:id` returned HTTP 000 for all-but-one delete per burst.** Consistent across 3 runs. Determined the deletes actually commit asynchronously (ghosts cleared ~5 min later with no further calls); likely a missing index on child-table `agent_id` FK columns making the `ON DELETE CASCADE` slow enough to reset the connection. Filed as BUG-018.
- **`post-bot-alert.sh` returned Discord 400 (invalid JSON)** on a message containing a Unicode em-dash/arrow — the helper does not JSON-escape non-ASCII. Resolved by resending ASCII-only. Helper bug noted.
- **Plink first-connect to IX server failed in batch mode** (host key not cached). Resolved by pinning the presented fingerprint with `-hostkey`.
- **Parent claudetools push rejected (non-fast-forward)** during the BUG-018 commit — resolved by the Gitea Agent with `git pull --rebase` then push (no force).

## Configuration Changes

Created:
- `wiki/systems/jupiter-docker-templating.md` — Jupiter container inventory + recreate plan + execution log (gururmm-agent done, device-id fix, ghost-check notes).
- `.claude/memory/reference_rmm_agent_runs_in_systemd_sandbox.md` (+ MEMORY.md index pointer).
- `clients/ezfastautoglass/jon-amazon-email-instructions.txt` — client email draft (still untracked at save time; will be committed by sync).
- On Jupiter: `/boot/config/plugins/dockerMan/templates-user/my-gururmm-agent.xml`.
- On GURU-KALI: `/etc/systemd/system/gururmm-agent.service.d/override.conf` (`ReadWritePaths=/var/lib/gururmm`) — applied by GURU-KALI's session.

Modified:
- `projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md` — added BUG-018 (gururmm commit `c0c0119`).
- Jupiter: `gururmm-agent` container recreated twice (managed label; then + `/var/lib/gururmm` mapping). Backup `/root/gururmm-agent.inspect.bak.json`.
- GuruRMM DB: GURU-KALI ghost agent rows deleted (11 → 1 keeper `9bca5090`).

Closed (coord todos, status=done): `7eb7e60a`, `5ada8ff4`, `04497b97` (all dupes of BUG-016/017/018).

## Credentials & Secrets

- **IX cPanel server** — vault `infrastructure/ix-server.sops.yaml`: host 172.16.3.10 (ext 72.194.62.5), port 22, root. SSH host key fingerprint `SHA256:GZYP/o5XUoRtFRCv1iGjxmqGfQoEsMuiNQBJucoJUh8`. WHM 2087 / cPanel 2083.
- **Jupiter** — vault `infrastructure/jupiter-unraid-primary.sops.yaml`: 172.16.3.20:22 root.
- **GuruRMM API** — vault `infrastructure/gururmm-server.sops.yaml` (admin email/password). GuruRMM Postgres: `gururmm` @ 172.16.3.30:5432 (vault `projects/gururmm/database.sops.yaml`).
- **Syncro** — vault `msp-tools/syncro.sops.yaml` (Mike key, user_id 1735).
- No new secrets created. `jon@ezfastautoglass.com` mailbox password NOT retrieved/created (would be needed if Jon opts for POP3 pull).

## Infrastructure & Servers

- **Jupiter** 172.16.3.20 — Unraid primary container host. Compose Manager plugin installed. Compose files: gitea `/mnt/cache/appdata/gitea/docker-compose.yml`, seafile `/mnt/user0/SeaFile/DockerCompose/docker-compose.yml`. gururmm-agent: net=host, image `localhost:3000/azcomputerguru/gururmm-agent:latest`.
- **GURU-KALI** — Kali Linux box, GuruRMM Linux agent via systemd `gururmm-agent.service` (PID stable, no docker). `/etc/gururmm/agent.toml` → `wss://rmm-api.azcomputerguru.com/ws`. Stable device-id `ec975630-d297-4df9-bcb5-a445c65b648d`, agent_id `9bca5090-...`.
- **IX cPanel server** 172.16.3.10 (ext 72.194.62.5), Rocky Linux WHM/cPanel. ezfastautoglass.com: cPanel acct `ezfastautoglass`, MX self (SPF `v=spf1 +a +mx +ip4:72.194.62.5 ~all`). Mailbox `jon@` forwards to `jshailer1@gmail.com` (+ local copy, 2,822 msgs). SRS disabled (srs=0). AutoSSL cert covers `mail.ezfastautoglass.com` (valid to 2026-07-29); POP3S 995 / IMAPS 993 listening.
- **GuruRMM** server 172.16.3.30:3001; coord API 172.16.3.30:8001.

## Commands & Outputs

- gururmm-agent recreate (Jupiter): `docker run -d --name=gururmm-agent --network=host --restart=unless-stopped -e GURURMM_CONFIG=/config/config.toml -l net.unraid.docker.managed=dockerman -v /dev/kvm:/dev/kvm:ro -v /proc:/proc:ro -v /sys:/sys:ro -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock:ro -v /mnt/user/appdata/gururmm:/config -v /mnt/user/appdata/gururmm/lib:/var/lib/gururmm --entrypoint /usr/local/bin/gururmm-agent localhost:3000/azcomputerguru/gururmm-agent:latest run`
- GURU-KALI root cause (its journal): `WARN Failed to persist device ID: Read-only file system (os error 30)`; mount `/var/lib/gururmm ext4 rw` (bind) while sandbox `/` ro — systemd `ProtectSystem=strict` + `ReadWritePaths` missing `/var/lib/gururmm`.
- Ghost purge: `DELETE /api/agents/:id` returned HTTP 000 (reset) for most; ghosts committed async. Final DB state: 1 GURU-KALI row (keeper), 84 agents total.
- Exim (IX): `zgrep -ih ezfastautoglass /var/log/exim_mainlog* | grep -i amazon` → all `<=` from `smtp-out.amazonses.com`, SpamAssassin "NOT spam", `=> jon ... Saved` + `=> jshailer1@gmail.com ... 250 OK`.
- Syncro ticket create: POST /tickets (customer_id 35547225, problem_type Email) → #32359 id 111854503; initial-issue comment 415836801.

## Pending / Incomplete Tasks

- **Jupiter templating (held):** youtube-sync-test, radio-archive (low); binhex-radarr/sonarr, MariaDB-Official (templates exist, reconcile); seafile stack + gitea (adopt into Compose Manager); npm (recreate from existing template). All backup-first, in a maintenance window. Discourse left alone.
- **ezfastautoglass:** send Jon the drafted instructions; optionally enable SRS in WHM Exim (server-wide); if Jon wants POP3 pull, provide/reset `jon@` mailbox password securely + drop the forward to avoid duplicates. Syncro #32359 open.
- **GuruRMM bugs open in roadmap:** BUG-016 (agent ReadWritePaths — workaround only on GURU-KALI; installer fix pending, fleet-wide), BUG-017 (device_id churn — defense-in-depth), BUG-018 (DELETE endpoint / FK index).
- **C2 (#4):** beast→5070 H.264 cross-GPU test still deferred.
- Optional: harden `post-bot-alert.sh` to JSON-escape non-ASCII.

## Reference Information

- Commits: gururmm `c0c0119` (BUG-018), claudetools `0925582` (submodule bump). Submodule was at `e3d6a46` (KALI's BUG-016/017).
- Syncro: ticket #32359 = https://computerguru.syncromsp.com/tickets/111854503 ; customer EZ Fast Auto Glass id 35547225 (Jon Shailer, jon@ezfastautoglass.com / jshailer1@gmail.com).
- Coord messages: GURU-KALI handoff `23b095d0`, KALI reply `d91406ce`, purge confirm `f2ee93b6`.
- Plan doc: `wiki/systems/jupiter-docker-templating.md`. Bug tracker: `projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md` (BUG-NNN sections).
- POP3 settings for Jon: server `mail.ezfastautoglass.com`, port 995, SSL, user `jon@ezfastautoglass.com`.

---

## Update: 09:44 PT — MSP360 License Report & Feature Request

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-BEAST-ROG
- **Role:** admin

## Session Summary

Mike requested a license report for MSPBackups (MSP360 Managed Backup Service). The MSP360 API at `api.mspbackups.com` was queried using credentials from the vault. DNS resolution for the API hostname fails on BEAST's local DNS server (`fddf:7d20:a256:8::1`); this was worked around by resolving the hostname via Google DNS (8.8.8.8) using curl's `--resolve` flag with the returned IP `52.6.7.137`. An auth token was obtained via `POST /api/Provider/Login`, then three endpoints were queried in sequence: `/api/Licenses`, `/api/Companies`, and `/api/Users`. The Users endpoint provided the company association for each user ID, enabling a full cross-reference of which licenses (by type and computer name) are attached to which client company.

The report covered 35 companies with active licenses. Most held standard Server licenses (one per monitored machine). Notable exceptions included Dataforth (7 licenses: 5x Server, 1x MS SQL Server, 1x VM Server), Glaztech Industries (3 licenses: 1x MS SQL Server, 1x VM Server, 1x expired trial Server), and Jimmy Company (1x active Server + 9x pool/undeployed Server licenses). Three trial licenses were flagged as expired (Brett Interiors, Glaztech SBS, Len's Auto LAB-SVR). All MS Exchange and unallocated licenses were confirmed unused.

Mike then asked whether licensing could be modified via the API. All write-capable endpoints were probed: `POST /api/Licenses/Revoke` and `POST /api/Licenses/Release` were confirmed functional (both require `LicenseID` and `UserID`). Sub-routes including Assign, Transfer, Move, Pool, Count, and Update all responded to GET but returned null or 405 for POST — no assignment capability exists via API. Companies and Users endpoints support POST/PUT for create/update operations.

Mike requested a GuruRMM feature request to automate MSP360 license release on agent decommission. The feature-request skill was invoked, which performed full codebase research, classified the feature via Ollama (qwen3.6), generated a comprehensive specification via Ollama (qwen3:14b), and created SPEC-023. The spec was committed and pushed to the GuruRMM repo, and the submodule pointer was updated in the ClaudeTools parent repo. Mike declined Syncro logging.

## Key Decisions

- Used `--resolve api.mspbackups.com:443:52.6.7.137` with curl as the DNS workaround rather than hardcoding the IP, so the TLS certificate still validates against the hostname.
- Cross-referenced all three MSP360 endpoints (Licenses + Companies + Users) to produce the company-level report — the Licenses endpoint alone only has UserID, not company name.
- SPEC-023 design: Release fires best-effort on agent delete (never blocks deletion); failures are audit-logged to a new DB table rather than surfaced as blocking errors.
- Confidence guard added: only agents with `manually_verified = true` or `mapping_confidence = 'high'` trigger auto-release, preventing false releases on ambiguous hostname matches.
- A dedicated `reqwest::Client` with `hickory-resolver` pointing to 8.8.8.8/8.8.4.4 was specified rather than patching the global HTTP client, since the DNS issue is specific to the MSP360 integration.

## Problems Encountered

- `api.mspbackups.com` would not resolve via BEAST's local DNS (`fddf:7d20:a256:8::1`). Python `socket.gethostbyname()` also failed. Resolved by using curl's `--resolve` flag after obtaining the IP via `nslookup api.mspbackups.com 8.8.8.8`. IP resolved to `52.6.7.137`.
- Initial login attempt with `curl -X POST` returned an empty body. Adding `--resolve` with the IP from `nslookup 8.8.8.8` produced a valid token response. Root cause: DNS failure silently returned no connection rather than an error.

## Configuration Changes

- Created: `projects/msp-tools/guru-rmm/docs/specs/SPEC-023-msp360-license-release-on-decommission.md`
- Modified: `projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md` (added SPEC-023 under Integrations > MSP360)

## Credentials & Secrets

- Vault path accessed: `msp-tools/msp360-api.sops.yaml`
  - `credentials.login`: API username (API-specific, not portal login)
  - `credentials.password`: API password
  - Auth flow: POST `/api/Provider/Login` → Bearer token (14-day expiry)

## Infrastructure & Servers

- MSP360 API: `https://api.mspbackups.com` → resolved IP `52.6.7.137` (via 8.8.8.8)
- Auth token obtained: `uiEqi8Ln...` (issued 2026-06-01 16:24 UTC, expires 2026-06-15)
- BEAST local DNS: `fddf:7d20:a256:8::1` — does NOT resolve `api.mspbackups.com`

## Commands & Outputs

```bash
# DNS workaround — resolve via Google DNS, then use --resolve flag
IP=$(nslookup api.mspbackups.com 8.8.8.8 | grep -E '^Address:' | tail -1 | awk '{print $2}')
# IP = 52.6.7.137

# Login
curl -s --resolve "api.mspbackups.com:443:52.6.7.137" \
  -X POST "https://api.mspbackups.com/api/Provider/Login" \
  -H "Content-Type: application/json" \
  -d '{"UserName": "kY9PvDdWki", "Password": "..."}'
# Returns: {"access_token":"...","expires_in":1209599,...}

# License write endpoints confirmed:
# POST /api/Licenses/Revoke  — requires LicenseID, UserID
# POST /api/Licenses/Release — requires LicenseID, UserID
# All others (Assign, Transfer, Move, Pool, etc.) — GET only, return null
```

## Pending / Incomplete Tasks

- SPEC-023 is in Proposed status — needs review and sprint planning assignment
- The DNS fix for BEAST (dedicated resolver in MSP360 HTTP client) is a prerequisite for SPEC-023 implementation and also fixes the existing silent failure in the backup sync job
- 3 expired trial licenses may warrant cleanup in MSP360 portal: Brett Interiors (Server, exp 2026-05-01), Glaztech SBS (Server, exp 2026-05-23), Len's Auto LAB-SVR (Server, exp 2026-05-01)
- Jimmy Company has 9 undeployed pool Server licenses — worth confirming whether intentional

## Reference Information

- SPEC-023: `projects/msp-tools/guru-rmm/docs/specs/SPEC-023-msp360-license-release-on-decommission.md`
- FEATURE_ROADMAP.md: `projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md`
- MSP360 API base: `https://api.mspbackups.com`
- MSP360 vault: `msp-tools/msp360-api.sops.yaml`
- MSP360 release endpoint: `POST /api/Licenses/Release` (params: `LicenseID`, `UserID`)
- MSP360 revoke endpoint: `POST /api/Licenses/Revoke` (params: `LicenseID`, `UserID`)

---

## Update: 11:08 PT — MSP360 Storage Usage, vland Deletion, Rohrbach Retention

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-BEAST-ROG
- **Role:** admin

## Session Summary

Mike asked whether storage usage by customer or computer was available. The Users endpoint already contained `SpaceUsed` and `DestinationList[].CurrentVolume` fields from the prior pull — no additional API calls were needed to retrieve storage data. A Python cross-reference against company names produced a full per-company storage breakdown sorted descending by total usage. Grand total across all company-assigned accounts was 60.68 TB. Three accounts with no company assigned but significant data were flagged: `admin@azcomputerguru.com` (13.84 TB), `vland@airyoptics.com` (11.43 TB), and `mike@azcomputerguru.com` (399 GB). Notable findings included Stamback Services storing 2.73 TB exclusively to Local-E (no cloud), and Martell and Associates splitting across Local-E and B2.

Mike identified `vland@airyoptics.com` (Airy Optics) as a former client and requested full account and data deletion. All deletion sub-routes — `DELETE /api/Users/Delete`, `DELETE /api/Users/Disable`, `DELETE /api/Users/Archive`, `DELETE /api/Users/Deactivate`, and `DELETE /api/Users/{id}` directly — returned HTTP 400 `"Not Acceptable personal user"`. This restriction applies to accounts with `LicenseManagmentMode: 1` (personal/standalone users not enrolled under a company). A fallback `PUT /api/Users` with `{"ID": "...", "Enabled": false}` succeeded (HTTP 200), disabling the account. Mike then reported the MSP360 portal was already showing the account status as "Deleting" — one of the DELETE calls queued the deletion server-side despite returning an error body in the response.

Mike then asked about retention settings for Rohrbach backups. The Monitoring endpoint confirmed two active Rohrbach entries: `rohrbach` on MIKE-THINK running a "Files Plan" (PlanType 3) daily at 9am, and `susan-think` running an image-based plan. Plan detail endpoints (`/api/Plan/{id}`, `/api/Users/{id}/Plan`, `/api/BackupPlan/{id}`) all returned 404. The only endpoint that responded for plan-level access was `/api/Computers/{hid}/Plans`, which returned HTTP 400 with `"Remote Management API methods are not enabled for your account"` — indicating plan configuration data requires a separate add-on tier. Mike declined to pursue enabling it or checking the portal.

## Key Decisions

- Used cached Users data from the prior pull for storage report rather than re-fetching — data was fresh from the same session.
- Attempted all available deletion patterns before concluding the API blocks personal user deletion — did not assume failure from the first error response.
- Left vland account in disabled state after portal confirmed deletion was already in progress — no need to re-enable or clean up.
- Did not attempt to modify plan retention settings — scope was read-only (check only), and the endpoint isn't accessible without the Remote Management API add-on.

## Problems Encountered

- All MSP360 delete endpoints returned `"Not Acceptable personal user"` for `vland@airyoptics.com`. Root cause: account is `LicenseManagmentMode: 1` (personal user), which the provider API cannot delete programmatically. Worked around by disabling via PUT; portal completed the deletion independently.
- `/api/Computers/{hid}/Plans` returned 400 `"Remote Management API methods are not enabled"` rather than 404 — endpoint exists but requires an add-on not currently active on the MSP360 account.

## Configuration Changes

None — read-only session after the vland disable operation.

## Credentials & Secrets

- Vault path accessed: `msp-tools/msp360-api.sops.yaml` (same credentials as prior update, no new secrets)

## Infrastructure & Servers

- MSP360 API: `https://api.mspbackups.com` → `52.6.7.137` (via 8.8.8.8 DNS workaround)
- vland account: `98f0c6ae-0090-4d6e-acc0-cceeed5ac60b` — disabled via API, deletion initiated in portal
- Rohrbach MIKE-THINK computer HID: `{32C6F83D-A4D6-4E5E-8DA0-698F6215A9A7}`, PlanId: `5c13b9a4-d48c-4347-bd28-ff457963bc9d`
- Rohrbach susan-think computer HID: `{652C686F-C5CA-4A05-A8C2-FD15BA5193B7}`, PlanId: `80c6bd7e-fcda-4c0a-846e-2e0237fbf6d1`

## Commands & Outputs

```bash
# Storage from Users endpoint — SpaceUsed field, grouped by Company
# Grand total: 60.68 TB across all company-assigned accounts

# vland disable (succeeded HTTP 200)
PUT /api/Users  {"ID": "98f0c6ae-0090-4d6e-acc0-cceeed5ac60b", "Enabled": false}

# vland direct delete (HTTP 400, but queued deletion server-side)
DELETE /api/Users/98f0c6ae-0090-4d6e-acc0-cceeed5ac60b
# Response: {"Message":"Not Acceptable personal user"}
# Portal showed status: "Deleting" — deletion was queued despite error body

# Retention probe — blocked by missing add-on
GET /api/Computers/{hid}/Plans?userId=...
# Response: {"Message":"Remote Management API methods are not enabled for your account"}
```

## Pending / Incomplete Tasks

- vland deletion is in progress in MSP360 portal — 11.43 TB of B2Storage data being purged. No follow-up needed unless deletion fails.
- Rohrbach retention settings remain unchecked — requires either MSP360 portal manual check or enabling Remote Management API add-on on the account.
- Stamback Services: 2.73 TB on Local-E only (no cloud) — worth flagging to client if not intentional.
- admin@azcomputerguru.com has 13.84 TB in B2Storage with no company assigned — should confirm this is expected (likely ACG internal/test data).

## Reference Information

- MSP360 storage totals by company: see session log above
- vland UserID: `98f0c6ae-0090-4d6e-acc0-cceeed5ac60b`
- MSP360 API DNS workaround: `--resolve api.mspbackups.com:443:52.6.7.137`
- Remote Management API: requires add-on — toggle in MSP360 console Settings if needed
- Rohrbach Monitoring: Files Plan (MIKE-THINK) + Image-Based (susan-think), both healthy as of 2026-06-01

---

## Update: 20:28 PDT — Session recovery toolset + wiki-on-save (Sonnet)

### Session Summary

Recovered lost context for the Lone Star Electrical Sophos removal (LS-1/LS-2): the ~May 28-29 work had never been saved to a session log and survived only in a gitignored Ollama draft (`.claude/tmp/ollama_prompt.txt`) and coordinator message `8a5cb25c`. Reassembled the full picture, reconstructed it into `clients/lonestar-electrical/session-logs/2026-05-29-sophos-removal.md`, recompiled the Lone Star wiki article (added the inherited-Sophos kernel-driver tamper-protection removal pattern), and sent a complete handoff to Howard via coordinator message `689cfb7c` including the WinRE completion procedure.

Diagnosed why that work was invisible to search: never `/save`d, so it lived only in a gitignored temp file and the coord-message DB — neither indexed by GrepAI nor in git. To close the gap, built a session-recovery toolset: `recover_session.py` (engine that parses a Claude Code transcript JSONL, classifies substantive/saved, extracts a verbatim command/config/reference trail, drafts prose), `detect_orphaned_sessions.py` (scheduled scanner that auto-builds banner-marked logs for substantive/unsaved transcripts, commits them, pings #bot-alerts), `/recover` (manual command), a scheduled-task registrar, `RECOVERY.md`, and a memory entry. Registered the scheduled task on GURU-5070 (logon + every 4h).

Code-reviewed the toolset: fixed a Critical bug (staging the gitignored ledger aborted `git add`, so nothing ever committed), High issues (write ledger only after a successful push; per-uuid idempotency; `--max` cap), and a submodule-routing bug (recovered logs must not land inside the guru-rmm/guru-connect submodules — route to root `session-logs/`). Ran a reviewed backfill of 12 historical orphans; standout recovery was the Peaceful Spirit RADIUS/VPN buildout (full verbatim command trail). Moved the Peaceful Spirit recovered log into `clients/peaceful-spirit/`, cross-linked it with the existing manual 2026-05-10 log (it is the primary-source transcript of the crashed session that log reconstructed second-hand), and corrected its machine attribution to DESKTOP-0O8A1RL.

Integrated wiki maintenance into `/save`: a pre-sync Phase 3 that recompiles the worked-on client/project article so it ships in the same commit. Iterated the design from refresh-only to full recompile (per user), then switched the drafting engine from Ollama qwen3 to a Sonnet subagent for better prose quality and no local-Ollama dependency. Verified the `/save` wiki-recompile end-to-end with a Lone Star-scoped test save.

### Key Decisions

- Recovered logs are auto-built but banner-marked `[RECOVERED -- UNVERIFIED]`; Python extracts verbatim commands/IPs/SHAs while the model drafts only prose — no hallucinated command can enter a saved log.
- The ledger (`.claude/state/recovered-sessions.json`) is machine-local and gitignored, never staged; recovered uuids are marked only after a successful push so failed runs retry.
- Recovered logs are never written inside submodules; project scopes whose dir is a submodule route to root `session-logs/`.
- `/save` full-recompiles the worked-on wiki article before sync (skips general/root scope); softfails to a surgical refresh so a save is never blocked.
- Wiki drafting moved off Ollama to a Sonnet subagent (`model: "sonnet"`); refresh mode stays surgical/no-model.
- GrepAI was deliberately NOT pointed at raw transcripts (too noisy/large); recovery distills them into `session-logs/`, which GrepAI already indexes.

### Problems Encountered

- Sophos/Lone Star work unsearchable — never `/save`d; recovered from temp draft + coord message. This was the root cause that motivated the whole session.
- Code review found a Critical: `git add` of the gitignored ledger aborted atomically, silently disabling all auto-commits. Fixed by never staging the ledger.
- Reviewed backfill caught recovered logs landing inside the guru-rmm/guru-connect submodules; moved 4 out to root and patched the engine's `compute_output_path`.
- The recovery engine stamps the current machine (GURU-5070) in the User block, not the machine where the work happened (DESKTOP-0O8A1RL for the 2026-05-10 PST session). Corrected manually; noted as a known limitation of recovered logs.

### Configuration Changes

- [created] `.claude/scripts/recover_session.py`, `.claude/scripts/detect_orphaned_sessions.py`, `.claude/scripts/register-orphan-detector.ps1`
- [created] `.claude/commands/recover.md`, `.claude/RECOVERY.md`, `.claude/memory/feedback_session_recovery.md` (+ MEMORY.md pointer)
- [modified] `.claude/commands/save.md` (Phase 3 wiki recompile via Sonnet subagent, before sync)
- [modified] `.claude/commands/wiki-compile.md` (Sonnet-subagent synthesis; dropped Ollama)
- [modified] `.gitignore` (`.claude/state/`)
- [created] `clients/lonestar-electrical/session-logs/2026-05-29-sophos-removal.md`, `2026-06-01-session.md`
- [modified] `wiki/clients/lonestar-electrical.md` (Sophos pattern + recompile)
- [created] 12 recovered backfill logs across `session-logs/`, `clients/`, `projects/gururmm-agent/`
- [moved] Peaceful Spirit recovered log -> `clients/peaceful-spirit/session-logs/` (+ cross-link in `2026-05-10-session.md`)

### Scheduled Tasks

- Registered `ClaudeTools - Orphaned Session Detector` on GURU-5070 (AtLogOn + every 4h).

### Pending / Incomplete Tasks

- Howard to complete the offline WinRE Sophos removal on LS-1/LS-2, then `SophosZap --confirm`; verify the drafted Syncro ticket exists before logging time.
- The 12 backfilled recovered logs are UNVERIFIED — review and clear banners as time permits.
- The Ollama -> Sonnet wiki draft path is wired but not yet exercised live (integration verified Claude-direct; the Sonnet subagent draft itself is untested on a real recompile).

### Reference Information

- Commits: `eed3ece` (toolset), `59397e8` (submodule fix), `aa9bd26` (12-log backfill), `5afb781` (save+recompile test), `c893d3e` (Sonnet subagent), `f44a96b`/`2a5476f` (save wiki phases), `4651990` (Lone Star wiki+log), `07c86c7` (PST move)
- Coordinator messages: `689cfb7c` (Howard handoff), `8a5cb25c` (WinRE commands source)
- Scheduled task: `ClaudeTools - Orphaned Session Detector` (GURU-5070)
- Ledger: `.claude/state/recovered-sessions.json` (machine-local, gitignored)
- Transcripts: `~/.claude/projects/D--claudetools/*.jsonl`