diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index cb346c38..2d209de9 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -217,3 +217,4 @@ - [TECH03L systemprofile shortcut corruption](project_tech03l_systemprofile_shortcut_corruption.md) — Auto-Claude "opens then closes" = .lnk pointing at nonexistent systemprofile path; repoint, do not debug the app - [Claude tui fullscreen crash](feedback_claude_tui_fullscreen_crash.md) — "flicker-free" prompt writes tui=fullscreen; instant-exit crash loop on some Windows terminals; set tui=default (reinstall does NOT fix) - [Backup targets never guessed](feedback_backup_targets_never_guessed.md) — billed-vs-running mismatches are billing questions for Mike/Winter; never infer/deploy backup targets +- [Fire #dev-alerts for client/RMM work](feedback_fire_dev_alerts_for_client_work.md) — Howard+Mike watch #dev-alerts Discord for live visibility into what techs/sessions touch. Fire post-bot-alert.sh ([DEV]/[RMM]/[DEPLOY]/[GURURMM] prefix -> #dev-alerts) at the START of any RMM/Dev or active-client-machine action. No CLAUDE.md rule yet — this memory is the rule. diff --git a/.claude/memory/feedback_fire_dev_alerts_for_client_work.md b/.claude/memory/feedback_fire_dev_alerts_for_client_work.md new file mode 100644 index 00000000..e6440e6c --- /dev/null +++ b/.claude/memory/feedback_fire_dev_alerts_for_client_work.md @@ -0,0 +1,29 @@ +--- +name: feedback_fire_dev_alerts_for_client_work +description: Fire a #dev-alerts Discord post (post-bot-alert.sh with an [RMM]/[DEV]/[DEPLOY]/[BUILD]/[GURURMM] prefix) at the START of any RMM/Dev or active-client-machine action — Howard+Mike watch that channel for live visibility into what techs/sessions are touching +metadata: + type: feedback +--- + +Howard and Mike watch the **#dev-alerts** Discord channel to see, in real time, what the other techs +(and Claude sessions) are working on. Any substantive action on **RMM/Dev** or an **ACTIVE CLIENT +machine** MUST post a #dev-alerts heads-up — ideally at the START of the work, not just after. + +**What counts:** SSH into a client server, restarting/reconfiguring services, editing boot/config on a +live box, RMM remote commands, agent/server/dashboard deploys, channel promotions, anything with real +blast radius on production/client infra. + +**Why:** without it a session can be hand-editing a live client box (e.g. Lonestar Electrical's Unraid +`/boot/config/go`, cycling libvirt) with ZERO visibility to the team — exactly what happened +2026-07-06, which Mike flagged: "all of the actions you've taken today should have fired the discord +alerts." + +**How to apply:** +`bash .claude/scripts/post-bot-alert.sh "[DEV] — . — Claude @ ()"` +Prefixes `[RMM] [DEPLOY] [DEV] [BUILD] [GURURMM] [SMARTBADGE-WATCH]` auto-route to the PRIVATE +#dev-alerts (Howard+Mike only, channel 1509998508198068484); everything else → #bot-alerts (whole +team). It soft-fails (best-effort), so it never blocks the work. Channel IDs + routing live in +`.claude/scripts/post-bot-alert.sh`. + +As of 2026-07-06 there is NO CLAUDE.md rule enforcing this — treat this memory AS the rule until a core +rule/hook lands (worth proposing one). Related: [[discord-dm]] skill for direct DMs. diff --git a/clients/lonestar-electrical/session-logs/2026-07/2026-07-06-mike-unraid-vm-recovery.md b/clients/lonestar-electrical/session-logs/2026-07/2026-07-06-mike-unraid-vm-recovery.md new file mode 100644 index 00000000..d5a5f863 --- /dev/null +++ b/clients/lonestar-electrical/session-logs/2026-07/2026-07-06-mike-unraid-vm-recovery.md @@ -0,0 +1,140 @@ +# Session: Lonestar Unraid VM recovery + root cause — plus GuruRMM promotions + dev-alerts gap + +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Session Summary + +Mike reported the Lonestar Electrical Unraid server's `/VMs` page (192.168.120.177) was "empty +again" — a recurring problem on this box only, while his other Unraid boxes are stable. Connected +via VPN + PuTTY/plink (system OpenSSH lacks non-interactive password auth; plink `-batch` with the +vaulted root password works because the host key was already cached from prior PuTTY use). Diagnosed: +the "Windows 11" VM (= LONESTAR-VM, a4d39a9d) domain, its 64 GiB vdisk, nvram, and swtpm are all +intact; `libvirt.img` mounts read-only fine and contains the domain; cache SSD (Crucial MX500) SMART +PASSED. The actual failure: `libvirt.img` was NOT mounted at `/etc/libvirt`, so libvirtd came up with +zero domains. `docker.img` (same cache/system share, via `/mnt/cache`) mounted; libvirt (via +`/mnt/user` FUSE) did not. + +Recovered the VM live but hit several stale-state artifacts from a broken boot: `/etc/libvirt` was a +plain dir (not a mountpoint), `/var/run/libvirt/libvirt-sock` was a stale empty DIRECTORY (blocking +libvirtd's socket bind), and `/dev/kvm` was ALSO a stale directory (blocking KVM). Cleared each, +manually mounted the image, started libvirtd via `rc.libvirt`, loaded KVM modules + recreated +`/dev/kvm`, and started "Windows 11" — now running with autostart enabled. + +Mike then asked to multi-AI the DURABLE fix ("you've done all this before and it's not clean"). +Briefed Gemini 3.1 Pro (agy) + Grok. Gemini gave a detailed answer but had two wrong assumptions that +live verification caught: `rc.domain` does not exist on 7.1.4, and this box runs MONOLITHIC libvirtd +(not modular). Grok returned only a preamble (subagent-mode flakiness), no usable answer. The real +root cause came from reading `/boot/config/go`: a leftover hand-hack labeled "LibVirt Fix (final +2026-06-10)" that, every boot (`sleep 5`), writes `/etc/libvirt/*.conf` into the UNMOUNTED +`/etc/libvirt` and manually starts libvirtd BEFORE emhttpd can mount `libvirt.img` — hijacking libvirt +so emhttpd's mount then fails and the VMs vanish. A prior "fix" was the cause; that is why it only +affects this box and why past repairs never stayed clean. + +Applied the durable fix: rewrote `/boot/config/go` to remove the hijack (stock Unraid libvirt boot; +backup `go.bak-2026-07-06`), and pinned `IMAGE_FILE` to `/mnt/cache/...` (bypass the FUSE path; +backup `domain.cfg.bak-2026-07-06`). Verified the image's own qemu.conf is self-sufficient (VM runs +under it). Per Mike, hand-patched to get the VM up NOW; clean reboot scheduled tonight to reach the +proper stock state. Earlier in the session also: promoted GuruRMM agent 0.6.77 beta->stable (flipped +`.channel` files as guru) and dashboard beta->prod (via `promote-dashboard.sh`, live), and zipped 21 +GURU-5070 `.claude/tmp/` promotion candidates into `promotion-candidates/` + coord-messaged Beast. +Finally: Mike flagged that all this client-machine + RMM/Dev work should have fired #dev-alerts Discord +posts (Howard+Mike watch that channel for live tech visibility) — posted retroactive alerts and made +it a standing rule. + +## Key Decisions + +- **Root cause = the `/boot/config/go` libvirt hijack, not storage/FUSE.** Removing the leftover + 2026-06-10 hand-hack is THE durable fix (let Unraid boot libvirt the stock way via emhttpd). +- **Hand-patch now, clean reboot tonight** (Mike's call) — the piecemeal recovery is emhttpd-unaware; + tonight's reboot with the hijack removed resets to the proper stock init and validates the fix. +- **Kept `IMAGE_FILE` -> /mnt/cache** as belt-and-suspenders even though the go-hack was the primary + cause (same underlying file; direct pool path avoids any FUSE-timing edge). +- **plink `-batch` over system ssh** — no sshpass on GURU-5070; plink `-batch -pw` works with the + cached host key (Git-Bash `echo y |` does NOT reach plink's console prompt → hangs). +- **Fire #dev-alerts for client/RMM work going forward** — saved as a feedback memory (rule) since no + CLAUDE.md rule enforces it; a real auto-hook can't tell "this SSH is a client box," so it's a + judgment rule. + +## Problems Encountered + +- **`/VMs` empty / libvirt.img not mounted** — root cause the `/boot/config/go` hijack (above); + removed it. Immediate recovery: mount image + clear stale sock/kvm dirs + rc.libvirt start + virsh + start. +- **Stale directories where nodes/sockets belong** — `/var/run/libvirt/libvirt-sock` and `/dev/kvm` + were empty DIRECTORIES (from the broken boot), blocking libvirtd socket + KVM. `rmdir` + let module + recreate `/dev/kvm` (crw 10,232); rc.libvirt then created the proper `libvirt-sock` socket. +- **Gemini wrong on rc.domain + modular libvirt** — caught by verifying `ls /etc/rc.d/rc.domain` + (absent) and rc.libvirt (uses monolithic libvirtd) BEFORE executing. Grok returned no usable answer. +- **Git-Bash mangling** — remote `/etc/...` paths mangled to `C:/Program`; fixed with + `MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*'`. Also the `/tmp` write hook blocked a `/tmp/cand.txt`. +- **No #dev-alerts fired for the day's client/RMM/Dev work** (Mike's correction) — posted retroactive + alerts, saved a feedback memory, logged the correction. + +## Configuration Changes + +Lonestar Unraid (192.168.120.177), live: +- `/boot/config/go` — REWROTE: removed the "LibVirt Fix (final 2026-06-10)" hijack block; kept emhttp + start + XDG_CACHE_HOME. Backup `/boot/config/go.bak-2026-07-06`. +- `/boot/config/domain.cfg` — `IMAGE_FILE` `/mnt/user/system/libvirt/libvirt.img` -> + `/mnt/cache/system/libvirt/libvirt.img`. Backup `/boot/config/domain.cfg.bak-2026-07-06`. +- Runtime (reset by tonight's reboot): manually mounted libvirt.img at /etc/libvirt; recreated + /dev/kvm; started libvirtd via rc.libvirt; `virsh start "Windows 11"` + `virsh autostart`. +GuruRMM (build server 172.16.3.30, as guru): +- `/var/www/gururmm/downloads/*0.6.77*.channel` -> "stable" (agent promotion). +- Dashboard beta -> prod via promote-dashboard.sh (backup dashboard-20260704-204039). +ClaudeTools repo: +- NEW `promotion-candidates/tmp-candidates-GURU-5070-2026-07-06.zip` + README.md (for Beast). +- NEW `.claude/memory/feedback_fire_dev_alerts_for_client_work.md` + MEMORY.md index. + +## Credentials & Secrets + +- Lonestar Unraid root (used this session): user `root`, password `Gptf*77ttb123!@#-lonestar` — + ALREADY vaulted at `clients/lonestar-electrical/unraid-server.sops.yaml` (fields + `credentials.root_password`, `credentials.ssh_user`). No new secrets created. Note the vault entry's + `ip: 172.16.1.188` is STALE (from when the box was at the ACG office); live IP is 192.168.120.177. + +## Infrastructure & Servers + +- **Lonestar Unraid** `Tower` — 192.168.120.177 (Warren LAN 192.168.120.0/24, reached via client VPN). + Unraid 7.1.4, kernel 6.12.24, libvirt 11.2.0 (monolithic libvirtd). Array: 2 XFS data disks + parity + (STARTED). Cache: single btrfs on Crucial MX500 500GB (`/dev/sdb1`). No UPS active. VM "Windows 11" + (= LONESTAR-VM, agent a4d39a9d, 192.168.120.197), vdisk `/mnt/user/domains/Windows 11/vdisk1.img`. + Only Docker container: `gururmm-agent` (crash-looping -> host RMM agent offline; another session + handling it). RMM agent IDs seen: host `00b48379-...` (offline), vault-noted `e827f798-...`. +- GuruRMM build server 172.16.3.30; downloads `/var/www/gururmm/downloads/`; dashboards + `/var/www/gururmm/dashboard{,-beta}` (guru-owned). Beast build host `guru@100.101.122.4`. +- Discord alert channels: #dev-alerts (private, Howard+Mike) `1509998508198068484`; #bot-alerts + `624710699771232265`. Poster: `.claude/scripts/post-bot-alert.sh` (auto-routes [RMM]/[DEV]/[DEPLOY]/ + [BUILD]/[GURURMM] -> #dev-alerts). + +## Commands & Outputs + +- SSH: `MSYS_NO_PATHCONV=1 "C:/Program Files/PuTTY/plink.exe" -ssh -batch -pw '' root@192.168.120.177 ""`. +- Recovery: `rmdir /var/run/libvirt/libvirt-sock`; `mount -o loop /mnt/cache/system/libvirt/libvirt.img /etc/libvirt`; `rmdir /dev/kvm; modprobe -r kvm_intel kvm; modprobe kvm_intel`; `/etc/rc.d/rc.libvirt start`; `virsh start "Windows 11"; virsh autostart "Windows 11"`. +- Key error resolved: `virNetSocketNewListenUNIX: Failed to bind '/var/run/libvirt/libvirt-sock': Address already in use` = the path was a stale DIRECTORY; `unsupported ... virt type 'kvm'` = `/dev/kvm` was a stale directory. +- Dev-alerts: `bash .claude/scripts/post-bot-alert.sh "[DEV]/[GURURMM] ..."` -> #dev-alerts (msg ids 1523773910238101595, 1523773916412383272). + +## Pending / Incomplete Tasks + +- **Reboot the Unraid box tonight** (Mike) — validate it comes up clean with the hijack removed + (emhttpd mounts libvirt.img, VM autostarts). Optionally enable Settings -> Syslog Server -> "Mirror + syslog to flash" first so any failure's boot log survives (Unraid syslog is RAM-only; Jun 30's + evidence had already rotated out). +- **gururmm-agent Docker container crash-loop** on the Unraid host — another session is handling. +- **RMM offline-alerting gap** — LONESTAR-VM + the Unraid host agent were offline ~6 days with no + alert; separate from the dev-alerts visibility gap (that one is the tech-action Discord channel). +- **Optional:** add a CLAUDE.md core rule to enforce firing #dev-alerts on RMM/Dev/client-machine + actions (currently only the feedback memory covers it). +- Update the Lonestar Unraid wiki: live IP is 192.168.120.177 (vault IP 172.16.1.188 is stale). + +## Reference Information + +- Vault: `clients/lonestar-electrical/unraid-server.sops.yaml`. +- Backups on the box: `/boot/config/go.bak-2026-07-06`, `/boot/config/domain.cfg.bak-2026-07-06`. +- GuruRMM shipped v0.6.77 (stable); dashboard prod backup `dashboard-20260704-204039`. +- Repo: `promotion-candidates/tmp-candidates-GURU-5070-2026-07-06.zip`; coord msg to Beast + `cfba46f7...` + todo `a676fc2b...`. +- Diag transcript saved locally: `promotion-candidates/.lonestar-unraid-diag.txt` (gitignored dotfile). diff --git a/errorlog.md b/errorlog.md index a312f041..1cb6ca0e 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,8 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-06 | GURU-5070 | dev-alerts/visibility | [correction] did substantial RMM/Dev + live-client-machine work (Lonestar Unraid boot-config edits, service cycling, v0.6.77 deploy, promotions) WITHOUT firing #dev-alerts; Howard+Mike lost live visibility. Fire post-bot-alert.sh at the START of such work. [ctx: ref=feedback_fire_dev_alerts_for_client_work] + 2026-07-06 | GURU-BEAST-ROG | remediation-tool | onboard-tenant: failed to acquire Tenant Admin token [ctx: tenant=5c53ae9f-7071-4248-b834-8685b646450f exit=3] (x2) 2026-07-06 | Howard-Home | vault/windows-pro-mak | [correction] assumed vault key labeled 'Windows Pro MAK' was plain Pro; correct is Pro FOR WORKSTATIONS MAK - slmgr /ipk auto-upgrades edition to ProfessionalWorkstation [ctx: host=ACG-Tech03L partial=HMJ4K]