azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from GURU-5070 at 2026-07-06 12:39:35

Browse Source 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-06 12:39:35
This commit is contained in:
Mike Swanson
2026-07-06 12:39:59 -07:00
parent c3ad79d9fe
commit a2a9356a02
4 changed files with 172 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

140
clients/lonestar-electrical/session-logs/2026-07/2026-07-06-mike-unraid-vm-recovery.md Normal file
View File

@@ -0,0 +1,140 @@
# Session: Lonestar Unraid VM recovery + root cause — plus GuruRMM promotions + dev-alerts gap
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
## Session Summary
Mike reported the Lonestar Electrical Unraid server's `/VMs` page (192.168.120.177) was "empty
again" — a recurring problem on this box only, while his other Unraid boxes are stable. Connected
via VPN + PuTTY/plink (system OpenSSH lacks non-interactive password auth; plink `-batch` with the
vaulted root password works because the host key was already cached from prior PuTTY use). Diagnosed:
the "Windows 11" VM (= LONESTAR-VM, a4d39a9d) domain, its 64 GiB vdisk, nvram, and swtpm are all
intact; `libvirt.img` mounts read-only fine and contains the domain; cache SSD (Crucial MX500) SMART
PASSED. The actual failure: `libvirt.img` was NOT mounted at `/etc/libvirt`, so libvirtd came up with
zero domains. `docker.img` (same cache/system share, via `/mnt/cache`) mounted; libvirt (via
`/mnt/user` FUSE) did not.
Recovered the VM live but hit several stale-state artifacts from a broken boot: `/etc/libvirt` was a
plain dir (not a mountpoint), `/var/run/libvirt/libvirt-sock` was a stale empty DIRECTORY (blocking
libvirtd's socket bind), and `/dev/kvm` was ALSO a stale directory (blocking KVM). Cleared each,
manually mounted the image, started libvirtd via `rc.libvirt`, loaded KVM modules + recreated
`/dev/kvm`, and started "Windows 11" — now running with autostart enabled.
Mike then asked to multi-AI the DURABLE fix ("you've done all this before and it's not clean").
Briefed Gemini 3.1 Pro (agy) + Grok. Gemini gave a detailed answer but had two wrong assumptions that
live verification caught: `rc.domain` does not exist on 7.1.4, and this box runs MONOLITHIC libvirtd
(not modular). Grok returned only a preamble (subagent-mode flakiness), no usable answer. The real
root cause came from reading `/boot/config/go`: a leftover hand-hack labeled "LibVirt Fix (final
2026-06-10)" that, every boot (`sleep 5`), writes `/etc/libvirt/*.conf` into the UNMOUNTED
`/etc/libvirt` and manually starts libvirtd BEFORE emhttpd can mount `libvirt.img` — hijacking libvirt
so emhttpd's mount then fails and the VMs vanish. A prior "fix" was the cause; that is why it only
affects this box and why past repairs never stayed clean.
Applied the durable fix: rewrote `/boot/config/go` to remove the hijack (stock Unraid libvirt boot;
backup `go.bak-2026-07-06`), and pinned `IMAGE_FILE` to `/mnt/cache/...` (bypass the FUSE path;
backup `domain.cfg.bak-2026-07-06`). Verified the image's own qemu.conf is self-sufficient (VM runs
under it). Per Mike, hand-patched to get the VM up NOW; clean reboot scheduled tonight to reach the
proper stock state. Earlier in the session also: promoted GuruRMM agent 0.6.77 beta->stable (flipped
`.channel` files as guru) and dashboard beta->prod (via `promote-dashboard.sh`, live), and zipped 21
GURU-5070 `.claude/tmp/` promotion candidates into `promotion-candidates/` + coord-messaged Beast.
Finally: Mike flagged that all this client-machine + RMM/Dev work should have fired #dev-alerts Discord
posts (Howard+Mike watch that channel for live tech visibility) — posted retroactive alerts and made
it a standing rule.
## Key Decisions
- **Root cause = the `/boot/config/go` libvirt hijack, not storage/FUSE.** Removing the leftover
2026-06-10 hand-hack is THE durable fix (let Unraid boot libvirt the stock way via emhttpd).
- **Hand-patch now, clean reboot tonight** (Mike's call) — the piecemeal recovery is emhttpd-unaware;
tonight's reboot with the hijack removed resets to the proper stock init and validates the fix.
- **Kept `IMAGE_FILE` -> /mnt/cache** as belt-and-suspenders even though the go-hack was the primary
cause (same underlying file; direct pool path avoids any FUSE-timing edge).
- **plink `-batch` over system ssh** — no sshpass on GURU-5070; plink `-batch -pw` works with the
cached host key (Git-Bash `echo y |` does NOT reach plink's console prompt → hangs).
- **Fire #dev-alerts for client/RMM work going forward** — saved as a feedback memory (rule) since no
CLAUDE.md rule enforces it; a real auto-hook can't tell "this SSH is a client box," so it's a
judgment rule.
## Problems Encountered
- **`/VMs` empty / libvirt.img not mounted** — root cause the `/boot/config/go` hijack (above);
removed it. Immediate recovery: mount image + clear stale sock/kvm dirs + rc.libvirt start + virsh
start.
- **Stale directories where nodes/sockets belong** — `/var/run/libvirt/libvirt-sock` and `/dev/kvm`
were empty DIRECTORIES (from the broken boot), blocking libvirtd socket + KVM. `rmdir` + let module
recreate `/dev/kvm` (crw 10,232); rc.libvirt then created the proper `libvirt-sock` socket.
- **Gemini wrong on rc.domain + modular libvirt** — caught by verifying `ls /etc/rc.d/rc.domain`
(absent) and rc.libvirt (uses monolithic libvirtd) BEFORE executing. Grok returned no usable answer.
- **Git-Bash mangling** — remote `/etc/...` paths mangled to `C:/Program`; fixed with
`MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*'`. Also the `/tmp` write hook blocked a `/tmp/cand.txt`.
- **No #dev-alerts fired for the day's client/RMM/Dev work** (Mike's correction) — posted retroactive
alerts, saved a feedback memory, logged the correction.
## Configuration Changes
Lonestar Unraid (192.168.120.177), live:
- `/boot/config/go` — REWROTE: removed the "LibVirt Fix (final 2026-06-10)" hijack block; kept emhttp
start + XDG_CACHE_HOME. Backup `/boot/config/go.bak-2026-07-06`.
- `/boot/config/domain.cfg` — `IMAGE_FILE` `/mnt/user/system/libvirt/libvirt.img` ->
`/mnt/cache/system/libvirt/libvirt.img`. Backup `/boot/config/domain.cfg.bak-2026-07-06`.
- Runtime (reset by tonight's reboot): manually mounted libvirt.img at /etc/libvirt; recreated
/dev/kvm; started libvirtd via rc.libvirt; `virsh start "Windows 11"` + `virsh autostart`.
GuruRMM (build server 172.16.3.30, as guru):
- `/var/www/gururmm/downloads/*0.6.77*.channel` -> "stable" (agent promotion).
- Dashboard beta -> prod via promote-dashboard.sh (backup dashboard-20260704-204039).
ClaudeTools repo:
- NEW `promotion-candidates/tmp-candidates-GURU-5070-2026-07-06.zip` + README.md (for Beast).
- NEW `.claude/memory/feedback_fire_dev_alerts_for_client_work.md` + MEMORY.md index.
## Credentials & Secrets
- Lonestar Unraid root (used this session): user `root`, password `Gptf*77ttb123!@#-lonestar` —
ALREADY vaulted at `clients/lonestar-electrical/unraid-server.sops.yaml` (fields
`credentials.root_password`, `credentials.ssh_user`). No new secrets created. Note the vault entry's
`ip: 172.16.1.188` is STALE (from when the box was at the ACG office); live IP is 192.168.120.177.
## Infrastructure & Servers
- **Lonestar Unraid** `Tower` — 192.168.120.177 (Warren LAN 192.168.120.0/24, reached via client VPN).
Unraid 7.1.4, kernel 6.12.24, libvirt 11.2.0 (monolithic libvirtd). Array: 2 XFS data disks + parity
(STARTED). Cache: single btrfs on Crucial MX500 500GB (`/dev/sdb1`). No UPS active. VM "Windows 11"
(= LONESTAR-VM, agent a4d39a9d, 192.168.120.197), vdisk `/mnt/user/domains/Windows 11/vdisk1.img`.
Only Docker container: `gururmm-agent` (crash-looping -> host RMM agent offline; another session
handling it). RMM agent IDs seen: host `00b48379-...` (offline), vault-noted `e827f798-...`.
- GuruRMM build server 172.16.3.30; downloads `/var/www/gururmm/downloads/`; dashboards
`/var/www/gururmm/dashboard{,-beta}` (guru-owned). Beast build host `guru@100.101.122.4`.
- Discord alert channels: #dev-alerts (private, Howard+Mike) `1509998508198068484`; #bot-alerts
`624710699771232265`. Poster: `.claude/scripts/post-bot-alert.sh` (auto-routes [RMM]/[DEV]/[DEPLOY]/
[BUILD]/[GURURMM] -> #dev-alerts).
## Commands & Outputs
- SSH: `MSYS_NO_PATHCONV=1 "C:/Program Files/PuTTY/plink.exe" -ssh -batch -pw '<pw>' root@192.168.120.177 "<cmd>"`.
- Recovery: `rmdir /var/run/libvirt/libvirt-sock`; `mount -o loop /mnt/cache/system/libvirt/libvirt.img /etc/libvirt`; `rmdir /dev/kvm; modprobe -r kvm_intel kvm; modprobe kvm_intel`; `/etc/rc.d/rc.libvirt start`; `virsh start "Windows 11"; virsh autostart "Windows 11"`.
- Key error resolved: `virNetSocketNewListenUNIX: Failed to bind '/var/run/libvirt/libvirt-sock': Address already in use` = the path was a stale DIRECTORY; `unsupported ... virt type 'kvm'` = `/dev/kvm` was a stale directory.
- Dev-alerts: `bash .claude/scripts/post-bot-alert.sh "[DEV]/[GURURMM] ..."` -> #dev-alerts (msg ids 1523773910238101595, 1523773916412383272).
## Pending / Incomplete Tasks
- **Reboot the Unraid box tonight** (Mike) — validate it comes up clean with the hijack removed
(emhttpd mounts libvirt.img, VM autostarts). Optionally enable Settings -> Syslog Server -> "Mirror
syslog to flash" first so any failure's boot log survives (Unraid syslog is RAM-only; Jun 30's
evidence had already rotated out).
- **gururmm-agent Docker container crash-loop** on the Unraid host — another session is handling.
- **RMM offline-alerting gap** — LONESTAR-VM + the Unraid host agent were offline ~6 days with no
alert; separate from the dev-alerts visibility gap (that one is the tech-action Discord channel).
- **Optional:** add a CLAUDE.md core rule to enforce firing #dev-alerts on RMM/Dev/client-machine
actions (currently only the feedback memory covers it).
- Update the Lonestar Unraid wiki: live IP is 192.168.120.177 (vault IP 172.16.1.188 is stale).
## Reference Information
- Vault: `clients/lonestar-electrical/unraid-server.sops.yaml`.
- Backups on the box: `/boot/config/go.bak-2026-07-06`, `/boot/config/domain.cfg.bak-2026-07-06`.
- GuruRMM shipped v0.6.77 (stable); dashboard prod backup `dashboard-20260704-204039`.
- Repo: `promotion-candidates/tmp-candidates-GURU-5070-2026-07-06.zip`; coord msg to Beast
`cfba46f7...` + todo `a676fc2b...`.
- Diag transcript saved locally: `promotion-candidates/.lonestar-unraid-diag.txt` (gitignored dotfile).