azcomputerguru/claudetools
1
0
Fork 0
Code Issues 58 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
db10206aeaf7d83bb9c0b3c2ba40cf48735c3687
claudetools/clients/lonestar-electrical/session-logs/2026-07/2026-07-06-mike-unraid-vm-recovery.md
Mike Swanson a2a9356a02 sync: auto-sync from GURU-5070 at 2026-07-06 12:39:35 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-06 12:39:35
2026-07-06 12:39:59 -07:00

9.8 KiB
Raw Blame History

Session: Lonestar Unraid VM recovery + root cause — plus GuruRMM promotions + dev-alerts gap

User

  • User: Mike Swanson (mike)
  • Machine: GURU-5070
  • Role: admin

Session Summary

Mike reported the Lonestar Electrical Unraid server's /VMs page (192.168.120.177) was "empty again" — a recurring problem on this box only, while his other Unraid boxes are stable. Connected via VPN + PuTTY/plink (system OpenSSH lacks non-interactive password auth; plink -batch with the vaulted root password works because the host key was already cached from prior PuTTY use). Diagnosed: the "Windows 11" VM (= LONESTAR-VM, a4d39a9d) domain, its 64 GiB vdisk, nvram, and swtpm are all intact; libvirt.img mounts read-only fine and contains the domain; cache SSD (Crucial MX500) SMART PASSED. The actual failure: libvirt.img was NOT mounted at /etc/libvirt, so libvirtd came up with zero domains. docker.img (same cache/system share, via /mnt/cache) mounted; libvirt (via /mnt/user FUSE) did not.

Recovered the VM live but hit several stale-state artifacts from a broken boot: /etc/libvirt was a plain dir (not a mountpoint), /var/run/libvirt/libvirt-sock was a stale empty DIRECTORY (blocking libvirtd's socket bind), and /dev/kvm was ALSO a stale directory (blocking KVM). Cleared each, manually mounted the image, started libvirtd via rc.libvirt, loaded KVM modules + recreated /dev/kvm, and started "Windows 11" — now running with autostart enabled.

Mike then asked to multi-AI the DURABLE fix ("you've done all this before and it's not clean"). Briefed Gemini 3.1 Pro (agy) + Grok. Gemini gave a detailed answer but had two wrong assumptions that live verification caught: rc.domain does not exist on 7.1.4, and this box runs MONOLITHIC libvirtd (not modular). Grok returned only a preamble (subagent-mode flakiness), no usable answer. The real root cause came from reading /boot/config/go: a leftover hand-hack labeled "LibVirt Fix (final 2026-06-10)" that, every boot (sleep 5), writes /etc/libvirt/*.conf into the UNMOUNTED /etc/libvirt and manually starts libvirtd BEFORE emhttpd can mount libvirt.img — hijacking libvirt so emhttpd's mount then fails and the VMs vanish. A prior "fix" was the cause; that is why it only affects this box and why past repairs never stayed clean.

Applied the durable fix: rewrote /boot/config/go to remove the hijack (stock Unraid libvirt boot; backup go.bak-2026-07-06), and pinned IMAGE_FILE to /mnt/cache/... (bypass the FUSE path; backup domain.cfg.bak-2026-07-06). Verified the image's own qemu.conf is self-sufficient (VM runs under it). Per Mike, hand-patched to get the VM up NOW; clean reboot scheduled tonight to reach the proper stock state. Earlier in the session also: promoted GuruRMM agent 0.6.77 beta->stable (flipped .channel files as guru) and dashboard beta->prod (via promote-dashboard.sh, live), and zipped 21 GURU-5070 .claude/tmp/ promotion candidates into promotion-candidates/ + coord-messaged Beast. Finally: Mike flagged that all this client-machine + RMM/Dev work should have fired #dev-alerts Discord posts (Howard+Mike watch that channel for live tech visibility) — posted retroactive alerts and made it a standing rule.

Key Decisions

  • Root cause = the /boot/config/go libvirt hijack, not storage/FUSE. Removing the leftover 2026-06-10 hand-hack is THE durable fix (let Unraid boot libvirt the stock way via emhttpd).
  • Hand-patch now, clean reboot tonight (Mike's call) — the piecemeal recovery is emhttpd-unaware; tonight's reboot with the hijack removed resets to the proper stock init and validates the fix.
  • Kept IMAGE_FILE -> /mnt/cache as belt-and-suspenders even though the go-hack was the primary cause (same underlying file; direct pool path avoids any FUSE-timing edge).
  • plink -batch over system ssh — no sshpass on GURU-5070; plink -batch -pw works with the cached host key (Git-Bash echo y | does NOT reach plink's console prompt → hangs).
  • Fire #dev-alerts for client/RMM work going forward — saved as a feedback memory (rule) since no CLAUDE.md rule enforces it; a real auto-hook can't tell "this SSH is a client box," so it's a judgment rule.

Problems Encountered

  • /VMs empty / libvirt.img not mounted — root cause the /boot/config/go hijack (above); removed it. Immediate recovery: mount image + clear stale sock/kvm dirs + rc.libvirt start + virsh start.
  • Stale directories where nodes/sockets belong — /var/run/libvirt/libvirt-sock and /dev/kvm were empty DIRECTORIES (from the broken boot), blocking libvirtd socket + KVM. rmdir + let module recreate /dev/kvm (crw 10,232); rc.libvirt then created the proper libvirt-sock socket.
  • Gemini wrong on rc.domain + modular libvirt — caught by verifying ls /etc/rc.d/rc.domain (absent) and rc.libvirt (uses monolithic libvirtd) BEFORE executing. Grok returned no usable answer.
  • Git-Bash mangling — remote /etc/... paths mangled to C:/Program; fixed with MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*'. Also the /tmp write hook blocked a /tmp/cand.txt.
  • No #dev-alerts fired for the day's client/RMM/Dev work (Mike's correction) — posted retroactive alerts, saved a feedback memory, logged the correction.

Configuration Changes

Lonestar Unraid (192.168.120.177), live:

  • /boot/config/go — REWROTE: removed the "LibVirt Fix (final 2026-06-10)" hijack block; kept emhttp start + XDG_CACHE_HOME. Backup /boot/config/go.bak-2026-07-06.
  • /boot/config/domain.cfg — IMAGE_FILE /mnt/user/system/libvirt/libvirt.img -> /mnt/cache/system/libvirt/libvirt.img. Backup /boot/config/domain.cfg.bak-2026-07-06.
  • Runtime (reset by tonight's reboot): manually mounted libvirt.img at /etc/libvirt; recreated /dev/kvm; started libvirtd via rc.libvirt; virsh start "Windows 11" + virsh autostart. GuruRMM (build server 172.16.3.30, as guru):
  • /var/www/gururmm/downloads/*0.6.77*.channel -> "stable" (agent promotion).
  • Dashboard beta -> prod via promote-dashboard.sh (backup dashboard-20260704-204039). ClaudeTools repo:
  • NEW promotion-candidates/tmp-candidates-GURU-5070-2026-07-06.zip + README.md (for Beast).
  • NEW .claude/memory/feedback_fire_dev_alerts_for_client_work.md + MEMORY.md index.

Credentials & Secrets

  • Lonestar Unraid root (used this session): user root, password Gptf*77ttb123!@#-lonestar — ALREADY vaulted at clients/lonestar-electrical/unraid-server.sops.yaml (fields credentials.root_password, credentials.ssh_user). No new secrets created. Note the vault entry's ip: 172.16.1.188 is STALE (from when the box was at the ACG office); live IP is 192.168.120.177.

Infrastructure & Servers

  • Lonestar Unraid Tower — 192.168.120.177 (Warren LAN 192.168.120.0/24, reached via client VPN). Unraid 7.1.4, kernel 6.12.24, libvirt 11.2.0 (monolithic libvirtd). Array: 2 XFS data disks + parity (STARTED). Cache: single btrfs on Crucial MX500 500GB (/dev/sdb1). No UPS active. VM "Windows 11" (= LONESTAR-VM, agent a4d39a9d, 192.168.120.197), vdisk /mnt/user/domains/Windows 11/vdisk1.img. Only Docker container: gururmm-agent (crash-looping -> host RMM agent offline; another session handling it). RMM agent IDs seen: host 00b48379-... (offline), vault-noted e827f798-....
  • GuruRMM build server 172.16.3.30; downloads /var/www/gururmm/downloads/; dashboards /var/www/gururmm/dashboard{,-beta} (guru-owned). Beast build host guru@100.101.122.4.
  • Discord alert channels: #dev-alerts (private, Howard+Mike) 1509998508198068484; #bot-alerts 624710699771232265. Poster: .claude/scripts/post-bot-alert.sh (auto-routes [RMM]/[DEV]/[DEPLOY]/ [BUILD]/[GURURMM] -> #dev-alerts).

Commands & Outputs

  • SSH: MSYS_NO_PATHCONV=1 "C:/Program Files/PuTTY/plink.exe" -ssh -batch -pw '<pw>' root@192.168.120.177 "<cmd>".
  • Recovery: rmdir /var/run/libvirt/libvirt-sock; mount -o loop /mnt/cache/system/libvirt/libvirt.img /etc/libvirt; rmdir /dev/kvm; modprobe -r kvm_intel kvm; modprobe kvm_intel; /etc/rc.d/rc.libvirt start; virsh start "Windows 11"; virsh autostart "Windows 11".
  • Key error resolved: virNetSocketNewListenUNIX: Failed to bind '/var/run/libvirt/libvirt-sock': Address already in use = the path was a stale DIRECTORY; unsupported ... virt type 'kvm' = /dev/kvm was a stale directory.
  • Dev-alerts: bash .claude/scripts/post-bot-alert.sh "[DEV]/[GURURMM] ..." -> #dev-alerts (msg ids 1523773910238101595, 1523773916412383272).

Pending / Incomplete Tasks

  • Reboot the Unraid box tonight (Mike) — validate it comes up clean with the hijack removed (emhttpd mounts libvirt.img, VM autostarts). Optionally enable Settings -> Syslog Server -> "Mirror syslog to flash" first so any failure's boot log survives (Unraid syslog is RAM-only; Jun 30's evidence had already rotated out).
  • gururmm-agent Docker container crash-loop on the Unraid host — another session is handling.
  • RMM offline-alerting gap — LONESTAR-VM + the Unraid host agent were offline ~6 days with no alert; separate from the dev-alerts visibility gap (that one is the tech-action Discord channel).
  • Optional: add a CLAUDE.md core rule to enforce firing #dev-alerts on RMM/Dev/client-machine actions (currently only the feedback memory covers it).
  • Update the Lonestar Unraid wiki: live IP is 192.168.120.177 (vault IP 172.16.1.188 is stale).

Reference Information

  • Vault: clients/lonestar-electrical/unraid-server.sops.yaml.
  • Backups on the box: /boot/config/go.bak-2026-07-06, /boot/config/domain.cfg.bak-2026-07-06.
  • GuruRMM shipped v0.6.77 (stable); dashboard prod backup dashboard-20260704-204039.
  • Repo: promotion-candidates/tmp-candidates-GURU-5070-2026-07-06.zip; coord msg to Beast cfba46f7... + todo a676fc2b....
  • Diag transcript saved locally: promotion-candidates/.lonestar-unraid-diag.txt (gitignored dotfile).
Reference in New Issue View Git Blame Copy Permalink