From da2aa609901f9b925241781afbfe090061435130 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Tue, 7 Jul 2026 06:39:15 -0700 Subject: [PATCH] sync: auto-sync from GURU-BEAST-ROG at 2026-07-07 06:38:06 Author: Mike Swanson Machine: GURU-BEAST-ROG Timestamp: 2026-07-07 06:38:06 --- .../2026-07-07-mike-vwp-infra-esxi-qbs.md | 154 ++++++++++++++++++ errorlog.md | 2 + 2 files changed, 156 insertions(+) create mode 100644 clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-infra-esxi-qbs.md diff --git a/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-infra-esxi-qbs.md b/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-infra-esxi-qbs.md new file mode 100644 index 00000000..19e872e5 --- /dev/null +++ b/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-infra-esxi-qbs.md @@ -0,0 +1,154 @@ +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-BEAST-ROG +- **Role:** admin + +## Session Summary + +Continued from the VWP M365 remediation into a broad VWP (Valley Wide Plastering) infrastructure day plus +a GuruRMM agent-build discovery. First, the Lonestar Electrical Unraid GuruRMM agent was found offline; root +cause is a **glibc mismatch** — agent v0.6.59 is linked against GLIBC_2.39 but the agent's Docker container +is Debian 12 (glibc 2.36), so the binary won't load. This is a GuruRMM agent build regression that puts +every Linux agent on Debian-12/older-glibc hosts at risk (Lonestar's box dropped off 2026-07-05). + +Checked VWP server-room temperature via out-of-band controllers: the Dell R640 (VWP-QBS) iDRAC inlet read +27C (~81F, warm/top of ASHRAE range but healthy); the HP DL360 Gen10 (ESXi host) iLO showed all zones +26-40C. Then checked the ESXi host (192.168.3.24) and found **all 9 VMs powered off** because the **ESXi +license had expired** (reverted to 00000-00000-... after a config/NVRAM loss). Recovered it over SSH: +restarted hostd, reset the evaluation license to a fresh 60 days, powered on all 9 VMs (confirmed running), +and — per Mike's direction — installed a persistent monthly **auto-reset cron** (reset script on the +"Tesst" VMFS-6 datastore, re-installed on boot via rc.local.d + auto-backup). SSH from Windows required a +compiled-askpass workaround (Posh-SSH couldn't install; Server 2025 strict TLS; ESXi uses +keyboard-interactive auth). + +Mike confirmed Broadcom killed SMB/free-ESXi licensing, so the plan is to **migrate the ESXi VMs to +Hyper-V** (VWP-HYPERV1). Assessed capacity: VWP-HYPERV1 has 112 vCPU / 255 GB RAM / ~10 TB free vs ~8.6 TB +of ESXi VM storage — fits. Began a proper server inventory to find redundancy. AD assessment showed +**VWP_AD_Srvr (VWP_ADSRVR, 192.168.0.25) is a live DC holding the Schema + Domain-Naming FSMO roles**, so +retiring it requires FSMO transfer to VWP-DC1 then graceful demotion (not a power-off). RMM coverage is +12/19 servers, and the 12 with agents are exactly the live ones (the 7 without map to stale/dead AD +objects). Compared SERVER19 (live 2012 R2 RDS server) vs WIN-SERVER19-2 (a dead, abandoned 2019-eval +rebuild) — confirmed SERVER19 is the keeper and WIN-SERVER19-2 is a stale AD object to clean up. + +Finally, VWP-QBS (physical PowerEdge R640, the QuickBooks + RDS-license server) had been offline since +03:30. Its iDRAC showed hardware **On + healthy, no SEL faults**, but the OS was fully unresponsive (no +ping/RDP/WinRM/SMB) — a hung OS. Captured the console screen (iDRAC's non-standard libav JPEG would not +render locally), then **force-rebooted via iDRAC**. It POSTed (BootProgress=OSRunning) but stayed +unresponsive ~15 min — most likely **autochk/chkdsk on the dirty QuickBooks volume** after the hard reset. +Monitoring for recovery; will pull event logs for the original hang cause once it is back. + +## Key Decisions + +- **ESXi eval-reset + persistent auto-reset cron** (Mike-authorized). Flagged the EULA-compliance issue and + recommended the free vSphere Hypervisor license; Mike noted Broadcom discontinued SMB/free ESXi, so the + cron is a bridge and Hyper-V migration is the real exit. +- **Reset script on the VMFS datastore, not the boot bank** — this host has recurring NVRAM/bootbank + corruption, so the datastore is the durable location; only the boot-time re-install hook lives in + local.sh. +- **Compiled a C# askpass.exe on the fly** for non-interactive Windows->ESXi SSH (Posh-SSH install failed: + no PSGallery; SSH_ASKPASS .cmd not runnable). Plus `PreferredAuthentications=keyboard-interactive` since + ESXi does not offer plain "password". +- **iDRAC/iLO queried from a Server 2019 box (VWP-DC1) via curl** — Server 2025 (VWP-HYPERV1) strict TLS + cannot negotiate with the BMCs' older ciphers. +- **VWP_AD_Srvr retirement = FSMO transfer + graceful demotion**, not power-off (it holds Schema + + Domain-Naming master and is a live GC). +- **Force-rebooted VWP-QBS** — the OS was hung and unresponsive; a hard reset via iDRAC was the only option + (accepted the chkdsk-on-dirty-volume consequence). + +## Problems Encountered + +- **ESXi license expired** (00000-00000-...) after config/NVRAM loss -> all VMs off, power-on returned + vim.fault.LicenseExpired. Fixed by resetting the eval (cp /etc/vmware/.#license.cfg -> license.cfg + + hostd restart) -> expirationHours=1440. +- **Posh-SSH install failed** (no PSGallery/internet) -> compiled askpass.exe via csc + Windows OpenSSH. +- **ESXi SSH "Permission denied (publickey,keyboard-interactive)"** -> it does not offer "password"; used + `-o PreferredAuthentications=keyboard-interactive,password`. +- **Server 2025 TLS** could not reach iDRAC/iLO ("underlying connection closed") -> ran from VWP-DC1 (2019) + and/or curl. +- **ps-encoded size limits** (refuse >6000 encoded; cmd.exe 8191 command-line limit) -> split scripts, + used the one-line `ServerCertificateValidationCallback={$true}` cert bypass, hardcoded values. +- **Datastore auto-pick grabbed BOOTBANK1** (bad) -> hardcoded the "Tesst" VMFS-6 UUID mount. +- **CLPH placeholder in local.sh** did not get sed-substituted in-script -> fixed with a hardcoded sed. +- **iDRAC screenshot** returns a minimal libav JPEG that PIL and GDI+ both reject; ExportServerScreenShot + is flaky -> could not render the console screen locally. +- **VWP-QBS OS hung** (03:30) -> force-reboot; still recovering (likely chkdsk) at session end. + +## Configuration Changes + +VWP ESXi host (192.168.3.24), via SSH: +- Reset `/etc/vmware/license.cfg` from the eval template (`.#license.cfg`) — 60-day eval restored. +- Created `/vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh` (eval auto-reset). +- Added cron `10 3 1 * * ` to `/var/spool/cron/crontabs/root`. +- Added boot re-install hook to `/etc/rc.local.d/local.sh` + ran `/sbin/auto-backup.sh`. +- Stored the reset path at `/etc/vmware/.acg-reseteval-path`. + +VWP-QBS (physical R640): force reboot via iDRAC Redfish ComputerSystem.Reset ForceRestart. + +ClaudeTools: this session log (no code changes this session). + +## Credentials & Secrets + +- **VWP ESXi root** — vault `clients/vwp/esxi.sops.yaml` field `credentials.password`. Host 192.168.3.24. +- **VWP QBS iDRAC root** — vault `clients/vwp/quickbooks-server-idrac.sops.yaml` `credentials.password`. + iDRAC 192.168.3.189 (R640, Service Tag C84TTQ2). +- **VWP HP iLO Administrator** — vault `clients/vwp/hp-ilo.sops.yaml` `credentials.password`. iLO + 172.16.9.125. +- **Lonestar Unraid root** — `Gptf*77ttb123!@#-lonestar`, vault `clients/lonestar-electrical/unraid-server.sops.yaml` + `credentials.root_password` (already vaulted). +- ESXi license: none on record; was expired eval — reset to eval (60d). No paid/free key available + (Broadcom discontinued SMB/free ESXi). + +## Infrastructure & Servers + +- **VWP domain:** VWP.US. **DCs:** VWP_ADSRVR (192.168.0.25 — Schema + Domain-Naming FSMO, GC, ESXi VM), + VWP-DC1 (172.16.9.2 — PDC/RID/Infrastructure FSMO, GC, Hyper-V VM). Both Server 2019, replicating clean. +- **VWP ESXi** — HP ProLiant DL360 Gen10 (SN MXQ80400X4), ESXi 8.0, mgmt 192.168.3.24 (VLAN 99), SSH :22, + iLO 172.16.9.125. Datastore **"Tesst"** VMFS-6, UUID `5869d65c-c5ff4b86-13c0-9009d0143403`, 14 TB / 8.6 TB + used. 9 VMs (all now powered on): Server-97, WINFilrSrvr, VWP-SERVER, SERVER 19, VWIN7-2-PC, VWP_AD_Srvr, + VWP-FIN, WIN-Acct, WIN-AD2. +- **VWP-HYPERV1** — 172.16.9.184, Server 2025, Hyper-V host: 112 logical CPU, 255 GB RAM, C:\VHD ~10 TB free. + Runs VWP-DC1 + VWP-FILES. +- **VWP-QBS** — physical PowerEdge R640, iDRAC 192.168.3.189, OS IP 172.16.9.169, Server 2022. QuickBooks + server + RDS license server (`vwp-qbs.vwp.us`). Hung 03:30, force-rebooted, recovering. +- RMM agents (ids): VWP-HYPERV1 `bdc3e142-1eb3-4034-8ac2-300f46dbc1d9`, VWP-DC1 + `8eefbba6-28cf-4e3f-8fda-11900e0ac302`, VWP-QBS `f3386f0a-b3ee-417e-ace9-995cc1d0662b`. +- Lonestar Unraid agent `bfcfbc739d23` (id `00b48379-ec09-4396-9d8f-31a1ee8b4ce3`), offline (glibc). + +## Commands & Outputs + +- ESXi eval reset: `cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg; /etc/init.d/hostd restart` -> + license --show `expirationHours=1440`. +- Non-interactive Win->ESXi SSH: compile `askpass.exe` (`Add-Type ... -OutputType ConsoleApplication` + printing env `XPW`), set `SSH_ASKPASS`/`SSH_ASKPASS_REQUIRE=force`, `ssh -o + PreferredAuthentications=keyboard-interactive,password`. +- iDRAC/BMC Redfish must run from a 2019 box via curl (`curl.exe -sk -H "Authorization: Basic "`) — + Server 2025 .NET TLS fails. +- VWP-QBS iDRAC: PowerState=On, Health=OK, BootProgress=OSRunning, SEL clean (last event Feb 2025). +- ESXi power-on before reset: `Power on failed: (vim.fault.LicenseExpired) ... licenseKey = "Invalid + License: 00000-00000-00000-00000-00000"`. + +## Pending / Incomplete Tasks + +- **VWP-QBS (IN PROGRESS):** monitor for recovery (likely chkdsk); once up, verify QuickBooks + RDS + licensing services, pull System/Application event logs around 03:30 for the original hang root cause. If + still down in ~45 min or console shows a true hang, treat as unstable hardware/OS. +- **VWP_AD_Srvr retirement:** pre-flight (DNS/DHCP not handing out .0.25; dcdiag VWP-DC1) -> transfer Schema + + Domain-Naming FSMO to VWP-DC1 -> graceful demote -> verify -> delete VM. Consider a 2nd Hyper-V DC for + redundancy. +- **Server redundancy cleanup:** stale AD objects to remove — WIN-SERVER19-2, WIN-SVR-1, VWP-AD2, + WIN-FIN2-OLD, SERVER3, VWP-BUSVR. Verify ORDERS2 (recent logon, no RMM — gap or decommission). +- **VMware -> Hyper-V migration:** StarWind V2V Converter; finalize keep-list, pick pilot VM. Eval cron is + the bridge until migrated. +- **GuruRMM Linux agent glibc regression:** rebuild agent static/musl or target glibc 2.36; scan fleet for + other offline Linux agents; redeploy Lonestar Unraid container; update its vault agent_id (stale + e827f798 -> 00b48379). +- Identity-tool hardening (from prior save) still pending: consent-audit misattribution + friction log. + +## Reference Information + +- iDRAC screenshot Redfish action: `POST /redfish/v1/Dell/Managers/iDRAC.Embedded.1/DellLCService/Actions/ + DellLCService.ExportServerScreenShot` body `{"FileType":"ServerScreenShot"}` (returns base64 JPEG). +- ESXi reset action: `POST /redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset` + `{"ResetType":"ForceRestart"}`. +- Eval-reset cron: `10 3 1 * * /vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh`. +- Prior session log: `session-logs/2026-07/2026-07-06-mike-vwp-m365-remediation-gururmm-console.md`. diff --git a/errorlog.md b/errorlog.md index 6d5e8634..4ed69e17 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,8 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-07 | GURU-BEAST-ROG | rmm/ilo-redfish | [friction] embedded double-quotes in curl.exe -d JSON body stripped by PowerShell native arg passing (MalformedJSON from iLO); fix = write body to temp file via [char]34, --data-binary @file [ctx: ref=feedback_windows_quote_stripping] + 2026-07-07 | Howard-Home | rmm/windows-edition-upgrade | [correction] hand-rolled changepk Home->Pro instead of following reference_windows_edition_upgrade_rmm; changepk does NOT auto-reboot (flips edition live + sets RebootPending) so the long offline was the flaky RMM agent, not a reboot in progress - must trigger Restart-Computer -Force explicitly [ctx: host=DALLAS ref=reference_windows_edition_upgrade_rmm] 2026-07-06 | Howard-Home | ssh/gururmm-server | [friction] vault infrastructure/gururmm-server.sops.yaml lists username/password + 'Sudo password same as SSH', but 172.16.3.30 SSH is publickey-only (password auth disabled) and no fleet key authenticates guru/root/mike/howard from workstations. Watching build logs / running promote-dashboard.sh must be done on the box. Fix: add SSH pubkey to guru@.30 or correct the vault note. [ctx: ref=infrastructure/gururmm-server.sops.yaml host=172.16.3.30]