sync: auto-sync from GURU-BEAST-ROG at 2026-07-07 06:38:06

Author: Mike Swanson
Machine: GURU-BEAST-ROG
Timestamp: 2026-07-07 06:38:06

clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-infra-esxi-qbs.md

@@ -0,0 +1,154 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-BEAST-ROG
+- **Role:** admin
+
+## Session Summary
+
+Continued from the VWP M365 remediation into a broad VWP (Valley Wide Plastering) infrastructure day plus
+a GuruRMM agent-build discovery. First, the Lonestar Electrical Unraid GuruRMM agent was found offline; root
+cause is a **glibc mismatch** — agent v0.6.59 is linked against GLIBC_2.39 but the agent's Docker container
+is Debian 12 (glibc 2.36), so the binary won't load. This is a GuruRMM agent build regression that puts
+every Linux agent on Debian-12/older-glibc hosts at risk (Lonestar's box dropped off 2026-07-05).
+
+Checked VWP server-room temperature via out-of-band controllers: the Dell R640 (VWP-QBS) iDRAC inlet read
+27C (~81F, warm/top of ASHRAE range but healthy); the HP DL360 Gen10 (ESXi host) iLO showed all zones
+26-40C. Then checked the ESXi host (192.168.3.24) and found **all 9 VMs powered off** because the **ESXi
+license had expired** (reverted to 00000-00000-... after a config/NVRAM loss). Recovered it over SSH:
+restarted hostd, reset the evaluation license to a fresh 60 days, powered on all 9 VMs (confirmed running),
+and — per Mike's direction — installed a persistent monthly **auto-reset cron** (reset script on the
+"Tesst" VMFS-6 datastore, re-installed on boot via rc.local.d + auto-backup). SSH from Windows required a
+compiled-askpass workaround (Posh-SSH couldn't install; Server 2025 strict TLS; ESXi uses
+keyboard-interactive auth).
+
+Mike confirmed Broadcom killed SMB/free-ESXi licensing, so the plan is to **migrate the ESXi VMs to
+Hyper-V** (VWP-HYPERV1). Assessed capacity: VWP-HYPERV1 has 112 vCPU / 255 GB RAM / ~10 TB free vs ~8.6 TB
+of ESXi VM storage — fits. Began a proper server inventory to find redundancy. AD assessment showed
+**VWP_AD_Srvr (VWP_ADSRVR, 192.168.0.25) is a live DC holding the Schema + Domain-Naming FSMO roles**, so
+retiring it requires FSMO transfer to VWP-DC1 then graceful demotion (not a power-off). RMM coverage is
+12/19 servers, and the 12 with agents are exactly the live ones (the 7 without map to stale/dead AD
+objects). Compared SERVER19 (live 2012 R2 RDS server) vs WIN-SERVER19-2 (a dead, abandoned 2019-eval
+rebuild) — confirmed SERVER19 is the keeper and WIN-SERVER19-2 is a stale AD object to clean up.
+
+Finally, VWP-QBS (physical PowerEdge R640, the QuickBooks + RDS-license server) had been offline since
+03:30. Its iDRAC showed hardware **On + healthy, no SEL faults**, but the OS was fully unresponsive (no
+ping/RDP/WinRM/SMB) — a hung OS. Captured the console screen (iDRAC's non-standard libav JPEG would not
+render locally), then **force-rebooted via iDRAC**. It POSTed (BootProgress=OSRunning) but stayed
+unresponsive ~15 min — most likely **autochk/chkdsk on the dirty QuickBooks volume** after the hard reset.
+Monitoring for recovery; will pull event logs for the original hang cause once it is back.
+
+## Key Decisions
+
+- **ESXi eval-reset + persistent auto-reset cron** (Mike-authorized). Flagged the EULA-compliance issue and
+  recommended the free vSphere Hypervisor license; Mike noted Broadcom discontinued SMB/free ESXi, so the
+  cron is a bridge and Hyper-V migration is the real exit.
+- **Reset script on the VMFS datastore, not the boot bank** — this host has recurring NVRAM/bootbank
+  corruption, so the datastore is the durable location; only the boot-time re-install hook lives in
+  local.sh.
+- **Compiled a C# askpass.exe on the fly** for non-interactive Windows->ESXi SSH (Posh-SSH install failed:
+  no PSGallery; SSH_ASKPASS .cmd not runnable). Plus `PreferredAuthentications=keyboard-interactive` since
+  ESXi does not offer plain "password".
+- **iDRAC/iLO queried from a Server 2019 box (VWP-DC1) via curl** — Server 2025 (VWP-HYPERV1) strict TLS
+  cannot negotiate with the BMCs' older ciphers.
+- **VWP_AD_Srvr retirement = FSMO transfer + graceful demotion**, not power-off (it holds Schema +
+  Domain-Naming master and is a live GC).
+- **Force-rebooted VWP-QBS** — the OS was hung and unresponsive; a hard reset via iDRAC was the only option
+  (accepted the chkdsk-on-dirty-volume consequence).
+
+## Problems Encountered
+
+- **ESXi license expired** (00000-00000-...) after config/NVRAM loss -> all VMs off, power-on returned
+  vim.fault.LicenseExpired. Fixed by resetting the eval (cp /etc/vmware/.#license.cfg -> license.cfg +
+  hostd restart) -> expirationHours=1440.
+- **Posh-SSH install failed** (no PSGallery/internet) -> compiled askpass.exe via csc + Windows OpenSSH.
+- **ESXi SSH "Permission denied (publickey,keyboard-interactive)"** -> it does not offer "password"; used
+  `-o PreferredAuthentications=keyboard-interactive,password`.
+- **Server 2025 TLS** could not reach iDRAC/iLO ("underlying connection closed") -> ran from VWP-DC1 (2019)
+  and/or curl.
+- **ps-encoded size limits** (refuse >6000 encoded; cmd.exe 8191 command-line limit) -> split scripts,
+  used the one-line `ServerCertificateValidationCallback={$true}` cert bypass, hardcoded values.
+- **Datastore auto-pick grabbed BOOTBANK1** (bad) -> hardcoded the "Tesst" VMFS-6 UUID mount.
+- **CLPH placeholder in local.sh** did not get sed-substituted in-script -> fixed with a hardcoded sed.
+- **iDRAC screenshot** returns a minimal libav JPEG that PIL and GDI+ both reject; ExportServerScreenShot
+  is flaky -> could not render the console screen locally.
+- **VWP-QBS OS hung** (03:30) -> force-reboot; still recovering (likely chkdsk) at session end.
+
+## Configuration Changes
+
+VWP ESXi host (192.168.3.24), via SSH:
+- Reset `/etc/vmware/license.cfg` from the eval template (`.#license.cfg`) — 60-day eval restored.
+- Created `/vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh` (eval auto-reset).
+- Added cron `10 3 1 * * <reset-eval.sh>` to `/var/spool/cron/crontabs/root`.
+- Added boot re-install hook to `/etc/rc.local.d/local.sh` + ran `/sbin/auto-backup.sh`.
+- Stored the reset path at `/etc/vmware/.acg-reseteval-path`.
+
+VWP-QBS (physical R640): force reboot via iDRAC Redfish ComputerSystem.Reset ForceRestart.
+
+ClaudeTools: this session log (no code changes this session).
+
+## Credentials & Secrets
+
+- **VWP ESXi root** — vault `clients/vwp/esxi.sops.yaml` field `credentials.password`. Host 192.168.3.24.
+- **VWP QBS iDRAC root** — vault `clients/vwp/quickbooks-server-idrac.sops.yaml` `credentials.password`.
+  iDRAC 192.168.3.189 (R640, Service Tag C84TTQ2).
+- **VWP HP iLO Administrator** — vault `clients/vwp/hp-ilo.sops.yaml` `credentials.password`. iLO
+  172.16.9.125.
+- **Lonestar Unraid root** — `Gptf*77ttb123!@#-lonestar`, vault `clients/lonestar-electrical/unraid-server.sops.yaml`
+  `credentials.root_password` (already vaulted).
+- ESXi license: none on record; was expired eval — reset to eval (60d). No paid/free key available
+  (Broadcom discontinued SMB/free ESXi).
+
+## Infrastructure & Servers
+
+- **VWP domain:** VWP.US. **DCs:** VWP_ADSRVR (192.168.0.25 — Schema + Domain-Naming FSMO, GC, ESXi VM),
+  VWP-DC1 (172.16.9.2 — PDC/RID/Infrastructure FSMO, GC, Hyper-V VM). Both Server 2019, replicating clean.
+- **VWP ESXi** — HP ProLiant DL360 Gen10 (SN MXQ80400X4), ESXi 8.0, mgmt 192.168.3.24 (VLAN 99), SSH :22,
+  iLO 172.16.9.125. Datastore **"Tesst"** VMFS-6, UUID `5869d65c-c5ff4b86-13c0-9009d0143403`, 14 TB / 8.6 TB
+  used. 9 VMs (all now powered on): Server-97, WINFilrSrvr, VWP-SERVER, SERVER 19, VWIN7-2-PC, VWP_AD_Srvr,
+  VWP-FIN, WIN-Acct, WIN-AD2.
+- **VWP-HYPERV1** — 172.16.9.184, Server 2025, Hyper-V host: 112 logical CPU, 255 GB RAM, C:\VHD ~10 TB free.
+  Runs VWP-DC1 + VWP-FILES.
+- **VWP-QBS** — physical PowerEdge R640, iDRAC 192.168.3.189, OS IP 172.16.9.169, Server 2022. QuickBooks
+  server + RDS license server (`vwp-qbs.vwp.us`). Hung 03:30, force-rebooted, recovering.
+- RMM agents (ids): VWP-HYPERV1 `bdc3e142-1eb3-4034-8ac2-300f46dbc1d9`, VWP-DC1
+  `8eefbba6-28cf-4e3f-8fda-11900e0ac302`, VWP-QBS `f3386f0a-b3ee-417e-ace9-995cc1d0662b`.
+- Lonestar Unraid agent `bfcfbc739d23` (id `00b48379-ec09-4396-9d8f-31a1ee8b4ce3`), offline (glibc).
+
+## Commands & Outputs
+
+- ESXi eval reset: `cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg; /etc/init.d/hostd restart` ->
+  license --show `expirationHours=1440`.
+- Non-interactive Win->ESXi SSH: compile `askpass.exe` (`Add-Type ... -OutputType ConsoleApplication`
+  printing env `XPW`), set `SSH_ASKPASS`/`SSH_ASKPASS_REQUIRE=force`, `ssh -o
+  PreferredAuthentications=keyboard-interactive,password`.
+- iDRAC/BMC Redfish must run from a 2019 box via curl (`curl.exe -sk -H "Authorization: Basic <b64>"`) —
+  Server 2025 .NET TLS fails.
+- VWP-QBS iDRAC: PowerState=On, Health=OK, BootProgress=OSRunning, SEL clean (last event Feb 2025).
+- ESXi power-on before reset: `Power on failed: (vim.fault.LicenseExpired) ... licenseKey = "Invalid
+  License: 00000-00000-00000-00000-00000"`.
+
+## Pending / Incomplete Tasks
+
+- **VWP-QBS (IN PROGRESS):** monitor for recovery (likely chkdsk); once up, verify QuickBooks + RDS
+  licensing services, pull System/Application event logs around 03:30 for the original hang root cause. If
+  still down in ~45 min or console shows a true hang, treat as unstable hardware/OS.
+- **VWP_AD_Srvr retirement:** pre-flight (DNS/DHCP not handing out .0.25; dcdiag VWP-DC1) -> transfer Schema
+  + Domain-Naming FSMO to VWP-DC1 -> graceful demote -> verify -> delete VM. Consider a 2nd Hyper-V DC for
+  redundancy.
+- **Server redundancy cleanup:** stale AD objects to remove — WIN-SERVER19-2, WIN-SVR-1, VWP-AD2,
+  WIN-FIN2-OLD, SERVER3, VWP-BUSVR. Verify ORDERS2 (recent logon, no RMM — gap or decommission).
+- **VMware -> Hyper-V migration:** StarWind V2V Converter; finalize keep-list, pick pilot VM. Eval cron is
+  the bridge until migrated.
+- **GuruRMM Linux agent glibc regression:** rebuild agent static/musl or target glibc 2.36; scan fleet for
+  other offline Linux agents; redeploy Lonestar Unraid container; update its vault agent_id (stale
+  e827f798 -> 00b48379).
+- Identity-tool hardening (from prior save) still pending: consent-audit misattribution + friction log.
+
+## Reference Information
+
+- iDRAC screenshot Redfish action: `POST /redfish/v1/Dell/Managers/iDRAC.Embedded.1/DellLCService/Actions/
+  DellLCService.ExportServerScreenShot` body `{"FileType":"ServerScreenShot"}` (returns base64 JPEG).
+- ESXi reset action: `POST /redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset`
+  `{"ResetType":"ForceRestart"}`.
+- Eval-reset cron: `10 3 1 * * /vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh`.
+- Prior session log: `session-logs/2026-07/2026-07-06-mike-vwp-m365-remediation-gururmm-console.md`.

errorlog.md

@@ -19,6 +19,8 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
 
 <!-- Append entries below this line -->
 
+2026-07-07 | GURU-BEAST-ROG | rmm/ilo-redfish | [friction] embedded double-quotes in curl.exe -d JSON body stripped by PowerShell native arg passing (MalformedJSON from iLO); fix = write body to temp file via [char]34, --data-binary @file [ctx: ref=feedback_windows_quote_stripping]
+
 2026-07-07 | Howard-Home | rmm/windows-edition-upgrade | [correction] hand-rolled changepk Home->Pro instead of following reference_windows_edition_upgrade_rmm; changepk does NOT auto-reboot (flips edition live + sets RebootPending) so the long offline was the flaky RMM agent, not a reboot in progress - must trigger Restart-Computer -Force explicitly [ctx: host=DALLAS ref=reference_windows_edition_upgrade_rmm]
 
 2026-07-06 | Howard-Home | ssh/gururmm-server | [friction] vault infrastructure/gururmm-server.sops.yaml lists username/password + 'Sudo password same as SSH', but 172.16.3.30 SSH is publickey-only (password auth disabled) and no fleet key authenticates guru/root/mike/howard from workstations. Watching build logs / running promote-dashboard.sh must be done on the box. Fix: add SSH pubkey to guru@.30 or correct the vault note. [ctx: ref=infrastructure/gururmm-server.sops.yaml host=172.16.3.30]