## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-BEAST-ROG
- **Role:** admin

## Session Summary

Continued from the VWP M365 remediation into a broad VWP (Valley Wide Plastering) infrastructure day plus
a GuruRMM agent-build discovery. First, the Lonestar Electrical Unraid GuruRMM agent was found offline; root
cause is a **glibc mismatch** — agent v0.6.59 is linked against GLIBC_2.39 but the agent's Docker container
is Debian 12 (glibc 2.36), so the binary won't load. This is a GuruRMM agent build regression that puts
every Linux agent on Debian-12/older-glibc hosts at risk (Lonestar's box dropped off 2026-07-05).

Checked VWP server-room temperature via out-of-band controllers: the Dell R640 (VWP-QBS) iDRAC inlet read
27C (~81F, warm/top of ASHRAE range but healthy); the HP DL360 Gen10 (ESXi host) iLO showed all zones
26-40C. Then checked the ESXi host (192.168.3.24) and found **all 9 VMs powered off** because the **ESXi
license had expired** (reverted to 00000-00000-... after a config/NVRAM loss). Recovered it over SSH:
restarted hostd, reset the evaluation license to a fresh 60 days, powered on all 9 VMs (confirmed running),
and — per Mike's direction — installed a persistent monthly **auto-reset cron** (reset script on the
"Tesst" VMFS-6 datastore, re-installed on boot via rc.local.d + auto-backup). SSH from Windows required a
compiled-askpass workaround (Posh-SSH couldn't install; Server 2025 strict TLS; ESXi uses
keyboard-interactive auth).

Mike confirmed Broadcom killed SMB/free-ESXi licensing, so the plan is to **migrate the ESXi VMs to
Hyper-V** (VWP-HYPERV1). Assessed capacity: VWP-HYPERV1 has 112 vCPU / 255 GB RAM / ~10 TB free vs ~8.6 TB
of ESXi VM storage — fits. Began a proper server inventory to find redundancy. AD assessment showed
**VWP_AD_Srvr (VWP_ADSRVR, 192.168.0.25) is a live DC holding the Schema + Domain-Naming FSMO roles**, so
retiring it requires FSMO transfer to VWP-DC1 then graceful demotion (not a power-off). RMM coverage is
12/19 servers, and the 12 with agents are exactly the live ones (the 7 without map to stale/dead AD
objects). Compared SERVER19 (live 2012 R2 RDS server) vs WIN-SERVER19-2 (a dead, abandoned 2019-eval
rebuild) — confirmed SERVER19 is the keeper and WIN-SERVER19-2 is a stale AD object to clean up.

Finally, VWP-QBS (physical PowerEdge R640, the QuickBooks + RDS-license server) had been offline since
03:30. Its iDRAC showed hardware **On + healthy, no SEL faults**, but the OS was fully unresponsive (no
ping/RDP/WinRM/SMB) — a hung OS. Captured the console screen (iDRAC's non-standard libav JPEG would not
render locally), then **force-rebooted via iDRAC**. It POSTed (BootProgress=OSRunning) but stayed
unresponsive ~15 min — most likely **autochk/chkdsk on the dirty QuickBooks volume** after the hard reset.
Monitoring for recovery; will pull event logs for the original hang cause once it is back.

## Key Decisions

- **ESXi eval-reset + persistent auto-reset cron** (Mike-authorized). Flagged the EULA-compliance issue and
  recommended the free vSphere Hypervisor license; Mike noted Broadcom discontinued SMB/free ESXi, so the
  cron is a bridge and Hyper-V migration is the real exit.
- **Reset script on the VMFS datastore, not the boot bank** — this host has recurring NVRAM/bootbank
  corruption, so the datastore is the durable location; only the boot-time re-install hook lives in
  local.sh.
- **Compiled a C# askpass.exe on the fly** for non-interactive Windows->ESXi SSH (Posh-SSH install failed:
  no PSGallery; SSH_ASKPASS .cmd not runnable). Plus `PreferredAuthentications=keyboard-interactive` since
  ESXi does not offer plain "password".
- **iDRAC/iLO queried from a Server 2019 box (VWP-DC1) via curl** — Server 2025 (VWP-HYPERV1) strict TLS
  cannot negotiate with the BMCs' older ciphers.
- **VWP_AD_Srvr retirement = FSMO transfer + graceful demotion**, not power-off (it holds Schema +
  Domain-Naming master and is a live GC).
- **Force-rebooted VWP-QBS** — the OS was hung and unresponsive; a hard reset via iDRAC was the only option
  (accepted the chkdsk-on-dirty-volume consequence).

## Problems Encountered

- **ESXi license expired** (00000-00000-...) after config/NVRAM loss -> all VMs off, power-on returned
  vim.fault.LicenseExpired. Fixed by resetting the eval (cp /etc/vmware/.#license.cfg -> license.cfg +
  hostd restart) -> expirationHours=1440.
- **Posh-SSH install failed** (no PSGallery/internet) -> compiled askpass.exe via csc + Windows OpenSSH.
- **ESXi SSH "Permission denied (publickey,keyboard-interactive)"** -> it does not offer "password"; used
  `-o PreferredAuthentications=keyboard-interactive,password`.
- **Server 2025 TLS** could not reach iDRAC/iLO ("underlying connection closed") -> ran from VWP-DC1 (2019)
  and/or curl.
- **ps-encoded size limits** (refuse >6000 encoded; cmd.exe 8191 command-line limit) -> split scripts,
  used the one-line `ServerCertificateValidationCallback={$true}` cert bypass, hardcoded values.
- **Datastore auto-pick grabbed BOOTBANK1** (bad) -> hardcoded the "Tesst" VMFS-6 UUID mount.
- **CLPH placeholder in local.sh** did not get sed-substituted in-script -> fixed with a hardcoded sed.
- **iDRAC screenshot** returns a minimal libav JPEG that PIL and GDI+ both reject; ExportServerScreenShot
  is flaky -> could not render the console screen locally.
- **VWP-QBS OS hung** (03:30) -> force-reboot; still recovering (likely chkdsk) at session end.

## Configuration Changes

VWP ESXi host (192.168.3.24), via SSH:
- Reset `/etc/vmware/license.cfg` from the eval template (`.#license.cfg`) — 60-day eval restored.
- Created `/vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh` (eval auto-reset).
- Added cron `10 3 1 * * <reset-eval.sh>` to `/var/spool/cron/crontabs/root`.
- Added boot re-install hook to `/etc/rc.local.d/local.sh` + ran `/sbin/auto-backup.sh`.
- Stored the reset path at `/etc/vmware/.acg-reseteval-path`.

VWP-QBS (physical R640): force reboot via iDRAC Redfish ComputerSystem.Reset ForceRestart.

ClaudeTools: this session log (no code changes this session).

## Credentials & Secrets

- **VWP ESXi root** — vault `clients/vwp/esxi.sops.yaml` field `credentials.password`. Host 192.168.3.24.
- **VWP QBS iDRAC root** — vault `clients/vwp/quickbooks-server-idrac.sops.yaml` `credentials.password`.
  iDRAC 192.168.3.189 (R640, Service Tag C84TTQ2).
- **VWP HP iLO Administrator** — vault `clients/vwp/hp-ilo.sops.yaml` `credentials.password`. iLO
  172.16.9.125.
- **Lonestar Unraid root** — `Gptf*77ttb123!@#-lonestar`, vault `clients/lonestar-electrical/unraid-server.sops.yaml`
  `credentials.root_password` (already vaulted).
- ESXi license: none on record; was expired eval — reset to eval (60d). No paid/free key available
  (Broadcom discontinued SMB/free ESXi).

## Infrastructure & Servers

- **VWP domain:** VWP.US. **DCs:** VWP_ADSRVR (192.168.0.25 — Schema + Domain-Naming FSMO, GC, ESXi VM),
  VWP-DC1 (172.16.9.2 — PDC/RID/Infrastructure FSMO, GC, Hyper-V VM). Both Server 2019, replicating clean.
- **VWP ESXi** — HP ProLiant DL360 Gen10 (SN MXQ80400X4), ESXi 8.0, mgmt 192.168.3.24 (VLAN 99), SSH :22,
  iLO 172.16.9.125. Datastore **"Tesst"** VMFS-6, UUID `5869d65c-c5ff4b86-13c0-9009d0143403`, 14 TB / 8.6 TB
  used. 9 VMs (all now powered on): Server-97, WINFilrSrvr, VWP-SERVER, SERVER 19, VWIN7-2-PC, VWP_AD_Srvr,
  VWP-FIN, WIN-Acct, WIN-AD2.
- **VWP-HYPERV1** — 172.16.9.184, Server 2025, Hyper-V host: 112 logical CPU, 255 GB RAM, C:\VHD ~10 TB free.
  Runs VWP-DC1 + VWP-FILES.
- **VWP-QBS** — physical PowerEdge R640, iDRAC 192.168.3.189, OS IP 172.16.9.169, Server 2022. QuickBooks
  server + RDS license server (`vwp-qbs.vwp.us`). Hung 03:30, force-rebooted, recovering.
- RMM agents (ids): VWP-HYPERV1 `bdc3e142-1eb3-4034-8ac2-300f46dbc1d9`, VWP-DC1
  `8eefbba6-28cf-4e3f-8fda-11900e0ac302`, VWP-QBS `f3386f0a-b3ee-417e-ace9-995cc1d0662b`.
- Lonestar Unraid agent `bfcfbc739d23` (id `00b48379-ec09-4396-9d8f-31a1ee8b4ce3`), offline (glibc).

## Commands & Outputs

- ESXi eval reset: `cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg; /etc/init.d/hostd restart` ->
  license --show `expirationHours=1440`.
- Non-interactive Win->ESXi SSH: compile `askpass.exe` (`Add-Type ... -OutputType ConsoleApplication`
  printing env `XPW`), set `SSH_ASKPASS`/`SSH_ASKPASS_REQUIRE=force`, `ssh -o
  PreferredAuthentications=keyboard-interactive,password`.
- iDRAC/BMC Redfish must run from a 2019 box via curl (`curl.exe -sk -H "Authorization: Basic <b64>"`) —
  Server 2025 .NET TLS fails.
- VWP-QBS iDRAC: PowerState=On, Health=OK, BootProgress=OSRunning, SEL clean (last event Feb 2025).
- ESXi power-on before reset: `Power on failed: (vim.fault.LicenseExpired) ... licenseKey = "Invalid
  License: 00000-00000-00000-00000-00000"`.

## Pending / Incomplete Tasks

- **VWP-QBS (IN PROGRESS):** monitor for recovery (likely chkdsk); once up, verify QuickBooks + RDS
  licensing services, pull System/Application event logs around 03:30 for the original hang root cause. If
  still down in ~45 min or console shows a true hang, treat as unstable hardware/OS.
- **VWP_AD_Srvr retirement:** pre-flight (DNS/DHCP not handing out .0.25; dcdiag VWP-DC1) -> transfer Schema
  + Domain-Naming FSMO to VWP-DC1 -> graceful demote -> verify -> delete VM. Consider a 2nd Hyper-V DC for
  redundancy.
- **Server redundancy cleanup:** stale AD objects to remove — WIN-SERVER19-2, WIN-SVR-1, VWP-AD2,
  WIN-FIN2-OLD, SERVER3, VWP-BUSVR. Verify ORDERS2 (recent logon, no RMM — gap or decommission).
- **VMware -> Hyper-V migration:** StarWind V2V Converter; finalize keep-list, pick pilot VM. Eval cron is
  the bridge until migrated.
- **GuruRMM Linux agent glibc regression:** rebuild agent static/musl or target glibc 2.36; scan fleet for
  other offline Linux agents; redeploy Lonestar Unraid container; update its vault agent_id (stale
  e827f798 -> 00b48379).
- Identity-tool hardening (from prior save) still pending: consent-audit misattribution + friction log.

## Reference Information

- iDRAC screenshot Redfish action: `POST /redfish/v1/Dell/Managers/iDRAC.Embedded.1/DellLCService/Actions/
  DellLCService.ExportServerScreenShot` body `{"FileType":"ServerScreenShot"}` (returns base64 JPEG).
- ESXi reset action: `POST /redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset`
  `{"ResetType":"ForceRestart"}`.
- Eval-reset cron: `10 3 1 * * /vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh`.
- Prior session log: `session-logs/2026-07/2026-07-06-mike-vwp-m365-remediation-gururmm-console.md`.
