sync: auto-sync from GURU-BEAST-ROG at 2026-07-07 06:38:06
Author: Mike Swanson Machine: GURU-BEAST-ROG Timestamp: 2026-07-07 06:38:06
This commit is contained in:
@@ -0,0 +1,154 @@
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** GURU-BEAST-ROG
|
||||
- **Role:** admin
|
||||
|
||||
## Session Summary
|
||||
|
||||
Continued from the VWP M365 remediation into a broad VWP (Valley Wide Plastering) infrastructure day plus
|
||||
a GuruRMM agent-build discovery. First, the Lonestar Electrical Unraid GuruRMM agent was found offline; root
|
||||
cause is a **glibc mismatch** — agent v0.6.59 is linked against GLIBC_2.39 but the agent's Docker container
|
||||
is Debian 12 (glibc 2.36), so the binary won't load. This is a GuruRMM agent build regression that puts
|
||||
every Linux agent on Debian-12/older-glibc hosts at risk (Lonestar's box dropped off 2026-07-05).
|
||||
|
||||
Checked VWP server-room temperature via out-of-band controllers: the Dell R640 (VWP-QBS) iDRAC inlet read
|
||||
27C (~81F, warm/top of ASHRAE range but healthy); the HP DL360 Gen10 (ESXi host) iLO showed all zones
|
||||
26-40C. Then checked the ESXi host (192.168.3.24) and found **all 9 VMs powered off** because the **ESXi
|
||||
license had expired** (reverted to 00000-00000-... after a config/NVRAM loss). Recovered it over SSH:
|
||||
restarted hostd, reset the evaluation license to a fresh 60 days, powered on all 9 VMs (confirmed running),
|
||||
and — per Mike's direction — installed a persistent monthly **auto-reset cron** (reset script on the
|
||||
"Tesst" VMFS-6 datastore, re-installed on boot via rc.local.d + auto-backup). SSH from Windows required a
|
||||
compiled-askpass workaround (Posh-SSH couldn't install; Server 2025 strict TLS; ESXi uses
|
||||
keyboard-interactive auth).
|
||||
|
||||
Mike confirmed Broadcom killed SMB/free-ESXi licensing, so the plan is to **migrate the ESXi VMs to
|
||||
Hyper-V** (VWP-HYPERV1). Assessed capacity: VWP-HYPERV1 has 112 vCPU / 255 GB RAM / ~10 TB free vs ~8.6 TB
|
||||
of ESXi VM storage — fits. Began a proper server inventory to find redundancy. AD assessment showed
|
||||
**VWP_AD_Srvr (VWP_ADSRVR, 192.168.0.25) is a live DC holding the Schema + Domain-Naming FSMO roles**, so
|
||||
retiring it requires FSMO transfer to VWP-DC1 then graceful demotion (not a power-off). RMM coverage is
|
||||
12/19 servers, and the 12 with agents are exactly the live ones (the 7 without map to stale/dead AD
|
||||
objects). Compared SERVER19 (live 2012 R2 RDS server) vs WIN-SERVER19-2 (a dead, abandoned 2019-eval
|
||||
rebuild) — confirmed SERVER19 is the keeper and WIN-SERVER19-2 is a stale AD object to clean up.
|
||||
|
||||
Finally, VWP-QBS (physical PowerEdge R640, the QuickBooks + RDS-license server) had been offline since
|
||||
03:30. Its iDRAC showed hardware **On + healthy, no SEL faults**, but the OS was fully unresponsive (no
|
||||
ping/RDP/WinRM/SMB) — a hung OS. Captured the console screen (iDRAC's non-standard libav JPEG would not
|
||||
render locally), then **force-rebooted via iDRAC**. It POSTed (BootProgress=OSRunning) but stayed
|
||||
unresponsive ~15 min — most likely **autochk/chkdsk on the dirty QuickBooks volume** after the hard reset.
|
||||
Monitoring for recovery; will pull event logs for the original hang cause once it is back.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **ESXi eval-reset + persistent auto-reset cron** (Mike-authorized). Flagged the EULA-compliance issue and
|
||||
recommended the free vSphere Hypervisor license; Mike noted Broadcom discontinued SMB/free ESXi, so the
|
||||
cron is a bridge and Hyper-V migration is the real exit.
|
||||
- **Reset script on the VMFS datastore, not the boot bank** — this host has recurring NVRAM/bootbank
|
||||
corruption, so the datastore is the durable location; only the boot-time re-install hook lives in
|
||||
local.sh.
|
||||
- **Compiled a C# askpass.exe on the fly** for non-interactive Windows->ESXi SSH (Posh-SSH install failed:
|
||||
no PSGallery; SSH_ASKPASS .cmd not runnable). Plus `PreferredAuthentications=keyboard-interactive` since
|
||||
ESXi does not offer plain "password".
|
||||
- **iDRAC/iLO queried from a Server 2019 box (VWP-DC1) via curl** — Server 2025 (VWP-HYPERV1) strict TLS
|
||||
cannot negotiate with the BMCs' older ciphers.
|
||||
- **VWP_AD_Srvr retirement = FSMO transfer + graceful demotion**, not power-off (it holds Schema +
|
||||
Domain-Naming master and is a live GC).
|
||||
- **Force-rebooted VWP-QBS** — the OS was hung and unresponsive; a hard reset via iDRAC was the only option
|
||||
(accepted the chkdsk-on-dirty-volume consequence).
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **ESXi license expired** (00000-00000-...) after config/NVRAM loss -> all VMs off, power-on returned
|
||||
vim.fault.LicenseExpired. Fixed by resetting the eval (cp /etc/vmware/.#license.cfg -> license.cfg +
|
||||
hostd restart) -> expirationHours=1440.
|
||||
- **Posh-SSH install failed** (no PSGallery/internet) -> compiled askpass.exe via csc + Windows OpenSSH.
|
||||
- **ESXi SSH "Permission denied (publickey,keyboard-interactive)"** -> it does not offer "password"; used
|
||||
`-o PreferredAuthentications=keyboard-interactive,password`.
|
||||
- **Server 2025 TLS** could not reach iDRAC/iLO ("underlying connection closed") -> ran from VWP-DC1 (2019)
|
||||
and/or curl.
|
||||
- **ps-encoded size limits** (refuse >6000 encoded; cmd.exe 8191 command-line limit) -> split scripts,
|
||||
used the one-line `ServerCertificateValidationCallback={$true}` cert bypass, hardcoded values.
|
||||
- **Datastore auto-pick grabbed BOOTBANK1** (bad) -> hardcoded the "Tesst" VMFS-6 UUID mount.
|
||||
- **CLPH placeholder in local.sh** did not get sed-substituted in-script -> fixed with a hardcoded sed.
|
||||
- **iDRAC screenshot** returns a minimal libav JPEG that PIL and GDI+ both reject; ExportServerScreenShot
|
||||
is flaky -> could not render the console screen locally.
|
||||
- **VWP-QBS OS hung** (03:30) -> force-reboot; still recovering (likely chkdsk) at session end.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
VWP ESXi host (192.168.3.24), via SSH:
|
||||
- Reset `/etc/vmware/license.cfg` from the eval template (`.#license.cfg`) — 60-day eval restored.
|
||||
- Created `/vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh` (eval auto-reset).
|
||||
- Added cron `10 3 1 * * <reset-eval.sh>` to `/var/spool/cron/crontabs/root`.
|
||||
- Added boot re-install hook to `/etc/rc.local.d/local.sh` + ran `/sbin/auto-backup.sh`.
|
||||
- Stored the reset path at `/etc/vmware/.acg-reseteval-path`.
|
||||
|
||||
VWP-QBS (physical R640): force reboot via iDRAC Redfish ComputerSystem.Reset ForceRestart.
|
||||
|
||||
ClaudeTools: this session log (no code changes this session).
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- **VWP ESXi root** — vault `clients/vwp/esxi.sops.yaml` field `credentials.password`. Host 192.168.3.24.
|
||||
- **VWP QBS iDRAC root** — vault `clients/vwp/quickbooks-server-idrac.sops.yaml` `credentials.password`.
|
||||
iDRAC 192.168.3.189 (R640, Service Tag C84TTQ2).
|
||||
- **VWP HP iLO Administrator** — vault `clients/vwp/hp-ilo.sops.yaml` `credentials.password`. iLO
|
||||
172.16.9.125.
|
||||
- **Lonestar Unraid root** — `Gptf*77ttb123!@#-lonestar`, vault `clients/lonestar-electrical/unraid-server.sops.yaml`
|
||||
`credentials.root_password` (already vaulted).
|
||||
- ESXi license: none on record; was expired eval — reset to eval (60d). No paid/free key available
|
||||
(Broadcom discontinued SMB/free ESXi).
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **VWP domain:** VWP.US. **DCs:** VWP_ADSRVR (192.168.0.25 — Schema + Domain-Naming FSMO, GC, ESXi VM),
|
||||
VWP-DC1 (172.16.9.2 — PDC/RID/Infrastructure FSMO, GC, Hyper-V VM). Both Server 2019, replicating clean.
|
||||
- **VWP ESXi** — HP ProLiant DL360 Gen10 (SN MXQ80400X4), ESXi 8.0, mgmt 192.168.3.24 (VLAN 99), SSH :22,
|
||||
iLO 172.16.9.125. Datastore **"Tesst"** VMFS-6, UUID `5869d65c-c5ff4b86-13c0-9009d0143403`, 14 TB / 8.6 TB
|
||||
used. 9 VMs (all now powered on): Server-97, WINFilrSrvr, VWP-SERVER, SERVER 19, VWIN7-2-PC, VWP_AD_Srvr,
|
||||
VWP-FIN, WIN-Acct, WIN-AD2.
|
||||
- **VWP-HYPERV1** — 172.16.9.184, Server 2025, Hyper-V host: 112 logical CPU, 255 GB RAM, C:\VHD ~10 TB free.
|
||||
Runs VWP-DC1 + VWP-FILES.
|
||||
- **VWP-QBS** — physical PowerEdge R640, iDRAC 192.168.3.189, OS IP 172.16.9.169, Server 2022. QuickBooks
|
||||
server + RDS license server (`vwp-qbs.vwp.us`). Hung 03:30, force-rebooted, recovering.
|
||||
- RMM agents (ids): VWP-HYPERV1 `bdc3e142-1eb3-4034-8ac2-300f46dbc1d9`, VWP-DC1
|
||||
`8eefbba6-28cf-4e3f-8fda-11900e0ac302`, VWP-QBS `f3386f0a-b3ee-417e-ace9-995cc1d0662b`.
|
||||
- Lonestar Unraid agent `bfcfbc739d23` (id `00b48379-ec09-4396-9d8f-31a1ee8b4ce3`), offline (glibc).
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
- ESXi eval reset: `cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg; /etc/init.d/hostd restart` ->
|
||||
license --show `expirationHours=1440`.
|
||||
- Non-interactive Win->ESXi SSH: compile `askpass.exe` (`Add-Type ... -OutputType ConsoleApplication`
|
||||
printing env `XPW`), set `SSH_ASKPASS`/`SSH_ASKPASS_REQUIRE=force`, `ssh -o
|
||||
PreferredAuthentications=keyboard-interactive,password`.
|
||||
- iDRAC/BMC Redfish must run from a 2019 box via curl (`curl.exe -sk -H "Authorization: Basic <b64>"`) —
|
||||
Server 2025 .NET TLS fails.
|
||||
- VWP-QBS iDRAC: PowerState=On, Health=OK, BootProgress=OSRunning, SEL clean (last event Feb 2025).
|
||||
- ESXi power-on before reset: `Power on failed: (vim.fault.LicenseExpired) ... licenseKey = "Invalid
|
||||
License: 00000-00000-00000-00000-00000"`.
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **VWP-QBS (IN PROGRESS):** monitor for recovery (likely chkdsk); once up, verify QuickBooks + RDS
|
||||
licensing services, pull System/Application event logs around 03:30 for the original hang root cause. If
|
||||
still down in ~45 min or console shows a true hang, treat as unstable hardware/OS.
|
||||
- **VWP_AD_Srvr retirement:** pre-flight (DNS/DHCP not handing out .0.25; dcdiag VWP-DC1) -> transfer Schema
|
||||
+ Domain-Naming FSMO to VWP-DC1 -> graceful demote -> verify -> delete VM. Consider a 2nd Hyper-V DC for
|
||||
redundancy.
|
||||
- **Server redundancy cleanup:** stale AD objects to remove — WIN-SERVER19-2, WIN-SVR-1, VWP-AD2,
|
||||
WIN-FIN2-OLD, SERVER3, VWP-BUSVR. Verify ORDERS2 (recent logon, no RMM — gap or decommission).
|
||||
- **VMware -> Hyper-V migration:** StarWind V2V Converter; finalize keep-list, pick pilot VM. Eval cron is
|
||||
the bridge until migrated.
|
||||
- **GuruRMM Linux agent glibc regression:** rebuild agent static/musl or target glibc 2.36; scan fleet for
|
||||
other offline Linux agents; redeploy Lonestar Unraid container; update its vault agent_id (stale
|
||||
e827f798 -> 00b48379).
|
||||
- Identity-tool hardening (from prior save) still pending: consent-audit misattribution + friction log.
|
||||
|
||||
## Reference Information
|
||||
|
||||
- iDRAC screenshot Redfish action: `POST /redfish/v1/Dell/Managers/iDRAC.Embedded.1/DellLCService/Actions/
|
||||
DellLCService.ExportServerScreenShot` body `{"FileType":"ServerScreenShot"}` (returns base64 JPEG).
|
||||
- ESXi reset action: `POST /redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset`
|
||||
`{"ResetType":"ForceRestart"}`.
|
||||
- Eval-reset cron: `10 3 1 * * /vmfs/volumes/5869d65c-c5ff4b86-13c0-9009d0143403/.acg/reset-eval.sh`.
|
||||
- Prior session log: `session-logs/2026-07/2026-07-06-mike-vwp-m365-remediation-gururmm-console.md`.
|
||||
@@ -19,6 +19,8 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
|
||||
|
||||
<!-- Append entries below this line -->
|
||||
|
||||
2026-07-07 | GURU-BEAST-ROG | rmm/ilo-redfish | [friction] embedded double-quotes in curl.exe -d JSON body stripped by PowerShell native arg passing (MalformedJSON from iLO); fix = write body to temp file via [char]34, --data-binary @file [ctx: ref=feedback_windows_quote_stripping]
|
||||
|
||||
2026-07-07 | Howard-Home | rmm/windows-edition-upgrade | [correction] hand-rolled changepk Home->Pro instead of following reference_windows_edition_upgrade_rmm; changepk does NOT auto-reboot (flips edition live + sets RebootPending) so the long offline was the flaky RMM agent, not a reboot in progress - must trigger Restart-Computer -Force explicitly [ctx: host=DALLAS ref=reference_windows_edition_upgrade_rmm]
|
||||
|
||||
2026-07-06 | Howard-Home | ssh/gururmm-server | [friction] vault infrastructure/gururmm-server.sops.yaml lists username/password + 'Sudo password same as SSH', but 172.16.3.30 SSH is publickey-only (password auth disabled) and no fleet key authenticates guru/root/mike/howard from workstations. Watching build logs / running promote-dashboard.sh must be done on the box. Fix: add SSH pubkey to guru@.30 or correct the vault note. [ctx: ref=infrastructure/gururmm-server.sops.yaml host=172.16.3.30]
|
||||
|
||||
Reference in New Issue
Block a user