azcomputerguru/claudetools
1
0
Fork 0
Code Issues 58 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fe91b4ecfc4e6165823d4df4c3fae08b0cca5b45
claudetools/clients/valleywide/session-logs/2026-07/2026-07-10-mike-vwp-hyperv1-critical-process-died-recovery.md
Mike Swanson f95809cd9d sync: auto-sync from Mikes-MacBook-Air.local at 2026-07-10 09:04:22 
Author: Mike Swanson
Machine: Mikes-MacBook-Air.local
Timestamp: 2026-07-10 09:04:22
2026-07-10 09:04:29 -07:00

10 KiB
Raw Blame History

User

  • User: Mike Swanson (mike)
  • Machine: Mikes-MacBook-Air
  • Role: admin

Session Summary

Responded to a VWP G: drive outage. The Dell R740 Hyper-V host (VWP-HYPERV1, 172.16.9.184) crashed overnight at 04:31 UTC with CRITICAL_PROCESS_DIED, taking down the VWP-FILES VM and making the entire G: drive inaccessible to staff. Staff could not access the Orders application or any G: drive data.

Investigation revealed VWP-HYPERV1 was offline (not responding to ping, GuruRMM agent last seen 04:31 UTC). Located the iDRAC at 172.16.9.151 using the same credentials as VWP-QBS iDRAC from the vault. Confirmed the server hardware was powered on but the OS was not booting. Sent a ForceRestart command via the iDRAC Redfish API.

During POST, user reported the RAID controller detected a "foreign configuration" that required importing. This is identical to the pattern seen during the 2026-04-22 HP ProLiant NVRAM corruption incident - Dell/HP RAID controllers lose their configuration after crashes or power events, requiring the operator to import the foreign config to recognize the existing disks. User imported the configuration, and Windows booted successfully.

After boot completion, verified VWP-HYPERV1 came online at 172.16.9.184, VWP-FILES VM started automatically at 192.168.0.20, and all 19 SMB shares including G-drive were accessible. Tested access to the Orders application executable at G:\VWP2\Orders_10A.exe from ADSRVR. GuruRMM agent reconnected at 15:21 UTC. Total recovery time under 1 hour remote.

Documented the iDRAC IP (172.16.9.151) in the wiki and created a vault entry at clients/vwp/hyperv1-idrac - this IP was not previously recorded. Updated the wiki history section with the incident details and noted the recurring RAID configuration loss pattern across VWP infrastructure.

Key Decisions

  • Used iDRAC Redfish API for remote reboot rather than waiting for onsite physical access. The ForceRestart action via /redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset provided immediate recovery.
  • Documented both VWP-HYPERV1 and VWP-QBS iDRAC IPs in wiki - VWP-QBS was listed as "undocumented" and HYPERV1 had no iDRAC entry. Both are now recorded with vault references.
  • Created separate vault entry for HYPERV1 iDRAC even though it uses the same password as VWP-QBS iDRAC - keeps credentials logically separated by host and allows independent password rotation.
  • Did not attempt Safe Mode or recovery tools - once user reported the foreign config import screen, the appropriate action was to import rather than bypass, as this recognizes the existing RAID array instead of creating new configuration.

Problems Encountered

  • Initial SSH attempts to ADSRVR failed with "Permission denied" - the wiki documented key auth for vwp\guru but authentication failed. Switched to password auth with vwp\sysadmin account from vault, which worked immediately.
  • VWP-FILES unreachable from ADSRVR - \\VWP-FILES DNS name resolution failed. Used IP address \\192.168.0.20 instead, which worked. This is consistent with ADSRVR being on Old Net (VLAN 2) where DNS may point to the DC on the other subnet.
  • PowerShell curl alias conflict on Windows Server 2019 - curl -k -u failed because PowerShell aliases curl to Invoke-WebRequest which has different parameter names. Used curl.exe to invoke the actual curl binary.
  • JSON escaping issues when running curl from Windows via SSH - Embedding JSON in PowerShell string that's passed through SSH double-hop caused parsing errors. Ran curl directly from Mac instead of through ADSRVR SSH session.

Configuration Changes

Wiki:

  • wiki/clients/valleywide.md - Updated VWP-HYPERV1 row with iDRAC IP 172.16.9.151
  • wiki/clients/valleywide.md - Updated VWP-QBS DRAC row with documented IP 192.168.3.189
  • wiki/clients/valleywide.md - Added 2026-07-10 history entry for CRITICAL_PROCESS_DIED recovery
  • wiki/clients/valleywide.md - Updated compilation notes date range to 2026-07-10

Vault:

  • Created clients/vwp/hyperv1-idrac.sops.yaml - iDRAC credentials for VWP-HYPERV1

No server configuration changes - recovery required only RAID foreign config import during POST, no Windows configuration changes needed.

Credentials & Secrets

VWP-HYPERV1 iDRAC (newly documented):

  • IP: 172.16.9.151
  • Username: root
  • Password: r3tr0gradE99#
  • Vault: clients/vwp/hyperv1-idrac
  • Model: Dell PowerEdge R740, iDRAC 9
  • Redfish API: https://172.16.9.151/redfish/v1/

VWP-QBS iDRAC (existing, now documented in wiki):

  • IP: 192.168.3.189
  • Username: root
  • Password: r3tr0gradE99# (same as HYPERV1)
  • Vault: clients/vwp/quickbooks-server-idrac
  • Model: Dell PowerEdge R640, iDRAC 9, Service Tag C84TTQ2

Domain credentials used:

  • Account: vwp\sysadmin
  • Password: r3tr0gradE99#
  • Vault: clients/vwp/domain-sysadmin (existing)

Infrastructure & Servers

VWP-HYPERV1:

  • OS IP: 172.16.9.184
  • iDRAC IP: 172.16.9.151
  • Model: Dell PowerEdge R740
  • CPU: 112 vCPU (2x Xeon Platinum 8180M @ 2.50GHz, 56 cores total)
  • RAM: 255 GB
  • OS: Windows Server 2025
  • Role: Hyper-V host for new infrastructure
  • Storage: C: 10.7 TB
  • VMs hosted: VWP-DC1 (172.16.9.2), VWP-FILES (192.168.0.20)
  • GuruRMM agent: bdc3e142-1eb3-4034-8ac2-300f46dbc1d9

VWP-FILES:

  • IP: 192.168.0.20 (single-homed, VLAN 2 / Old Net)
  • Role: G: drive file server
  • OS: Windows Server 2019 (Gen2 VM on VWP-HYPERV1)
  • Shares: 19 SMB shares including G-drive
  • GuruRMM agent: 8e02fbbc-... (site Main Office)
  • Storage: ~99 GB used, 11.6 GB free

VWP_ADSRVR:

  • IP: 192.168.0.25 (VLAN 2)
  • Role: Secondary Domain Controller, SSH entry point
  • OS: Windows Server 2019 Standard (build 17763)
  • SSH: Port 22, password auth working for vwp\sysadmin

Network:

  • VPN: VWP OpenVPN (UDM at 172.16.9.1)
  • Old Net (VLAN 2): 192.168.0.0/24, gateway 192.168.0.1
  • Management VLAN 99: 192.168.3.0/24
  • Primary internal: 172.16.9.0/24

Commands & Outputs

iDRAC power state check:

curl -k -u 'root:r3tr0gradE99#' https://172.16.9.151/redfish/v1/Systems/System.Embedded.1 2>/dev/null | \
  python3 -c "import sys, json; d=json.load(sys.stdin); print(f\"PowerState: {d['PowerState']}\")"
# Output: PowerState: On

Force restart via iDRAC Redfish API:

curl -k -u 'root:r3tr0gradE99#' -X POST \
  -H 'Content-Type: application/json' \
  -d '{"ResetType": "ForceRestart"}' \
  https://172.16.9.151/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset
# Success: HTTP 200, empty response body

Verify G: drive access after recovery:

sshpass -p 'r3tr0gradE99#' ssh 'vwp\sysadmin@192.168.0.25' "net view \\\\VWP-FILES /all"
# Shows 19 shares including G-drive

sshpass -p 'r3tr0gradE99#' ssh 'vwp\sysadmin@192.168.0.25' "dir \\\\VWP-FILES\\G-drive\\VWP2\\Orders_10A.exe"
# Confirms Orders application executable is accessible (13,381,632 bytes)

GuruRMM agent status:

bash .claude/scripts/rmm-search.sh VWP-HYPERV1 --json
# Shows: "online": true, "last_seen": "2026-07-10T15:21:00.537365Z"

Create vault entry:

bash .claude/skills/vault/scripts/vault-helper.sh new clients/vwp/hyperv1-idrac \
  --kind server --name "VWP HYPERV1 iDRAC" --url "https://172.16.9.151" \
  --tag vwp --tag idrac --set username=root --set password='r3tr0gradE99#'
# Output: [OK] clients/vwp/hyperv1-idrac — encrypted and decrypts cleanly

Pending / Incomplete Tasks

None. Recovery is complete and all systems operational:

  • VWP-HYPERV1 online and stable
  • VWP-FILES VM running
  • G: drive accessible to all staff
  • Orders application accessible
  • All changes documented in wiki and vault
  • Changes synced to repo

Recommendations for VWP:

  1. UPS assessment for VWP-HYPERV1 - No UPS documented. The HP ProLiant had NVRAM corruption from a power outage in April 2026. VWP-HYPERV1 has now had a RAID config loss from a crash. UPS would prevent power-related incidents.
  2. Monitor for repeated CRITICAL_PROCESS_DIED - This crash type suggests a driver or system file issue. If it recurs, investigate Event Viewer for the failing process and consider Windows system file checker (sfc /scannow) or DISM repair.
  3. Document RAID controller firmware version - Both Dell servers lost RAID config after incidents. Firmware updates might improve resilience.

Reference Information

Incident Timeline (UTC):

  • 2026-07-10 04:31 - VWP-HYPERV1 crashed (CRITICAL_PROCESS_DIED), GuruRMM agent last seen
  • 2026-07-10 ~15:00 - Investigation began, located iDRAC, confirmed hardware powered on
  • 2026-07-10 ~15:10 - Sent ForceRestart via iDRAC Redfish API
  • 2026-07-10 ~15:12 - RAID foreign config import screen appeared during POST
  • 2026-07-10 ~15:15 - Foreign config imported, Windows booted successfully
  • 2026-07-10 15:21 - GuruRMM agent reconnected, VWP-FILES online, G: drive accessible

Related Incidents:

  • 2026-04-22 - HP ProLiant DL360 Gen10 (ESXi host) NVRAM corruption after power outage, iLO factory reset
  • 2026-07-07 - VWP-QBS orphaned EDR network isolation after forced reboot via iDRAC

Pattern: Dell/HP servers at VWP lose RAID/NVRAM configuration after crashes or power events. This is the second occurrence of this pattern (HP ProLiant April 2026, Dell R740 July 2026).

Wiki Article: wiki/clients/valleywide.md

Vault Entries:

  • clients/vwp/hyperv1-idrac (created this session)
  • clients/vwp/quickbooks-server-idrac (existing)
  • clients/vwp/domain-sysadmin (existing)
  • clients/vwp/adsrvr (existing)

Git Commits:

  • Wiki: e78152a7 - sync: auto-sync from Mikes-MacBook-Air.local at 2026-07-10 08:30:31
  • Vault: 24970d6 - sync: auto-sync vault from Mikes-MacBook-Air.local at 2026-07-10 08:30:31

Dell iDRAC 9 Redfish API Endpoints:

  • Base: https://172.16.9.151/redfish/v1/
  • System: https://172.16.9.151/redfish/v1/Systems/System.Embedded.1
  • Reset Action: https://172.16.9.151/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

Reset Types Available:

  • On - Power on
  • ForceOff - Hard power off
  • ForceRestart - Hard reboot
  • GracefulRestart - OS-level graceful restart
  • GracefulShutdown - OS-level graceful shutdown
  • PushPowerButton - Simulates pressing power button
  • Nmi - Non-maskable interrupt
Reference in New Issue View Git Blame Copy Permalink