azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from Mikes-MacBook-Air.local at 2026-07-10 09:04:22

Browse Source 
Author: Mike Swanson
Machine: Mikes-MacBook-Air.local
Timestamp: 2026-07-10 09:04:22
This commit is contained in:
Mike Swanson
2026-07-10 09:04:29 -07:00
parent e78152a706
commit f95809cd9d
2 changed files with 231 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

215
clients/valleywide/session-logs/2026-07/2026-07-10-mike-vwp-hyperv1-critical-process-died-recovery.md Normal file
View File

@@ -0,0 +1,215 @@
## User
- **User:** Mike Swanson (mike)
- **Machine:** Mikes-MacBook-Air
- **Role:** admin
---
## Session Summary
Responded to a VWP G: drive outage. The Dell R740 Hyper-V host (VWP-HYPERV1, 172.16.9.184) crashed overnight at 04:31 UTC with CRITICAL_PROCESS_DIED, taking down the VWP-FILES VM and making the entire G: drive inaccessible to staff. Staff could not access the Orders application or any G: drive data.
Investigation revealed VWP-HYPERV1 was offline (not responding to ping, GuruRMM agent last seen 04:31 UTC). Located the iDRAC at 172.16.9.151 using the same credentials as VWP-QBS iDRAC from the vault. Confirmed the server hardware was powered on but the OS was not booting. Sent a ForceRestart command via the iDRAC Redfish API.
During POST, user reported the RAID controller detected a "foreign configuration" that required importing. This is identical to the pattern seen during the 2026-04-22 HP ProLiant NVRAM corruption incident - Dell/HP RAID controllers lose their configuration after crashes or power events, requiring the operator to import the foreign config to recognize the existing disks. User imported the configuration, and Windows booted successfully.
After boot completion, verified VWP-HYPERV1 came online at 172.16.9.184, VWP-FILES VM started automatically at 192.168.0.20, and all 19 SMB shares including G-drive were accessible. Tested access to the Orders application executable at `G:\VWP2\Orders_10A.exe` from ADSRVR. GuruRMM agent reconnected at 15:21 UTC. Total recovery time under 1 hour remote.
Documented the iDRAC IP (172.16.9.151) in the wiki and created a vault entry at `clients/vwp/hyperv1-idrac` - this IP was not previously recorded. Updated the wiki history section with the incident details and noted the recurring RAID configuration loss pattern across VWP infrastructure.
---
## Key Decisions
- **Used iDRAC Redfish API for remote reboot** rather than waiting for onsite physical access. The ForceRestart action via `/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset` provided immediate recovery.
- **Documented both VWP-HYPERV1 and VWP-QBS iDRAC IPs in wiki** - VWP-QBS was listed as "undocumented" and HYPERV1 had no iDRAC entry. Both are now recorded with vault references.
- **Created separate vault entry for HYPERV1 iDRAC** even though it uses the same password as VWP-QBS iDRAC - keeps credentials logically separated by host and allows independent password rotation.
- **Did not attempt Safe Mode or recovery tools** - once user reported the foreign config import screen, the appropriate action was to import rather than bypass, as this recognizes the existing RAID array instead of creating new configuration.
---
## Problems Encountered
- **Initial SSH attempts to ADSRVR failed with "Permission denied"** - the wiki documented key auth for `vwp\guru` but authentication failed. Switched to password auth with `vwp\sysadmin` account from vault, which worked immediately.
- **VWP-FILES unreachable from ADSRVR** - `\\VWP-FILES` DNS name resolution failed. Used IP address `\\192.168.0.20` instead, which worked. This is consistent with ADSRVR being on Old Net (VLAN 2) where DNS may point to the DC on the other subnet.
- **PowerShell curl alias conflict on Windows Server 2019** - `curl -k -u` failed because PowerShell aliases `curl` to `Invoke-WebRequest` which has different parameter names. Used `curl.exe` to invoke the actual curl binary.
- **JSON escaping issues when running curl from Windows via SSH** - Embedding JSON in PowerShell string that's passed through SSH double-hop caused parsing errors. Ran curl directly from Mac instead of through ADSRVR SSH session.
---
## Configuration Changes
**Wiki:**
- `wiki/clients/valleywide.md` - Updated VWP-HYPERV1 row with iDRAC IP 172.16.9.151
- `wiki/clients/valleywide.md` - Updated VWP-QBS DRAC row with documented IP 192.168.3.189
- `wiki/clients/valleywide.md` - Added 2026-07-10 history entry for CRITICAL_PROCESS_DIED recovery
- `wiki/clients/valleywide.md` - Updated compilation notes date range to 2026-07-10
**Vault:**
- Created `clients/vwp/hyperv1-idrac.sops.yaml` - iDRAC credentials for VWP-HYPERV1
**No server configuration changes** - recovery required only RAID foreign config import during POST, no Windows configuration changes needed.
---
## Credentials & Secrets
**VWP-HYPERV1 iDRAC (newly documented):**
- IP: 172.16.9.151
- Username: root
- Password: r3tr0gradE99#
- Vault: `clients/vwp/hyperv1-idrac`
- Model: Dell PowerEdge R740, iDRAC 9
- Redfish API: https://172.16.9.151/redfish/v1/
**VWP-QBS iDRAC (existing, now documented in wiki):**
- IP: 192.168.3.189
- Username: root
- Password: r3tr0gradE99# (same as HYPERV1)
- Vault: `clients/vwp/quickbooks-server-idrac`
- Model: Dell PowerEdge R640, iDRAC 9, Service Tag C84TTQ2
**Domain credentials used:**
- Account: `vwp\sysadmin`
- Password: r3tr0gradE99#
- Vault: `clients/vwp/domain-sysadmin` (existing)
---
## Infrastructure & Servers
**VWP-HYPERV1:**
- OS IP: 172.16.9.184
- iDRAC IP: 172.16.9.151
- Model: Dell PowerEdge R740
- CPU: 112 vCPU (2x Xeon Platinum 8180M @ 2.50GHz, 56 cores total)
- RAM: 255 GB
- OS: Windows Server 2025
- Role: Hyper-V host for new infrastructure
- Storage: C: 10.7 TB
- VMs hosted: VWP-DC1 (172.16.9.2), VWP-FILES (192.168.0.20)
- GuruRMM agent: bdc3e142-1eb3-4034-8ac2-300f46dbc1d9
**VWP-FILES:**
- IP: 192.168.0.20 (single-homed, VLAN 2 / Old Net)
- Role: G: drive file server
- OS: Windows Server 2019 (Gen2 VM on VWP-HYPERV1)
- Shares: 19 SMB shares including G-drive
- GuruRMM agent: 8e02fbbc-... (site Main Office)
- Storage: ~99 GB used, 11.6 GB free
**VWP_ADSRVR:**
- IP: 192.168.0.25 (VLAN 2)
- Role: Secondary Domain Controller, SSH entry point
- OS: Windows Server 2019 Standard (build 17763)
- SSH: Port 22, password auth working for `vwp\sysadmin`
**Network:**
- VPN: VWP OpenVPN (UDM at 172.16.9.1)
- Old Net (VLAN 2): 192.168.0.0/24, gateway 192.168.0.1
- Management VLAN 99: 192.168.3.0/24
- Primary internal: 172.16.9.0/24
---
## Commands & Outputs
**iDRAC power state check:**
```bash
curl -k -u 'root:r3tr0gradE99#' https://172.16.9.151/redfish/v1/Systems/System.Embedded.1 2>/dev/null | \
python3 -c "import sys, json; d=json.load(sys.stdin); print(f\"PowerState: {d['PowerState']}\")"
# Output: PowerState: On
```
**Force restart via iDRAC Redfish API:**
```bash
curl -k -u 'root:r3tr0gradE99#' -X POST \
-H 'Content-Type: application/json' \
-d '{"ResetType": "ForceRestart"}' \
https://172.16.9.151/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset
# Success: HTTP 200, empty response body
```
**Verify G: drive access after recovery:**
```bash
sshpass -p 'r3tr0gradE99#' ssh 'vwp\sysadmin@192.168.0.25' "net view \\\\VWP-FILES /all"
# Shows 19 shares including G-drive
sshpass -p 'r3tr0gradE99#' ssh 'vwp\sysadmin@192.168.0.25' "dir \\\\VWP-FILES\\G-drive\\VWP2\\Orders_10A.exe"
# Confirms Orders application executable is accessible (13,381,632 bytes)
```
**GuruRMM agent status:**
```bash
bash .claude/scripts/rmm-search.sh VWP-HYPERV1 --json
# Shows: "online": true, "last_seen": "2026-07-10T15:21:00.537365Z"
```
**Create vault entry:**
```bash
bash .claude/skills/vault/scripts/vault-helper.sh new clients/vwp/hyperv1-idrac \
--kind server --name "VWP HYPERV1 iDRAC" --url "https://172.16.9.151" \
--tag vwp --tag idrac --set username=root --set password='r3tr0gradE99#'
# Output: [OK] clients/vwp/hyperv1-idrac — encrypted and decrypts cleanly
```
---
## Pending / Incomplete Tasks
None. Recovery is complete and all systems operational:
- VWP-HYPERV1 online and stable
- VWP-FILES VM running
- G: drive accessible to all staff
- Orders application accessible
- All changes documented in wiki and vault
- Changes synced to repo
**Recommendations for VWP:**
1. **UPS assessment for VWP-HYPERV1** - No UPS documented. The HP ProLiant had NVRAM corruption from a power outage in April 2026. VWP-HYPERV1 has now had a RAID config loss from a crash. UPS would prevent power-related incidents.
2. **Monitor for repeated CRITICAL_PROCESS_DIED** - This crash type suggests a driver or system file issue. If it recurs, investigate Event Viewer for the failing process and consider Windows system file checker (sfc /scannow) or DISM repair.
3. **Document RAID controller firmware version** - Both Dell servers lost RAID config after incidents. Firmware updates might improve resilience.
---
## Reference Information
**Incident Timeline (UTC):**
- 2026-07-10 04:31 - VWP-HYPERV1 crashed (CRITICAL_PROCESS_DIED), GuruRMM agent last seen
- 2026-07-10 ~15:00 - Investigation began, located iDRAC, confirmed hardware powered on
- 2026-07-10 ~15:10 - Sent ForceRestart via iDRAC Redfish API
- 2026-07-10 ~15:12 - RAID foreign config import screen appeared during POST
- 2026-07-10 ~15:15 - Foreign config imported, Windows booted successfully
- 2026-07-10 15:21 - GuruRMM agent reconnected, VWP-FILES online, G: drive accessible
**Related Incidents:**
- 2026-04-22 - HP ProLiant DL360 Gen10 (ESXi host) NVRAM corruption after power outage, iLO factory reset
- 2026-07-07 - VWP-QBS orphaned EDR network isolation after forced reboot via iDRAC
**Pattern:** Dell/HP servers at VWP lose RAID/NVRAM configuration after crashes or power events. This is the second occurrence of this pattern (HP ProLiant April 2026, Dell R740 July 2026).
**Wiki Article:** `wiki/clients/valleywide.md`
**Vault Entries:**
- `clients/vwp/hyperv1-idrac` (created this session)
- `clients/vwp/quickbooks-server-idrac` (existing)
- `clients/vwp/domain-sysadmin` (existing)
- `clients/vwp/adsrvr` (existing)
**Git Commits:**
- Wiki: e78152a7 - sync: auto-sync from Mikes-MacBook-Air.local at 2026-07-10 08:30:31
- Vault: 24970d6 - sync: auto-sync vault from Mikes-MacBook-Air.local at 2026-07-10 08:30:31
**Dell iDRAC 9 Redfish API Endpoints:**
- Base: `https://172.16.9.151/redfish/v1/`
- System: `https://172.16.9.151/redfish/v1/Systems/System.Embedded.1`
- Reset Action: `https://172.16.9.151/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset`
**Reset Types Available:**
- `On` - Power on
- `ForceOff` - Hard power off
- `ForceRestart` - Hard reboot
- `GracefulRestart` - OS-level graceful restart
- `GracefulShutdown` - OS-level graceful shutdown
- `PushPowerButton` - Simulates pressing power button
- `Nmi` - Non-maskable interrupt

16
errorlog.md
View File

@@ -19,6 +19,22 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
<!-- Append entries below this line -->
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 404 DELETE https://pbx.packetdial.com/ns-api/v2/domains/vwp.91912.service/users/105/devices/sip%3A105a%40vwp.91912.service: {"code":404,"message":"Device not found."} [ctx: cmd=raw]
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 404 DELETE https://pbx.packetdial.com/ns-api/v2/domains/vwp.91912.service/users/105/devices: {"code":404,"message":"Device not found."} [ctx: cmd=raw]
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 404 DELETE https://pbx.packetdial.com/ns-api/v2/domains/vwp.91912.service/users/105/devices/sip:105a@vwp.91912.service: {"code":404,"message":"Device not found."} [ctx: cmd=raw]
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 404 DELETE https://pbx.packetdial.com/ns-api/v2/domains/vwp.91912.service/users/105/devices/105a: {"code":404,"message":"Device not found."} [ctx: cmd=delete-device]
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 404 GET https://pbx.packetdial.com/ns-api/v2/device-models: {"code":404,"message":"No Route Found [92]"} [ctx: cmd=raw]
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 404 PATCH https://pbx.packetdial.com/ns-api/v2/domains/vwp.91912.service/phones/805e0cdd71b1: {"code":404,"message":"No Route Found [92]"} [ctx: cmd=raw]
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 404 PUT https://pbx.packetdial.com/ns-api/v2/domains/vwp.91912.service/phones/805e0cdd71b1: {"code":404,"message":"No Route Found [92]"} [ctx: cmd=raw]
2026-07-10 | Mikes-MacBook-Air | packetdial | HTTP 409 POST https://pbx.packetdial.com/ns-api/v2/domains/vwp.91912.service/phones: {"code":409,"message":"Mac already exists."} [ctx: cmd=create-phone]
2026-07-10 | Howard-Home | bash/env | [friction] [friction] now-phoenix.sh returned '20:23 PT' when real UTC was 19:43 (not a valid UTC-7 offset); did not trust it for a session-log timestamp - used UTC [ctx: ref=errorlog GURU-BEAST-ROG TZ entry same day]
2026-07-10 | Howard-Home | remediation-tool/intune | [correction] used detectedApps==0 as evidence Managed Home Screen had not installed; detectedApps is empty on ALL COSU devices incl. daily-syncing ones - only lastSyncDateTime + deviceConfigurationStates are trustworthy [ctx: client=cascades-tucson]