3.9 KiB
3.9 KiB
2026-04-22 — Valleywide HP Server NVRAM Corruption Emergency
User
- User: Mike Swanson (mike)
- Machine: Mikes-MacBook-Air.local
- Role: admin
Ticket Information
- Type: Emergency onsite
- Priority: Critical
- Status: In Progress
- Arrival: 0935 MST
Issue Summary
Multiple server issues following power outage at Valleywide:
- HP ProLiant Server (SN: MXQ80400X4): Non-volatile memory corruption, BIOS/iLO reset required
- Dell Server (VWP-QBS): Boot retry loop, resolved via DRAC manual boot
- XenServer (Older Dell): OFFLINE - investigating (CRITICAL)
Timeline
0935 - Arrival Onsite
- HP Server SN: MXQ80400X4
- Issue: Non-Volatile Memory Corruption
- Cause: Power outage
- Impact: BIOS/UEFI reset to factory defaults
0935-[IN PROGRESS] - Recovery Actions
HP ProLiant Server (SN: MXQ80400X4):
BIOS/UEFI Reconfiguration:
- Factory reset required due to NVRAM corruption
- Reconfigured BIOS settings
- Restored boot order
- Re-enabled virtualization settings
iLO (Integrated Lights-Out) Reset:
- [WARNING] iLO was reset to factory defaults due to BIOS reset
- iLO credentials will need to be re-entered
- Network configuration may need restoration
- Remote management temporarily unavailable until iLO reconfigured
VM Status:
- [OK] All VMs running
- Hypervisor operational after BIOS reconfiguration
- No VM data loss reported
Dell Server (VWP-QBS) - Separate Boot Issue:
Boot Retry Loop:
- VWP-QBS (Dell physical server, 172.16.9.169) stuck at "Boot Retry" screen
- Accessed via DRAC (Dell Remote Access Controller)
- Forced manual boot device selection -> Windows Boot Manager
- [OK] Server booted successfully
- [OK] Server appears to be functioning normally now
- Likely related to power outage affecting boot order/configuration
- NOTE: VWP-QBS is NOT a VM - it's a separate physical Dell server
XenServer (Older Dell) - OFFLINE:
Status:
- [CRITICAL] XenServer offline
- Impact: Server3 VM unavailable
- Investigating cause (likely power outage related)
- Checking hardware status, boot sequence, and hypervisor state
- Dell server - older hardware
Next Steps
CRITICAL:
- Restore XenServer (currently investigating offline status)
- Verify Server3 VM status once XenServer restored
High Priority:
- Complete onsite work (timer running)
- Reconfigure HP iLO settings (credentials, network)
- Document iLO IP address and credentials
- Verify all server settings match pre-incident configuration
Follow-up:
- Test remote management access (iLO, DRAC)
- Update server documentation with serial numbers and DRAC IPs
- Create follow-up preventive measures (UPS assessment critical)
Server Information
HP ProLiant Server:
- Serial Number: MXQ80400X4
- Model: [TO BE DOCUMENTED]
- Role: VM Host (runs VWP_ADSRVR and other VMs)
- Location: Valleywide onsite
- Status: Reconfigured, operational
HP iLO Management:
- Status: Reset to factory defaults
- IP: [TO BE RECONFIGURED]
- Credentials: [TO BE RESET]
Dell Server (VWP-QBS):
- Model: Dell (with DRAC)
- Role: QuickBooks Server, RDS Host (Windows Server 2022)
- IP: 172.16.9.169
- Location: Valleywide onsite
- Status: Boot issue resolved, operational
- NOTE: Physical server, NOT a VM
Dell DRAC Management:
- Status: Functional (used to force manual boot)
- IP: [TO BE DOCUMENTED]
XenServer (Older Dell):
- Model: Dell (older hardware)
- Role: VM Host for Server3
- Location: Valleywide onsite
- Status: OFFLINE - INVESTIGATING
- Impact: Server3 VM unavailable
Notes
- Power outage caused NVRAM corruption - rare but critical failure
- Quick recovery due to all VMs remaining intact
- iLO reconfiguration required for remote management
- Consider UPS assessment as preventive measure
Work Status: COMPLETE - Invoiced Resolved: 2026-04-22 ~12:00 MST
Message for Howard
Yealink password: n7*!O0qx&$IB$83*
— Mike, 2026-04-22