139 lines
3.9 KiB
Markdown
139 lines
3.9 KiB
Markdown
# 2026-04-22 — Valleywide HP Server NVRAM Corruption Emergency
|
|
|
|
## User
|
|
- **User:** Mike Swanson (mike)
|
|
- **Machine:** Mikes-MacBook-Air.local
|
|
- **Role:** admin
|
|
|
|
## Ticket Information
|
|
- **Type:** Emergency onsite
|
|
- **Priority:** Critical
|
|
- **Status:** In Progress
|
|
- **Arrival:** 0935 MST
|
|
|
|
## Issue Summary
|
|
|
|
Multiple server issues following power outage at Valleywide:
|
|
- HP ProLiant Server (SN: MXQ80400X4): Non-volatile memory corruption, BIOS/iLO reset required
|
|
- Dell Server (VWP-QBS): Boot retry loop, resolved via DRAC manual boot
|
|
- **XenServer (Older Dell): OFFLINE - investigating (CRITICAL)**
|
|
|
|
## Timeline
|
|
|
|
### 0935 - Arrival Onsite
|
|
- HP Server SN: MXQ80400X4
|
|
- Issue: Non-Volatile Memory Corruption
|
|
- Cause: Power outage
|
|
- Impact: BIOS/UEFI reset to factory defaults
|
|
|
|
### 0935-[IN PROGRESS] - Recovery Actions
|
|
|
|
**HP ProLiant Server (SN: MXQ80400X4):**
|
|
|
|
**BIOS/UEFI Reconfiguration:**
|
|
- Factory reset required due to NVRAM corruption
|
|
- Reconfigured BIOS settings
|
|
- Restored boot order
|
|
- Re-enabled virtualization settings
|
|
|
|
**iLO (Integrated Lights-Out) Reset:**
|
|
- [WARNING] iLO was reset to factory defaults due to BIOS reset
|
|
- iLO credentials will need to be re-entered
|
|
- Network configuration may need restoration
|
|
- Remote management temporarily unavailable until iLO reconfigured
|
|
|
|
**VM Status:**
|
|
- [OK] All VMs running
|
|
- Hypervisor operational after BIOS reconfiguration
|
|
- No VM data loss reported
|
|
|
|
**Dell Server (VWP-QBS) - Separate Boot Issue:**
|
|
|
|
**Boot Retry Loop:**
|
|
- VWP-QBS (Dell physical server, 172.16.9.169) stuck at "Boot Retry" screen
|
|
- Accessed via DRAC (Dell Remote Access Controller)
|
|
- Forced manual boot device selection -> Windows Boot Manager
|
|
- [OK] Server booted successfully
|
|
- [OK] Server appears to be functioning normally now
|
|
- Likely related to power outage affecting boot order/configuration
|
|
- NOTE: VWP-QBS is NOT a VM - it's a separate physical Dell server
|
|
|
|
**XenServer (Older Dell) - OFFLINE:**
|
|
|
|
**Status:**
|
|
- [CRITICAL] XenServer offline
|
|
- Impact: Server3 VM unavailable
|
|
- Investigating cause (likely power outage related)
|
|
- Checking hardware status, boot sequence, and hypervisor state
|
|
- Dell server - older hardware
|
|
|
|
## Next Steps
|
|
|
|
**CRITICAL:**
|
|
- [ ] **Restore XenServer** (currently investigating offline status)
|
|
- [ ] **Verify Server3 VM status** once XenServer restored
|
|
|
|
**High Priority:**
|
|
- [ ] Complete onsite work (timer running)
|
|
- [ ] Reconfigure HP iLO settings (credentials, network)
|
|
- [ ] Document iLO IP address and credentials
|
|
- [ ] Verify all server settings match pre-incident configuration
|
|
|
|
**Follow-up:**
|
|
- [ ] Test remote management access (iLO, DRAC)
|
|
- [ ] Update server documentation with serial numbers and DRAC IPs
|
|
- [ ] Create follow-up preventive measures (UPS assessment critical)
|
|
|
|
## Server Information
|
|
|
|
**HP ProLiant Server:**
|
|
- Serial Number: MXQ80400X4
|
|
- Model: [TO BE DOCUMENTED]
|
|
- Role: VM Host (runs VWP_ADSRVR and other VMs)
|
|
- Location: Valleywide onsite
|
|
- Status: Reconfigured, operational
|
|
|
|
**HP iLO Management:**
|
|
- Status: Reset to factory defaults
|
|
- IP: [TO BE RECONFIGURED]
|
|
- Credentials: [TO BE RESET]
|
|
|
|
**Dell Server (VWP-QBS):**
|
|
- Model: Dell (with DRAC)
|
|
- Role: QuickBooks Server, RDS Host (Windows Server 2022)
|
|
- IP: 172.16.9.169
|
|
- Location: Valleywide onsite
|
|
- Status: Boot issue resolved, operational
|
|
- NOTE: Physical server, NOT a VM
|
|
|
|
**Dell DRAC Management:**
|
|
- Status: Functional (used to force manual boot)
|
|
- IP: [TO BE DOCUMENTED]
|
|
|
|
**XenServer (Older Dell):**
|
|
- Model: Dell (older hardware)
|
|
- Role: VM Host for Server3
|
|
- Location: Valleywide onsite
|
|
- Status: **OFFLINE - INVESTIGATING**
|
|
- Impact: Server3 VM unavailable
|
|
|
|
## Notes
|
|
|
|
- Power outage caused NVRAM corruption - rare but critical failure
|
|
- Quick recovery due to all VMs remaining intact
|
|
- iLO reconfiguration required for remote management
|
|
- Consider UPS assessment as preventive measure
|
|
|
|
---
|
|
|
|
**Work Status:** COMPLETE - Invoiced
|
|
**Resolved:** 2026-04-22 ~12:00 MST
|
|
|
|
---
|
|
|
|
## Message for Howard
|
|
|
|
Yealink password: n7*!O0qx&$IB$83*
|
|
|
|
— Mike, 2026-04-22
|