# 2026-04-22 — Valleywide HP Server NVRAM Corruption Emergency

## User
- **User:** Mike Swanson (mike)
- **Machine:** Mikes-MacBook-Air.local
- **Role:** admin

## Ticket Information
- **Type:** Emergency onsite
- **Priority:** Critical
- **Status:** In Progress
- **Arrival:** 0935 MST

## Issue Summary

Multiple server issues following power outage at Valleywide:
- HP ProLiant Server (SN: MXQ80400X4): Non-volatile memory corruption, BIOS/iLO reset required
- Dell Server (VWP-QBS): Boot retry loop, resolved via DRAC manual boot
- **XenServer (Older Dell): OFFLINE - investigating (CRITICAL)**

## Timeline

### 0935 - Arrival Onsite
- HP Server SN: MXQ80400X4
- Issue: Non-Volatile Memory Corruption
- Cause: Power outage
- Impact: BIOS/UEFI reset to factory defaults

### 0935-[IN PROGRESS] - Recovery Actions

**HP ProLiant Server (SN: MXQ80400X4):**

**BIOS/UEFI Reconfiguration:**
- Factory reset required due to NVRAM corruption
- Reconfigured BIOS settings
- Restored boot order
- Re-enabled virtualization settings

**iLO (Integrated Lights-Out) Reset:**
- [WARNING] iLO was reset to factory defaults due to BIOS reset
- iLO credentials will need to be re-entered
- Network configuration may need restoration
- Remote management temporarily unavailable until iLO reconfigured

**VM Status:**
- [OK] All VMs running
- Hypervisor operational after BIOS reconfiguration
- No VM data loss reported

**Dell Server (VWP-QBS) - Separate Boot Issue:**

**Boot Retry Loop:**
- VWP-QBS (Dell physical server, 172.16.9.169) stuck at "Boot Retry" screen
- Accessed via DRAC (Dell Remote Access Controller)
- Forced manual boot device selection -> Windows Boot Manager
- [OK] Server booted successfully
- [OK] Server appears to be functioning normally now
- Likely related to power outage affecting boot order/configuration
- NOTE: VWP-QBS is NOT a VM - it's a separate physical Dell server

**XenServer (Older Dell) - OFFLINE:**

**Status:**
- [CRITICAL] XenServer offline
- Impact: Server3 VM unavailable
- Investigating cause (likely power outage related)
- Checking hardware status, boot sequence, and hypervisor state
- Dell server - older hardware

## Next Steps

**CRITICAL:**
- [ ] **Restore XenServer** (currently investigating offline status)
- [ ] **Verify Server3 VM status** once XenServer restored

**High Priority:**
- [ ] Complete onsite work (timer running)
- [ ] Reconfigure HP iLO settings (credentials, network)
- [ ] Document iLO IP address and credentials
- [ ] Verify all server settings match pre-incident configuration

**Follow-up:**
- [ ] Test remote management access (iLO, DRAC)
- [ ] Update server documentation with serial numbers and DRAC IPs
- [ ] Create follow-up preventive measures (UPS assessment critical)

## Server Information

**HP ProLiant Server:**
- Serial Number: MXQ80400X4
- Model: [TO BE DOCUMENTED]
- Role: VM Host (runs VWP_ADSRVR and other VMs)
- Location: Valleywide onsite
- Status: Reconfigured, operational

**HP iLO Management:**
- Status: Reset to factory defaults
- IP: [TO BE RECONFIGURED]
- Credentials: [TO BE RESET]

**Dell Server (VWP-QBS):**
- Model: Dell (with DRAC)
- Role: QuickBooks Server, RDS Host (Windows Server 2022)
- IP: 172.16.9.169
- Location: Valleywide onsite
- Status: Boot issue resolved, operational
- NOTE: Physical server, NOT a VM

**Dell DRAC Management:**
- Status: Functional (used to force manual boot)
- IP: [TO BE DOCUMENTED]

**XenServer (Older Dell):**
- Model: Dell (older hardware)
- Role: VM Host for Server3
- Location: Valleywide onsite
- Status: **OFFLINE - INVESTIGATING**
- Impact: Server3 VM unavailable

## Notes

- Power outage caused NVRAM corruption - rare but critical failure
- Quick recovery due to all VMs remaining intact
- iLO reconfiguration required for remote management
- Consider UPS assessment as preventive measure

---

**Work Status:** COMPLETE - Invoiced
**Resolved:** 2026-04-22 ~12:00 MST

---

## Message for Howard

Yealink password: n7*!O0qx&$IB$83*

— Mike, 2026-04-22