sync: auto-sync from HOWARD-HOME at 2026-06-22 18:54:25
Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-22 18:54:25
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# Runbook — Cascades Planned Power Outage (2026-06-23, 05:30–09:00)
|
||||
|
||||
- **Client:** Cascades of Tucson
|
||||
- **Event:** Building-wide electrical work; **no power 05:30–09:00 MST, 2026-06-23**.
|
||||
- **Authored:** 2026-06-22 by Howard Enos (Howard-Home).
|
||||
- **Why this runbook exists:** Avoid a repeat of the **2026-06-17 unplanned outage**, where a
|
||||
*dirty* pfSense power-loss caused duplicate dhcpd + a 2nd-floor switch with one-way L2 forwarding
|
||||
+ a Cox-modem-reboot scramble. This outage is **planned**, so we shut down **clean** and skip all of it.
|
||||
Incident basis: `clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md`.
|
||||
|
||||
## Roles
|
||||
- **Howard (remote):** graceful shutdowns at ~05:20; monitors + remediates bring-up from 09:00.
|
||||
- **John Trozzi (onsite):** physical power-on of CS-SERVER + Synology at ~09:00 once power is confirmed stable.
|
||||
- **Standing rule:** no Cascades prod change without explicit per-change go (memory `feedback_cascades` #4).
|
||||
|
||||
## Decision: CLEAN graceful shutdown, do NOT let the UPS drain
|
||||
- CS-SERVER has a **DEGRADED RAID-1 OS mirror on a single old spindle** — a dirty power-loss is the
|
||||
worst-case data-integrity event. Graceful shutdown is mandatory.
|
||||
- Once everything is soft-off the UPS load collapses, so it may **not** drain in 3.5h → devices won't
|
||||
see a power-loss→restore event → they **stay off**. So we plan on **John powering devices on at 09:00**;
|
||||
AC-recovery/auto-restart settings are only a backstop.
|
||||
|
||||
---
|
||||
|
||||
## PRE-FLIGHT — VERIFIED 2026-06-22 ~18:50 (read-only)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| CS-SERVER cloud backup last-run | **[OK] Success @ 2026-06-22 00:11** — MSP360 FFI file backup, Full consistency check PASSED, 578.2 GB scanned, 6.81 GB uploaded, **0 failed / 0 errors**. (File-level FFI, not bare-metal image — image-based remains a separate roadmap item.) |
|
||||
| CS-SERVER iDRAC "AC Power Recovery" (backstop) | **[OK] On** (verified via OMSA `omreport chassis biossetup` → "AC Power Recovery Mode: On" — self-boots when AC returns) |
|
||||
| Synology "Restart automatically after power failure" (backstop) | **[OK] Enabled** (DSM API `SYNO.Core.Hardware.PowerRecovery` → `rc_power_config: true`) |
|
||||
| CS-SERVER GuruRMM agent (resolve live) | **[OK]** `c39f1de7-d5b6-45ae-b132-e06977ab1713`, online (last seen 01:46Z 6/23) |
|
||||
| Remote access paths (for the 05:20 shutdowns) | **[OK] pfSense** via OpenVPN tunnel (up 4d, single clean dhcpd). **[OK] Synology** DSM API over tunnel (auth OK, reachable). **[OK] CS-SERVER** via RMM cloud. |
|
||||
| What is on battery-backed UPS outlets (pfSense was on surge-only 6/17 — Mike moved it) | **TODO — John/onsite** confirm pfSense + core/PoE switches on battery side |
|
||||
|
||||
---
|
||||
|
||||
## SHUTDOWN SEQUENCE (start 05:28) — dependents first, gateway last
|
||||
> Start at **05:28**. All three target devices are on the UPS, so the battery carries them through the
|
||||
> 05:30 cut — the graceful shutdowns complete on UPS power even if they run slightly past 5:30. Kick off
|
||||
> CS-SERVER FIRST (slowest to come down); pfSense last.
|
||||
|
||||
1. **CS-SERVER** (192.168.2.254) — graceful Windows shutdown via `rmm` (backup already verified).
|
||||
DC/DNS/DHCP-role/Hyper-V/file+print server — the fragile box (degraded RAID). Best-effort stop
|
||||
Hyper-V VMs first, then shut down:
|
||||
```powershell
|
||||
Get-VM | Where-Object State -eq 'Running' | Stop-VM -Force -ErrorAction SilentlyContinue
|
||||
Stop-Computer -Force
|
||||
```
|
||||
(Dispatch via RMM `command_type: powershell`. Confirm the agent goes offline = down.)
|
||||
2. **Synology cascadesDS** (192.168.0.120) — graceful DSM shutdown via API (auth verified tonight):
|
||||
`SYNO.API.Auth login` → `SYNO.Core.System method=shutdown` (over the OpenVPN tunnel).
|
||||
3. **UniFi switches/APs** — drop with building power (resilient; no special action). Note UPS-backed ones.
|
||||
4. **pfSense** (192.168.0.1) — clean shutdown **last**, so DHCP/routing stays up while the rest drain:
|
||||
`bash .claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson run "shutdown -p now"`
|
||||
|
||||
---
|
||||
|
||||
## POWER-ON SEQUENCE (~09:00, John presses buttons, Howard monitors) — bottom-up
|
||||
|
||||
1. **pfSense first.** Verify single dhcpd and WAN up:
|
||||
- `pgrep -f "dhcpd -user" | wc -l` → must be **1** (not 2 — the 6/17 failure).
|
||||
- WAN up: dpinger WAN_DHCP + WANCOAX_DHCP healthy.
|
||||
- **If WAN does NOT establish → reboot the Cox modem** (the missed post-restore step on 6/17).
|
||||
2. **Switching/APs re-adopt** (core → distribution → APs). UniFi is SLOW here — watch the UOS controller
|
||||
(172.16.3.29, site `va6iba3v`) until **12/12 switches + 77/77 APs** report connected.
|
||||
3. **CS-SERVER** boots → verify AD/DNS, DHCP role, Hyper-V VMs, file + print shares. Then **Synology**.
|
||||
4. **Straggler sweep:** power-cycle any printer/POS/IoT that booted into a DHCP-down window and cached a
|
||||
disconnected state. **Known: kitchen thermal printer (iPad POS ticket printer).** These are invisible
|
||||
to a DHCP scan (they stop requesting) — expect a few "won't connect" reports, each fixed by a power-cycle.
|
||||
|
||||
---
|
||||
|
||||
## WATCH-LIST (the 2026-06-17 casualties — confirm each recovers)
|
||||
- **Switch 2nd Floor #2** (USL24PB, `192.168.2.193`) — broke one-way L2 forwarding last time; floors 2/3/4
|
||||
hang off it. If those floors don't come back → **reset + re-adopt** the switch.
|
||||
- **Duplicate dhcpd** on pfSense — a clean shutdown should prevent it, but verify (step 1 above).
|
||||
- **Cox modem / WAN** — reboot if WAN doesn't re-establish.
|
||||
|
||||
## Access reference
|
||||
- pfSense: `clients/cascades-tucson/pfsense-firewall` · `pfsense-ssh.sh cascades-tucson run "<cmd>"` (logs PLAIN TEXT, not clog).
|
||||
- CS-SERVER: GuruRMM (`rmm`, resolve agent live) / ScreenConnect. iDRAC `192.168.2.65`.
|
||||
- Synology DSM: `http://192.168.0.120:5000` — vault `clients/cascades-tucson/synology-cascadesds`.
|
||||
- UOS controller: `https://172.16.3.29:11443`, site `va6iba3v` / `685f39068e65331c46ef6dd2`.
|
||||
</content>
|
||||
</invoke>
|
||||
Reference in New Issue
Block a user