# Runbook — Cascades Planned Power Outage (2026-06-23, 05:30–09:00)

- **Client:** Cascades of Tucson
- **Event:** Building-wide electrical work; **no power 05:30–09:00 MST, 2026-06-23**.
- **Authored:** 2026-06-22 by Howard Enos (Howard-Home).
- **Why this runbook exists:** Avoid a repeat of the **2026-06-17 unplanned outage**, where a
  *dirty* pfSense power-loss caused duplicate dhcpd + a 2nd-floor switch with one-way L2 forwarding
  + a Cox-modem-reboot scramble. This outage is **planned**, so we shut down **clean** and skip all of it.
  Incident basis: `clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md`.

## Roles
- **Howard (remote):** graceful shutdowns at ~05:20; monitors + remediates bring-up from 09:00.
- **John Trozzi (onsite):** physical power-on of CS-SERVER + Synology at ~09:00 once power is confirmed stable.
- **Standing rule:** no Cascades prod change without explicit per-change go (memory `feedback_cascades` #4).

## Decision: CLEAN graceful shutdown, do NOT let the UPS drain
- CS-SERVER has a **DEGRADED RAID-1 OS mirror on a single old spindle** — a dirty power-loss is the
  worst-case data-integrity event. Graceful shutdown is mandatory.
- Once everything is soft-off the UPS load collapses, so it may **not** drain in 3.5h → devices won't
  see a power-loss→restore event → they **stay off**. So we plan on **John powering devices on at 09:00**;
  AC-recovery/auto-restart settings are only a backstop.

---

## PRE-FLIGHT — VERIFIED 2026-06-22 ~18:50 (read-only)

| Check | Result |
|---|---|
| CS-SERVER cloud backup last-run | **[OK] Success @ 2026-06-22 00:11** — MSP360 FFI file backup, Full consistency check PASSED, 578.2 GB scanned, 6.81 GB uploaded, **0 failed / 0 errors**. (File-level FFI, not bare-metal image — image-based remains a separate roadmap item.) |
| CS-SERVER iDRAC "AC Power Recovery" (backstop) | **[OK] On** (verified via OMSA `omreport chassis biossetup` → "AC Power Recovery Mode: On" — self-boots when AC returns) |
| Synology "Restart automatically after power failure" (backstop) | **[OK] Enabled** (DSM API `SYNO.Core.Hardware.PowerRecovery` → `rc_power_config: true`) |
| CS-SERVER GuruRMM agent (resolve live) | **[OK]** `c39f1de7-d5b6-45ae-b132-e06977ab1713`, online (last seen 01:46Z 6/23) |
| Remote access paths (for the 05:20 shutdowns) | **[OK] pfSense** via OpenVPN tunnel (up 4d, single clean dhcpd). **[OK] Synology** DSM API over tunnel (auth OK, reachable). **[OK] CS-SERVER** via RMM cloud. |
| What is on battery-backed UPS outlets (pfSense was on surge-only 6/17 — Mike moved it) | **TODO — John/onsite** confirm pfSense + core/PoE switches on battery side |

---

## SHUTDOWN SEQUENCE — AUTOMATED + ARMED 2026-06-22 ~19:06
> **All three devices are armed to shut THEMSELVES down — no live action needed at fire time.**
> Each schedule is self-contained on its own box, so it fires even with the Howard-Home session idle
> and the OpenVPN tunnel dropped. All three are on the UPS, so the battery carries them cleanly through
> the 05:30 building cut. CS-SERVER + Synology at 05:28, pfSense (gateway) last at 05:30.

| Device | Fires (MST) | Mechanism | Armed/verified |
|---|---|---|---|
| **CS-SERVER** (192.168.2.254) | **05:28:00** | Windows one-time task `ACG-PlannedOutage-Shutdown` (runs as SYSTEM) → `C:\Windows\Temp\acg-outage-shutdown.ps1`: stop running Hyper-V VMs (only CS-QB) → wait 25s → `Stop-Computer -Force`. Run log: `C:\Windows\Temp\acg-outage-shutdown.run.log`. | Task "Ready", Next Run 6/23 5:28 AM; script syntax OK |
| **Synology** (192.168.0.120) | **05:28:00** | root detached `sleep → /sbin/poweroff` (systemd = clean DSM shutdown). Log: `/var/log/acg-sched-poweroff.log` | process live (pid 16009) |
| **pfSense** (192.168.0.1) | **05:30:00** | detached `sleep → /sbin/shutdown -p now`. Log: `/var/log/acg-sched-poweroff.log` | process live (pid 8793) |

- **UniFi switches/APs** — no action; they drop with building power at 05:30 (resilient).
- **Caveat:** the Synology + pfSense sleepers are in-memory — if either box reboots before 05:28 the arm
  is lost (CS-SERVER's task is persistent and survives reboot). Re-verify at ~05:25 (see below).

### Re-verify the arms (anytime before 05:28)
- CS-SERVER: RMM → `schtasks /query /tn "ACG-PlannedOutage-Shutdown" /v /fo LIST`
- pfSense: `pfsense-ssh.sh cascades-tucson run 'ps -axww | grep "shutdown -p now" | grep -v grep'`
- Synology: SSH (admin) → `sudo ps -ef | grep /sbin/poweroff | grep -v grep`

### CANCEL the shutdowns (if the outage is called off)
- CS-SERVER: RMM → `schtasks /delete /tn "ACG-PlannedOutage-Shutdown" /f`
- pfSense: `pfsense-ssh.sh cascades-tucson run 'pkill -f "shutdown -p now"; pkill -f "sleep 374"'`
- Synology: SSH (admin) → `echo "<pw>" | sudo -S sh -c 'pkill -f "/sbin/poweroff"; pkill -f "sleep 372"'`

---

## POWER-ON SEQUENCE (~09:00, John presses buttons, Howard monitors) — bottom-up

1. **pfSense first.** Verify single dhcpd and WAN up:
   - `pgrep -f "dhcpd -user" | wc -l` → must be **1** (not 2 — the 6/17 failure).
   - WAN up: dpinger WAN_DHCP + WANCOAX_DHCP healthy.
   - **If WAN does NOT establish → reboot the Cox modem** (the missed post-restore step on 6/17).
2. **Switching/APs re-adopt** (core → distribution → APs). UniFi is SLOW here — watch the UOS controller
   (172.16.3.29, site `va6iba3v`) until **12/12 switches + 77/77 APs** report connected.
3. **CS-SERVER** boots → verify AD/DNS, DHCP role, Hyper-V VMs, file + print shares. Then **Synology**.
4. **Straggler sweep:** power-cycle any printer/POS/IoT that booted into a DHCP-down window and cached a
   disconnected state. **Known: kitchen thermal printer (iPad POS ticket printer).** These are invisible
   to a DHCP scan (they stop requesting) — expect a few "won't connect" reports, each fixed by a power-cycle.

---

## WATCH-LIST (the 2026-06-17 casualties — confirm each recovers)
- **Switch 2nd Floor #2** (USL24PB, `192.168.2.193`) — broke one-way L2 forwarding last time; floors 2/3/4
  hang off it. If those floors don't come back → **reset + re-adopt** the switch.
- **Duplicate dhcpd** on pfSense — a clean shutdown should prevent it, but verify (step 1 above).
- **Cox modem / WAN** — reboot if WAN doesn't re-establish.

## Access reference
- pfSense: `clients/cascades-tucson/pfsense-firewall` · `pfsense-ssh.sh cascades-tucson run "<cmd>"` (logs PLAIN TEXT, not clog).
- CS-SERVER: GuruRMM (`rmm`, resolve agent live) / ScreenConnect. iDRAC `192.168.2.65`.
- Synology DSM: `http://192.168.0.120:5000` — vault `clients/cascades-tucson/synology-cascadesds`.
- UOS controller: `https://172.16.3.29:11443`, site `va6iba3v` / `685f39068e65331c46ef6dd2`.
</content>
</invoke>
