## User
- **User:** Howard Enos (howard)
- **Machine:** Howard-Home
- **Role:** tech

## Session Summary

Resumed the Cascades of Tucson planned-power-outage work at fire time on 2026-06-23.
The building had a scheduled electrical cut from 05:30-09:00 MST. The shutdown sequence
had been authored, automated, and armed the prior evening (2026-06-22 ~19:06) per the
runbook `clients/cascades-tucson/docs/runbooks/2026-06-23-planned-power-outage.md`; this
session was the fire-time verification that the arms executed cleanly.

Pulled the runbook back up to re-establish context. The plan: graceful self-shutdowns
on all three core devices (CS-SERVER, Synology, pfSense) so nothing takes a dirty
power-loss — critical because CS-SERVER runs a DEGRADED RAID-1 OS mirror on a single old
spindle, the worst-case box for an unclean cut. Each device was armed with a self-contained
local schedule so it fires independently of the Howard-Home session and the OpenVPN tunnel.

Attempted the "re-verify the arms" step but discovered via timezone math (UTC 12:30 =
05:30 MST, Phoenix UTC-7 no DST) that we were already AT fire time — past the
"verify before 05:28" window. The shutdowns had already executed. Confirmed clean
execution through the one independent channel that survives the site going dark: GuruRMM
cloud. CS-SERVER reported offline with last_seen 2026-06-23T12:29:49Z (05:29:49 MST) —
~1.5 min after its 05:28 scheduled task, exactly the expected graceful-shutdown lag
(stop CS-QB VM -> wait 25s -> Stop-Computer -Force). pfSense SSH was unreachable (exit 255),
consistent with it being the gateway/VPN endpoint and powering off. Synology had no remote
path (it sits behind the pfSense tunnel, now down) but was armed and verified the prior night.

Net result: the worst-case data-integrity risk (CS-SERVER) is positively confirmed handled.
We are now in the dark window with no remote visibility into the site by design — pfSense is
the OpenVPN endpoint, so once it powers off all in-site paths are gone; only RMM cloud reaches
CS-SERVER. Nothing further to send until the ~09:00 bring-up.

## Key Decisions

- Did NOT attempt to re-arm or re-send shutdown commands once timezone math showed we were past
  fire time — the arms had already fired and re-sending would be pointless/harmful.
- Treated the pfSense SSH exit-255 and the Synology no-path as EXPECTED (not errors): with pfSense
  down, the VPN endpoint is gone, so loss of in-site reachability is the designed outcome, not a fault.
- Relied on GuruRMM cloud as the single authoritative confirmation channel for CS-SERVER (the only box
  with an out-of-band path), since it is the highest-risk device (degraded RAID-1 mirror).
- Read-only checks only -> no #dev-alerts bot post (per rmm skill rule: alerts on writes only).

## Problems Encountered

- `pfsense-ssh.sh cascades-tucson run ...` returned exit 255 (SSH connect fail). Initially ambiguous;
  resolved by checking current time — it was 05:30:57 MST, i.e. pfSense had hit its 05:30 shutdown and/or
  the building cut had landed. Expected, not a fault.
- Git-Bash `TZ='America/Phoenix' date` printed GMT (no tzdata on this MSYS build). Worked around by
  computing MST manually from UTC (Phoenix = UTC-7, no DST): 12:30 UTC = 05:30 MST.

## Configuration Changes

- None. Read-only verification session. No files modified on endpoints or in the repo (this log only).

## Credentials & Secrets

- None discovered, created, or rotated. RMM auth via existing vault path
  `infrastructure/gururmm-server.sops.yaml` (admin-email / admin-password). pfSense via
  `clients/cascades-tucson/pfsense-firewall`. No new secrets.

## Infrastructure & Servers

- CS-SERVER: 192.168.2.254 (Windows), GuruRMM agent c39f1de7-d5b6-45ae-b132-e06977ab1713, agent ver 0.6.66.
  iDRAC 192.168.2.65. Confirmed offline last_seen 2026-06-23T12:29:49Z.
- Synology (CascadesDS): 192.168.0.120, DSM http://192.168.0.120:5000, vault
  clients/cascades-tucson/synology-cascadesds.
- pfSense: 192.168.0.1 (gateway + OpenVPN endpoint), pfSense Plus 25.07.
- UOS controller: https://172.16.3.29:11443, site va6iba3v / 685f39068e65331c46ef6dd2.
- GuruRMM API: http://172.16.3.30:3001.
- Outage window: 05:30-09:00 MST, 2026-06-23. Onsite: John Trozzi (physical power-on ~09:00).

## Commands & Outputs

- Time check: `date -u` -> 2026-06-23 12:30:57 UTC = 05:30:57 MST (Phoenix UTC-7).
- pfSense arm re-verify: `pfsense-ssh.sh cascades-tucson run 'ps -axww | grep ... shutdown -p now'`
  -> exit 255 (unreachable; expected — box down).
- CS-SERVER status (RMM): `curl $RMM/api/agents | jq 'select(.hostname=="CS-SERVER")'`
  -> `status=offline connected=null last_seen=2026-06-23T12:29:49.196769Z ver=0.6.66`.

## Pending / Incomplete Tasks

- BRING-UP at ~09:00 MST (John presses buttons, Howard monitors), bottom-up:
  1. pfSense first — verify SINGLE dhcpd (`pgrep -f "dhcpd -user" | wc -l` -> 1), WAN up
     (dpinger WAN_DHCP + WANCOAX_DHCP healthy). If WAN does NOT establish -> REBOOT THE COX MODEM
     (the missed post-restore step from the 6/17 unplanned outage).
  2. Switches/APs re-adopt — watch UOS controller until 12/12 switches + 77/77 APs connected.
  3. CS-SERVER boots -> verify AD/DNS, DHCP role, Hyper-V VMs (CS-QB), file + print shares. Then Synology.
  4. Straggler sweep: power-cycle printers/POS/IoT that cached a disconnected state. KNOWN: kitchen
     thermal printer (iPad POS ticket printer) — invisible to DHCP scan, fix by power-cycle.
- WATCH-LIST (6/17 casualties): Switch 2nd Floor #2 (USL24PB, 192.168.2.193) — one-way L2 break last time,
  floors 2/3/4 hang off it (reset + re-adopt if those floors don't return); duplicate dhcpd on pfSense;
  Cox modem / WAN.
- Open PRE-FLIGHT TODO (unverified): confirm pfSense + core/PoE switches are on the BATTERY side of the UPS
  (pfSense was surge-only on 6/17 until Mike moved it) — John/onsite to confirm.
- Optional: stand up a poll/loop ~08:55 to watch for WAN + controller recovery (offered, not yet set).

## Reference Information

- Runbook: clients/cascades-tucson/docs/runbooks/2026-06-23-planned-power-outage.md
- Incident basis (6/17 unplanned outage): clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md
- pfSense access: `.claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson run "<cmd>"`
- RMM auth bootstrap: `eval "$(bash .claude/scripts/rmm-auth.sh)"` -> $TOKEN, $RMM, $REPO_ROOT
- Standing rule: no Cascades prod change without explicit per-change go (memory feedback_cascades #4).
