azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from HOWARD-HOME at 2026-06-22 19:15:51

Browse Source 
Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-06-22 19:15:51
This commit is contained in:
Howard Enos
2026-06-22 19:16:22 -07:00
parent 86c789a7f9
commit 48b6c94b4a
3 changed files with 76 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
clients/cascades-tucson/docs/runbooks/2026-06-23-planned-power-outage.md
View File

@@ -35,24 +35,31 @@
---
## SHUTDOWN SEQUENCE (start 05:28) — dependents first, gateway last
> Start at **05:28**. All three target devices are on the UPS, so the battery carries them through the
> 05:30 cut — the graceful shutdowns complete on UPS power even if they run slightly past 5:30. Kick off
> CS-SERVER FIRST (slowest to come down); pfSense last.
## SHUTDOWN SEQUENCE — AUTOMATED + ARMED 2026-06-22 ~19:06
> **All three devices are armed to shut THEMSELVES down — no live action needed at fire time.**
> Each schedule is self-contained on its own box, so it fires even with the Howard-Home session idle
> and the OpenVPN tunnel dropped. All three are on the UPS, so the battery carries them cleanly through
> the 05:30 building cut. CS-SERVER + Synology at 05:28, pfSense (gateway) last at 05:30.
1. **CS-SERVER** (192.168.2.254) — graceful Windows shutdown via `rmm` (backup already verified).
DC/DNS/DHCP-role/Hyper-V/file+print server — the fragile box (degraded RAID). Best-effort stop
Hyper-V VMs first, then shut down:
```powershell
Get-VM | Where-Object State -eq 'Running' | Stop-VM -Force -ErrorAction SilentlyContinue
Stop-Computer -Force
```
(Dispatch via RMM `command_type: powershell`. Confirm the agent goes offline = down.)
2. **Synology cascadesDS** (192.168.0.120) — graceful DSM shutdown via API (auth verified tonight):
`SYNO.API.Auth login` → `SYNO.Core.System method=shutdown` (over the OpenVPN tunnel).
3. **UniFi switches/APs** — drop with building power (resilient; no special action). Note UPS-backed ones.
4. **pfSense** (192.168.0.1) — clean shutdown **last**, so DHCP/routing stays up while the rest drain:
`bash .claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson run "shutdown -p now"`
| Device | Fires (MST) | Mechanism | Armed/verified |
|---|---|---|---|
| **CS-SERVER** (192.168.2.254) | **05:28:00** | Windows one-time task `ACG-PlannedOutage-Shutdown` (runs as SYSTEM) → `C:\Windows\Temp\acg-outage-shutdown.ps1`: stop running Hyper-V VMs (only CS-QB) → wait 25s → `Stop-Computer -Force`. Run log: `C:\Windows\Temp\acg-outage-shutdown.run.log`. | Task "Ready", Next Run 6/23 5:28 AM; script syntax OK |
| **Synology** (192.168.0.120) | **05:28:00** | root detached `sleep → /sbin/poweroff` (systemd = clean DSM shutdown). Log: `/var/log/acg-sched-poweroff.log` | process live (pid 16009) |
| **pfSense** (192.168.0.1) | **05:30:00** | detached `sleep → /sbin/shutdown -p now`. Log: `/var/log/acg-sched-poweroff.log` | process live (pid 8793) |
- **UniFi switches/APs** — no action; they drop with building power at 05:30 (resilient).
- **Caveat:** the Synology + pfSense sleepers are in-memory — if either box reboots before 05:28 the arm
is lost (CS-SERVER's task is persistent and survives reboot). Re-verify at ~05:25 (see below).
### Re-verify the arms (anytime before 05:28)
- CS-SERVER: RMM → `schtasks /query /tn "ACG-PlannedOutage-Shutdown" /v /fo LIST`
- pfSense: `pfsense-ssh.sh cascades-tucson run 'ps -axww | grep "shutdown -p now" | grep -v grep'`
- Synology: SSH (admin) → `sudo ps -ef | grep /sbin/poweroff | grep -v grep`
### CANCEL the shutdowns (if the outage is called off)
- CS-SERVER: RMM → `schtasks /delete /tn "ACG-PlannedOutage-Shutdown" /f`
- pfSense: `pfsense-ssh.sh cascades-tucson run 'pkill -f "shutdown -p now"; pkill -f "sleep 374"'`
- Synology: SSH (admin) → `echo "<pw>" | sudo -S sh -c 'pkill -f "/sbin/poweroff"; pkill -f "sleep 372"'`
---