azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

wiki: compile cascades-tucson (full) — 2026-06-23 planned power outage

Browse Source 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Howard Enos
2026-06-23 05:45:34 -07:00
parent beb8a36ff2
commit 3e414c1572
2 changed files with 22 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
wiki/clients/cascades-tucson.md
View File

@@ -2,8 +2,8 @@
type: client
name: cascades-tucson
display_name: Cascades of Tucson
last_compiled: 2026-06-20
compiled_by: GURU-5070/claude-main
last_compiled: 2026-06-23
compiled_by: HOWARD-HOME/claude-main
sources:
- session-logs/2026-03-24-session.md
- session-logs/2026-03-31-session.md
@@ -86,6 +86,8 @@ sources:
- clients/cascades-tucson/session-logs/2026-06/2026-06-19-howard-cascades-rf-night-capstone.md
- clients/cascades-tucson/session-logs/2026-06/2026-06-19-howard-voice-vlan-migration-complete-and-vertical-handoff.md
- clients/cascades-tucson/docs/network/2026-06-19-vertical-5ghz-lock-request.md
- clients/cascades-tucson/docs/runbooks/2026-06-23-planned-power-outage.md
- clients/cascades-tucson/session-logs/2026-06/2026-06-23-howard-cascades-planned-outage-shutdown-verify.md
backlinks:
- projects/gururmm
- wiki/systems/uos-server
@@ -153,10 +155,10 @@ Because per-user **Intune** never provisioned tenant-wide (`INTUNE_A = PendingIn
- Lupe Sanchez -- staff (DESKTOP-TRCIEJA). EOL workstation (Gateway ZX6971 AIO, i3-2120, 8 GB RAM, Win11 unsupported). **Decision 2026-06-18: replace machine** (dual-AV + EOL hardware causing slow Excel; no remediation on current box). GuruRMM agent `c9bf1a2d-bfdc-401e-9cc8-f9e90bb19587` (resolve live by hostname; UUIDs change on re-enroll).
- **Syncro contact emails (authoritative):** ashley.jensen@, jd.martin@, crystal.rodriguez@, John.trozzi@, meredith.kuhn@, accounting@/accountingassistant@cascadestucson.com.
- **Billing rate:** $175/hr all labor (prepaid block customer)
- **Hours remaining:** **48.75 hrs as of 2026-06-20 (live Syncro).** Most recent draw: 7h remote+onsite 2026-06-19 voice VLAN + RF optimization (ticket #32444, 55.75->48.75). Prior: 0.5h remote 2026-06-12 shared mailboxes (ticket #32417, 56.25->55.75); 0.5h remote 2026-06-10 Meredith locked Word doc (ticket #32403, 56.75->56.25). Always live-check via `GET /customers/20149445` before billing.
- **Hours remaining:** **48.75 hrs as of 2026-06-23 (live Syncro -- unchanged since 2026-06-20; the 2026-06-23 planned outage is monitoring, not yet billed).** Most recent draw: 7h remote+onsite 2026-06-19 voice VLAN + RF optimization (ticket #32444, 55.75->48.75). Prior: 0.5h remote 2026-06-12 shared mailboxes (ticket #32417, 56.25->55.75); 0.5h remote 2026-06-10 Meredith locked Word doc (ticket #32403, 56.75->56.25). Always live-check via `GET /customers/20149445` before billing.
- **Syncro customer ID:** 20149445
- **Managed devices (Syncro):** 29 (live 2026-06-20)
- **Active tickets:** 0 open Syncro tickets as of 2026-06-20. See Active Work for open non-ticketed projects.
- **Managed devices (Syncro):** 29 (live 2026-06-23)
- **Active tickets:** 0 open Syncro tickets as of 2026-06-23. See Active Work for open non-ticketed projects.
- #110680053 / #32303 -- Entra / domain migration project. Status: **Invoiced** as of 2026-06-05. Plan: `C:\Users\Howard\.claude\plans\wise-discovering-panda.md`
- #109412123 -- Entra setup project (verify status)
- #32403 -- Meredith locked Word doc (0.5h remote, billed 2026-06-10, Invoiced)
@@ -185,6 +187,8 @@ Because per-user **Intune** never provisioned tenant-wide (`INTUNE_A = PendingIn
**[WARN] Power outage (2026-06-17):** Building power outage took the entire Cascades network down. Root cause: pfSense was plugged into the **surge-only side of the UPS** (no battery) -- it hard-powered-off uncleanly. ZFS survived. Dirty boot caused a **duplicate dhcpd** and a **2nd-floor switch (USL24PB, 192.168.2.193) with one-way L2 forwarding** blocking DHCP OFFERs. Howard killed the duplicate dhcpd remotely; Mike re-seated pfSense onto battery outlets, restored config from on-box auto-backup (12:20 version, VLAN30 intact), reset+re-adopted Switch 2nd Floor #2. Network fully restored. Post-recovery casualties: devices that booted during DHCP-down window cached disconnected state (kitchen thermal printer fixed by power-cycle). Incident report: `clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md`.
**[INFO] Planned power outage (2026-06-23, 05:30-09:00 MST) -- clean shutdown executed:** Building-wide electrical work scheduled a 3.5h power cut. To avoid a repeat of the 6/17 dirty-shutdown damage (and because CS-SERVER's OS mirror is degraded), all three core devices were armed the prior evening (2026-06-22 ~19:06) to **shut THEMSELVES down** on self-contained local schedules -- CS-SERVER (Windows task `ACG-PlannedOutage-Shutdown` -> stop CS-QB VM -> `Stop-Computer`, 05:28), Synology (`/sbin/poweroff`, 05:28), pfSense (`shutdown -p now`, 05:30) -- so they fired independent of any remote session or the OpenVPN tunnel, with the UPS carrying them through the 05:30 cut. **Verified clean (2026-06-23 05:31 MST):** CS-SERVER confirmed offline via GuruRMM cloud at last_seen 05:29:49 MST (the one out-of-band channel; expected ~1.5 min graceful-shutdown lag); pfSense + Synology unreachable as expected (pfSense is the VPN endpoint -- once down, all in-site paths drop). Pre-flight (2026-06-22) verified: cloud backup last full SUCCESS @ 6/22 00:11 (0 errors); iDRAC AC Power Recovery ON + Synology auto-restart ON (boot backstops); John Trozzi onsite for physical power-on ~09:00. Bring-up is bottom-up: pfSense first (verify SINGLE dhcpd, WAN up, reboot Cox modem if WAN fails) -> switches/APs re-adopt (12/12 + 77/77) -> CS-SERVER -> Synology -> straggler sweep. Runbook: `clients/cascades-tucson/docs/runbooks/2026-06-23-planned-power-outage.md`.
### Email & Identity
- **M365 tenant:** cascadestucson.com | Tenant ID: `207fa277-e9d8-4eb7-ada1-1064d2221498`
@@ -439,7 +443,10 @@ Full design: `docs/network/phase1-voice-qos-design.md`. Status DESIGN -- nothing
Syncro live pull 2026-06-20: **0 open tickets.**
**Non-Syncro follow-ups open as of 2026-06-20:**
**Non-Syncro follow-ups open as of 2026-06-23:**
- **[TODAY 2026-06-23 ~09:00] Planned-outage bring-up + monitoring.** Power returns ~09:00 MST; John Trozzi powers on CS-SERVER + Synology. Howard monitors bottom-up: pfSense (verify SINGLE dhcpd `pgrep -f "dhcpd -user" | wc -l`==1, WAN up -- **reboot Cox modem if WAN doesn't establish**, the missed 6/17 step) -> switches/APs re-adopt (watch UOS controller for 12/12 switches + 77/77 APs) -> CS-SERVER (AD/DNS, DHCP, Hyper-V CS-QB, shares) -> Synology -> straggler sweep (known: kitchen thermal printer). **Watch-list (6/17 casualties):** Switch 2nd Floor #2 (USL24PB 192.168.2.193, one-way L2 break -- reset+re-adopt if floors 2/3/4 don't return); duplicate dhcpd. Clean shutdown verified at 05:31 (CS-SERVER offline via RMM cloud). Runbook: `docs/runbooks/2026-06-23-planned-power-outage.md`.
- **[OPEN -- from runbook pre-flight] Confirm pfSense + core/PoE switches are on the BATTERY side of the UPS.** pfSense was on surge-only on 6/17 until Mike moved it; the other gear's battery-vs-surge placement was still "TODO -- John/onsite" at the 2026-06-22 pre-flight. Verify onsite.
- **[URGENT] Order replacement workstation for Lupe Sanchez (DESKTOP-TRCIEJA).** Decision made 2026-06-18. EOL Gateway ZX6971 / i3-2120 / 8 GB / Win11-unsupported. On new machine: provision GuruRMM + Bitdefender only; do NOT carry over the Datto stack.
- **[URGENT] Rotate exposed Synology Cloud Signin Portal credential.** Vault commit 1fbc0e1 committed it plaintext; encrypted go-forward but credential is exposed in git history. Also verify MDM service account + WiFi CSCNet from that same commit were never plaintext.
@@ -537,11 +544,19 @@ Syncro live pull 2026-06-20: **0 open tickets.**
| 2026-06-18 | **Power outage follow-ups: OpenVPN flapping root-caused (--inactive timeout, not a fault); kitchen printer straggler resolved by power-cycle.** |
| 2026-06-19 | **PRODUCTION RF OPTIMIZATION APPLIED (autonomous 2 AM window) -- 5 GHz retry HALVED.** 2.4 power -> MEDIUM on 47 radios (over-thinning fix + MemCare off full power; per-AP targeting). CSCNet BSS-transition ON. 6 GHz attempted but BLOCKED (`Wpa3MandatoryFor6GHzBand`). Blind non-DFS 5 GHz reshuffle tried, failed, rolled back. Howard's correction: scan FIRST, decide from data. Full channel survey (74/74 APs) proved DFS channels here 4-5x cleaner (2-3%) than non-DFS (ch149=12%, ch157=28%). Data-driven clean-DFS plan (8 DFS 40MHz channels, per-AP cleanest + neighbor graph-color, 0 co-channel) applied to 72 non-mesh APs. **Result: 5 GHz retry 8.7->3.8 avg (median 8.2->2.1), satisfaction median 99, all 72 APs holding DFS, 0 radar vacates.** `survey-report.py` added; `channel-plan.sh` made data-driven. |
| 2026-06-19 | **Voice VLAN migration COMPLETE (29/29 Poly) + band-selection diagnosis + Vertical 5 GHz handoff.** Howard walked the building, re-keyed all remaining Poly handsets to voice PPSK. Per-phone re-look: most phones on clean 5 GHz (Lauren .202: 2.4/50% -> 5GHz/12%), but several stuck on 2.4 despite -50 to -60 dBm signal -- controller band-steering not holding Poly OUI on 5 GHz. Phone-side fix: **5 GHz-only lock request sent to Richard Turner (Vertical)**, awaiting response = the last voice item. Kitchen server phone bad (pulled by John); Bistro phone relocated to Kitchen; Bistro now has no phone (replacement pending). Billed ticket #32444 (7h: 4 onsite + 3 remote), block 55.75->48.75. |
| 2026-06-23 | **Planned power outage (05:30-09:00 MST) -- clean shutdown executed + verified.** Building electrical work; to avoid the 6/17 dirty-shutdown damage (and given CS-SERVER's degraded OS mirror), all three core devices were armed 6/22 ~19:06 to self-shut-down on local schedules (CS-SERVER task 05:28, Synology 05:28, pfSense 05:30) -- firing independent of any remote session/tunnel, UPS carrying them through the cut. Verified clean at 05:31: CS-SERVER offline via RMM cloud (last_seen 05:29:49 MST); pfSense/Synology unreachable as expected (pfSense = VPN endpoint). Pre-flight confirmed cloud backup last full SUCCESS (0 errors), iDRAC AC-recovery + Synology auto-restart backstops ON. Bring-up (~09:00, John onsite) pending. Runbook: `docs/runbooks/2026-06-23-planned-power-outage.md`. |
---
## Compilation Notes
**2026-06-23 recompile (HOWARD-HOME/claude-main) changes vs. prior (2026-06-20, GURU-5070):**
- Surgical/additive full recompile -- the prior compile was current; the only new knowledge was the 2026-06-23 planned power outage. All other sections preserved verbatim.
- Billing re-verified live (Syncro): 48.75 hrs / 29 devices / 0 open tickets -- unchanged since 2026-06-20; "as of" dates advanced to 2026-06-23. Outage day is monitoring, not yet billed.
- Infrastructure: added [INFO] planned-outage block (clean self-shutdown armed 6/22, executed + verified clean 6/23 05:31).
- Active Work: added [TODAY] bring-up/monitoring item + [OPEN] UPS battery-side verification (from runbook pre-flight).
- History Highlights: added 2026-06-23 planned-outage entry. Sources: added the runbook + the 2026-06-23 session log.
**2026-06-20 recompile (GURU-5070/claude-main) changes vs. prior (2026-06-19, HOWARD-HOME):**
- Billing updated: 48.75 hrs as of 2026-06-20 (Syncro authoritative); ticket #32444 (7h) reflected in block balance and ticket list.
- Infrastructure > Network > Wireless RF section updated: replaced stale "OVER-THINNED (as of 2026-06-17)" and "NOT applied (pending go-ahead)" narrative with the actual applied 2026-06-19 state (2.4 Medium, 5 GHz clean DFS 40MHz, results).

2
wiki/index.md
View File

@@ -18,7 +18,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks.
| Article | Summary | Last Compiled |
|---|---|---|
| [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, **48.75 hrs remaining** (live 2026-06-20); senior living; active domain migration + HIPAA caregiver-lockdown project (GPOs deployed; Entra Hybrid Join + CA allow-list + ALIS SSO model proven); single DC (CS-SERVER) on aging R610, OS RAID-1 degraded 2026-06-15 (data-loss risk; cloud backup started); **Voice VLAN 30 migration COMPLETE 2026-06-19** (~38 devices: 29 Poly + 8 AudioCodes + desktop; awaiting Vertical to set Poly 5GHz-only); **UniFi RF optimized 2026-06-19** (77 U7-Pro APs/~587 clients: 2.4GHz power->Medium on 47 radios + 5GHz clean-DFS 40MHz channel plan -> 5GHz retry halved; 6GHz blocked by WPA3 on PPSK SSID); Syncro 0 open tickets | 2026-06-20 |
| [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, **48.75 hrs remaining** (live 2026-06-23); senior living; active domain migration + HIPAA caregiver-lockdown project (GPOs deployed; Entra Hybrid Join + CA allow-list + ALIS SSO model proven); single DC (CS-SERVER) on aging R610, OS RAID-1 degraded 2026-06-15 (data-loss risk; cloud backup started); **Planned power outage 2026-06-23** clean self-shutdown executed + verified (bring-up ~09:00, John onsite); **Voice VLAN 30 migration COMPLETE 2026-06-19** (~38 devices: 29 Poly + 8 AudioCodes + desktop; awaiting Vertical to set Poly 5GHz-only); **UniFi RF optimized 2026-06-19** (77 U7-Pro APs/~587 clients: 2.4GHz power->Medium on 47 radios + 5GHz clean-DFS 40MHz channel plan -> 5GHz retry halved; 6GHz blocked by WPA3 on PPSK SSID); Syncro 0 open tickets | 2026-06-23 |
| [Dataforth Corporation](clients/dataforth.md) | Prepaid block ~$2,099/mo, **31.5 hrs remaining** (live 2026-06-23); signal-conditioning manufacturer; 64 DOS test stations; 2025 ransomware recovery + incomplete file restore (migration-gap audit); 2026-03 phishing + MFA rollout; test-datasheet pipeline (DSCA cert publish via Hoffman API + testdatadb UI on AD2); mail stack INKY->Mailprotector CloudFilter->EXO; FreePBX 17 outage fixed 2026-06-08/09 (qualify_frequency=0; no RTP-forward); shares-ACL project (all open to staff; Phase 2 target-state strawman drafted 2026-06-22); Syncro asset reconciliation 2026-06-02; GuruRMM fleet ~45; Bitdefender phase-off | 2026-06-23 |
| [Instrumental Music Center](clients/instrumental-music-center.md) | Prepaid block $175/hr, 12.5 hrs remaining; music retail/repair; AIMsi POS on SQL Server 2019; phantom DC causing slow logons; GuruRMM enrolled (IMC1) | 2026-05-24 |
| [Jimmy Company](clients/jimmy.md) | Break-fix, $150/hr; single aging workstation BLASTER2 (Win10 22H2 EOL, i5-3470/3.8GB — replace); backups the recurring theme (QuickBooks data); onboarded to GuruRMM 2026-06-19 (RDP NLA + Kaseya removal + cleanup); MSP360 local backup drive full, 90-day retention set, space reclaim pending in console (cloud B2 healthy) | 2026-06-19 |