94 lines
12 KiB
Markdown
94 lines
12 KiB
Markdown
# Cascades — Poly phone-drop root cause + whole-network smoothing plan (IN PROGRESS, no changes applied)
|
||
|
||
- **Date:** 2026-06-17
|
||
- **Machine:** Howard-Home
|
||
- **Client:** Cascades of Tucson
|
||
- **Status:** Investigation complete; remediation PLANNED + dry-run verified; **NOTHING APPLIED**. Resume at "Pending / Next steps".
|
||
|
||
## User
|
||
- **User:** Howard Enos (howard)
|
||
- **Machine:** Howard-Home
|
||
- **Role:** tech
|
||
|
||
## Session Summary
|
||
|
||
Started as a CS-SERVER failing-drive review (separate log: `2026-06-17-howard-cs-server-drive-review-and-spike-question.md`), pivoted to a Cascades WiFi "noon network spike" question, then to a live complaint that the **Poly WiFi phones are dropping** ("around 102"). Ran a deep, multi-angle investigation against the UniFi controller (UOS `172.16.3.29`, Cascades site `va6iba3v` / `685f39068e65331c46ef6dd2`) via `.claude/scripts/uos-mongo.sh`, and pfSense (`192.168.0.1`, Plus 25.07) via `unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson` (admin SSH; VPN already up — pfSense reachable). All read-only / dry-run.
|
||
|
||
**Root cause of the phone drops = the INTENTIONAL pfSense reboot on 6/16 at 22:38:12 MST.** Built a per-phone drop-rate timeline from `ace_stat.stat_hourly` (o:'user', `duration` field is in MILLISECONDS): the 30 Poly phones run ~99.9% connected every day EXCEPT 6/16 (uptime 97.95%, 28 drop-hours) — one fleet-wide event where 28 of 30 phones each dropped exactly once, on every floor including **Floors 5/6 which the radio work left untouched**. Only a gateway-level event (the pfSense reboot, kern.boottime = Tue Jun 16 22:38:12 2026) explains all-floors-at-once; the 2.4 radio apply cannot (it only reprovisions touched APs). Today (6/17) the phones are back to ~99.77% / 1 blip. Howard confirmed the reboot was intentional → phone mystery CLOSED. The complaint came in today but today's data is healthy; goal reframed to **smooth the entire network for all devices**.
|
||
|
||
Corrected several of my own earlier wrong reads during the session (logged honestly): (1) the site-wide `sta_dhcp_failures` spike since 6/14 is DECOUPLED from the phones (phones were 99.99% on 6/14–15); (2) DHCP is HEALTHY — my "empty DHCP log" was a tooling error: pfSense 25.07 uses PLAIN-TEXT logs, I was reading with `clog` (old binary format); read directly, dhcpd.log shows 1241 ACK / 1 NAK / 0 no-free-leases; (3) band steering + min-RSSI are PRE-EXISTING, not the new trigger; (4) the per-room /28 subnets are INTENTIONAL HIPAA isolation and are healthy (fullest 12/13), not the cause.
|
||
|
||
Produced a prioritized whole-network smoothing plan, dry-ran the radio changes, and answered Howard's questions on client limits and the residents bandwidth cap. Howard set a hard rule: **do not change anything on Cascades prod infra without discussing it + explicit per-change go** (saved to memory `feedback_cascades.md` rule #4 + MEMORY.md). Saving now to survive context condensation, then continuing.
|
||
|
||
## Key Decisions / Findings
|
||
|
||
- **Phone drop root cause = intentional 6/16 22:38 pfSense reboot** (one-time, transient; resolved). NOT the 2.4 radio work, NOT DHCP, NOT the 6/14 DHCP-metric spike.
|
||
- **DHCP server is healthy** (read logs DIRECTLY, not via clog). `sta_dhcp_failures` metric is client/WiFi-side (frames lost at 100% retry), not pfSense.
|
||
- **No WLAN client-device limit is set** (CSCNet/CSC ENT/Guest empty). Busiest AP = Dining Room 67 clients; median busy ~18; U7 Pro handles it. No limit needed; if a safety cap is wanted, ~127/radio. Not a drop cause.
|
||
- **40 Mbps residents cap = PER-CLIENT** (`Resident` user group 40 down/10 up), not a shared VLAN pipe. Generous; NOT a drop cause; raising it won't help drops. NOTE: CSCNet's default usergroup is actually 'Default' = 100/100 (id `685f39078e65331c46ef7edb`); `Resident` group is `685f39078e65331c46ef7edc` with enabled-flag `undefined` — VERIFY whether the 40 cap is actually active/which clients before touching.
|
||
- **Auto-action settings verified OFF/safe**: `network_optimization`=false, `ips`=IDS-only (non-blocking), `radio_ai.auto_enabled`=false, mesh=none (0 wireless uplinks), PMF/802.11r/v/k=off, group_rekey=0. No hidden auto-deauth.
|
||
- **Per-room /28 segmentation = intentional HIPAA L2 isolation** via CSCNet shared-PPSK (~230–242 per-key→room-VLAN mappings, rooms 101–631). Healthy occupancy (fullest 12/13). DO NOT flatten.
|
||
|
||
## Approved scope (Howard, this session)
|
||
|
||
- **DO Phase 0 + Phase 1.** Dry-run first to verify (done — clean).
|
||
- **SKIP item (e):** recover 2 down APs + 3 offline switches — requires onsite, deferred.
|
||
- **SKIP DFS channel change** for now ("if the logs don't show issues then it's fine").
|
||
- **KEEP band steering ON** ("I don't want band steering off just yet").
|
||
- **Turn on EVERY kind of logging** available (pfSense) to solve quickly.
|
||
- **Client limit:** none set; advised not needed (Howard asked for a number → ~127 if wanted).
|
||
- **Residents 40 Mbps cap:** advised per-client + not a drop cause; verify-before-change.
|
||
- **HARD RULE:** no changes without discussing + explicit per-change go; read-only/dry-run until then.
|
||
|
||
## The verified change-set (dry-run clean; NOT applied)
|
||
|
||
**Phase 1 — combined per-AP `radio_table` PUT (ONE bounce per AP — Howard's ask to avoid double-bounce):**
|
||
- 2.4 `ng` `tx_power_mode` → **medium** — ONLY the 42 radios currently at `low` (EXCLUDE the 24 disabled [would re-enable them] + ~10 Floor-5/6 `auto` radios). `apply-radio ... ng power medium` blanket = WRONG (hits all 77).
|
||
- 5 GHz `na` `ht` → **40** (80→40 MHz) — 76 radios (1 already 40). Clean, no disabled/wrong-direction hits.
|
||
- 5 GHz `na` `min_rssi` → **‑82** — ONLY the 69 radios currently at `-77` enabled (EXCLUDE the 5 currently OFF: 615/608/505/517/622 [would enable], and 108 at -89 [would tighten]).
|
||
- NO `na` channel/DFS change. NO band steering. NO client limit. NO cap change.
|
||
- Tooling note: `apply-radio.sh` does ONE band+setting per reprovision = multiple bounces. To get ONE bounce/AP, BUILD A COMBINED radio_table PUT (ng power + na ht + na min_rssi in a single PUT per AP, correctly scoped per above). RW write cred IS vaulted: `infrastructure/uos-server-network-api-rw` (so `--apply` is possible once gone-ahead). apply-radio rollback auto-saves to `.claude/tmp/apply-rollback-*.json`.
|
||
|
||
**Phase 0 — non-disruptive:**
|
||
- pfSense `ping-check` → off (all 240 DHCP pools currently have ping-check true; adds ~1s/lease, causes client-side DHCP timeouts = the failure metric). Reversible.
|
||
- Disable 3 AM AP firmware auto-upgrade (`mgmt.auto_upgrade=true, hour=3` — nightly AP reboots/drops).
|
||
- Turn on full pfSense logging: DHCP (already on), DNS resolver (unbound), firewall (filter, incl. default-block + resident-VLAN rules), system, gateway; BUMP log rotation/retention. CAVEAT: DNS-query + firewall-pass logging is high-volume → set sane rotation, dial back chattiest after we catch the issue.
|
||
|
||
## Problems Encountered (my own corrections — for linting)
|
||
|
||
- **clog vs plain-text logs (pfSense 25.07):** read DHCP/system logs with `clog` → returned empty → I wrongly concluded "DHCP log empty / phones not doing DHCP." 25.07 logs are PLAIN TEXT; use `tail`/`grep` directly. Real data: dhcpd.log healthy (1241 ACK/1 NAK). WASTED a hypothesis. (Worth an errorlog `--friction`.)
|
||
- **stat duration units:** `ace_stat.stat_hourly` o:'user' `duration` is MILLISECONDS, not seconds — first uptime calc was inflated 1000×. Corrected.
|
||
- Conflated two metrics (site-wide DHCP-failures vs per-phone drops) → Howard caught it; they're decoupled (different populations, different dates).
|
||
|
||
## Infrastructure & Servers
|
||
|
||
- **UniFi UOS controller** `172.16.3.29`; Cascades site name `va6iba3v`, id `685f39068e65331c46ef6dd2`. Access: `infrastructure/uos-server-ssh-key` + `.claude/scripts/uos-mongo.sh` (db `ace` config, `ace_stat` history). Write cred: `infrastructure/uos-server-network-api-rw` (vaulted); read: `infrastructure/uos-server-network-api`.
|
||
- **Cascades pfSense** `192.168.0.1`, Plus 25.07-RELEASE. Admin SSH cred `clients/cascades-tucson/pfsense-firewall`. Access via `bash .claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson audit|dhcp|run "<cmd>"`. VPN (OpenVPN Connect) already up — pfSense reachable from Howard-Home. kern.boottime 2026-06-16 22:38:12 MST.
|
||
- **WiFi:** 77 APs (mostly U7PRO), ~573–788 clients. CSCNet = shared-PPSK SSID (wlanconf `685f39078e65331c46ef7ee5`, networkconf `685f39078e65331c46ef8ac4`, ~242 PPSK keys), bands 2g+5g (no 6e), `no2ghz_oui=true` (band steering ON), usergroup `Default`(100/100). Other SSIDs: CSC ENT (`...7ee4`), Guest (`...7ee6`, VLAN 50 isolated), element (`...7ee3`).
|
||
- **Poly phones:** OUI `48:25:67`, 30 seen (~22 active fleet), 26/30 on 5 GHz (na), 4 on 2.4 (ng), on VLAN 20 (`10.0.20.x`, network "Internal"). AudioCodes (8, wired) + Vertical-Remote desktop are the wired voice gear.
|
||
- **pfSense DHCP:** 240 pools, 236 are /28 (13-host) per-room HIPAA VLANs; VLAN 20 = `10.0.20.0/24` range .50–.239 (190 hosts, 61 active), ping-check on ALL pools, lease 7200/86400. ISC dhcpd (Kea dormant).
|
||
- **RF prior work (other session, in repo):** overnight 6/17 applied 24 of 76 2.4 radios DISABLED + 42 set to Low (~6 dBm); Floors 5/6 + mesh untouched. Over-thinned (retry 17→23.4%, satisfaction 39→30). Recommend Low→Medium. Reports: `reports/2026-06-16-unifi-full-audit.md`, `reports/2026-06-16-2.4ghz-remediation-runbook.md`. 5 GHz: 55/77 on DFS, 76/77 at 80 MHz; audit found 0 radar events (DFS low-risk). 2 APs down (108 + dup); 3 switches offline (2nd-Flr#2, 4th-Flr#2, USW Pro Max 16); ~25–34 gig ports linked at 100M.
|
||
|
||
## Commands & Outputs (key)
|
||
|
||
- Per-phone drop timeline: `ace_stat.stat_hourly` o:'user', oid in Poly macs, `duration` (MS) per hour → uptime%. 6/16 = 97.95% / 28 drop-hrs (the event); all other days 99.9%+.
|
||
- pfSense boot: `sysctl -n kern.boottime` → 1781674692 = Tue Jun 16 22:38:12 2026 MST; uptime 18:13 (single reboot).
|
||
- DHCP health (DIRECT read, NOT clog): `grep -c DHCPACK /var/log/dhcpd.log` → 1241; DHCPNAK → 1; "no free leases" → 0. `file /var/log/dhcpd.log` → ASCII text.
|
||
- Dry-runs (all gated, no --apply): `apply-radio cascades ng power medium` (42 low correct; 24 disabled + 10 auto WRONG to include), `apply-radio cascades na width 40` (76 change clean), `apply-radio cascades na minrssi -82` (69 at -77 correct; 5 OFF + 108 WRONG).
|
||
- Coord: posted hold-note to other radio session (msg id `5690f1b8...`).
|
||
|
||
## Pending / Next steps (RESUME HERE)
|
||
|
||
1. **Decide Phase 1 window** — it kicks every client off each AP as it reprovisions. Recommend per-zone (floor-by-floor) in an evening window (e.g. after 8pm), NOT all-at-once during the day. Howard to choose window (or accept brief per-AP blips now). NOT yet answered.
|
||
2. **Get explicit go**, then:
|
||
- **Phase 0** (safe anytime): apply ping-check off + disable 3 AM auto-upgrade + enable full pfSense logging (with rotation).
|
||
- **Phase 1** (windowed): build + apply the COMBINED per-AP radio_table PUT (ng power medium [42] + na ht 40 [76] + na min_rssi -82 [69]), scoped exactly as above, per-zone, validate live with watch-ap before/after.
|
||
3. Deferred: DFS change (revisit only if new logs show DFS issues), band steering (Howard wants kept ON for now), onsite item (e) down APs/switches, VLAN 30 voice cutover, residents 40 Mbps cap (verify active first), the 6/14 site-wide DHCP-failure metric (other clients — re-check with now-readable logs).
|
||
4. HARD RULE: discuss + explicit per-change go before ANY apply. Read-only/dry-run only otherwise.
|
||
|
||
## Reference
|
||
|
||
- Memory: `feedback_cascades.md` rule #4 (no prod changes w/o discussing); `project_cascades_isolated_vlan_pattern.md` (the /28 HIPAA design + pfSense PHP API path).
|
||
- Phone-mystery sibling log: `2026-06-17-howard-cs-server-drive-review-and-spike-question.md`.
|
||
- Client wiki: `wiki/clients/cascades-tucson.md`.
|