claudetools/clients/cascades-tucson/session-logs/2026-06/2026-06-17-howard-cascades-poly-phone-drops-network-smoothing.md

Cascades — Poly phone-drop root cause + whole-network smoothing plan (IN PROGRESS, no changes applied)

• Date: 2026-06-17
• Machine: Howard-Home
• Client: Cascades of Tucson
• Status: Investigation complete; remediation PLANNED + dry-run verified; NOTHING APPLIED. Resume at "Pending / Next steps".

User

• User: Howard Enos (howard)
• Machine: Howard-Home
• Role: tech

Session Summary

Started as a CS-SERVER failing-drive review (separate log: 2026-06-17-howard-cs-server-drive-review-and-spike-question.md), pivoted to a Cascades WiFi "noon network spike" question, then to a live complaint that the Poly WiFi phones are dropping ("around 102"). Ran a deep, multi-angle investigation against the UniFi controller (UOS 172.16.3.29, Cascades site va6iba3v / 685f39068e65331c46ef6dd2) via .claude/scripts/uos-mongo.sh, and pfSense (192.168.0.1, Plus 25.07) via unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson (admin SSH; VPN already up — pfSense reachable). All read-only / dry-run.

Root cause of the phone drops = the INTENTIONAL pfSense reboot on 6/16 at 22:38:12 MST. Built a per-phone drop-rate timeline from ace_stat.stat_hourly (o:'user', duration field is in MILLISECONDS): the 30 Poly phones run ~99.9% connected every day EXCEPT 6/16 (uptime 97.95%, 28 drop-hours) — one fleet-wide event where 28 of 30 phones each dropped exactly once, on every floor including Floors 5/6 which the radio work left untouched. Only a gateway-level event (the pfSense reboot, kern.boottime = Tue Jun 16 22:38:12 2026) explains all-floors-at-once; the 2.4 radio apply cannot (it only reprovisions touched APs). Today (6/17) the phones are back to ~99.77% / 1 blip. Howard confirmed the reboot was intentional → phone mystery CLOSED. The complaint came in today but today's data is healthy; goal reframed to smooth the entire network for all devices.

Corrected several of my own earlier wrong reads during the session (logged honestly): (1) the site-wide sta_dhcp_failures spike since 6/14 is DECOUPLED from the phones (phones were 99.99% on 6/14–15); (2) DHCP is HEALTHY — my "empty DHCP log" was a tooling error: pfSense 25.07 uses PLAIN-TEXT logs, I was reading with clog (old binary format); read directly, dhcpd.log shows 1241 ACK / 1 NAK / 0 no-free-leases; (3) band steering + min-RSSI are PRE-EXISTING, not the new trigger; (4) the per-room /28 subnets are INTENTIONAL HIPAA isolation and are healthy (fullest 12/13), not the cause.

Produced a prioritized whole-network smoothing plan, dry-ran the radio changes, and answered Howard's questions on client limits and the residents bandwidth cap. Howard set a hard rule: do not change anything on Cascades prod infra without discussing it + explicit per-change go (saved to memory feedback_cascades.md rule #4 + MEMORY.md). Saving now to survive context condensation, then continuing.

Key Decisions / Findings

• Phone drop root cause = intentional 6/16 22:38 pfSense reboot (one-time, transient; resolved). NOT the 2.4 radio work, NOT DHCP, NOT the 6/14 DHCP-metric spike.
• DHCP server is healthy (read logs DIRECTLY, not via clog). sta_dhcp_failures metric is client/WiFi-side (frames lost at 100% retry), not pfSense.
• No WLAN client-device limit is set (CSCNet/CSC ENT/Guest empty). Busiest AP = Dining Room 67 clients; median busy ~18; U7 Pro handles it. No limit needed; if a safety cap is wanted, ~127/radio. Not a drop cause.
• 40 Mbps residents cap = PER-CLIENT (Resident user group 40 down/10 up), not a shared VLAN pipe. Generous; NOT a drop cause; raising it won't help drops. NOTE: CSCNet's default usergroup is actually 'Default' = 100/100 (id 685f39078e65331c46ef7edb); Resident group is 685f39078e65331c46ef7edc with enabled-flag undefined — VERIFY whether the 40 cap is actually active/which clients before touching.
• Auto-action settings verified OFF/safe: network_optimization=false, ips=IDS-only (non-blocking), radio_ai.auto_enabled=false, mesh=none (0 wireless uplinks), PMF/802.11r/v/k=off, group_rekey=0. No hidden auto-deauth.
• Per-room /28 segmentation = intentional HIPAA L2 isolation via CSCNet shared-PPSK (~230–242 per-key→room-VLAN mappings, rooms 101–631). Healthy occupancy (fullest 12/13). DO NOT flatten.

Approved scope (Howard, this session)

• DO Phase 0 + Phase 1. Dry-run first to verify (done — clean).
• SKIP item (e): recover 2 down APs + 3 offline switches — requires onsite, deferred.
• SKIP DFS channel change for now ("if the logs don't show issues then it's fine").
• KEEP band steering ON ("I don't want band steering off just yet").
• Turn on EVERY kind of logging available (pfSense) to solve quickly.
• Client limit: none set; advised not needed (Howard asked for a number → ~127 if wanted).
• Residents 40 Mbps cap: advised per-client + not a drop cause; verify-before-change.
• HARD RULE: no changes without discussing + explicit per-change go; read-only/dry-run until then.

The verified change-set (dry-run clean; NOT applied)

Phase 1 — combined per-AP radio_table PUT (ONE bounce per AP — Howard's ask to avoid double-bounce):

• 2.4 ng tx_power_mode → medium — ONLY the 42 radios currently at low (EXCLUDE the 24 disabled [would re-enable them] + ~10 Floor-5/6 auto radios). apply-radio ... ng power medium blanket = WRONG (hits all 77).
• 5 GHz na ht → 40 (80→40 MHz) — 76 radios (1 already 40). Clean, no disabled/wrong-direction hits.
• 5 GHz na min_rssi → ‑82 — ONLY the 69 radios currently at -77 enabled (EXCLUDE the 5 currently OFF: 615/608/505/517/622 [would enable], and 108 at -89 [would tighten]).
• NO na channel/DFS change. NO band steering. NO client limit. NO cap change.
• Tooling note: apply-radio.sh does ONE band+setting per reprovision = multiple bounces. To get ONE bounce/AP, BUILD A COMBINED radio_table PUT (ng power + na ht + na min_rssi in a single PUT per AP, correctly scoped per above). RW write cred IS vaulted: infrastructure/uos-server-network-api-rw (so --apply is possible once gone-ahead). apply-radio rollback auto-saves to .claude/tmp/apply-rollback-*.json.

Phase 0 — non-disruptive:

• pfSense ping-check → off (all 240 DHCP pools currently have ping-check true; adds ~1s/lease, causes client-side DHCP timeouts = the failure metric). Reversible.
• Disable 3 AM AP firmware auto-upgrade (mgmt.auto_upgrade=true, hour=3 — nightly AP reboots/drops).
• Turn on full pfSense logging: DHCP (already on), DNS resolver (unbound), firewall (filter, incl. default-block + resident-VLAN rules), system, gateway; BUMP log rotation/retention. CAVEAT: DNS-query + firewall-pass logging is high-volume → set sane rotation, dial back chattiest after we catch the issue.

Problems Encountered (my own corrections — for linting)

• clog vs plain-text logs (pfSense 25.07): read DHCP/system logs with clog → returned empty → I wrongly concluded "DHCP log empty / phones not doing DHCP." 25.07 logs are PLAIN TEXT; use tail/grep directly. Real data: dhcpd.log healthy (1241 ACK/1 NAK). WASTED a hypothesis. (Worth an errorlog --friction.)
• stat duration units: ace_stat.stat_hourly o:'user' duration is MILLISECONDS, not seconds — first uptime calc was inflated 1000×. Corrected.
• Conflated two metrics (site-wide DHCP-failures vs per-phone drops) → Howard caught it; they're decoupled (different populations, different dates).

Infrastructure & Servers

• UniFi UOS controller 172.16.3.29; Cascades site name va6iba3v, id 685f39068e65331c46ef6dd2. Access: infrastructure/uos-server-ssh-key + .claude/scripts/uos-mongo.sh (db ace config, ace_stat history). Write cred: infrastructure/uos-server-network-api-rw (vaulted); read: infrastructure/uos-server-network-api.
• Cascades pfSense 192.168.0.1, Plus 25.07-RELEASE. Admin SSH cred clients/cascades-tucson/pfsense-firewall. Access via bash .claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson audit|dhcp|run "<cmd>". VPN (OpenVPN Connect) already up — pfSense reachable from Howard-Home. kern.boottime 2026-06-16 22:38:12 MST.
• WiFi: 77 APs (mostly U7PRO), ~573–788 clients. CSCNet = shared-PPSK SSID (wlanconf 685f39078e65331c46ef7ee5, networkconf 685f39078e65331c46ef8ac4, ~242 PPSK keys), bands 2g+5g (no 6e), no2ghz_oui=true (band steering ON), usergroup Default(100/100). Other SSIDs: CSC ENT (...7ee4), Guest (...7ee6, VLAN 50 isolated), element (...7ee3).
• Poly phones: OUI 48:25:67, 30 seen (~22 active fleet), 26/30 on 5 GHz (na), 4 on 2.4 (ng), on VLAN 20 (10.0.20.x, network "Internal"). AudioCodes (8, wired) + Vertical-Remote desktop are the wired voice gear.
• pfSense DHCP: 240 pools, 236 are /28 (13-host) per-room HIPAA VLANs; VLAN 20 = 10.0.20.0/24 range .50–.239 (190 hosts, 61 active), ping-check on ALL pools, lease 7200/86400. ISC dhcpd (Kea dormant).
• RF prior work (other session, in repo): overnight 6/17 applied 24 of 76 2.4 radios DISABLED + 42 set to Low (~6 dBm); Floors 5/6 + mesh untouched. Over-thinned (retry 17→23.4%, satisfaction 39→30). Recommend Low→Medium. Reports: reports/2026-06-16-unifi-full-audit.md, reports/2026-06-16-2.4ghz-remediation-runbook.md. 5 GHz: 55/77 on DFS, 76/77 at 80 MHz; audit found 0 radar events (DFS low-risk). 2 APs down (108 + dup); 3 switches offline (2nd-Flr#2, 4th-Flr#2, USW Pro Max 16); ~25–34 gig ports linked at 100M.

Commands & Outputs (key)

• Per-phone drop timeline: ace_stat.stat_hourly o:'user', oid in Poly macs, duration (MS) per hour → uptime%. 6/16 = 97.95% / 28 drop-hrs (the event); all other days 99.9%+.
• pfSense boot: sysctl -n kern.boottime → 1781674692 = Tue Jun 16 22:38:12 2026 MST; uptime 18:13 (single reboot).
• DHCP health (DIRECT read, NOT clog): grep -c DHCPACK /var/log/dhcpd.log → 1241; DHCPNAK → 1; "no free leases" → 0. file /var/log/dhcpd.log → ASCII text.
• Dry-runs (all gated, no --apply): apply-radio cascades ng power medium (42 low correct; 24 disabled + 10 auto WRONG to include), apply-radio cascades na width 40 (76 change clean), apply-radio cascades na minrssi -82 (69 at -77 correct; 5 OFF + 108 WRONG).
• Coord: posted hold-note to other radio session (msg id 5690f1b8...).

Pending / Next steps (RESUME HERE)

1. Decide Phase 1 window — it kicks every client off each AP as it reprovisions. Recommend per-zone (floor-by-floor) in an evening window (e.g. after 8pm), NOT all-at-once during the day. Howard to choose window (or accept brief per-AP blips now). NOT yet answered.
2. Get explicit go, then:
   • Phase 0 (safe anytime): apply ping-check off + disable 3 AM auto-upgrade + enable full pfSense logging (with rotation).
   • Phase 1 (windowed): build + apply the COMBINED per-AP radio_table PUT (ng power medium [42] + na ht 40 [76] + na min_rssi -82 [69]), scoped exactly as above, per-zone, validate live with watch-ap before/after.
3. Deferred: DFS change (revisit only if new logs show DFS issues), band steering (Howard wants kept ON for now), onsite item (e) down APs/switches, VLAN 30 voice cutover, residents 40 Mbps cap (verify active first), the 6/14 site-wide DHCP-failure metric (other clients — re-check with now-readable logs).
4. HARD RULE: discuss + explicit per-change go before ANY apply. Read-only/dry-run only otherwise.

Reference

• Memory: feedback_cascades.md rule #4 (no prod changes w/o discussing); project_cascades_isolated_vlan_pattern.md (the /28 HIPAA design + pfSense PHP API path).
• Phone-mystery sibling log: 2026-06-17-howard-cs-server-drive-review-and-spike-question.md.
• Client wiki: wiki/clients/cascades-tucson.md.