claudetools/clients/cascades-tucson/session-logs/2026-06/2026-06-15-howard-cascades-wifi-rf-audit.md

User

• User: Howard Enos (howard)
• Machine: Howard-Home
• Role: tech

Session Summary

Resumed the Cascades wireless investigation that had stalled on 2026-05-16 (that session was read-only via the cloud API and blocked from per-AP RF data). The day's earlier syncs delivered the unblock: Mike (GURU-5070) vaulted infrastructure/uos-server-ssh-key + clients/cascades-tucson/unifi-ap-ssh and shipped a purpose-built unifi-wifi skill (audit/model-rank/optimize/apply/watch scripts + methodology references). Controller access via the vaulted key was verified, then the full live audit ran against the Cascades site (685f39068e65331c46ef6dd2) on the UOS controller (172.16.3.29).

The audit confirmed and quantified the 05-16 hypothesis with real controller data: 77 U7-Pro APs, all running 2.4GHz at auto (~full) power, 20MHz. 2.4 airtime (cu_total) is 74–94% busy on 75 radios with 61–81% of that being pure interference, serving ~1 client each, at TX-retry rates of 40–65% and single-digit AP satisfaction on the worst (209 sat=1/retr=65%, 139 sat=2/retr=49%, CC Bridge retr=48%). Neighbor-BSSID density is catastrophic on 2.4 (ch6=33,370, ch1=19,274, ch11=16,580). 5GHz is on 80MHz width on 76/77 APs (kills spatial reuse), biased to the busy upper channels (149/157). 6GHz is active on 75 radios but nearly empty of clients. 6 APs have 2.4 min-RSSI OFF (615, 608, 505, 517, 622, salon); 4 are off the 1/6/11 plan on auto (128, 108, 108U7-Pro, salon).

Diagnosis of "bad for SOME users": experience splits by band. Clients on 5/6GHz are fine; clients that land or stick on 2.4GHz (legacy phones, medical/IoT, poor band-steerers, or anything held by a min-RSSI-OFF AP from across the building) hit the saturated 2.4 radios with 40–65% retransmits and near-zero satisfaction.

Late in the session Howard raised the military-base / DFS factor — Cascades is in Tucson near Davis-Monthan AFB (+ TUS airport radar). This reverses the earlier "bias 5GHz toward DFS" recommendation: DFS channels (UNII-2/2e, ch52–144) will see frequent radar-detection events forcing channel-vacate + CAC silence, producing exactly the intermittent per-area dropouts reported. Revised plan uses non-DFS only (UNII-1 36–48 + UNII-3 149–161), which makes 40MHz width and 6GHz steering more important.

Key Decisions

• Used the unifi-wifi skill end-to-end (audit-site, model-rank, optimize-radios, apply-radio dry-run) rather than ad-hoc Mongo queries — it encodes the multi-model methodology and coverage-safe model.
• 5GHz channel plan revised to AVOID DFS due to proximity to Davis-Monthan AFB. Non-DFS UNII-1 + UNII-3 only; verify empirically against the controller's radar-detection event history before reconsidering DFS.
• No changes applied — writes are intentionally gated until a read-WRITE controller admin is vaulted (infrastructure/uos-server-network-api-rw) and --apply is passed. Confirmed the apply path works via dry-run (Floor 3: 17 radios auto->low with rollback values captured).
• Rollout will be per-zone (one floor at a time) with live before/after validation, never site-wide at once.

Problems Encountered

• Controller SSH key not on Howard-Home and not in vault (earlier in day). Tested both vaulted keys (gururmm-server-physical, openclaw-fleet) against root@172.16.3.29 — both denied; OC-5070/OC-Mac unreachable over Tailscale, fleet key denied on OC-Beast. Resolved by coord-requesting Mike to vault the UOS key; he did (infrastructure/uos-server-ssh-key), picked up on sync.
• Cloud Site Manager API insufficient (re-confirmed). Device objects carry no site field and no RF/channel/power/uplink data — only online/offline + firmware. Cannot drive RF tuning. The Mongo-via-SSH path (now available) is required.

Configuration Changes

• None to Cascades infra. Read-only audit + dry-run only.
• No repo files edited beyond this session log.

Credentials & Secrets

• infrastructure/uos-server-ssh-key (vaulted by Mike this day) — root SSH key for the UOS controller 172.16.3.29; used by uos-mongo.sh / unifi-wifi scripts. This is the DATA plane (read), not an API write session.
• clients/cascades-tucson/unifi-ap-ssh (vaulted by Mike this day) — device-auth cred for SSHing directly into Cascades APs (used by watch-ap.sh; needs site VPN for L3 reach to 192.168.2.x/3.x).
• Needed, not yet created: infrastructure/uos-server-network-api-rw (read-write controller admin) to apply radio changes; infrastructure/uos-server-network-api (read-only admin) to wire live-stats Plane 2 validation.

Infrastructure & Servers

• UOS controller: 172.16.3.29 (Rocky 9 VM "Unifi" on Jupiter 172.16.3.20); UniFi-OS HTTPS on 11443 (not 8443). Mongo ace on 127.0.0.1:27117 inside rootless podman uosserver. Cascades site_id 685f39068e65331c46ef6dd2.
• Cascades wireless: 77 U7-Pro APs, ~550 clients. Firewall = pfSense 192.168.0.1 (site VPN endpoint; .ovpn comes from pfSense OpenVPN Client Export, NOT UniFi). APs on 192.168.2.x/3.x.
• Location/RF: Tucson, near Davis-Monthan AFB + TUS radar → DFS unreliable.

Commands & Outputs

bash .claude/scripts/uos-mongo.sh --sites | grep -i casc        # 685f39068e65331c46ef6dd2  Cascades (access OK)
bash .claude/skills/unifi-wifi/scripts/audit-site.sh cascades    # config + neighbor-density + flags
bash .claude/skills/unifi-wifi/scripts/model-rank.sh cascades 7 ng
bash .claude/skills/unifi-wifi/scripts/optimize-radios.sh cascades 14 ng   # power-down 74, disable 0, keep 1
bash .claude/skills/unifi-wifi/scripts/apply-radio.sh cascades ng power low --zone "Floor 3"  # DRY-RUN: 17 radios auto->low

Key audit output: 2.4 cu_total 74–94% / interf 61–81% / ~1 client; retry 40–65%; ch6=33,370 neighbors; 5GHz 80MHz on 76/77; 6GHz active 75 but empty; min-RSSI OFF on 615/608/505/517/622/salon.

Pending / Incomplete Tasks

• Vault read-WRITE controller admin infrastructure/uos-server-network-api-rw (blocks applying any radio change). Candidate: coord-request Mike.
• Vault read-only infrastructure/uos-server-network-api to wire live-stats Plane 2 (before/after cu_total/satisfaction validation).
• Apply Phase A (2.4 power-down to Low) per-zone with live validation, once RW cred exists.
• 5GHz: 80->40MHz width; non-DFS channel plan (UNII-1 36–48 + UNII-3 149–161); 6GHz steering for capable clients.
• Min data rates (kill 1–11Mbps, 2.4 floor 12/24Mbps); set 2.4 min-RSSI -75/-76 on the 6 OFF APs; pin 4 off-plan APs to 1/6/11.
• Pull controller radar-detection event history to empirically confirm DFS unusability.
• Secondary: fix the non-working ".ovpn / Download configuration" — likely pfSense OpenVPN Client Export (192.168.0.1), not UniFi. Needed for watch-ap.sh live validation.
• AP 108 offline — KNOWN, needs a new cable run (per Howard); ignore for now. Also a stale duplicate controller object (108 vs 108U7 Pro) to clean up later.

Reference Information

• Coord message to GURU-5070 re UOS key: id a4b385ad-4fbb-4097-a066-099622080055; backstop todo 3bd12a14-2b51-4c11-8f76-3f835b07e8dc (--user mike).
• unifi-wifi skill: .claude/skills/unifi-wifi/ (methodology.md, data-access.md, interference-model.md).
• Prior wireless log: clients/cascades-tucson/session-logs/2026-05-16-howard-wireless-diagnostic.md.
• UOS system wiki: wiki/systems/uos-server.md.

---

Update: 20:40 PT — RW cred arrived, live (Plane 2) re-look, live-stats.sh accuracy fixes

Mike vaulted the RW controller admin (infrastructure/uos-server-network-api-rw) within ~30 min of the request, so the live Network API (Plane 2) became available. Re-audited Cascades live and, in doing so, found + fixed accuracy bugs in unifi-wifi/scripts/live-stats.sh that had skewed the earlier read.

live-stats.sh bugs fixed (held a coord lock; messaged Mike e8be889f)

1. stat/device output hard-capped at head -60 -> only ~15 of 77 APs were shown (we'd been judging the whole site from a 15-AP sample). Removed the cap; verified 77/77 now.
2. satisfaction was read per-radio (always -1 on this controller). DEVICE-level satisfaction IS populated -> switched to d['satisfaction'].
3. tx_retries was the raw cumulative counter (scales with traffic, misleading). Switched to radio_table_stats.tx_retries_pct (a true rate).
4. --clients was [:40] unsorted (hid 90% of 574 clients). Now sorts worst-by-satisfaction and prints signal/noise/retry%/satisfaction_reason.
5. RF-neighbor table left as a documented TODO in the header (not a single API field; must be built from the rogue BSSID cross-ref vs each AP's vap_table). It's what unlocks confident radio DISABLES; until then power-down/channel/width are the safe levers.

Corrected diagnosis (the data fix changed a conclusion)

• Accurate avg retry RATE: 2.4GHz 11.2% > 5GHz clear 9.0% ~= 5GHz DFS 8.4%. My mid-session claim that "5GHz/DFS is now the #1 problem" was an ARTIFACT of the raw counter + 15-AP sample and is WITHDRAWN. On the rate, DFS is NOT retrying worse than clear channels.
• 2.4GHz is the primary pain band (highest retry; 27 of the 40 worst clients are on 2.4, retry 11-42%, mostly IoT/legacy: Ring cams, robotic cleaner, smart plugs, EPSON printer, Poly phone, handheld scanners, Watch). The original 2.4 power-down/prune plan stands as #1.
• DFS = resilience risk, not throughput killer. 55/77 5GHz radios on DFS near Davis-Monthan = radar-vacate exposure (client drops), worth moving off DFS for stability, but not urgent for performance.
• 6GHz still dead (1 client of 574) — top untapped clean/non-DFS capacity; steering remains a key opportunity.
• AP-level satisfaction 95-100 across the fleet — network is healthy on average; pain is in the client tail = consistent with "bad for SOME users."

Cascades live snapshot (2026-06-15 ~20:30 PT)

• 77/77 APs reporting; 574 wireless clients. Band split: na (5GHz) ~87 / ng (2.4) ~21 sampled per pull / 6e ~1.
• 2.4 cu_total 69-94% live (saturation confirmed). 5GHz cu_total mostly <40%.
• Worst clients: RingStickupCam (sat 60), Galaxy-A32-5G (60), DIRECTV (71), Samsung on ch1 (76, retry 42%).

Updated next steps

• Build Floor-pilot DRY-RUN: 2.4 power-down to Low on one floor (e.g. Floor 4), validate live cu_total/retry% before+after via the fixed live-stats.sh. Get explicit go before --apply.
• Implement the AP-to-AP RF-neighbor table (rogue BSSID x vap_table) to enable safe DISABLEs.
• 6GHz steering plan; 5GHz 80->40MHz + non-DFS channel plan (resilience).
• Coord msgs this update: RW-cred request 6b98282f (+todo cbb355ef); live-stats fix e8be889f.
• pfSense .ovpn (Howard handling) — needed for per-AP watch-ap.sh live stream.