azcomputerguru/claudetools
1
0
Fork 0
Code Issues 58 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
60d0a2bf8759a5c48fb5784b76c5686c468c335b
claudetools/clients/cascades-tucson/session-logs/2026-06/2026-06-15-howard-cascades-wifi-rf-audit.md
Howard Enos 60d0a2bf87 sync: auto-sync from HOWARD-HOME at 2026-06-15 23:24:56 
Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-06-15 23:24:56
2026-06-15 23:25:05 -07:00

21 KiB
Raw Blame History

User

  • User: Howard Enos (howard)
  • Machine: Howard-Home
  • Role: tech

Session Summary

Resumed the Cascades wireless investigation that had stalled on 2026-05-16 (that session was read-only via the cloud API and blocked from per-AP RF data). The day's earlier syncs delivered the unblock: Mike (GURU-5070) vaulted infrastructure/uos-server-ssh-key + clients/cascades-tucson/unifi-ap-ssh and shipped a purpose-built unifi-wifi skill (audit/model-rank/optimize/apply/watch scripts + methodology references). Controller access via the vaulted key was verified, then the full live audit ran against the Cascades site (685f39068e65331c46ef6dd2) on the UOS controller (172.16.3.29).

The audit confirmed and quantified the 05-16 hypothesis with real controller data: 77 U7-Pro APs, all running 2.4GHz at auto (~full) power, 20MHz. 2.4 airtime (cu_total) is 74–94% busy on 75 radios with 61–81% of that being pure interference, serving ~1 client each, at TX-retry rates of 40–65% and single-digit AP satisfaction on the worst (209 sat=1/retr=65%, 139 sat=2/retr=49%, CC Bridge retr=48%). Neighbor-BSSID density is catastrophic on 2.4 (ch6=33,370, ch1=19,274, ch11=16,580). 5GHz is on 80MHz width on 76/77 APs (kills spatial reuse), biased to the busy upper channels (149/157). 6GHz is active on 75 radios but nearly empty of clients. 6 APs have 2.4 min-RSSI OFF (615, 608, 505, 517, 622, salon); 4 are off the 1/6/11 plan on auto (128, 108, 108U7-Pro, salon).

Diagnosis of "bad for SOME users": experience splits by band. Clients on 5/6GHz are fine; clients that land or stick on 2.4GHz (legacy phones, medical/IoT, poor band-steerers, or anything held by a min-RSSI-OFF AP from across the building) hit the saturated 2.4 radios with 40–65% retransmits and near-zero satisfaction.

Late in the session Howard raised the military-base / DFS factor — Cascades is in Tucson near Davis-Monthan AFB (+ TUS airport radar). This reverses the earlier "bias 5GHz toward DFS" recommendation: DFS channels (UNII-2/2e, ch52–144) will see frequent radar-detection events forcing channel-vacate + CAC silence, producing exactly the intermittent per-area dropouts reported. Revised plan uses non-DFS only (UNII-1 36–48 + UNII-3 149–161), which makes 40MHz width and 6GHz steering more important.

Key Decisions

  • Used the unifi-wifi skill end-to-end (audit-site, model-rank, optimize-radios, apply-radio dry-run) rather than ad-hoc Mongo queries — it encodes the multi-model methodology and coverage-safe model.
  • 5GHz channel plan revised to AVOID DFS due to proximity to Davis-Monthan AFB. Non-DFS UNII-1 + UNII-3 only; verify empirically against the controller's radar-detection event history before reconsidering DFS.
  • No changes applied — writes are intentionally gated until a read-WRITE controller admin is vaulted (infrastructure/uos-server-network-api-rw) and --apply is passed. Confirmed the apply path works via dry-run (Floor 3: 17 radios auto->low with rollback values captured).
  • Rollout will be per-zone (one floor at a time) with live before/after validation, never site-wide at once.

Problems Encountered

  • Controller SSH key not on Howard-Home and not in vault (earlier in day). Tested both vaulted keys (gururmm-server-physical, openclaw-fleet) against root@172.16.3.29 — both denied; OC-5070/OC-Mac unreachable over Tailscale, fleet key denied on OC-Beast. Resolved by coord-requesting Mike to vault the UOS key; he did (infrastructure/uos-server-ssh-key), picked up on sync.
  • Cloud Site Manager API insufficient (re-confirmed). Device objects carry no site field and no RF/channel/power/uplink data — only online/offline + firmware. Cannot drive RF tuning. The Mongo-via-SSH path (now available) is required.

Configuration Changes

  • None to Cascades infra. Read-only audit + dry-run only.
  • No repo files edited beyond this session log.

Credentials & Secrets

  • infrastructure/uos-server-ssh-key (vaulted by Mike this day) — root SSH key for the UOS controller 172.16.3.29; used by uos-mongo.sh / unifi-wifi scripts. This is the DATA plane (read), not an API write session.
  • clients/cascades-tucson/unifi-ap-ssh (vaulted by Mike this day) — device-auth cred for SSHing directly into Cascades APs (used by watch-ap.sh; needs site VPN for L3 reach to 192.168.2.x/3.x).
  • Needed, not yet created: infrastructure/uos-server-network-api-rw (read-write controller admin) to apply radio changes; infrastructure/uos-server-network-api (read-only admin) to wire live-stats Plane 2 validation.

Infrastructure & Servers

  • UOS controller: 172.16.3.29 (Rocky 9 VM "Unifi" on Jupiter 172.16.3.20); UniFi-OS HTTPS on 11443 (not 8443). Mongo ace on 127.0.0.1:27117 inside rootless podman uosserver. Cascades site_id 685f39068e65331c46ef6dd2.
  • Cascades wireless: 77 U7-Pro APs, ~550 clients. Firewall = pfSense 192.168.0.1 (site VPN endpoint; .ovpn comes from pfSense OpenVPN Client Export, NOT UniFi). APs on 192.168.2.x/3.x.
  • Location/RF: Tucson, near Davis-Monthan AFB + TUS radar → DFS unreliable.

Commands & Outputs

bash .claude/scripts/uos-mongo.sh --sites | grep -i casc        # 685f39068e65331c46ef6dd2  Cascades (access OK)
bash .claude/skills/unifi-wifi/scripts/audit-site.sh cascades    # config + neighbor-density + flags
bash .claude/skills/unifi-wifi/scripts/model-rank.sh cascades 7 ng
bash .claude/skills/unifi-wifi/scripts/optimize-radios.sh cascades 14 ng   # power-down 74, disable 0, keep 1
bash .claude/skills/unifi-wifi/scripts/apply-radio.sh cascades ng power low --zone "Floor 3"  # DRY-RUN: 17 radios auto->low

Key audit output: 2.4 cu_total 74–94% / interf 61–81% / ~1 client; retry 40–65%; ch6=33,370 neighbors; 5GHz 80MHz on 76/77; 6GHz active 75 but empty; min-RSSI OFF on 615/608/505/517/622/salon.

Pending / Incomplete Tasks

  • Vault read-WRITE controller admin infrastructure/uos-server-network-api-rw (blocks applying any radio change). Candidate: coord-request Mike.
  • Vault read-only infrastructure/uos-server-network-api to wire live-stats Plane 2 (before/after cu_total/satisfaction validation).
  • Apply Phase A (2.4 power-down to Low) per-zone with live validation, once RW cred exists.
  • 5GHz: 80->40MHz width; non-DFS channel plan (UNII-1 36–48 + UNII-3 149–161); 6GHz steering for capable clients.
  • Min data rates (kill 1–11Mbps, 2.4 floor 12/24Mbps); set 2.4 min-RSSI -75/-76 on the 6 OFF APs; pin 4 off-plan APs to 1/6/11.
  • Pull controller radar-detection event history to empirically confirm DFS unusability.
  • Secondary: fix the non-working ".ovpn / Download configuration" — likely pfSense OpenVPN Client Export (192.168.0.1), not UniFi. Needed for watch-ap.sh live validation.
  • AP 108 offline — KNOWN, needs a new cable run (per Howard); ignore for now. Also a stale duplicate controller object (108 vs 108U7 Pro) to clean up later.

Reference Information

  • Coord message to GURU-5070 re UOS key: id a4b385ad-4fbb-4097-a066-099622080055; backstop todo 3bd12a14-2b51-4c11-8f76-3f835b07e8dc (--user mike).
  • unifi-wifi skill: .claude/skills/unifi-wifi/ (methodology.md, data-access.md, interference-model.md).
  • Prior wireless log: clients/cascades-tucson/session-logs/2026-05-16-howard-wireless-diagnostic.md.
  • UOS system wiki: wiki/systems/uos-server.md.

Update: 20:40 PT — RW cred arrived, live (Plane 2) re-look, live-stats.sh accuracy fixes

Mike vaulted the RW controller admin (infrastructure/uos-server-network-api-rw) within ~30 min of the request, so the live Network API (Plane 2) became available. Re-audited Cascades live and, in doing so, found + fixed accuracy bugs in unifi-wifi/scripts/live-stats.sh that had skewed the earlier read.

live-stats.sh bugs fixed (held a coord lock; messaged Mike e8be889f)

  1. stat/device output hard-capped at head -60 -> only ~15 of 77 APs were shown (we'd been judging the whole site from a 15-AP sample). Removed the cap; verified 77/77 now.
  2. satisfaction was read per-radio (always -1 on this controller). DEVICE-level satisfaction IS populated -> switched to d['satisfaction'].
  3. tx_retries was the raw cumulative counter (scales with traffic, misleading). Switched to radio_table_stats.tx_retries_pct (a true rate).
  4. --clients was [:40] unsorted (hid 90% of 574 clients). Now sorts worst-by-satisfaction and prints signal/noise/retry%/satisfaction_reason.
  5. RF-neighbor table left as a documented TODO in the header (not a single API field; must be built from the rogue BSSID cross-ref vs each AP's vap_table). It's what unlocks confident radio DISABLES; until then power-down/channel/width are the safe levers.

Corrected diagnosis (the data fix changed a conclusion)

  • Accurate avg retry RATE: 2.4GHz 11.2% > 5GHz clear 9.0% ~= 5GHz DFS 8.4%. My mid-session claim that "5GHz/DFS is now the #1 problem" was an ARTIFACT of the raw counter + 15-AP sample and is WITHDRAWN. On the rate, DFS is NOT retrying worse than clear channels.
  • 2.4GHz is the primary pain band (highest retry; 27 of the 40 worst clients are on 2.4, retry 11-42%, mostly IoT/legacy: Ring cams, robotic cleaner, smart plugs, EPSON printer, Poly phone, handheld scanners, Watch). The original 2.4 power-down/prune plan stands as #1.
  • DFS = resilience risk, not throughput killer. 55/77 5GHz radios on DFS near Davis-Monthan = radar-vacate exposure (client drops), worth moving off DFS for stability, but not urgent for performance.
  • 6GHz still dead (1 client of 574) — top untapped clean/non-DFS capacity; steering remains a key opportunity.
  • AP-level satisfaction 95-100 across the fleet — network is healthy on average; pain is in the client tail = consistent with "bad for SOME users."

Cascades live snapshot (2026-06-15 ~20:30 PT)

  • 77/77 APs reporting; 574 wireless clients. Band split: na (5GHz) ~87 / ng (2.4) ~21 sampled per pull / 6e ~1.
  • 2.4 cu_total 69-94% live (saturation confirmed). 5GHz cu_total mostly <40%.
  • Worst clients: RingStickupCam (sat 60), Galaxy-A32-5G (60), DIRECTV (71), Samsung on ch1 (76, retry 42%).

Updated next steps

  • Build Floor-pilot DRY-RUN: 2.4 power-down to Low on one floor (e.g. Floor 4), validate live cu_total/retry% before+after via the fixed live-stats.sh. Get explicit go before --apply.
  • Implement the AP-to-AP RF-neighbor table (rogue BSSID x vap_table) to enable safe DISABLEs.
  • 6GHz steering plan; 5GHz 80->40MHz + non-DFS channel plan (resilience).
  • Coord msgs this update: RW-cred request 6b98282f (+todo cbb355ef); live-stats fix e8be889f.
  • pfSense .ovpn (Howard handling) — needed for per-AP watch-ap.sh live stream.

Update: 22:05 PT — site VPN up (split-tunnel), direct-AP cross-validation, watch-ap.sh fixed

Howard got the Cascades OpenVPN running. The pushed config is full-tunnel (pfSense redirect-gateway) which killed his local internet + Tailscale (and my tool calls, since they run on his machine). Fixed with a split-tunnel client profile: C:\Users\Howard\Documents\ComputerGuru Connect v2\Files\cascades-splittunnel.ovpn — copy of the working profile + pull-filter ignore "redirect-gateway", pull-filter ignore "dhcp-option DNS", route 192.168.0.0 255.255.252.0, and the VPN creds embedded inline (<auth-user-pass> = vault clients/cascades-tucson/pfsense-openvpn-howard, user Howard). After import+connect: internet + Tailscale + direct AP reach all work simultaneously.

Rescan + ground-truth verification (Howard: "test until sure there are no issues")

  • Controller rescan: 77/77 APs, config identical to prior pull — no drift, scripts stable.
  • Cross-validated controller data against AP ground truth (direct SSH to APs, mca-dump + iw survey) on TWO APs (132 @192.168.2.34, 335 @192.168.3.46): controller cu_total/clients match the radios' own numbers within polling jitter (AP132 2.4: controller 81% / mca 87% / iw survey 92% busy, -95dBm noise; AP335: 2.4 controller 81% vs mca 79%, 5GHz 10%=10% exact, clients 2=2 exact). Conclusion: the controller is accurate; the diagnosis stands on solid data.
  • New concrete fact: U7-Pro 2.4 radios report min_txpower 6 / max 23 dBm → "Low" = 6 dBm (aggressive cell shrink — good for the power-down plan).

watch-ap.sh fixed (2 bugs found by testing; validated live; Mike notified b0cf24a5)

  1. Hard sshpass dependency → failed on Windows (no sshpass). Added SSH_ASKPASS fallback (temp askpass helper + SSH_ASKPASS_REQUIRE=force); uses sshpass if present. Fallback needs ssh from PATH = MSYS ssh on Windows (system OpenSSH can't exec a shell askpass, CreateProcessW err 193); Linux/macOS system ssh is fine.
  2. Radio list included virtual APs (wifi0ap0..) + mld-wifi0 → duplicate lines AND sh: malformed ?: operator (hyphen in mld-wifi0 broke ${ACT_mld-wifi0:-0}). Filter now grep -E '^(wifi[0-9]+|ath[0-9]+)$'. Validated: clean 3-line/tick (wifi0 ch6 92%, wifi1 8%, wifi2 0%).
    • Minor (not fixed, interactive use is fine): script's grep|head block-buffers when piped non-interactively; would need stdbuf -oL to be pipeable.

Issues found this round (none in the data; environment/VPN only)

  • 192.168.0.0/24 home-LAN overlap: Howard's home LAN is 192.168.0.x, so Cascades devices on 192.168.0.x (pfSense 192.168.0.1, cascadesDS 192.168.0.120) are shadowed/unreachable over the VPN from Howard-Home. APs on 192.168.2.x/3.x are unaffected. Use Tailscale for controller; for pfSense/NAS over VPN would need a home renumber or a more specific route trick.
  • OpenVPN Connect DCO↔TAP failover intermittently drops the 192.168.2.x route mid-session (caused two transient watch-ap timeouts). Community OpenVPN GUI is the more stable client if it recurs.

State / next

  • live-stats.sh + watch-ap.sh both fixed + validated; controller data verified accurate.
  • NEXT (per Howard): one more clean retest pass, then build the AP-to-AP RF-neighbor table (now feasible via direct AP iw scan/scan_radio_table) to unlock safe radio DISABLEs; then the Floor-4 2.4 power-down pilot with live before/after validation.
  • Coord this update: watch-ap.sh fix b0cf24a5.

Update: 23:03 PT — RF-neighbor table DEFINITIVELY unobtainable; survey-dump is the usable substitute

Exhaustively investigated whether an AP-to-AP RF-neighbor table (which of OUR managed APs each AP hears + RSSI) is obtainable, to enable data-driven radio DISABLE decisions. Cross-checked against two rounds of UniFi's own AI guidance. Conclusion: NOT obtainable on this deployment by any safe/supported method. Stop chasing it.

Why it's impossible here (all paths tested)

  • rogue collection AND Network API stat/rogueap (7,354 records): both FILTER OUT our managed APs — only foreign BSSIDs (our OUI 0c:ea:14 = 2 hits, CSCNet = 0, is_ubnt = 2). By design.
  • AP iw dev wifiX scan: firmware-blocked — "Operation not permitted (-1)" even as uid=0(root), on all radios incl. an idle 6GHz radio. UniFi locks raw scanning on operating radios.
  • iw scan dump (cached): empty. device.scan_radio_table: empty []. No spectrum/neighbor collection exists. hostapd_cli show_neighbor (802.11k): FAIL on every VAP.
  • mca-cli-op: interactive CLI (bare invocation spews prompts — feed </dev/null). Subcommand info works (firmware 8.6.11.18870, U7-Pro) but exposes NO neighbor/RF data; wireless/radio/ status return nothing useful.
  • Channel AI / channelplan: stale (last run 2026-04-06); exposes only resulting channel lists, not the neighbor graph. UniFi consumes neighbor data internally but exposes it nowhere.
  • Roam graph (ace_stat.wifi_connectivity_event): too sparse (50 clients, 6 edges) — Cascades devices are largely stationary (senior living), so roams can't prove coverage redundancy.
  • A disruptive controller spectrum-scan would only return per-channel RF ENERGY, not a managed-AP BSSID list — so even disruptive testing won't yield the neighbor table. NOT worth running.

NET: automated radio DISABLES are not data-supportable at Cascades. optimize-radios.sh "disable=0" is correct. Any disables must be MANUAL (UniFi UI RF Environment view + on-site).

The usable WIN: non-disruptive per-AP per-channel RF environment via iw survey dump

iw dev wifiX survey dump returns, per AP, per-channel noise floor + airtime busy% across the WHOLE band (not just in-use) — already populated by background scanning, zero disruption. This is real measured RF data that supports a DATA-DRIVEN CHANNEL PLAN. Sample AP 132 (Rec Room):

  • 2.4 GHz: ch1 89%, ch6 95% (noise -65dBm), ch11 96% — every channel saturated.
  • 5 GHz: UNII-1/low (36-48) 1-20% = cleanest; ch157 31% = congested → confirms bias 5GHz LOW.
  • 6 GHz: ~0% across nearly all channels → wide-open lane, confirms 6GHz steering is the big win.

Decisions / state

  • Do NOT pursue the AP-neighbor table or disruptive spectrum-scan further (definitively closed).
  • Channel plan can be MEASURED from survey dump (non-disruptive) instead of inferred.
  • Two safe high-value moves remain: (1) 2.4 GHz power-down per-zone; (2) measured non-DFS channel plan + 6GHz steering.

Next steps

  • Build survey-collect.sh (unifi-wifi skill): sweep all 77 APs' iw survey dump (non-disruptive) → fleet-wide measured per-channel busy%/noise map → drive the channel plan.
  • Floor-4 2.4 GHz power-down pilot with live before/after (watch-ap.sh + live-stats.sh).
  • Record the neighbor-data dead-end + survey-dump method in skill docs (interference-model.md, watch-ap.sh/live-stats.sh headers) so it is never re-chased.
  • (Howard) has more information to add — pending.
  • Confirmed env: firmware 8.6.11.18870 on U7-Pro; AP device-auth SSH = uid 0 (root); split-tunnel VPN gives AP reach on 192.168.2.x/3.x (192.168.0.x shadowed by home LAN).

Update: 23:24 PT — BREAKTHROUGH: AP-to-AP SNR matrix IS obtainable; neighbor-collect.sh built; disables now data-supportable

REVERSES the earlier "AP-neighbor table unobtainable / disables not data-supportable" conclusion. The data DOES exist — not via any controller API/DB (all of which filter our managed APs) but on each AP in /proc, populated NON-DISRUPTIVELY by UniFi's background RRM scanning:

  • /proc/ui_neighbor/ess_ap_list — full list of managed APs this AP hears (serial/band/channel)
  • /proc/ui_neighbor/ssid/<n> — same neighbors WITH SNR (per scan-vap, per band) Discovered via strings $(which mcad) | grep neighbor → /proc/ui_neighbor/ess_ap_list + load_ssid_neighbor_cache. /proc/ui_neighbor exists on every UniFi AP → fleet-generic.

NEW: .claude/skills/unifi-wifi/scripts/neighbor-collect.sh (built + validated, Mike notified 3dbe2437)

Logs into controller for the mac/bssid->AP-name map, SSHes each online AP (sshpass OR SSH_ASKPASS fallback), reads /proc/ui_neighbor, maps neighbors to AP names, emits the AP-to-AP SNR adjacency matrix + a redundancy summary (APs heard by >=2 neighbors at SNR>=N = power-down/disable candidates). Usage: neighbor-collect.sh <site> [ap-ssh-vault-path] [snr_min]. Run FOREGROUND. Validated on Cascades: 74/74 APs reporting. Physically sensible (115<->116 SNR 63, 121<->221 SNR 65 = adjacent). Redundancy: 73/74 APs have >=2 strong (SNR>=20) 2.4GHz neighbors, 72/74 on 5GHz, 65/74 on 6GHz → nearly every AP is coverage-redundant on 2.4 → aggressive 2.4 pruning is data-supported, and individual radio DISABLES are now decidable from data.

Three bugs found+fixed building it (apply to any loop-over-AP script)

  1. Python wrote the AP-list temp file in Windows TEXT mode -> CRLF -> bash read left \r on the IP -> ssh "hostname contains invalid characters" on EVERY AP (the "reachable 0" symptom). Fix: open(...,newline='\n') + ip="${ip%$'\r'}" strip in the read loop.
  2. Must run FOREGROUND — a fully detached background process can't spawn the SSH_ASKPASS helper (both background runs got reachable 0; foreground after the CRLF fix got 74).
  3. ssh in a while read loop needs </dev/null or it consumes the loop's stdin.

Skill capability assessment (Howard: "must work for ALL clients / any unifi issue")

For WiFi/RF, the skill now gathers essentially everything obtainable, all site-parameterized (works for any of the 49 UOS sites): config (uos-mongo), live per-AP+per-client (live-stats), interference (audit-site + neighbor-collect), per-AP ground truth (watch-ap: mca-dump/iw survey), the AP-to-AP SNR matrix (neighbor-collect), and gated apply (apply-radio). Per-client gap: watch-ap/neighbor-collect default the AP device-auth cred to cascades — other clients need their own clients/<x>/unifi-ap-ssh vaulted (overridable via arg). Out of current scope (it's unifi-WIFI): switch/PoE, gateway/WAN/firewall, adoption — reachable via the same access layer (uos-mongo/controller API/device SSH) but not yet wrapped as scripts.

Pending follow-ups

  • Wire neighbor-collect redundancy output into optimize-radios.sh (data-backed disables).
  • Fold survey-dump (per-channel busy/noise) + dmesg DFS-radar recipes into the skill as reusable collectors (the two ad-hoc datasets from this session).
  • Per-client AP device-auth creds for other clients when extending beyond Cascades.
  • Floor-4 2.4 power-down pilot (still the next live change; nothing applied yet).
  • Coord this update: neighbor-collect.sh announce 3dbe2437.
Reference in New Issue View Git Blame Copy Permalink