azcomputerguru/claudetools
1
0
Fork 0
Code Issues 58 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
40ff8accd3d053854e2efed4b3f3faa7388fec98
claudetools/clients/cascades-tucson/session-logs/2026-06/2026-06-15-howard-cascades-wifi-rf-audit.md
Howard Enos 2fea35714b sync: auto-sync from HOWARD-HOME at 2026-06-17 00:09:45 
Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-06-17 00:09:45
2026-06-17 00:09:53 -07:00

56 KiB
Raw Blame History

User

  • User: Howard Enos (howard)
  • Machine: Howard-Home
  • Role: tech

Session Summary

Resumed the Cascades wireless investigation that had stalled on 2026-05-16 (that session was read-only via the cloud API and blocked from per-AP RF data). The day's earlier syncs delivered the unblock: Mike (GURU-5070) vaulted infrastructure/uos-server-ssh-key + clients/cascades-tucson/unifi-ap-ssh and shipped a purpose-built unifi-wifi skill (audit/model-rank/optimize/apply/watch scripts + methodology references). Controller access via the vaulted key was verified, then the full live audit ran against the Cascades site (685f39068e65331c46ef6dd2) on the UOS controller (172.16.3.29).

The audit confirmed and quantified the 05-16 hypothesis with real controller data: 77 U7-Pro APs, all running 2.4GHz at auto (~full) power, 20MHz. 2.4 airtime (cu_total) is 74–94% busy on 75 radios with 61–81% of that being pure interference, serving ~1 client each, at TX-retry rates of 40–65% and single-digit AP satisfaction on the worst (209 sat=1/retr=65%, 139 sat=2/retr=49%, CC Bridge retr=48%). Neighbor-BSSID density is catastrophic on 2.4 (ch6=33,370, ch1=19,274, ch11=16,580). 5GHz is on 80MHz width on 76/77 APs (kills spatial reuse), biased to the busy upper channels (149/157). 6GHz is active on 75 radios but nearly empty of clients. 6 APs have 2.4 min-RSSI OFF (615, 608, 505, 517, 622, salon); 4 are off the 1/6/11 plan on auto (128, 108, 108U7-Pro, salon).

Diagnosis of "bad for SOME users": experience splits by band. Clients on 5/6GHz are fine; clients that land or stick on 2.4GHz (legacy phones, medical/IoT, poor band-steerers, or anything held by a min-RSSI-OFF AP from across the building) hit the saturated 2.4 radios with 40–65% retransmits and near-zero satisfaction.

Late in the session Howard raised the military-base / DFS factor — Cascades is in Tucson near Davis-Monthan AFB (+ TUS airport radar). This reverses the earlier "bias 5GHz toward DFS" recommendation: DFS channels (UNII-2/2e, ch52–144) will see frequent radar-detection events forcing channel-vacate + CAC silence, producing exactly the intermittent per-area dropouts reported. Revised plan uses non-DFS only (UNII-1 36–48 + UNII-3 149–161), which makes 40MHz width and 6GHz steering more important.

Key Decisions

  • Used the unifi-wifi skill end-to-end (audit-site, model-rank, optimize-radios, apply-radio dry-run) rather than ad-hoc Mongo queries — it encodes the multi-model methodology and coverage-safe model.
  • 5GHz channel plan revised to AVOID DFS due to proximity to Davis-Monthan AFB. Non-DFS UNII-1 + UNII-3 only; verify empirically against the controller's radar-detection event history before reconsidering DFS.
  • No changes applied — writes are intentionally gated until a read-WRITE controller admin is vaulted (infrastructure/uos-server-network-api-rw) and --apply is passed. Confirmed the apply path works via dry-run (Floor 3: 17 radios auto->low with rollback values captured).
  • Rollout will be per-zone (one floor at a time) with live before/after validation, never site-wide at once.

Problems Encountered

  • Controller SSH key not on Howard-Home and not in vault (earlier in day). Tested both vaulted keys (gururmm-server-physical, openclaw-fleet) against root@172.16.3.29 — both denied; OC-5070/OC-Mac unreachable over Tailscale, fleet key denied on OC-Beast. Resolved by coord-requesting Mike to vault the UOS key; he did (infrastructure/uos-server-ssh-key), picked up on sync.
  • Cloud Site Manager API insufficient (re-confirmed). Device objects carry no site field and no RF/channel/power/uplink data — only online/offline + firmware. Cannot drive RF tuning. The Mongo-via-SSH path (now available) is required.

Configuration Changes

  • None to Cascades infra. Read-only audit + dry-run only.
  • No repo files edited beyond this session log.

Credentials & Secrets

  • infrastructure/uos-server-ssh-key (vaulted by Mike this day) — root SSH key for the UOS controller 172.16.3.29; used by uos-mongo.sh / unifi-wifi scripts. This is the DATA plane (read), not an API write session.
  • clients/cascades-tucson/unifi-ap-ssh (vaulted by Mike this day) — device-auth cred for SSHing directly into Cascades APs (used by watch-ap.sh; needs site VPN for L3 reach to 192.168.2.x/3.x).
  • Needed, not yet created: infrastructure/uos-server-network-api-rw (read-write controller admin) to apply radio changes; infrastructure/uos-server-network-api (read-only admin) to wire live-stats Plane 2 validation.

Infrastructure & Servers

  • UOS controller: 172.16.3.29 (Rocky 9 VM "Unifi" on Jupiter 172.16.3.20); UniFi-OS HTTPS on 11443 (not 8443). Mongo ace on 127.0.0.1:27117 inside rootless podman uosserver. Cascades site_id 685f39068e65331c46ef6dd2.
  • Cascades wireless: 77 U7-Pro APs, ~550 clients. Firewall = pfSense 192.168.0.1 (site VPN endpoint; .ovpn comes from pfSense OpenVPN Client Export, NOT UniFi). APs on 192.168.2.x/3.x.
  • Location/RF: Tucson, near Davis-Monthan AFB + TUS radar → DFS unreliable.

Commands & Outputs

bash .claude/scripts/uos-mongo.sh --sites | grep -i casc        # 685f39068e65331c46ef6dd2  Cascades (access OK)
bash .claude/skills/unifi-wifi/scripts/audit-site.sh cascades    # config + neighbor-density + flags
bash .claude/skills/unifi-wifi/scripts/model-rank.sh cascades 7 ng
bash .claude/skills/unifi-wifi/scripts/optimize-radios.sh cascades 14 ng   # power-down 74, disable 0, keep 1
bash .claude/skills/unifi-wifi/scripts/apply-radio.sh cascades ng power low --zone "Floor 3"  # DRY-RUN: 17 radios auto->low

Key audit output: 2.4 cu_total 74–94% / interf 61–81% / ~1 client; retry 40–65%; ch6=33,370 neighbors; 5GHz 80MHz on 76/77; 6GHz active 75 but empty; min-RSSI OFF on 615/608/505/517/622/salon.

Pending / Incomplete Tasks

  • Vault read-WRITE controller admin infrastructure/uos-server-network-api-rw (blocks applying any radio change). Candidate: coord-request Mike.
  • Vault read-only infrastructure/uos-server-network-api to wire live-stats Plane 2 (before/after cu_total/satisfaction validation).
  • Apply Phase A (2.4 power-down to Low) per-zone with live validation, once RW cred exists.
  • 5GHz: 80->40MHz width; non-DFS channel plan (UNII-1 36–48 + UNII-3 149–161); 6GHz steering for capable clients.
  • Min data rates (kill 1–11Mbps, 2.4 floor 12/24Mbps); set 2.4 min-RSSI -75/-76 on the 6 OFF APs; pin 4 off-plan APs to 1/6/11.
  • Pull controller radar-detection event history to empirically confirm DFS unusability.
  • Secondary: fix the non-working ".ovpn / Download configuration" — likely pfSense OpenVPN Client Export (192.168.0.1), not UniFi. Needed for watch-ap.sh live validation.
  • AP 108 offline — KNOWN, needs a new cable run (per Howard); ignore for now. Also a stale duplicate controller object (108 vs 108U7 Pro) to clean up later.

Reference Information

  • Coord message to GURU-5070 re UOS key: id a4b385ad-4fbb-4097-a066-099622080055; backstop todo 3bd12a14-2b51-4c11-8f76-3f835b07e8dc (--user mike).
  • unifi-wifi skill: .claude/skills/unifi-wifi/ (methodology.md, data-access.md, interference-model.md).
  • Prior wireless log: clients/cascades-tucson/session-logs/2026-05-16-howard-wireless-diagnostic.md.
  • UOS system wiki: wiki/systems/uos-server.md.

Update: 20:40 PT — RW cred arrived, live (Plane 2) re-look, live-stats.sh accuracy fixes

Mike vaulted the RW controller admin (infrastructure/uos-server-network-api-rw) within ~30 min of the request, so the live Network API (Plane 2) became available. Re-audited Cascades live and, in doing so, found + fixed accuracy bugs in unifi-wifi/scripts/live-stats.sh that had skewed the earlier read.

live-stats.sh bugs fixed (held a coord lock; messaged Mike e8be889f)

  1. stat/device output hard-capped at head -60 -> only ~15 of 77 APs were shown (we'd been judging the whole site from a 15-AP sample). Removed the cap; verified 77/77 now.
  2. satisfaction was read per-radio (always -1 on this controller). DEVICE-level satisfaction IS populated -> switched to d['satisfaction'].
  3. tx_retries was the raw cumulative counter (scales with traffic, misleading). Switched to radio_table_stats.tx_retries_pct (a true rate).
  4. --clients was [:40] unsorted (hid 90% of 574 clients). Now sorts worst-by-satisfaction and prints signal/noise/retry%/satisfaction_reason.
  5. RF-neighbor table left as a documented TODO in the header (not a single API field; must be built from the rogue BSSID cross-ref vs each AP's vap_table). It's what unlocks confident radio DISABLES; until then power-down/channel/width are the safe levers.

Corrected diagnosis (the data fix changed a conclusion)

  • Accurate avg retry RATE: 2.4GHz 11.2% > 5GHz clear 9.0% ~= 5GHz DFS 8.4%. My mid-session claim that "5GHz/DFS is now the #1 problem" was an ARTIFACT of the raw counter + 15-AP sample and is WITHDRAWN. On the rate, DFS is NOT retrying worse than clear channels.
  • 2.4GHz is the primary pain band (highest retry; 27 of the 40 worst clients are on 2.4, retry 11-42%, mostly IoT/legacy: Ring cams, robotic cleaner, smart plugs, EPSON printer, Poly phone, handheld scanners, Watch). The original 2.4 power-down/prune plan stands as #1.
  • DFS = resilience risk, not throughput killer. 55/77 5GHz radios on DFS near Davis-Monthan = radar-vacate exposure (client drops), worth moving off DFS for stability, but not urgent for performance.
  • 6GHz still dead (1 client of 574) — top untapped clean/non-DFS capacity; steering remains a key opportunity.
  • AP-level satisfaction 95-100 across the fleet — network is healthy on average; pain is in the client tail = consistent with "bad for SOME users."

Cascades live snapshot (2026-06-15 ~20:30 PT)

  • 77/77 APs reporting; 574 wireless clients. Band split: na (5GHz) ~87 / ng (2.4) ~21 sampled per pull / 6e ~1.
  • 2.4 cu_total 69-94% live (saturation confirmed). 5GHz cu_total mostly <40%.
  • Worst clients: RingStickupCam (sat 60), Galaxy-A32-5G (60), DIRECTV (71), Samsung on ch1 (76, retry 42%).

Updated next steps

  • Build Floor-pilot DRY-RUN: 2.4 power-down to Low on one floor (e.g. Floor 4), validate live cu_total/retry% before+after via the fixed live-stats.sh. Get explicit go before --apply.
  • Implement the AP-to-AP RF-neighbor table (rogue BSSID x vap_table) to enable safe DISABLEs.
  • 6GHz steering plan; 5GHz 80->40MHz + non-DFS channel plan (resilience).
  • Coord msgs this update: RW-cred request 6b98282f (+todo cbb355ef); live-stats fix e8be889f.
  • pfSense .ovpn (Howard handling) — needed for per-AP watch-ap.sh live stream.

Update: 22:05 PT — site VPN up (split-tunnel), direct-AP cross-validation, watch-ap.sh fixed

Howard got the Cascades OpenVPN running. The pushed config is full-tunnel (pfSense redirect-gateway) which killed his local internet + Tailscale (and my tool calls, since they run on his machine). Fixed with a split-tunnel client profile: C:\Users\Howard\Documents\ComputerGuru Connect v2\Files\cascades-splittunnel.ovpn — copy of the working profile + pull-filter ignore "redirect-gateway", pull-filter ignore "dhcp-option DNS", route 192.168.0.0 255.255.252.0, and the VPN creds embedded inline (<auth-user-pass> = vault clients/cascades-tucson/pfsense-openvpn-howard, user Howard). After import+connect: internet + Tailscale + direct AP reach all work simultaneously.

Rescan + ground-truth verification (Howard: "test until sure there are no issues")

  • Controller rescan: 77/77 APs, config identical to prior pull — no drift, scripts stable.
  • Cross-validated controller data against AP ground truth (direct SSH to APs, mca-dump + iw survey) on TWO APs (132 @192.168.2.34, 335 @192.168.3.46): controller cu_total/clients match the radios' own numbers within polling jitter (AP132 2.4: controller 81% / mca 87% / iw survey 92% busy, -95dBm noise; AP335: 2.4 controller 81% vs mca 79%, 5GHz 10%=10% exact, clients 2=2 exact). Conclusion: the controller is accurate; the diagnosis stands on solid data.
  • New concrete fact: U7-Pro 2.4 radios report min_txpower 6 / max 23 dBm → "Low" = 6 dBm (aggressive cell shrink — good for the power-down plan).

watch-ap.sh fixed (2 bugs found by testing; validated live; Mike notified b0cf24a5)

  1. Hard sshpass dependency → failed on Windows (no sshpass). Added SSH_ASKPASS fallback (temp askpass helper + SSH_ASKPASS_REQUIRE=force); uses sshpass if present. Fallback needs ssh from PATH = MSYS ssh on Windows (system OpenSSH can't exec a shell askpass, CreateProcessW err 193); Linux/macOS system ssh is fine.
  2. Radio list included virtual APs (wifi0ap0..) + mld-wifi0 → duplicate lines AND sh: malformed ?: operator (hyphen in mld-wifi0 broke ${ACT_mld-wifi0:-0}). Filter now grep -E '^(wifi[0-9]+|ath[0-9]+)$'. Validated: clean 3-line/tick (wifi0 ch6 92%, wifi1 8%, wifi2 0%).
    • Minor (not fixed, interactive use is fine): script's grep|head block-buffers when piped non-interactively; would need stdbuf -oL to be pipeable.

Issues found this round (none in the data; environment/VPN only)

  • 192.168.0.0/24 home-LAN overlap: Howard's home LAN is 192.168.0.x, so Cascades devices on 192.168.0.x (pfSense 192.168.0.1, cascadesDS 192.168.0.120) are shadowed/unreachable over the VPN from Howard-Home. APs on 192.168.2.x/3.x are unaffected. Use Tailscale for controller; for pfSense/NAS over VPN would need a home renumber or a more specific route trick.
  • OpenVPN Connect DCO↔TAP failover intermittently drops the 192.168.2.x route mid-session (caused two transient watch-ap timeouts). Community OpenVPN GUI is the more stable client if it recurs.

State / next

  • live-stats.sh + watch-ap.sh both fixed + validated; controller data verified accurate.
  • NEXT (per Howard): one more clean retest pass, then build the AP-to-AP RF-neighbor table (now feasible via direct AP iw scan/scan_radio_table) to unlock safe radio DISABLEs; then the Floor-4 2.4 power-down pilot with live before/after validation.
  • Coord this update: watch-ap.sh fix b0cf24a5.

Update: 23:03 PT — RF-neighbor table DEFINITIVELY unobtainable; survey-dump is the usable substitute

Exhaustively investigated whether an AP-to-AP RF-neighbor table (which of OUR managed APs each AP hears + RSSI) is obtainable, to enable data-driven radio DISABLE decisions. Cross-checked against two rounds of UniFi's own AI guidance. Conclusion: NOT obtainable on this deployment by any safe/supported method. Stop chasing it.

Why it's impossible here (all paths tested)

  • rogue collection AND Network API stat/rogueap (7,354 records): both FILTER OUT our managed APs — only foreign BSSIDs (our OUI 0c:ea:14 = 2 hits, CSCNet = 0, is_ubnt = 2). By design.
  • AP iw dev wifiX scan: firmware-blocked — "Operation not permitted (-1)" even as uid=0(root), on all radios incl. an idle 6GHz radio. UniFi locks raw scanning on operating radios.
  • iw scan dump (cached): empty. device.scan_radio_table: empty []. No spectrum/neighbor collection exists. hostapd_cli show_neighbor (802.11k): FAIL on every VAP.
  • mca-cli-op: interactive CLI (bare invocation spews prompts — feed </dev/null). Subcommand info works (firmware 8.6.11.18870, U7-Pro) but exposes NO neighbor/RF data; wireless/radio/ status return nothing useful.
  • Channel AI / channelplan: stale (last run 2026-04-06); exposes only resulting channel lists, not the neighbor graph. UniFi consumes neighbor data internally but exposes it nowhere.
  • Roam graph (ace_stat.wifi_connectivity_event): too sparse (50 clients, 6 edges) — Cascades devices are largely stationary (senior living), so roams can't prove coverage redundancy.
  • A disruptive controller spectrum-scan would only return per-channel RF ENERGY, not a managed-AP BSSID list — so even disruptive testing won't yield the neighbor table. NOT worth running.

NET: automated radio DISABLES are not data-supportable at Cascades. optimize-radios.sh "disable=0" is correct. Any disables must be MANUAL (UniFi UI RF Environment view + on-site).

The usable WIN: non-disruptive per-AP per-channel RF environment via iw survey dump

iw dev wifiX survey dump returns, per AP, per-channel noise floor + airtime busy% across the WHOLE band (not just in-use) — already populated by background scanning, zero disruption. This is real measured RF data that supports a DATA-DRIVEN CHANNEL PLAN. Sample AP 132 (Rec Room):

  • 2.4 GHz: ch1 89%, ch6 95% (noise -65dBm), ch11 96% — every channel saturated.
  • 5 GHz: UNII-1/low (36-48) 1-20% = cleanest; ch157 31% = congested → confirms bias 5GHz LOW.
  • 6 GHz: ~0% across nearly all channels → wide-open lane, confirms 6GHz steering is the big win.

Decisions / state

  • Do NOT pursue the AP-neighbor table or disruptive spectrum-scan further (definitively closed).
  • Channel plan can be MEASURED from survey dump (non-disruptive) instead of inferred.
  • Two safe high-value moves remain: (1) 2.4 GHz power-down per-zone; (2) measured non-DFS channel plan + 6GHz steering.

Next steps

  • Build survey-collect.sh (unifi-wifi skill): sweep all 77 APs' iw survey dump (non-disruptive) → fleet-wide measured per-channel busy%/noise map → drive the channel plan.
  • Floor-4 2.4 GHz power-down pilot with live before/after (watch-ap.sh + live-stats.sh).
  • Record the neighbor-data dead-end + survey-dump method in skill docs (interference-model.md, watch-ap.sh/live-stats.sh headers) so it is never re-chased.
  • (Howard) has more information to add — pending.
  • Confirmed env: firmware 8.6.11.18870 on U7-Pro; AP device-auth SSH = uid 0 (root); split-tunnel VPN gives AP reach on 192.168.2.x/3.x (192.168.0.x shadowed by home LAN).

Update: 23:24 PT — BREAKTHROUGH: AP-to-AP SNR matrix IS obtainable; neighbor-collect.sh built; disables now data-supportable

REVERSES the earlier "AP-neighbor table unobtainable / disables not data-supportable" conclusion. The data DOES exist — not via any controller API/DB (all of which filter our managed APs) but on each AP in /proc, populated NON-DISRUPTIVELY by UniFi's background RRM scanning:

  • /proc/ui_neighbor/ess_ap_list — full list of managed APs this AP hears (serial/band/channel)
  • /proc/ui_neighbor/ssid/<n> — same neighbors WITH SNR (per scan-vap, per band) Discovered via strings $(which mcad) | grep neighbor → /proc/ui_neighbor/ess_ap_list + load_ssid_neighbor_cache. /proc/ui_neighbor exists on every UniFi AP → fleet-generic.

NEW: .claude/skills/unifi-wifi/scripts/neighbor-collect.sh (built + validated, Mike notified 3dbe2437)

Logs into controller for the mac/bssid->AP-name map, SSHes each online AP (sshpass OR SSH_ASKPASS fallback), reads /proc/ui_neighbor, maps neighbors to AP names, emits the AP-to-AP SNR adjacency matrix + a redundancy summary (APs heard by >=2 neighbors at SNR>=N = power-down/disable candidates). Usage: neighbor-collect.sh <site> [ap-ssh-vault-path] [snr_min]. Run FOREGROUND. Validated on Cascades: 74/74 APs reporting. Physically sensible (115<->116 SNR 63, 121<->221 SNR 65 = adjacent). Redundancy: 73/74 APs have >=2 strong (SNR>=20) 2.4GHz neighbors, 72/74 on 5GHz, 65/74 on 6GHz → nearly every AP is coverage-redundant on 2.4 → aggressive 2.4 pruning is data-supported, and individual radio DISABLES are now decidable from data.

Three bugs found+fixed building it (apply to any loop-over-AP script)

  1. Python wrote the AP-list temp file in Windows TEXT mode -> CRLF -> bash read left \r on the IP -> ssh "hostname contains invalid characters" on EVERY AP (the "reachable 0" symptom). Fix: open(...,newline='\n') + ip="${ip%$'\r'}" strip in the read loop.
  2. Must run FOREGROUND — a fully detached background process can't spawn the SSH_ASKPASS helper (both background runs got reachable 0; foreground after the CRLF fix got 74).
  3. ssh in a while read loop needs </dev/null or it consumes the loop's stdin.

Skill capability assessment (Howard: "must work for ALL clients / any unifi issue")

For WiFi/RF, the skill now gathers essentially everything obtainable, all site-parameterized (works for any of the 49 UOS sites): config (uos-mongo), live per-AP+per-client (live-stats), interference (audit-site + neighbor-collect), per-AP ground truth (watch-ap: mca-dump/iw survey), the AP-to-AP SNR matrix (neighbor-collect), and gated apply (apply-radio). Per-client gap: watch-ap/neighbor-collect default the AP device-auth cred to cascades — other clients need their own clients/<x>/unifi-ap-ssh vaulted (overridable via arg). Out of current scope (it's unifi-WIFI): switch/PoE, gateway/WAN/firewall, adoption — reachable via the same access layer (uos-mongo/controller API/device SSH) but not yet wrapped as scripts.

Pending follow-ups

  • Wire neighbor-collect redundancy output into optimize-radios.sh (data-backed disables).
  • Fold survey-dump (per-channel busy/noise) + dmesg DFS-radar recipes into the skill as reusable collectors (the two ad-hoc datasets from this session).
  • Per-client AP device-auth creds for other clients when extending beyond Cascades.
  • Floor-4 2.4 power-down pilot (still the next live change; nothing applied yet).
  • Coord this update: neighbor-collect.sh announce 3dbe2437.

Update: 23:50 PT — gap 1 + gap 2 closed: survey-collect + dfs-check built; SKILL marked wifi-working / switches-WIP

GAP 1 (fold ad-hoc recipes into reusable collectors) — DONE, both validated on Cascades:

  • survey-collect.sh <site> [ap-ssh-vault-path] — per-AP per-channel busy%/noise from iw survey dump -> in-use busy% + cleanest non-DFS channels per band. 72 APs reported. Result: 2.4 saturated 94-97% on every channel; 5GHz cleanest non-DFS = ch36/48/153/161; 6GHz ~0%.
  • dfs-check.sh <site> [ap-ssh-vault-path] — per-AP dmesg radar history, PRECISE patterns (excludes the "cached ifindex" false-positive). Result: 55/75 APs on DFS, ZERO real radar events fleet-wide -> DFS empirically low-risk at Cascades despite Davis-Monthan proximity (confirms the retry-rate finding; the DFS concern was theoretical).

GAP 2 (scope) — DONE via SKILL.md Status block: WiFi monitoring+tuning = WORKING for every UOS site we monitor (audit-site, live-stats, model-rank, optimize-radios, survey-collect, dfs-check, neighbor-collect, watch-ap, apply-radio — all site-parameterized). Switches/PoE, gateway/WAN/ firewall, adoption = WIP (access layer reaches them; need dedicated collectors). Per-client: vault each client's own clients/<x>/unifi-ap-ssh, pass as the script arg.

CROSS-PLATFORM LESSON (baked into both new scripts): pass temp paths to python via ARGV — MSYS translates POSIX->Windows for python.exe; $TMP embedded in a python -c string is NOT translated and fails ("No such file or directory") on Windows. Use python - "$TMP/x" <<'PY' ... sys.argv[1].

Coord: collectors+status announce c3ccaa07. Next: wire neighbor-collect redundancy into optimize-radios.sh; Floor-4 2.4 power-down pilot (still nothing applied to live radios).

Update: 00:05 PT (06-16) — wired neighbor SNR matrix into optimize-radios.sh (disables now data-backed)

Closed the last follow-up: the AP-to-AP SNR matrix now DRIVES optimize-radios.sh disable decisions.

  • neighbor-collect.sh: NBR_JSON=<path> env now also emits machine-readable adjacency {ap:{band:{nbr:snr}}} (74 APs, ~21KB).
  • optimize-radios.sh: NEIGHBOR_JSON=<path> env now uses the measured bidirectional SNR overlap (A hears B AND B hears A, SNR>=NBR_SNR_MIN default 20, any band) as the coverage-adjacency source, replacing the sparse roam graph. Roam graph remains the fallback when NEIGHBOR_JSON unset.

Workflow: NBR_JSON=.claude/tmp/cascades-nbr.json bash .../neighbor-collect.sh cascades NEIGHBOR_JSON=.claude/tmp/cascades-nbr.json bash .../optimize-radios.sh cascades 14 ng

VALIDATED on Cascades 2.4GHz: roam-graph -> POWER-DOWN 74 / DISABLE 0; neighbor-SNR -> POWER-DOWN 65 / DISABLE 9 / KEEP 1, est. 621% interference-airtime removed, each disable listing its covering neighbors (127->128; 229->128; 248->348; 330->128; 445->347/348/247; 428->128; 622->505/615/517; Kitchen->Memcare TV room; salon->128). The disable capability we concluded was impossible is now data-backed for any site with a neighbor-collect run.

IMPL NOTE (bug fixed): injecting the full matrix as a JS object literal CRASHED the old mongo shell (SpiderMonkey GC-during-compile on a ~21KB literal). Fix: precompute bidirectional strong NAME-pairs in python, inject a compact ';'-joined flat STRING, parse in JS. (Pattern for any large data -> mongo-shell injection: flat string, not object literal.)

Still gated: apply-radio.sh has NO disable action (power/channel/width only); disables remain a reviewed, per-zone, live-validated MANUAL step. Coord: wire announce 68cae757. Remaining open: Floor-4 2.4 power-down pilot (still nothing applied to live radios).

Update: 2026-06-16 00:03 PT — skill-build focus: apply-radio generalized + write path validated + ROADMAP

Per Howard: pause the Cascades production fix; build the unifi-wifi skill into a fully working tool for any client. Used Cascades 0-client radios as the safe sandbox (6 GHz is ~0-client fleetwide; no whole-AP had 0 clients).

ADDED references/ROADMAP.md — full A/B/C/D plan: (A) apply side, (B) multi-client enablement, (C) switches/gateway WIP, (D) robustness. Cross-platform gotchas captured for future scripts.

GENERALIZED apply-radio.sh from power-only to 4 radio_table actions (same gated REST mechanism, field-generic via a compact FIELDS JSON shared by the dry-run preview and the apply python; rollback auto-saved): power low|medium|high|auto| -> tx_power_mode (+tx_power) width 20|40|80|160 -> ht channel |auto -> channel minrssi off|on|- -> min_rssi_enabled (+min_rssi)

WRITE PATH VALIDATED end-to-end (first real --apply): on Floor 6 0-client 6e radios (608, 622) applied ht 160->80, controller confirmed ht=80, then reverted to 160 (state restored). Dry-run previews for width/minrssi verified correct (incl. 615 U6-Pro correctly has no 6e radio).

radio_table field names confirmed: channel, ht (width), tx_power_mode/tx_power, min_rssi + min_rssi_enabled. NO radio enable/disable field -> DISABLE not implemented (ROADMAP A; discover via UI-toggle + device-JSON diff). min-data-rate/band-steering are wlanconf-level -> separate path.

SKILL.md apply section updated with all 4 actions + the validated-write-path note. Coord: 6aac1298. Next per roadmap: A (disable mechanism, wlanconf knobs, channel-plan apply) then B (per-client creds/VPN).

Update: 2026-06-16 00:24 PT — radio DISABLE mechanism found; apply-radio disable/enable + --ap (validated)

Howard toggled 6GHz off on AP 622 in the UI; diffed the device-JSON before/after. Disable mechanism: radio_table[].tx_power_mode = "disabled" (radio stays in table, channel/ht untouched). Re-enable = tx_power_mode "auto". Same field as power. (Other diffs were timestamps/cfgversion + JSON int/str noise.)

apply-radio.sh now supports actions: power | width | channel | minrssi | disable | enable, plus a new --ap "" single-AP filter (correct scope for disables). Arg parsing reworked so disable/enable take no value. VALIDATED full enable->disable->enable cycle on 622's 0-client 6GHz via --apply; 622 restored to ON (auto). Workflow: optimize-radios (NEIGHBOR_JSON) disable list -> apply-radio <site> <band> disable --ap <name> --apply one AP at a time, watch-ap validated.

SKILL.md + ROADMAP.md updated (disable now done). apply-side now covers all radio_table knobs; only wlanconf-level (min-data-rate, band-steering) remains -> future apply-wlan.sh. Coord: 6aac1298-> this: msg sent.

Update: 2026-06-16 00:40 PT — multi-client enablement + WLAN-level apply (wlanconf); both validated

MULTI-CLIENT (B): NEW scripts/sites.sh lists all ~49 UOS sites (APs/SW/GW) + per-client AP-cred readiness — the entry point for other clients. Controller-side scripts validated on other clients (Glabman 6 APs, Sonoran Glass 5 APs) with zero setup. AP-side scripts now print the exact vault command when a client's clients//unifi-ap-ssh is missing (controller-side still works). SKILL.md Multi-client section added. (Per-client AP creds vaulted as clients come online.)

WLAN-LEVEL APPLY (A2): NEW scripts/apply-wlan.sh (wlanconf, gated like apply-radio, --wlan filter): minrate ng|na auto|off| (kill 1-11Mbps: minrate ng 12) steer on|off (5GHz roaming_assistant) GOTCHA found+fixed: manual min rate only applies when minrate_setting_preference=manual (else controller auto-manages + silently reverts — first PUT looked [ok] but value stayed 1000). Script now sets the preference; 'minrate ... auto' restores controller management. WRITE PATH VALIDATED on a 0-client WLAN (Green Valley Computer Club): minrate ng 12 -> persisted (pref=manual, 12000) -> restored to auto.

apply-radio (radio_table: power all-states/width/channel/minrssi/disable/enable + --ap/--zone) and apply-wlan (wlanconf: minrate/steer) now cover the WiFi apply surface, all gated + rollback + validated on 0-client sandboxes. ROADMAP updated. Coord: this msg. Remaining: classic band-steering field, per-client creds/VPN, switches/gateway collectors (C).

Update: 2026-06-16 00:58 PT — band-steering field FOUND + wired into apply-wlan (validated)

Modern UniFi (Network 8.x) has NO classic 'bandsteering_mode'. Located its replacements by inspecting wlanconf + the setting collection (no UI toggle needed this time): no2ghz_oui -> the band-steering toggle (keep 5GHz-capable OUIs off 2.4) wlan_bands (["2g","5g"]) -> per-WLAN band membership (set ["5g"]/["5g","6e"] to force clients up) bss_transition -> 802.11v BSS-TM roaming_assistant_na_* -> 5GHz sticky-client kicker fast_roaming_enabled -> 802.11r (left out — risky for IoT/medical)

Wired into apply-wlan.sh as actions: bandsteer (no2ghz_oui) | bands (wlan_bands) | steer (roaming_assistant) | bsstm (bss_transition), alongside minrate. Validated on the 0-client Green Valley WLAN: bandsteer on->verify->off, bsstm toggle, bands 5g dry-run; restored to original (bss_transition=true).

Also surfaced setting.radio_ai (Channel AI: channels_na includes DFS, ht_modes_na=20/40, enabled=false)

  • setting.connectivity — possible future apply targets (pin the channel plan).

WiFi APPLY SURFACE NOW COMPLETE: apply-radio (power all-states/width/channel/minrssi/disable/enable, --zone/--ap) + apply-wlan (minrate/bandsteer/bands/steer/bsstm, --wlan), all gated + rollback + validated on 0-client sandboxes. Coord: this msg. Remaining (ROADMAP C): switch/PoE + gateway collectors.

Update: 2026-06-16 01:09 PT — WiFi control surface complete: full wlanconf knobs + device-lock + client controls

(a) DEVICE LOCK / CONTROL:

  • apply-wlan aps <ap-names|all> -> broadcasting_aps (resolves AP NAMES->MACs; restricts SSID to APs = the "lock device to AP" lever, since UniFi has NO native per-client AP pin). Validated dry-run: aps 622 --wlan CSCNet -> broadcasting_aps=[0c:ea:14:3e:65:a2].
  • apply-wlan macfilter off|allow|deny -> mac_filter_*. Dry-run validated.
  • NEW client-control.sh: block|unblock|kick a MAC via cmd/stamgr (gated). Validated block+unblock on dummy 00:00:00:00:00:01 (rc:ok), cleaned up.

(b) REMAINING wlanconf TUNING KNOBS added to apply-wlan: wlan(enable/disable), dtim (dtim_mode=custom gate), mcast(mcastenhance), bcfilter, rrm(802.11k), ftroam(802.11r, warns), isolation(l2_isolation), hidessid. dtim apply-validated on 0-client Green Valley (mode default->custom, reverted). Others via the proven gated REST path.

apply-wlan restructured: SITE resolved before the action case (so aps can map names->macs). WiFi APPLY SURFACE COMPLETE: apply-radio (radio_table) + apply-wlan (wlanconf, 15 actions) + client-control (cmd/stamgr), all gated + rollback + validated on 0-client sandboxes / dummy MAC. SKILL.md + ROADMAP updated. Coord: this msg. NEXT: switches/PoE collector (ROADMAP C).

Update: 2026-06-16 01:13 PT — switches: switch-audit.sh shipped (first non-WiFi collector, ROADMAP C)

NEW scripts/switch-audit.sh [--all-ports] (controller-side, read-only, any site): per switch -> PoE budget used/total, ports up/total, FLAGS: UNDERSPEED (gig-capable port linked 10/100 = FastEthernet uplink problem), DROPS (rate-based tx_dropped/tx_packets>2%), ERRORS (>1000), POE-FAULT (poe_enable & !poe_good), POE-BUDGET (>85%), + OFFLINE switches. port_table fields: speed/speed_caps/poe_*/rx_tx_errors.

VALIDATED on Cascades (12 switches): found ~25 ports linked at 100M but gig-capable (systematic cabling/NIC issue = the long-suspected FastEthernet uplink problem, now fleet-visible), offline switches (2nd Floor #2, 4th Floor #2, USW Pro Max 16), PoE budgets healthy, one real 4.1% tx-drop port (p38 on 1st Floor USW). Refined DROPS to rate-based (cumulative counters were noisy).

SKILL coverage now: WiFi (monitor+tune+full apply incl. device lock/client controls) + switch/PoE audit, all controller-side multi-client + AP-side where cred vaulted. ROADMAP remaining: gateway/WAN/firewall + adoption collectors. Coord: this msg.

Update: 2026-06-16 01:20 PT — gateway/WAN + site-health collector (gw-audit.sh); ROADMAP C largely done

NEW scripts/gw-audit.sh (controller-side, any site, read-only) via stat/health + gateway device: WAN status/IP/uplink/duplex + gw CPU/mem/uptime; internet(www) status/latency/drops/last-speedtest; health rollup (APs/switches/gw adopted vs disconnected vs pending, client user/guest/iot, firmware- upgradable); flags WAN/internet not-ok, latency>80/drops>5, WAN down, disconnected/pending, gw cpu>85/ mem>90. Third-party-firewall sites (num_gw=0) handled. Fixed: gw_name fallback to device name; drops threshold >5 (was >0, too sensitive).

Validated: Sonoran Glass (USG 3P [UGW3]) -> all healthy (cpu2%/mem21%, latency 14-24ms); Cascades (pfSense) -> "no UniFi gateway" + correctly caught 2 disconnected APs + 3 disconnected switches.

SKILL now end-to-end for any client: WiFi (monitor + tune + full apply incl device-lock/client-control), switch/PoE audit, gateway/WAN/site-health audit, sites.sh discovery. ROADMAP remaining: deeper firewall/VPN policy + adoption REMEDIATION (adopt/restart), channel-plan apply, per-client AP creds/VPN. Coord: this msg.

Update: 2026-06-16 07:26 PT — adoption remediation (device-control.sh) + channel-plan apply (channel-plan.sh)

NEW device-control.sh adopt|restart|provision|locate|unlocate|upgrade [--apply] (cmd/devmgr, gated, controller-side) — remediates what gw-audit flags. Validated locate+unlocate on AP 622 (rc:ok).

NEW channel-plan.sh ng|na [--apply] (NEIGHBOR_JSON + SURVEY_JSON): ng: graph-color 1/6/11 via the neighbor matrix (strong neighbors differ). na: per-AP lowest-cost NON-DFS channel = measured busy% (survey) + neighbor-collision penalty. Reports co-channel pairs before/after; gated apply per-AP rest/device + rollback. DRY-RUN validated on Cascades: ng 92->35 co-channel pairs (51 APs change); na 20->0 + ALL moved off DFS onto non-DFS (radar-safe) (71 APs). Not applied (production paused); PUT mechanism proven. Added SURVEY_JSON= emit to survey-collect.sh ({ap:{band:{channel:busy%}}}).

SKILL now: WiFi (monitor+tune+full apply+device-lock+client/device control+channel-plan) + switch/PoE audit + gateway/WAN/site-health + multi-client. ROADMAP nearly clear (deeper firewall/VPN policy + per-client AP creds/VPN remain). Coord: this msg.

Update: 2026-06-16 07:44 PT — robustness (ROADMAP D): monitor-run.sh + per-AP retry; gw-audit pfSense fix

Synced first to pick up Mike's gw-control.sh (eb87710, firewall/port-forward router actions — the "deeper firewall/VPN policy" item; no dup with my robustness work).

NEW scripts/monitor-run.sh <site|all> — cron-friendly controller-side read-only fleet health digest: per site -> gateway/WAN flags + switch/PoE flags + WiFi config flag count. Validated Sonoran (healthy)

  • Cascades (flags 2 disc APs / 3 disc switches / underspeed / firmware). Cron 'all' nightly.

VPN-flap resilience: neighbor-collect / survey-collect / dfs-check now RETRY per AP (3x, capture-to-var so a failed attempt never appends partial data; dfs-check distinguishes UNREACHABLE vs no-events). Validated neighbor-collect end-to-end (reachable 74/74, redundancy 73/74, JSON 74 APs - identical).

Fix: gw-audit no longer false-flags internet status=unknown on third-party-firewall sites (gated on num_gw).

SKILL.md + ROADMAP updated (D items done). Skill is feature-complete for monitoring+tuning+apply across WiFi/switch/gateway, multi-client, with scheduling + flap resilience. Remaining: per-client AP creds/VPN, read-only cred (Mike to create UI admin), gateway VPN-server/DHCP-DNS (Mike). Coord: this msg.

Update: 2026-06-16 13:12 PT — FIRST PRODUCTION CHANGE: Floor-4 2.4 power-down pilot + 445 outage/recovery learning

Baseline captured (Floor-4 2.4: 15 radios, avg cu_total 86%, avg retry 13.2%; AP 434 ground-truth 94% busy / hears 48 managed APs on 2.4). Applied: apply-radio cascades ng power low --zone "Floor 4" --apply. RESULT: 14/15 radios -> 6dBm (from ~23), clients retained (118 vs ~119) - no coverage loss. Rollback saved.

INCIDENT: AP 445 lagged (config=low but radio stayed 23dBm). Tried device-control 'provision' (cmd/devmgr force-provision) to nudge -> HTTP 400 AND it took 445 fully OFFLINE; Howard physically power-cycled the switch port to recover. 445 back online (booted to 30dBm; config still low, not converged

  • left alone, harmless).

FIX (Howard's directive - power-cycle the port is the recovery):

  • REMOVED 'provision'/force-provision from device-control.sh (unsafe on U7-Pro; refuses now).
  • ADDED 'poe-cycle <ap-name|mac>' - resolves AP uplink_mac + uplink_remote_port, POSTs cmd/devmgr power-cycle {mac:switch, port_idx} = remote PoE port cycle (the controller equivalent of re-seating the cable). Dry-run validated (445->Switch 4th Floor p7, 434->p5). errorlog.md updated. Coord -> Mike.

EXPECTATION SET: 2.4 cu_total won't plummet (channel is foreign-dominated, ~33k BSSIDs); power-down's win is cell-shrink + reduced SELF-interference + better client SNR, seen in retry%/consolidation over ~10-15 min, and it sets up the Phase-C disables. Next: settle ~15 min, re-snapshot Floor-4 retry% for before/after; decide on 445 (poe-cycle to apply low, or leave). Disables (445/428) still HELD.

Update: 2026-06-16 13:30 PT — Floor-4 pilot RESULT (15-min settle) + pfSense-access review + VPN flap

FLOOR-4 power-down before/after (controller live-stats, via Tailscale — unaffected by the Cascades VPN flap): avg 2.4 retry%: 13.2 -> 9.5 (~28% fewer retransmits); avg cu_total 86 -> 83% (small, channel is foreign-dominated as expected); clients RETAINED (no coverage loss). Big per-AP retry wins: 407 21->0, 4th Floor Atrium 27->0, 434 22->7; a few bounced up (403/428/421) = instantaneous-snapshot jitter as clients re-associate. Caveat: single 15-min snapshot is noisy; direction is positive + safe. Pilot = success. (445 still at 30dBm not converged — left alone.)

PFSENSE-ACCESS REVIEW (Howard asked, given pfSense+UniFi access): KEY GAP — Cascades' gateway/WAN/DHCP/ DNS/firewall is 100% pfSense; gw-audit/gw-control are UniFi-only (num_gw=0) so we've audited NONE of it. We have pfSense creds but NO path from Howard-Home: Cascades 192.168.0.0/24 (pfSense .0.1, NAS .0.120) is shadowed by Howard's HOME UniFi network (also 192.168.0.0/24; .0.1 = home UniFi console cert unifi.local). APs/CS-SERVER (.2.x/.3.x) work via the /22 tunnel route. Likely hidden non-RF contributors to "bad for some users": DHCP pool exhaustion (per-room /28 VLANs + 587 clients), DNS latency, dual-WAN (Cox Fiber+Coax) health/failover, inter-VLAN firewall. RF plan stays valid but don't declare "fixed" on RF alone. THE MISS (skill): gateway verbs are UniFi-REST only; our real topology is UniFi-behind-pfSense (Cascades, office, several clients) = Mike's ROADMAP §E pfSense compat layer = highest-value next build.

/32 ROUTE: added 192.168.0.1/32 via tunnel (192.168.10.1) to reach Cascades pfSense; REMOVED it again when the VPN started flapping (don't tie home gateway IP to an unstable tunnel). pfSense untouched throughout.

VPN FLAP: OpenVPN Connect DCO instability (DCO adapter Disconnected, TAP up, 4 OpenVPNConnect procs) - the same DCO/TAP issue seen earlier. Fix: disable DCO in OpenVPN Connect (force TAP datapath) OR switch to community OpenVPN GUI; also check duplicate-CN (cert connected from 2 devices = connect/disconnect loop).

Update: 13:42 PT (2026-06-16) — pfSense compat layer: started, backed out, aligned to Mike's design

Picked up "start the pfSense compat layer." Began a REST-API-backed pfsense-audit.sh sibling scaffold + a references/pfsense-driver.md design doc. On re-reading the repo, corrected course:

  • What Mike actually built vs proposed. gw-control.sh (commit eb87710) is UniFi gateway control — it calls /proxy/network/api/... (the UniFi controller REST), validated on Grabb's USG-3P. It does NOT touch pfSense. Commit 6998719 is a roadmap entry — ROADMAP §E "pfSense gateway compatibility layer," a design with explicit OPEN DECISIONS, not built code. So Mike designed the pfSense layer (with questions to resolve first); he did not build it.
  • I had pre-empted his open decisions. §E leaves two open: (1) backend = REST API package vs stock SSH easyrule/pfSsh.php; (2) where the abstraction lives = dispatch inside gw-audit.sh/gw-control.sh (keyed on num_gw=0) vs sibling pf-* scripts — Mike leans "inside." My scaffold unilaterally picked REST + a sibling script (the opposite structure).
  • Backed it out. Removed pfsense-audit.sh + pfsense-driver.md (never committed/pushed) rather than fork Mike's design. Released coord lock a34360aa. Messaged Mike (GURU-5070, id e402e783) with the two open decisions and my lean.
  • Howard's decision: "let us try what mike wanted and go that way unless we run into a real roadblock we can't work past." → REST API package backend, dispatched INSIDE gw-audit.sh/gw-control.sh (Mike's option 1 + his structural lean). Build to that next.

Net for this update: no code shipped; design alignment + coordination only. The WiFi/RF work (Floor-4 pilot applied + held, Phase-C disables held, steering/min-rate/channel-plan dry-run-validated) is unchanged from earlier in this log.

Next steps (pfSense layer, per Howard + Mike)

  1. Per-site setup helper: install pfSense-pkg-RESTAPI + create a read-only key + vault at clients/<slug>/pfsense-api (url+apikey). Cascades + ACG office have pfSense web creds vaulted already (clients/cascades-tucson/pfsense-firewall, infrastructure/pfsense-firewall) — need the API key added.
  2. Add a pfSense backend inside gw-audit.sh (dispatch on num_gw=0): WAN/gateway status via /api/v2/status/gateways, system, DHCP pool pressure. Then gw-control.sh verbs (pf-list/disable/ enable/set-ports, fw-disable/enable, block-ips) onto firewall/nat/port_forward + firewall/rule
    • easyrule-equivalent.
  3. Live validation pending a reachable pfSense (stable site VPN; mind the home-LAN .0.x shadow that currently masks Cascades pfSense from Howard-Home). Coordinate the build with Mike (his §E).

Update: 14:30 PT (2026-06-16) — pfSense compat layer built, then put ON HOLD (pfSense too old)

Built the pfSense gateway compatibility layer per Howard's decision (REST API package backend, dispatch INSIDE the existing gateway verbs — Mike's §E lean):

  • NEW scripts/pfsense-backend.sh — pfSense REST API driver (pfSense-pkg-RESTAPI v2, X-API-Key), same verbs as gw-control (audit, pf-list/disable/enable/delete/set-ports, fw-list/disable/enable, block-ips) + a setup helper. Writes --apply-gated, per-object rollback to .claude/tmp/, calls firewall/apply.
  • gw-audit.sh + gw-control.sh auto-dispatch to it when num_gw=0 AND a clients/<slug>/pfsense-api cred is vaulted (or --pfsense <slug>). Minimal diff to Mike's files (he edits them concurrently).
  • SKILL.md [PROPOSED]->[SCAFFOLDED]; ROADMAP §E open decisions marked resolved. Committed 1118594, lock released, Mike notified (msg f0c442cf).
  • Tested: syntax + setup/BLOCKED/no-cred paths + live gw-audit cascades (num_gw=0 -> pfSense hint, gating correct). NOT live-tested: the actual REST calls.

BLOCKER / ON HOLD (Howard): the RESTAPI package is third-party and the Cascades pfSense is too old to install it — it needs a firmware upgrade first. Live work is ON HOLD until the pfSense is upgraded. Tried to vault the API key but none exists yet (package not installed); pfSense host = 192.168.0.1 (web cred already at clients/cascades-tucson/pfsense-firewall). Also note 192.168.0.1 is shadowed from Howard-Home by the home UniFi .0.0/24 overlap, so first live validation must run from/through Cascades.

Filed coord todo (project unifi-wifi): upgrade Cascades pfSense -> install RESTAPI -> mint key -> vault clients/cascades-tucson/pfsense-api -> first live gw-audit cascades. Resume trigger + steps in ROADMAP §E.

Update: 17:02 PT (2026-06-16) — unifi-wifi skill health pass + 2.4 usage/coverage tools + tonight's runbook

Skill health pass

Verified all 19 unifi-wifi scripts. Controller-side read-only (sites, audit-site, switch-audit, live-stats, model-rank, optimize-radios, monitor-run, gw-audit) validated LIVE on Cascades; dry-run apply (apply-radio, apply-wlan, client-control, device-control) correct; AP-side mechanism (SSH + /proc/ui_neighbor) confirmed + full neighbor-collect (74-AP SNR sweep) -> channel-plan end-to-end. Fixes: optimize-radios stray mongo REPL echo (for(k in prof) numeric completion -> void 0); ASCII-cleaned printed output (em-dash/arrow/§ that rendered as mojibake on Windows console) across 10 scripts. Commit e1031ae.

Question chain: "can we see 2.4 device history -> which 2.4 radios to disable"

  1. radio-usage.sh (NEW, commit 659c3a2): per-AP 2.4 client history from ace_stat.stat_daily (~77d) — time-avg concurrent users (<radio>-num_sta_avg) + peak snapshot (ng-num_sta). Finding: only AP 108 (offline) has peak==0 over 77d; every other 2.4 radio takes real bursts at low avg -> confirms POWER-DOWN not disable. NEIGHBOR_JSON adds a safe-to-disable shortlist.
  2. radio-usage.sh --ap mode (commit 80384a7): per-DEVICE — devices on an AP's 2.4 (live stat/sta + recent assoc events) tagged steerable (dual-band / randomized MAC) vs legacy/IoT (fixed OUI). Data finding: wifi_connectivity_event retention on the UOS is ~1 DAY (current+recent, not long history); MAC-randomization heuristic fills the gap.
  3. coverage-thin.sh (NEW, commit f936224 + hint fix + mesh fix): the real deliverable — "which 2.4 radios to disable given over-coverage, by AP proximity." Greedy dominating-set on the 2.4 SNR layer; only ACTIVE-2.4 neighbors count as coverage. Exposed + avoided an optimize-radios flaw (band-AGNOSTIC adjacency counted the ng-DISABLED AP 128 as 'coverer' for 127/229/330/428 -> would be 2.4 holes; flagged to Mike).

Howard caught two hazards (pre-execution review) -> fixes

  • MESH: 2nd Floor Atrium is the wireless-mesh PARENT for CC Bridge + salon (backhaul ch36/5GHz); 206 U7 Pro carries 108. coverage-thin was mesh-blind and had listed 2nd Floor Atrium/CC Bridge/206 as disable targets. FIX: coverage-thin now fetches live uplink topology and auto-excludes wireless-mesh APs + their parents (108, 206 U7 Pro, 2nd Floor Atrium, CC Bridge, salon). resilient plan 34->33.
  • SSID/device pinning: verified NONE — CSC ENT/CSCNet/Guest broadcast on all APs (broadcasting_aps off), and UniFi doesn't lock clients to an AP -> disabling a radio orphans nothing; clients roam. No pinning risk.
  • Also fixed coverage-thin apply HINT: was emitting apply-radio ng disable --zone "<floor>" (would disable a WHOLE floor) -> now per --ap "<name>".

Tonight's runbook (committed 8c048c3)

clients/cascades-tucson/reports/2026-06-16-2.4ghz-remediation-runbook.md. Phases: 0 pre-flight+baseline; 1 power-down all non-mesh 2.4 -> Low (per-zone clean floors, per-AP on mesh-containing Floor 2/misc; re-disable 128 after Floor 1); 2 settle+re-measure (valid stop point); 3 disable 25 mesh-safe low-client radios per floor with gates (F1:7 F2:8 F3:5 F4:5); 4 wrap. Excluded: 7 high-client redundant (power-down instead: 3rd/4th Atrium,318,303,406-408,505,Dining Room); Memcare TV room (memcare hold); companion levers (min-rate/band-steer/ min-RSSI) deferred. Recovery: device-control poe-cycle (never force-provision). Rollback JSONs in .claude/tmp.

Open decision (awaiting Howard)

Scope for tonight: recommended = power-down-all + pilot ONE floor's disables, validate ~15min, expand if clean (vs all 25 at once). And whether I stay live during execution to run the per-floor validation gates.

Note: mesh data is from .claude/tmp/cascades-nbr.json (fresh tonight, gitignored) — regenerate via

neighbor-collect before execution if stale.

Update: 19:28 PT (2026-06-16) — home LAN renumber fixes VPN shadow; pfSense ruled out as WiFi cause; SSH backend

Home-LAN shadow RESOLVED (unblocked Cascades pfSense access)

Diagnosed why the Cascades VPN couldn't reach 192.168.0.x: Howard-Home was 192.168.0.0/24 with a UniFi gateway at 192.168.0.1 (cert unifi.local — NOT pfSense), colliding with Cascades' own 192.168.0.0/24 (pfSense .0.1). OS prefers the directly-connected local /24, so 192.168.0.x never crossed the tunnel (APs on 192.168.2.x/3.x worked). A /32 route can't fix it (.0.1 was Howard's own gateway). Howard renumbered Howard-Home to 10.137.42.0/24 (gw 10.137.42.1). Verified: ping 192.168.0.1 now replies over the tunnel (TTL 64, ~15ms); cert = CN=pfSense-685f277aa6886 / "Netgate pfSense Plus" (the REAL Cascades pfSense); NAS .120 reachable; controller-side (172.16.3.29) + AP reach (192.168.2.79) intact. Memory: howard-home-lan-shadow updated to RESOLVED.

pfSense investigated — NOT a WiFi factor (gateway ruled out)

SSH into Cascades pfSense (admin, shell works directly — no menu gotcha; pfSense Plus 25.07-RELEASE). Findings: DHCP NOT exhausted (0 "no free leases"; WiFi/AP pool 192.168.0.0/22 range .2.2-.3.254 cap ~507 at 270 active ~53%; 199 subnets total, mostly per-unit /28s; active backend ISC, Kea dormant); unbound DNS up; dual-WAN (WAN1 184.191.143.62/30, WAN2 72.211.21.217/27) both full-duplex, no gateway loss/down events; PF states 28-31k/790k; load 0.6; 10-day uptime. Minor: igc3/WAN2 1707 Ierrs+Coll = Intel I225/226 2.5G counter quirk (2.5G full-duplex active, no loss), not a fault. CONCLUSION: gateway/DHCP/DNS/WAN are not bottlenecking WiFi — the 2.4 GHz RF remediation is the sole fix. Added "pfSense health check (2026-06-16)" section to clients/cascades-tucson/reports/2026-06-16-unifi-full-audit.md.

pfSense compat layer: pivot to SSH backend, OFF HOLD (Mike's call)

Mike (relayed by Howard): "we don't need the RESTAPI — VPN + SSH gets the same data and makes changes." The earlier "pfSense too old for RESTAPI" premise was WRONG (it's 25.07, current). So the upgrade/package blocker is moot; layer off hold. Built scripts/pfsense-ssh.sh (audit | dhcp | run ""); cred from clients//pfsense-firewall; system OpenSSH via askpass; validated live on Cascades (the health data above came from it). ROADMAP §E + SKILL.md updated to the SSH decision; REST pfsense-backend.sh left dormant/optional. Closed obsolete coord todo afbe6a22 (upgrade-for-RESTAPI). Remaining = named gated CONTROL verbs over SSH (easyrule block-ips, pf/fw toggles) — Howard: "leave that up to Mike" (his §E). Commits: 58ecc5a (pfSense-ssh + health + off-hold), e42ad8f (memory).

Still open (the actual WiFi win)

Tonight's 2.4 remediation per the runbook (reports/2026-06-16-2.4ghz-remediation-runbook.md): scope (power-down-all + pilot one floor vs all 25) + whether Claude runs validation gates live during execution.

Update: 23:53 PT (2026-06-16) — 2.4 remediation EXECUTED (power-down + partial disables); PAUSED before Floor 2

Howard rebooted the Cascades pfSense, then gave the go to push the 2.4 changes. (Changes go through the UOS controller 172.16.3.29, NOT the pfSense/VPN — reboot didn't block them. Verified fleet healthy post-reboot: 77 adopted / 2 disconnected, unchanged.)

Phase 1 — 2.4 power-down to Low: COMPLETE (65 radios)

Applied auto->low across Floors 1,2,3 + misc (Floor 4 was already done). Floor 3 by-zone (17); Floors 1/2

  • misc PER-AP to skip mesh + offline + 128. Final ng state: low=65, auto=11 (correctly left: mesh parents 2nd Floor Atrium/206 U7 Pro, mesh children CC Bridge/salon, offline 108/108U7 Pro, EXCLUDED Floors 5/6 = 505/517/608/615/622), disabled=1 (128). 0 failures, no AP knocked offline. Initially missed 247 (Floor 2) and 4 Floor-1 stragglers (121/122/109/102) — see Problems; all fixed.

Phase 2 — re-measure (15-min settle): the cumulative WIN

Site-wide 2.4 averages (live-stats, ~74 radios): PRE power-down: cu_total 77% cu_interf 64% retry 12.2% clients 134 POST settle: cu_total 68% cu_interf 53% retry 13.8% clients 111 => cu_interf -11 pts, cu_total -9 pts site-wide (vs only ~4 pts from the Floor-4-only test earlier -> confirms the benefit is cumulative/site-wide). Retry +1.6 is midnight client-mix noise (fewer clients).

Phase 3 — disables: 15 of 23 done, PAUSED before Floor 2

Refreshed coverage-thin (MINCOV=2, mesh-aware) -> conservative set = 23 (low-client <2 avg, >=2 active coverers, non-mesh, non-memcare, Floors 1-4). Disabled (apply-radio ng disable --ap, spaced): Floor 4: 445 442 407 450 Floor 3: 342 317 304 336 Floor 1: 139 115 102 145 127 122 116 REMAINING (Howard said "lets pause"): Floor 2: 240 229 241 209 203 248 204 236 (8 APs). Fleet after disables: 77 adopted / 2 disconnected — disabling 2.4 just drops that radio; AP stays up on 5/6 GHz, every disabled radio's area keeps >=2 active 2.4 coverers. The force-provision hazard does NOT apply to disable (clean radio-off, unlike the 445 force-provision incident).

Other this turn

  • apply-wlan.sh BUG FIXED: wlan_bands token was "6e" but THIS controller stores "6g" (verified live on Guest SSID) -> 6 GHz enablement would have failed. Fixed (5g6g/6g/all). Commit 5ad25d1.
  • Runbook: folded in Phase 5 (5 GHz: 80->40 width on 76 radios + channel plan with DFS decision flagged -- DFS empirically CLEAN here so including clean-DFS gives ~20 ch vs ~5 non-DFS-only) and Phase 6 (6 GHz: ROOT CAUSE 6 GHz empty = production SSID CSCNet not on 6 GHz [bands 2g,5g]; add 6g + bss-transition).
  • Floors 5 & 6 EXCLUDED from the whole plan (Howard).

RESUME / next

  1. Finish Phase 3: disable Floor 2 (240 229 241 209 203 248 204 236), spaced, validate.
  2. Proper site-wide re-measure after settle (compare cu_interf again).
  3. Phase 5 (5 GHz width+channel) + Phase 6 (6 GHz CSCNet) when ready. ROLLBACK: re-enable a radio = apply-radio cascades ng enable --ap "" --apply; restore power = apply-radio cascades ng power auto --zone "Floor X" --apply. (apply-radio rollback JSON is per-call in .claude/tmp, overwritten each call -> for power-down revert just set power auto.)

Friction logged

apply-radio per-AP --apply loop: each call re-logs-in to the controller; rapid succession throttles and the write silently skips (no [ok]). Fix: space calls (sleep ~4) or add multi-AP/one-login support.

Update: 00:30 PT (2026-06-17) — Phase 3 disables COMPLETE (all 23 + 128); daytime re-check scheduled

Finished Floor 2 disables (240 229 241 209 203 248 204 236). Final 2.4 state: disabled=24, low=42, auto=11 (auto = mesh parents/children 2nd Floor Atrium/206/CC Bridge/salon + offline 108/108U7Pro + EXCLUDED Floors 5/6 = 505/517/608/615/622). Fleet 77 adopted / 2 disconnected — no AP offline.

Experienced improvement so far (midnight, low load — directional): active-radio-only site-wide 2.4 cu_interf 64% -> 48% (-16 pts, ~25% rel), cu_total 77% -> 66%. (NOTE: averaging ALL ng radios incl. the 24 disabled-reporting-0 misleadingly showed cu 45%/interf 33% -> always exclude disabled for a fair read.) Hourly time-of-day-controlled (same hr last night): retry 27.8->26.0, sat ~flat. Real client- experience delta needs BUSINESS-HOURS load -> daytime re-check scheduled.

Daytime re-check: CronCreate one-shot 9:33 AM 6/17 (reported session-only — NOT durable) + coord todo (reliable backstop) filed for Howard. Remaining: Phase 5 (5GHz 80->40 + channel) + Phase 6 (6GHz: add 6g to CSCNet + bss-tm). apply-wlan 6e->6g fix already committed (5ad25d1).

Reference in New Issue View Git Blame Copy Permalink