## User - **User:** Howard Enos (howard) - **Machine:** Howard-Home - **Role:** tech ## Session Summary Resumed the Cascades wireless investigation that had stalled on 2026-05-16 (that session was read-only via the cloud API and blocked from per-AP RF data). The day's earlier syncs delivered the unblock: Mike (GURU-5070) vaulted `infrastructure/uos-server-ssh-key` + `clients/cascades-tucson/unifi-ap-ssh` and shipped a purpose-built `unifi-wifi` skill (audit/model-rank/optimize/apply/watch scripts + methodology references). Controller access via the vaulted key was verified, then the full live audit ran against the Cascades site (`685f39068e65331c46ef6dd2`) on the UOS controller (172.16.3.29). The audit confirmed and quantified the 05-16 hypothesis with real controller data: 77 U7-Pro APs, all running 2.4GHz at auto (~full) power, 20MHz. 2.4 airtime (`cu_total`) is 74–94% busy on 75 radios with 61–81% of that being pure interference, serving ~1 client each, at TX-retry rates of 40–65% and single-digit AP satisfaction on the worst (209 sat=1/retr=65%, 139 sat=2/retr=49%, CC Bridge retr=48%). Neighbor-BSSID density is catastrophic on 2.4 (ch6=33,370, ch1=19,274, ch11=16,580). 5GHz is on 80MHz width on 76/77 APs (kills spatial reuse), biased to the busy upper channels (149/157). 6GHz is active on 75 radios but nearly empty of clients. 6 APs have 2.4 min-RSSI OFF (615, 608, 505, 517, 622, salon); 4 are off the 1/6/11 plan on auto (128, 108, 108U7-Pro, salon). Diagnosis of "bad for SOME users": experience splits by band. Clients on 5/6GHz are fine; clients that land or stick on 2.4GHz (legacy phones, medical/IoT, poor band-steerers, or anything held by a min-RSSI-OFF AP from across the building) hit the saturated 2.4 radios with 40–65% retransmits and near-zero satisfaction. Late in the session Howard raised the **military-base / DFS** factor — Cascades is in Tucson near Davis-Monthan AFB (+ TUS airport radar). This reverses the earlier "bias 5GHz toward DFS" recommendation: DFS channels (UNII-2/2e, ch52–144) will see frequent radar-detection events forcing channel-vacate + CAC silence, producing exactly the intermittent per-area dropouts reported. Revised plan uses non-DFS only (UNII-1 36–48 + UNII-3 149–161), which makes 40MHz width and 6GHz steering more important. ## Key Decisions - Used the `unifi-wifi` skill end-to-end (audit-site, model-rank, optimize-radios, apply-radio dry-run) rather than ad-hoc Mongo queries — it encodes the multi-model methodology and coverage-safe model. - **5GHz channel plan revised to AVOID DFS** due to proximity to Davis-Monthan AFB. Non-DFS UNII-1 + UNII-3 only; verify empirically against the controller's radar-detection event history before reconsidering DFS. - No changes applied — writes are intentionally gated until a read-WRITE controller admin is vaulted (`infrastructure/uos-server-network-api-rw`) and `--apply` is passed. Confirmed the apply path works via dry-run (Floor 3: 17 radios auto->low with rollback values captured). - Rollout will be per-zone (one floor at a time) with live before/after validation, never site-wide at once. ## Problems Encountered - **Controller SSH key not on Howard-Home and not in vault (earlier in day).** Tested both vaulted keys (gururmm-server-physical, openclaw-fleet) against root@172.16.3.29 — both denied; OC-5070/OC-Mac unreachable over Tailscale, fleet key denied on OC-Beast. Resolved by coord-requesting Mike to vault the UOS key; he did (`infrastructure/uos-server-ssh-key`), picked up on sync. - **Cloud Site Manager API insufficient (re-confirmed).** Device objects carry no site field and no RF/channel/power/uplink data — only online/offline + firmware. Cannot drive RF tuning. The Mongo-via-SSH path (now available) is required. ## Configuration Changes - None to Cascades infra. Read-only audit + dry-run only. - No repo files edited beyond this session log. ## Credentials & Secrets - **`infrastructure/uos-server-ssh-key`** (vaulted by Mike this day) — root SSH key for the UOS controller 172.16.3.29; used by `uos-mongo.sh` / unifi-wifi scripts. This is the DATA plane (read), not an API write session. - **`clients/cascades-tucson/unifi-ap-ssh`** (vaulted by Mike this day) — device-auth cred for SSHing directly into Cascades APs (used by `watch-ap.sh`; needs site VPN for L3 reach to 192.168.2.x/3.x). - **Needed, not yet created:** `infrastructure/uos-server-network-api-rw` (read-write controller admin) to apply radio changes; `infrastructure/uos-server-network-api` (read-only admin) to wire live-stats Plane 2 validation. ## Infrastructure & Servers - **UOS controller:** 172.16.3.29 (Rocky 9 VM "Unifi" on Jupiter 172.16.3.20); UniFi-OS HTTPS on **11443** (not 8443). Mongo `ace` on 127.0.0.1:27117 inside rootless podman `uosserver`. Cascades site_id `685f39068e65331c46ef6dd2`. - **Cascades wireless:** 77 U7-Pro APs, ~550 clients. Firewall = pfSense 192.168.0.1 (site VPN endpoint; `.ovpn` comes from pfSense OpenVPN Client Export, NOT UniFi). APs on 192.168.2.x/3.x. - **Location/RF:** Tucson, near Davis-Monthan AFB + TUS radar → DFS unreliable. ## Commands & Outputs ```bash bash .claude/scripts/uos-mongo.sh --sites | grep -i casc # 685f39068e65331c46ef6dd2 Cascades (access OK) bash .claude/skills/unifi-wifi/scripts/audit-site.sh cascades # config + neighbor-density + flags bash .claude/skills/unifi-wifi/scripts/model-rank.sh cascades 7 ng bash .claude/skills/unifi-wifi/scripts/optimize-radios.sh cascades 14 ng # power-down 74, disable 0, keep 1 bash .claude/skills/unifi-wifi/scripts/apply-radio.sh cascades ng power low --zone "Floor 3" # DRY-RUN: 17 radios auto->low ``` Key audit output: 2.4 cu_total 74–94% / interf 61–81% / ~1 client; retry 40–65%; ch6=33,370 neighbors; 5GHz 80MHz on 76/77; 6GHz active 75 but empty; min-RSSI OFF on 615/608/505/517/622/salon. ## Pending / Incomplete Tasks - [ ] Vault read-WRITE controller admin `infrastructure/uos-server-network-api-rw` (blocks applying any radio change). Candidate: coord-request Mike. - [ ] Vault read-only `infrastructure/uos-server-network-api` to wire live-stats Plane 2 (before/after cu_total/satisfaction validation). - [ ] Apply Phase A (2.4 power-down to Low) per-zone with live validation, once RW cred exists. - [ ] 5GHz: 80->40MHz width; non-DFS channel plan (UNII-1 36–48 + UNII-3 149–161); 6GHz steering for capable clients. - [ ] Min data rates (kill 1–11Mbps, 2.4 floor 12/24Mbps); set 2.4 min-RSSI -75/-76 on the 6 OFF APs; pin 4 off-plan APs to 1/6/11. - [ ] Pull controller radar-detection event history to empirically confirm DFS unusability. - [ ] Secondary: fix the non-working ".ovpn / Download configuration" — likely pfSense OpenVPN Client Export (192.168.0.1), not UniFi. Needed for watch-ap.sh live validation. - [ ] AP 108 offline — KNOWN, needs a new cable run (per Howard); ignore for now. Also a stale duplicate controller object (108 vs 108U7 Pro) to clean up later. ## Reference Information - Coord message to GURU-5070 re UOS key: id `a4b385ad-4fbb-4097-a066-099622080055`; backstop todo `3bd12a14-2b51-4c11-8f76-3f835b07e8dc` (--user mike). - unifi-wifi skill: `.claude/skills/unifi-wifi/` (methodology.md, data-access.md, interference-model.md). - Prior wireless log: `clients/cascades-tucson/session-logs/2026-05-16-howard-wireless-diagnostic.md`. - UOS system wiki: `wiki/systems/uos-server.md`. --- ## Update: 20:40 PT — RW cred arrived, live (Plane 2) re-look, live-stats.sh accuracy fixes Mike vaulted the RW controller admin (`infrastructure/uos-server-network-api-rw`) within ~30 min of the request, so the live Network API (Plane 2) became available. Re-audited Cascades live and, in doing so, found + fixed accuracy bugs in `unifi-wifi/scripts/live-stats.sh` that had skewed the earlier read. ### live-stats.sh bugs fixed (held a coord lock; messaged Mike e8be889f) 1. `stat/device` output hard-capped at `head -60` -> only ~15 of 77 APs were shown (we'd been judging the whole site from a 15-AP sample). Removed the cap; verified 77/77 now. 2. `satisfaction` was read per-radio (always `-1` on this controller). DEVICE-level satisfaction IS populated -> switched to `d['satisfaction']`. 3. `tx_retries` was the raw cumulative counter (scales with traffic, misleading). Switched to `radio_table_stats.tx_retries_pct` (a true rate). 4. `--clients` was `[:40]` unsorted (hid 90% of 574 clients). Now sorts worst-by-satisfaction and prints signal/noise/retry%/`satisfaction_reason`. 5. RF-neighbor table left as a documented TODO in the header (not a single API field; must be built from the `rogue` BSSID cross-ref vs each AP's `vap_table`). It's what unlocks confident radio DISABLES; until then power-down/channel/width are the safe levers. ### Corrected diagnosis (the data fix changed a conclusion) - Accurate avg retry RATE: **2.4GHz 11.2%** > 5GHz clear 9.0% ~= 5GHz DFS 8.4%. My mid-session claim that "5GHz/DFS is now the #1 problem" was an ARTIFACT of the raw counter + 15-AP sample and is WITHDRAWN. On the rate, DFS is NOT retrying worse than clear channels. - **2.4GHz is the primary pain band** (highest retry; 27 of the 40 worst clients are on 2.4, retry 11-42%, mostly IoT/legacy: Ring cams, robotic cleaner, smart plugs, EPSON printer, Poly phone, handheld scanners, Watch). The original 2.4 power-down/prune plan stands as #1. - **DFS = resilience risk, not throughput killer.** 55/77 5GHz radios on DFS near Davis-Monthan = radar-vacate exposure (client drops), worth moving off DFS for stability, but not urgent for performance. - **6GHz still dead (1 client of 574)** — top untapped clean/non-DFS capacity; steering remains a key opportunity. - **AP-level satisfaction 95-100 across the fleet** — network is healthy on average; pain is in the client tail = consistent with "bad for SOME users." ### Cascades live snapshot (2026-06-15 ~20:30 PT) - 77/77 APs reporting; 574 wireless clients. Band split: na (5GHz) ~87 / ng (2.4) ~21 sampled per pull / 6e ~1. - 2.4 cu_total 69-94% live (saturation confirmed). 5GHz cu_total mostly <40%. - Worst clients: RingStickupCam (sat 60), Galaxy-A32-5G (60), DIRECTV (71), Samsung on ch1 (76, retry 42%). ### Updated next steps - [ ] Build Floor-pilot DRY-RUN: 2.4 power-down to Low on one floor (e.g. Floor 4), validate live cu_total/retry% before+after via the fixed live-stats.sh. Get explicit go before `--apply`. - [ ] Implement the AP-to-AP RF-neighbor table (rogue BSSID x vap_table) to enable safe DISABLEs. - [ ] 6GHz steering plan; 5GHz 80->40MHz + non-DFS channel plan (resilience). - [ ] Coord msgs this update: RW-cred request 6b98282f (+todo cbb355ef); live-stats fix e8be889f. - [ ] pfSense `.ovpn` (Howard handling) — needed for per-AP watch-ap.sh live stream. --- ## Update: 22:05 PT — site VPN up (split-tunnel), direct-AP cross-validation, watch-ap.sh fixed Howard got the Cascades OpenVPN running. The pushed config is **full-tunnel** (pfSense `redirect-gateway`) which killed his local internet + Tailscale (and my tool calls, since they run on his machine). Fixed with a **split-tunnel client profile**: `C:\Users\Howard\Documents\ComputerGuru Connect v2\Files\cascades-splittunnel.ovpn` — copy of the working profile + `pull-filter ignore "redirect-gateway"`, `pull-filter ignore "dhcp-option DNS"`, `route 192.168.0.0 255.255.252.0`, and the VPN creds embedded inline (`` = vault `clients/cascades-tucson/pfsense-openvpn-howard`, user `Howard`). After import+connect: internet + Tailscale + direct AP reach all work simultaneously. ### Rescan + ground-truth verification (Howard: "test until sure there are no issues") - Controller rescan: **77/77 APs**, config identical to prior pull — no drift, scripts stable. - **Cross-validated controller data against AP ground truth** (direct SSH to APs, `mca-dump` + `iw survey`) on TWO APs (132 @192.168.2.34, 335 @192.168.3.46): controller cu_total/clients match the radios' own numbers within polling jitter (AP132 2.4: controller 81% / mca 87% / iw survey 92% busy, -95dBm noise; AP335: 2.4 controller 81% vs mca 79%, 5GHz 10%=10% exact, clients 2=2 exact). **Conclusion: the controller is accurate; the diagnosis stands on solid data.** - New concrete fact: U7-Pro 2.4 radios report `min_txpower 6 / max 23 dBm` → "Low" = 6 dBm (aggressive cell shrink — good for the power-down plan). ### watch-ap.sh fixed (2 bugs found by testing; validated live; Mike notified b0cf24a5) 1. Hard `sshpass` dependency → failed on Windows (no sshpass). Added **SSH_ASKPASS fallback** (temp askpass helper + `SSH_ASKPASS_REQUIRE=force`); uses sshpass if present. Fallback needs `ssh` from PATH = MSYS ssh on Windows (system OpenSSH can't exec a shell askpass, CreateProcessW err 193); Linux/macOS system ssh is fine. 2. Radio list included virtual APs (`wifi0ap0..`) + `mld-wifi0` → duplicate lines AND `sh: malformed ?: operator` (hyphen in `mld-wifi0` broke `${ACT_mld-wifi0:-0}`). Filter now `grep -E '^(wifi[0-9]+|ath[0-9]+)$'`. Validated: clean 3-line/tick (wifi0 ch6 92%, wifi1 8%, wifi2 0%). - Minor (not fixed, interactive use is fine): script's `grep|head` block-buffers when piped non-interactively; would need `stdbuf -oL` to be pipeable. ### Issues found this round (none in the data; environment/VPN only) - **192.168.0.0/24 home-LAN overlap:** Howard's home LAN is 192.168.0.x, so Cascades devices on 192.168.0.x (pfSense `192.168.0.1`, cascadesDS `192.168.0.120`) are **shadowed/unreachable over the VPN** from Howard-Home. APs on 192.168.2.x/3.x are unaffected. Use Tailscale for controller; for pfSense/NAS over VPN would need a home renumber or a more specific route trick. - **OpenVPN Connect DCO↔TAP failover** intermittently drops the 192.168.2.x route mid-session (caused two transient watch-ap timeouts). Community **OpenVPN GUI** is the more stable client if it recurs. ### State / next - live-stats.sh + watch-ap.sh both fixed + validated; controller data verified accurate. - NEXT (per Howard): one more clean retest pass, then build the **AP-to-AP RF-neighbor table** (now feasible via direct AP `iw scan`/scan_radio_table) to unlock safe radio DISABLEs; then the Floor-4 2.4 power-down pilot with live before/after validation. - Coord this update: watch-ap.sh fix b0cf24a5. --- ## Update: 23:03 PT — RF-neighbor table DEFINITIVELY unobtainable; survey-dump is the usable substitute Exhaustively investigated whether an AP-to-AP RF-neighbor table (which of OUR managed APs each AP hears + RSSI) is obtainable, to enable data-driven radio DISABLE decisions. Cross-checked against two rounds of UniFi's own AI guidance. **Conclusion: NOT obtainable on this deployment by any safe/supported method.** Stop chasing it. ### Why it's impossible here (all paths tested) - `rogue` collection AND Network API `stat/rogueap` (7,354 records): both FILTER OUT our managed APs — only foreign BSSIDs (our OUI 0c:ea:14 = 2 hits, CSCNet = 0, is_ubnt = 2). By design. - AP `iw dev wifiX scan`: **firmware-blocked — "Operation not permitted (-1)" even as uid=0(root)**, on all radios incl. an idle 6GHz radio. UniFi locks raw scanning on operating radios. - `iw scan dump` (cached): empty. `device.scan_radio_table`: empty []. No spectrum/neighbor collection exists. `hostapd_cli show_neighbor` (802.11k): FAIL on every VAP. - `mca-cli-op`: interactive CLI (bare invocation spews prompts — feed `` — same neighbors WITH SNR (per scan-vap, per band) Discovered via `strings $(which mcad) | grep neighbor` → `/proc/ui_neighbor/ess_ap_list` + `load_ssid_neighbor_cache`. `/proc/ui_neighbor` exists on every UniFi AP → fleet-generic. ### NEW: .claude/skills/unifi-wifi/scripts/neighbor-collect.sh (built + validated, Mike notified 3dbe2437) Logs into controller for the mac/bssid->AP-name map, SSHes each online AP (sshpass OR SSH_ASKPASS fallback), reads `/proc/ui_neighbor`, maps neighbors to AP names, emits the AP-to-AP SNR adjacency matrix + a redundancy summary (APs heard by >=2 neighbors at SNR>=N = power-down/disable candidates). Usage: `neighbor-collect.sh [ap-ssh-vault-path] [snr_min]`. Run FOREGROUND. **Validated on Cascades: 74/74 APs reporting.** Physically sensible (115<->116 SNR 63, 121<->221 SNR 65 = adjacent). Redundancy: **73/74 APs have >=2 strong (SNR>=20) 2.4GHz neighbors**, 72/74 on 5GHz, 65/74 on 6GHz → nearly every AP is coverage-redundant on 2.4 → aggressive 2.4 pruning is data-supported, and individual radio DISABLES are now decidable from data. ### Three bugs found+fixed building it (apply to any loop-over-AP script) 1. Python wrote the AP-list temp file in Windows TEXT mode -> CRLF -> bash `read` left `\r` on the IP -> ssh "hostname contains invalid characters" on EVERY AP (the "reachable 0" symptom). Fix: `open(...,newline='\n')` + `ip="${ip%$'\r'}"` strip in the read loop. 2. Must run FOREGROUND — a fully detached background process can't spawn the SSH_ASKPASS helper (both background runs got reachable 0; foreground after the CRLF fix got 74). 3. `ssh` in a `while read` loop needs `/unifi-ap-ssh` vaulted (overridable via arg). Out of current scope (it's unifi-WIFI): switch/PoE, gateway/WAN/firewall, adoption — reachable via the same access layer (uos-mongo/controller API/device SSH) but not yet wrapped as scripts. ### Pending follow-ups - [ ] Wire neighbor-collect redundancy output into optimize-radios.sh (data-backed disables). - [ ] Fold survey-dump (per-channel busy/noise) + dmesg DFS-radar recipes into the skill as reusable collectors (the two ad-hoc datasets from this session). - [ ] Per-client AP device-auth creds for other clients when extending beyond Cascades. - [ ] Floor-4 2.4 power-down pilot (still the next live change; nothing applied yet). - Coord this update: neighbor-collect.sh announce 3dbe2437. --- ## Update: 23:50 PT — gap 1 + gap 2 closed: survey-collect + dfs-check built; SKILL marked wifi-working / switches-WIP GAP 1 (fold ad-hoc recipes into reusable collectors) — DONE, both validated on Cascades: - **survey-collect.sh** ` [ap-ssh-vault-path]` — per-AP per-channel busy%/noise from `iw survey dump` -> in-use busy% + cleanest non-DFS channels per band. 72 APs reported. Result: 2.4 saturated 94-97% on every channel; 5GHz cleanest non-DFS = ch36/48/153/161; 6GHz ~0%. - **dfs-check.sh** ` [ap-ssh-vault-path]` — per-AP `dmesg` radar history, PRECISE patterns (excludes the "cached ifindex" false-positive). Result: **55/75 APs on DFS, ZERO real radar events fleet-wide** -> DFS empirically low-risk at Cascades despite Davis-Monthan proximity (confirms the retry-rate finding; the DFS concern was theoretical). GAP 2 (scope) — DONE via SKILL.md Status block: WiFi monitoring+tuning = **WORKING** for every UOS site we monitor (audit-site, live-stats, model-rank, optimize-radios, survey-collect, dfs-check, neighbor-collect, watch-ap, apply-radio — all site-parameterized). Switches/PoE, gateway/WAN/ firewall, adoption = **WIP** (access layer reaches them; need dedicated collectors). Per-client: vault each client's own `clients//unifi-ap-ssh`, pass as the script arg. CROSS-PLATFORM LESSON (baked into both new scripts): pass temp paths to python via ARGV — MSYS translates POSIX->Windows for python.exe; `$TMP` embedded in a `python -c` string is NOT translated and fails ("No such file or directory") on Windows. Use `python - "$TMP/x" <<'PY' ... sys.argv[1]`. Coord: collectors+status announce c3ccaa07. Next: wire neighbor-collect redundancy into optimize-radios.sh; Floor-4 2.4 power-down pilot (still nothing applied to live radios). --- ## Update: 00:05 PT (06-16) — wired neighbor SNR matrix into optimize-radios.sh (disables now data-backed) Closed the last follow-up: the AP-to-AP SNR matrix now DRIVES optimize-radios.sh disable decisions. - neighbor-collect.sh: `NBR_JSON=` env now also emits machine-readable adjacency `{ap:{band:{nbr:snr}}}` (74 APs, ~21KB). - optimize-radios.sh: `NEIGHBOR_JSON=` env now uses the measured bidirectional SNR overlap (A hears B AND B hears A, SNR>=NBR_SNR_MIN default 20, any band) as the coverage-adjacency source, replacing the sparse roam graph. Roam graph remains the fallback when NEIGHBOR_JSON unset. Workflow: NBR_JSON=.claude/tmp/cascades-nbr.json bash .../neighbor-collect.sh cascades NEIGHBOR_JSON=.claude/tmp/cascades-nbr.json bash .../optimize-radios.sh cascades 14 ng VALIDATED on Cascades 2.4GHz: roam-graph -> POWER-DOWN 74 / DISABLE 0; neighbor-SNR -> POWER-DOWN 65 / **DISABLE 9** / KEEP 1, est. 621% interference-airtime removed, each disable listing its covering neighbors (127->128; 229->128; 248->348; 330->128; 445->347/348/247; 428->128; 622->505/615/517; Kitchen->Memcare TV room; salon->128). The disable capability we concluded was impossible is now data-backed for any site with a neighbor-collect run. IMPL NOTE (bug fixed): injecting the full matrix as a JS object literal CRASHED the old mongo shell (SpiderMonkey GC-during-compile on a ~21KB literal). Fix: precompute bidirectional strong NAME-pairs in python, inject a compact ';'-joined flat STRING, parse in JS. (Pattern for any large data -> mongo-shell injection: flat string, not object literal.) Still gated: apply-radio.sh has NO disable action (power/channel/width only); disables remain a reviewed, per-zone, live-validated MANUAL step. Coord: wire announce 68cae757. Remaining open: Floor-4 2.4 power-down pilot (still nothing applied to live radios). --- ## Update: 2026-06-16 00:03 PT — skill-build focus: apply-radio generalized + write path validated + ROADMAP Per Howard: pause the Cascades production fix; build the unifi-wifi skill into a fully working tool for any client. Used Cascades **0-client radios as the safe sandbox** (6 GHz is ~0-client fleetwide; no whole-AP had 0 clients). ADDED references/ROADMAP.md — full A/B/C/D plan: (A) apply side, (B) multi-client enablement, (C) switches/gateway WIP, (D) robustness. Cross-platform gotchas captured for future scripts. GENERALIZED apply-radio.sh from power-only to 4 radio_table actions (same gated REST mechanism, field-generic via a compact FIELDS JSON shared by the dry-run preview and the apply python; rollback auto-saved): power low|medium|high|auto| -> tx_power_mode (+tx_power) width 20|40|80|160 -> ht channel |auto -> channel minrssi off|on|- -> min_rssi_enabled (+min_rssi) **WRITE PATH VALIDATED** end-to-end (first real --apply): on Floor 6 0-client 6e radios (608, 622) applied ht 160->80, controller confirmed ht=80, then reverted to 160 (state restored). Dry-run previews for width/minrssi verified correct (incl. 615 U6-Pro correctly has no 6e radio). radio_table field names confirmed: channel, ht (width), tx_power_mode/tx_power, min_rssi + min_rssi_enabled. NO radio enable/disable field -> DISABLE not implemented (ROADMAP A; discover via UI-toggle + device-JSON diff). min-data-rate/band-steering are wlanconf-level -> separate path. SKILL.md apply section updated with all 4 actions + the validated-write-path note. Coord: 6aac1298. Next per roadmap: A (disable mechanism, wlanconf knobs, channel-plan apply) then B (per-client creds/VPN). --- ## Update: 2026-06-16 00:24 PT — radio DISABLE mechanism found; apply-radio disable/enable + --ap (validated) Howard toggled 6GHz off on AP 622 in the UI; diffed the device-JSON before/after. **Disable mechanism: `radio_table[].tx_power_mode = "disabled"`** (radio stays in table, channel/ht untouched). Re-enable = `tx_power_mode "auto"`. Same field as power. (Other diffs were timestamps/cfgversion + JSON int/str noise.) apply-radio.sh now supports actions: power | width | channel | minrssi | **disable | enable**, plus a new **--ap ""** single-AP filter (correct scope for disables). Arg parsing reworked so disable/enable take no value. VALIDATED full enable->disable->enable cycle on 622's 0-client 6GHz via --apply; 622 restored to ON (auto). Workflow: optimize-radios (NEIGHBOR_JSON) disable list -> `apply-radio disable --ap --apply` one AP at a time, watch-ap validated. SKILL.md + ROADMAP.md updated (disable now done). apply-side now covers all radio_table knobs; only wlanconf-level (min-data-rate, band-steering) remains -> future apply-wlan.sh. Coord: 6aac1298-> this: msg sent. --- ## Update: 2026-06-16 00:40 PT — multi-client enablement + WLAN-level apply (wlanconf); both validated MULTI-CLIENT (B): NEW scripts/sites.sh lists all ~49 UOS sites (APs/SW/GW) + per-client AP-cred readiness — the entry point for other clients. Controller-side scripts validated on **other clients** (Glabman 6 APs, Sonoran Glass 5 APs) with zero setup. AP-side scripts now print the exact vault command when a client's clients//unifi-ap-ssh is missing (controller-side still works). SKILL.md Multi-client section added. (Per-client AP creds vaulted as clients come online.) WLAN-LEVEL APPLY (A2): NEW scripts/apply-wlan.sh (wlanconf, gated like apply-radio, --wlan filter): minrate ng|na auto|off| (kill 1-11Mbps: minrate ng 12) steer on|off (5GHz roaming_assistant) GOTCHA found+fixed: manual min rate only applies when minrate_setting_preference=manual (else controller auto-manages + silently reverts — first PUT looked [ok] but value stayed 1000). Script now sets the preference; 'minrate ... auto' restores controller management. WRITE PATH VALIDATED on a 0-client WLAN (Green Valley Computer Club): minrate ng 12 -> persisted (pref=manual, 12000) -> restored to auto. apply-radio (radio_table: power all-states/width/channel/minrssi/disable/enable + --ap/--zone) and apply-wlan (wlanconf: minrate/steer) now cover the WiFi apply surface, all gated + rollback + validated on 0-client sandboxes. ROADMAP updated. Coord: this msg. Remaining: classic band-steering field, per-client creds/VPN, switches/gateway collectors (C). --- ## Update: 2026-06-16 00:58 PT — band-steering field FOUND + wired into apply-wlan (validated) Modern UniFi (Network 8.x) has NO classic 'bandsteering_mode'. Located its replacements by inspecting wlanconf + the setting collection (no UI toggle needed this time): no2ghz_oui -> the band-steering toggle (keep 5GHz-capable OUIs off 2.4) wlan_bands (["2g","5g"]) -> per-WLAN band membership (set ["5g"]/["5g","6e"] to force clients up) bss_transition -> 802.11v BSS-TM roaming_assistant_na_* -> 5GHz sticky-client kicker fast_roaming_enabled -> 802.11r (left out — risky for IoT/medical) Wired into apply-wlan.sh as actions: bandsteer (no2ghz_oui) | bands (wlan_bands) | steer (roaming_assistant) | bsstm (bss_transition), alongside minrate. Validated on the 0-client Green Valley WLAN: bandsteer on->verify->off, bsstm toggle, bands 5g dry-run; restored to original (bss_transition=true). Also surfaced setting.radio_ai (Channel AI: channels_na includes DFS, ht_modes_na=20/40, enabled=false) + setting.connectivity — possible future apply targets (pin the channel plan). WiFi APPLY SURFACE NOW COMPLETE: apply-radio (power all-states/width/channel/minrssi/disable/enable, --zone/--ap) + apply-wlan (minrate/bandsteer/bands/steer/bsstm, --wlan), all gated + rollback + validated on 0-client sandboxes. Coord: this msg. Remaining (ROADMAP C): switch/PoE + gateway collectors. --- ## Update: 2026-06-16 01:09 PT — WiFi control surface complete: full wlanconf knobs + device-lock + client controls (a) DEVICE LOCK / CONTROL: - apply-wlan aps -> broadcasting_aps (resolves AP NAMES->MACs; restricts SSID to APs = the "lock device to AP" lever, since UniFi has NO native per-client AP pin). Validated dry-run: aps 622 --wlan CSCNet -> broadcasting_aps=[0c:ea:14:3e:65:a2]. - apply-wlan macfilter off|allow|deny -> mac_filter_*. Dry-run validated. - NEW client-control.sh: block|unblock|kick a MAC via cmd/stamgr (gated). Validated block+unblock on dummy 00:00:00:00:00:01 (rc:ok), cleaned up. (b) REMAINING wlanconf TUNING KNOBS added to apply-wlan: wlan(enable/disable), dtim (dtim_mode=custom gate), mcast(mcastenhance), bcfilter, rrm(802.11k), ftroam(802.11r, warns), isolation(l2_isolation), hidessid. dtim apply-validated on 0-client Green Valley (mode default->custom, reverted). Others via the proven gated REST path. apply-wlan restructured: SITE resolved before the action case (so aps can map names->macs). WiFi APPLY SURFACE COMPLETE: apply-radio (radio_table) + apply-wlan (wlanconf, 15 actions) + client-control (cmd/stamgr), all gated + rollback + validated on 0-client sandboxes / dummy MAC. SKILL.md + ROADMAP updated. Coord: this msg. NEXT: switches/PoE collector (ROADMAP C). --- ## Update: 2026-06-16 01:13 PT — switches: switch-audit.sh shipped (first non-WiFi collector, ROADMAP C) NEW scripts/switch-audit.sh [--all-ports] (controller-side, read-only, any site): per switch -> PoE budget used/total, ports up/total, FLAGS: UNDERSPEED (gig-capable port linked 10/100 = FastEthernet uplink problem), DROPS (rate-based tx_dropped/tx_packets>2%), ERRORS (>1000), POE-FAULT (poe_enable & !poe_good), POE-BUDGET (>85%), + OFFLINE switches. port_table fields: speed/speed_caps/poe_*/rx_tx_errors. VALIDATED on Cascades (12 switches): found ~25 ports linked at 100M but gig-capable (systematic cabling/NIC issue = the long-suspected FastEthernet uplink problem, now fleet-visible), offline switches (2nd Floor #2, 4th Floor #2, USW Pro Max 16), PoE budgets healthy, one real 4.1% tx-drop port (p38 on 1st Floor USW). Refined DROPS to rate-based (cumulative counters were noisy). SKILL coverage now: WiFi (monitor+tune+full apply incl. device lock/client controls) + switch/PoE audit, all controller-side multi-client + AP-side where cred vaulted. ROADMAP remaining: gateway/WAN/firewall + adoption collectors. Coord: this msg. --- ## Update: 2026-06-16 01:20 PT — gateway/WAN + site-health collector (gw-audit.sh); ROADMAP C largely done NEW scripts/gw-audit.sh (controller-side, any site, read-only) via stat/health + gateway device: WAN status/IP/uplink/duplex + gw CPU/mem/uptime; internet(www) status/latency/drops/last-speedtest; health rollup (APs/switches/gw adopted vs disconnected vs pending, client user/guest/iot, firmware- upgradable); flags WAN/internet not-ok, latency>80/drops>5, WAN down, disconnected/pending, gw cpu>85/ mem>90. Third-party-firewall sites (num_gw=0) handled. Fixed: gw_name fallback to device name; drops threshold >5 (was >0, too sensitive). Validated: Sonoran Glass (USG 3P [UGW3]) -> all healthy (cpu2%/mem21%, latency 14-24ms); Cascades (pfSense) -> "no UniFi gateway" + correctly caught 2 disconnected APs + 3 disconnected switches. SKILL now end-to-end for any client: WiFi (monitor + tune + full apply incl device-lock/client-control), switch/PoE audit, gateway/WAN/site-health audit, sites.sh discovery. ROADMAP remaining: deeper firewall/VPN policy + adoption REMEDIATION (adopt/restart), channel-plan apply, per-client AP creds/VPN. Coord: this msg. --- ## Update: 2026-06-16 07:26 PT — adoption remediation (device-control.sh) + channel-plan apply (channel-plan.sh) NEW device-control.sh adopt|restart|provision|locate|unlocate|upgrade [--apply] (cmd/devmgr, gated, controller-side) — remediates what gw-audit flags. Validated locate+unlocate on AP 622 (rc:ok). NEW channel-plan.sh ng|na [--apply] (NEIGHBOR_JSON + SURVEY_JSON): ng: graph-color 1/6/11 via the neighbor matrix (strong neighbors differ). na: per-AP lowest-cost NON-DFS channel = measured busy% (survey) + neighbor-collision penalty. Reports co-channel pairs before/after; gated apply per-AP rest/device + rollback. DRY-RUN validated on Cascades: ng 92->35 co-channel pairs (51 APs change); na 20->0 + ALL moved off DFS onto non-DFS (radar-safe) (71 APs). Not applied (production paused); PUT mechanism proven. Added SURVEY_JSON= emit to survey-collect.sh ({ap:{band:{channel:busy%}}}). SKILL now: WiFi (monitor+tune+full apply+device-lock+client/device control+channel-plan) + switch/PoE audit + gateway/WAN/site-health + multi-client. ROADMAP nearly clear (deeper firewall/VPN policy + per-client AP creds/VPN remain). Coord: this msg. --- ## Update: 2026-06-16 07:44 PT — robustness (ROADMAP D): monitor-run.sh + per-AP retry; gw-audit pfSense fix Synced first to pick up Mike's gw-control.sh (eb87710, firewall/port-forward router actions — the "deeper firewall/VPN policy" item; no dup with my robustness work). NEW scripts/monitor-run.sh — cron-friendly controller-side read-only fleet health digest: per site -> gateway/WAN flags + switch/PoE flags + WiFi config flag count. Validated Sonoran (healthy) + Cascades (flags 2 disc APs / 3 disc switches / underspeed / firmware). Cron 'all' nightly. VPN-flap resilience: neighbor-collect / survey-collect / dfs-check now RETRY per AP (3x, capture-to-var so a failed attempt never appends partial data; dfs-check distinguishes UNREACHABLE vs no-events). Validated neighbor-collect end-to-end (reachable 74/74, redundancy 73/74, JSON 74 APs - identical). Fix: gw-audit no longer false-flags internet status=unknown on third-party-firewall sites (gated on num_gw). SKILL.md + ROADMAP updated (D items done). Skill is feature-complete for monitoring+tuning+apply across WiFi/switch/gateway, multi-client, with scheduling + flap resilience. Remaining: per-client AP creds/VPN, read-only cred (Mike to create UI admin), gateway VPN-server/DHCP-DNS (Mike). Coord: this msg. --- ## Update: 2026-06-16 13:12 PT — FIRST PRODUCTION CHANGE: Floor-4 2.4 power-down pilot + 445 outage/recovery learning Baseline captured (Floor-4 2.4: 15 radios, avg cu_total 86%, avg retry 13.2%; AP 434 ground-truth 94% busy / hears 48 managed APs on 2.4). Applied: `apply-radio cascades ng power low --zone "Floor 4" --apply`. RESULT: 14/15 radios -> 6dBm (from ~23), clients retained (118 vs ~119) - no coverage loss. Rollback saved. INCIDENT: AP 445 lagged (config=low but radio stayed 23dBm). Tried device-control 'provision' (cmd/devmgr force-provision) to nudge -> HTTP 400 AND it took 445 fully OFFLINE; Howard physically power-cycled the switch port to recover. 445 back online (booted to 30dBm; config still low, not converged - left alone, harmless). FIX (Howard's directive - power-cycle the port is the recovery): - REMOVED 'provision'/force-provision from device-control.sh (unsafe on U7-Pro; refuses now). - ADDED 'poe-cycle ' - resolves AP uplink_mac + uplink_remote_port, POSTs cmd/devmgr power-cycle {mac:switch, port_idx} = remote PoE port cycle (the controller equivalent of re-seating the cable). Dry-run validated (445->Switch 4th Floor p7, 434->p5). errorlog.md updated. Coord -> Mike. EXPECTATION SET: 2.4 cu_total won't plummet (channel is foreign-dominated, ~33k BSSIDs); power-down's win is cell-shrink + reduced SELF-interference + better client SNR, seen in retry%/consolidation over ~10-15 min, and it sets up the Phase-C disables. Next: settle ~15 min, re-snapshot Floor-4 retry% for before/after; decide on 445 (poe-cycle to apply low, or leave). Disables (445/428) still HELD. --- ## Update: 2026-06-16 13:30 PT — Floor-4 pilot RESULT (15-min settle) + pfSense-access review + VPN flap FLOOR-4 power-down before/after (controller live-stats, via Tailscale — unaffected by the Cascades VPN flap): avg 2.4 retry%: 13.2 -> 9.5 (~28% fewer retransmits); avg cu_total 86 -> 83% (small, channel is foreign-dominated as expected); clients RETAINED (no coverage loss). Big per-AP retry wins: 407 21->0, 4th Floor Atrium 27->0, 434 22->7; a few bounced up (403/428/421) = instantaneous-snapshot jitter as clients re-associate. Caveat: single 15-min snapshot is noisy; direction is positive + safe. Pilot = success. (445 still at 30dBm not converged — left alone.) PFSENSE-ACCESS REVIEW (Howard asked, given pfSense+UniFi access): KEY GAP — Cascades' gateway/WAN/DHCP/ DNS/firewall is 100% pfSense; gw-audit/gw-control are UniFi-only (num_gw=0) so we've audited NONE of it. We have pfSense creds but NO path from Howard-Home: Cascades 192.168.0.0/24 (pfSense .0.1, NAS .0.120) is shadowed by Howard's HOME UniFi network (also 192.168.0.0/24; .0.1 = home UniFi console cert unifi.local). APs/CS-SERVER (.2.x/.3.x) work via the /22 tunnel route. Likely hidden non-RF contributors to "bad for some users": DHCP pool exhaustion (per-room /28 VLANs + 587 clients), DNS latency, dual-WAN (Cox Fiber+Coax) health/failover, inter-VLAN firewall. RF plan stays valid but don't declare "fixed" on RF alone. THE MISS (skill): gateway verbs are UniFi-REST only; our real topology is UniFi-behind-pfSense (Cascades, office, several clients) = Mike's ROADMAP §E pfSense compat layer = highest-value next build. /32 ROUTE: added 192.168.0.1/32 via tunnel (192.168.10.1) to reach Cascades pfSense; REMOVED it again when the VPN started flapping (don't tie home gateway IP to an unstable tunnel). pfSense untouched throughout. VPN FLAP: OpenVPN Connect DCO instability (DCO adapter Disconnected, TAP up, 4 OpenVPNConnect procs) - the same DCO/TAP issue seen earlier. Fix: disable DCO in OpenVPN Connect (force TAP datapath) OR switch to community OpenVPN GUI; also check duplicate-CN (cert connected from 2 devices = connect/disconnect loop). ## Update: 13:42 PT (2026-06-16) — pfSense compat layer: started, backed out, aligned to Mike's design Picked up "start the pfSense compat layer." Began a REST-API-backed `pfsense-audit.sh` **sibling** scaffold + a `references/pfsense-driver.md` design doc. On re-reading the repo, corrected course: - **What Mike actually built vs proposed.** `gw-control.sh` (commit `eb87710`) is **UniFi** gateway control — it calls `/proxy/network/api/...` (the UniFi controller REST), validated on Grabb's **USG-3P**. It does NOT touch pfSense. Commit `6998719` is a **roadmap entry** — ROADMAP §E "pfSense gateway compatibility layer," a design with explicit **OPEN DECISIONS**, not built code. So Mike *designed* the pfSense layer (with questions to resolve first); he did not build it. - **I had pre-empted his open decisions.** §E leaves two open: (1) backend = REST API package vs stock SSH `easyrule`/`pfSsh.php`; (2) where the abstraction lives = dispatch **inside** `gw-audit.sh`/`gw-control.sh` (keyed on `num_gw=0`) vs sibling `pf-*` scripts — Mike leans "inside." My scaffold unilaterally picked REST + a sibling script (the opposite structure). - **Backed it out.** Removed `pfsense-audit.sh` + `pfsense-driver.md` (never committed/pushed) rather than fork Mike's design. Released coord lock `a34360aa`. Messaged Mike (GURU-5070, id `e402e783`) with the two open decisions and my lean. - **Howard's decision:** "let us try what mike wanted and go that way unless we run into a real roadblock we can't work past." → **REST API package backend, dispatched INSIDE `gw-audit.sh`/`gw-control.sh`** (Mike's option 1 + his structural lean). Build to that next. **Net for this update:** no code shipped; design alignment + coordination only. The WiFi/RF work (Floor-4 pilot applied + held, Phase-C disables held, steering/min-rate/channel-plan dry-run-validated) is unchanged from earlier in this log. ### Next steps (pfSense layer, per Howard + Mike) 1. Per-site setup helper: install `pfSense-pkg-RESTAPI` + create a read-only key + vault at `clients//pfsense-api` (url+apikey). Cascades + ACG office have pfSense web creds vaulted already (`clients/cascades-tucson/pfsense-firewall`, `infrastructure/pfsense-firewall`) — need the API key added. 2. Add a pfSense backend **inside** `gw-audit.sh` (dispatch on `num_gw=0`): WAN/gateway status via `/api/v2/status/gateways`, system, DHCP pool pressure. Then `gw-control.sh` verbs (pf-list/disable/ enable/set-ports, fw-disable/enable, block-ips) onto `firewall/nat/port_forward` + `firewall/rule` + `easyrule`-equivalent. 3. Live validation pending a reachable pfSense (stable site VPN; mind the home-LAN .0.x shadow that currently masks Cascades pfSense from Howard-Home). Coordinate the build with Mike (his §E). ## Update: 14:30 PT (2026-06-16) — pfSense compat layer built, then put ON HOLD (pfSense too old) Built the pfSense gateway compatibility layer per Howard's decision (REST API package backend, dispatch INSIDE the existing gateway verbs — Mike's §E lean): - NEW `scripts/pfsense-backend.sh` — pfSense REST API driver (`pfSense-pkg-RESTAPI` v2, `X-API-Key`), same verbs as gw-control (audit, pf-list/disable/enable/delete/set-ports, fw-list/disable/enable, block-ips) + a `setup` helper. Writes `--apply`-gated, per-object rollback to `.claude/tmp/`, calls `firewall/apply`. - `gw-audit.sh` + `gw-control.sh` auto-dispatch to it when `num_gw=0` AND a `clients//pfsense-api` cred is vaulted (or `--pfsense `). Minimal diff to Mike's files (he edits them concurrently). - SKILL.md `[PROPOSED]`->`[SCAFFOLDED]`; ROADMAP §E open decisions marked resolved. Committed `1118594`, lock released, Mike notified (msg `f0c442cf`). - Tested: syntax + setup/BLOCKED/no-cred paths + live `gw-audit cascades` (num_gw=0 -> pfSense hint, gating correct). NOT live-tested: the actual REST calls. **BLOCKER / ON HOLD (Howard):** the RESTAPI package is third-party and the **Cascades pfSense is too old to install it** — it needs a **firmware upgrade first**. Live work is ON HOLD until the pfSense is upgraded. Tried to vault the API key but none exists yet (package not installed); pfSense host = `192.168.0.1` (web cred already at `clients/cascades-tucson/pfsense-firewall`). Also note `192.168.0.1` is shadowed from Howard-Home by the home UniFi `.0.0/24` overlap, so first live validation must run from/through Cascades. Filed coord todo (project unifi-wifi): upgrade Cascades pfSense -> install RESTAPI -> mint key -> vault `clients/cascades-tucson/pfsense-api` -> first live `gw-audit cascades`. Resume trigger + steps in ROADMAP §E. ## Update: 17:02 PT (2026-06-16) — unifi-wifi skill health pass + 2.4 usage/coverage tools + tonight's runbook ### Skill health pass Verified all 19 unifi-wifi scripts. Controller-side read-only (sites, audit-site, switch-audit, live-stats, model-rank, optimize-radios, monitor-run, gw-audit) validated LIVE on Cascades; dry-run apply (apply-radio, apply-wlan, client-control, device-control) correct; AP-side mechanism (SSH + /proc/ui_neighbor) confirmed + full neighbor-collect (74-AP SNR sweep) -> channel-plan end-to-end. Fixes: optimize-radios stray mongo REPL echo (`for(k in prof)` numeric completion -> `void 0`); ASCII-cleaned printed output (em-dash/arrow/§ that rendered as mojibake on Windows console) across 10 scripts. Commit e1031ae. ### Question chain: "can we see 2.4 device history -> which 2.4 radios to disable" 1. radio-usage.sh (NEW, commit 659c3a2): per-AP 2.4 client history from ace_stat.stat_daily (~77d) — time-avg concurrent users (`-num_sta_avg`) + peak snapshot (`ng-num_sta`). Finding: only AP 108 (offline) has peak==0 over 77d; every other 2.4 radio takes real bursts at low avg -> confirms POWER-DOWN not disable. NEIGHBOR_JSON adds a safe-to-disable shortlist. 2. radio-usage.sh --ap mode (commit 80384a7): per-DEVICE — devices on an AP's 2.4 (live stat/sta + recent assoc events) tagged steerable (dual-band / randomized MAC) vs legacy/IoT (fixed OUI). Data finding: wifi_connectivity_event retention on the UOS is ~1 DAY (current+recent, not long history); MAC-randomization heuristic fills the gap. 3. coverage-thin.sh (NEW, commit f936224 + hint fix + mesh fix): the real deliverable — "which 2.4 radios to disable given over-coverage, by AP proximity." Greedy dominating-set on the 2.4 SNR layer; only ACTIVE-2.4 neighbors count as coverage. Exposed + avoided an optimize-radios flaw (band-AGNOSTIC adjacency counted the ng-DISABLED AP 128 as 'coverer' for 127/229/330/428 -> would be 2.4 holes; flagged to Mike). ### Howard caught two hazards (pre-execution review) -> fixes - MESH: 2nd Floor Atrium is the wireless-mesh PARENT for CC Bridge + salon (backhaul ch36/5GHz); 206 U7 Pro carries 108. coverage-thin was mesh-blind and had listed 2nd Floor Atrium/CC Bridge/206 as disable targets. FIX: coverage-thin now fetches live uplink topology and auto-excludes wireless-mesh APs + their parents (108, 206 U7 Pro, 2nd Floor Atrium, CC Bridge, salon). resilient plan 34->33. - SSID/device pinning: verified NONE — CSC ENT/CSCNet/Guest broadcast on all APs (broadcasting_aps off), and UniFi doesn't lock clients to an AP -> disabling a radio orphans nothing; clients roam. No pinning risk. - Also fixed coverage-thin apply HINT: was emitting `apply-radio ng disable --zone ""` (would disable a WHOLE floor) -> now per `--ap ""`. ### Tonight's runbook (committed 8c048c3) clients/cascades-tucson/reports/2026-06-16-2.4ghz-remediation-runbook.md. Phases: 0 pre-flight+baseline; 1 power-down all non-mesh 2.4 -> Low (per-zone clean floors, per-AP on mesh-containing Floor 2/misc; re-disable 128 after Floor 1); 2 settle+re-measure (valid stop point); 3 disable 25 mesh-safe low-client radios per floor with gates (F1:7 F2:8 F3:5 F4:5); 4 wrap. Excluded: 7 high-client redundant (power-down instead: 3rd/4th Atrium,318,303,406-408,505,Dining Room); Memcare TV room (memcare hold); companion levers (min-rate/band-steer/ min-RSSI) deferred. Recovery: device-control poe-cycle (never force-provision). Rollback JSONs in .claude/tmp. ### Open decision (awaiting Howard) Scope for tonight: recommended = power-down-all + pilot ONE floor's disables, validate ~15min, expand if clean (vs all 25 at once). And whether I stay live during execution to run the per-floor validation gates. ### Note: mesh data is from .claude/tmp/cascades-nbr.json (fresh tonight, gitignored) — regenerate via neighbor-collect before execution if stale. ## Update: 19:28 PT (2026-06-16) — home LAN renumber fixes VPN shadow; pfSense ruled out as WiFi cause; SSH backend ### Home-LAN shadow RESOLVED (unblocked Cascades pfSense access) Diagnosed why the Cascades VPN couldn't reach 192.168.0.x: Howard-Home was 192.168.0.0/24 with a UniFi gateway at 192.168.0.1 (cert unifi.local — NOT pfSense), colliding with Cascades' own 192.168.0.0/24 (pfSense .0.1). OS prefers the directly-connected local /24, so 192.168.0.x never crossed the tunnel (APs on 192.168.2.x/3.x worked). A /32 route can't fix it (.0.1 was Howard's own gateway). **Howard renumbered Howard-Home to 10.137.42.0/24** (gw 10.137.42.1). Verified: ping 192.168.0.1 now replies over the tunnel (TTL 64, ~15ms); cert = CN=pfSense-685f277aa6886 / "Netgate pfSense Plus" (the REAL Cascades pfSense); NAS .120 reachable; controller-side (172.16.3.29) + AP reach (192.168.2.79) intact. Memory: [[howard-home-lan-shadow]] updated to RESOLVED. ### pfSense investigated — NOT a WiFi factor (gateway ruled out) SSH into Cascades pfSense (admin, shell works directly — no menu gotcha; pfSense Plus 25.07-RELEASE). Findings: DHCP NOT exhausted (0 "no free leases"; WiFi/AP pool 192.168.0.0/22 range .2.2-.3.254 cap ~507 at 270 active ~53%; 199 subnets total, mostly per-unit /28s; active backend ISC, Kea dormant); unbound DNS up; dual-WAN (WAN1 184.191.143.62/30, WAN2 72.211.21.217/27) both full-duplex, no gateway loss/down events; PF states 28-31k/790k; load 0.6; 10-day uptime. Minor: igc3/WAN2 1707 Ierrs+Coll = Intel I225/226 2.5G counter quirk (2.5G full-duplex active, no loss), not a fault. CONCLUSION: gateway/DHCP/DNS/WAN are not bottlenecking WiFi — the 2.4 GHz RF remediation is the sole fix. Added "pfSense health check (2026-06-16)" section to clients/cascades-tucson/reports/2026-06-16-unifi-full-audit.md. ### pfSense compat layer: pivot to SSH backend, OFF HOLD (Mike's call) Mike (relayed by Howard): "we don't need the RESTAPI — VPN + SSH gets the same data and makes changes." The earlier "pfSense too old for RESTAPI" premise was WRONG (it's 25.07, current). So the upgrade/package blocker is moot; layer off hold. Built scripts/pfsense-ssh.sh (audit | dhcp | run ""); cred from clients//pfsense-firewall; system OpenSSH via askpass; validated live on Cascades (the health data above came from it). ROADMAP §E + SKILL.md updated to the SSH decision; REST pfsense-backend.sh left dormant/optional. Closed obsolete coord todo afbe6a22 (upgrade-for-RESTAPI). Remaining = named gated CONTROL verbs over SSH (easyrule block-ips, pf/fw toggles) — Howard: "leave that up to Mike" (his §E). Commits: 58ecc5a (pfSense-ssh + health + off-hold), e42ad8f (memory). ### Still open (the actual WiFi win) Tonight's 2.4 remediation per the runbook (reports/2026-06-16-2.4ghz-remediation-runbook.md): scope (power-down-all + pilot one floor vs all 25) + whether Claude runs validation gates live during execution. ## Update: 23:53 PT (2026-06-16) — 2.4 remediation EXECUTED (power-down + partial disables); PAUSED before Floor 2 Howard rebooted the Cascades pfSense, then gave the go to push the 2.4 changes. (Changes go through the UOS controller 172.16.3.29, NOT the pfSense/VPN — reboot didn't block them. Verified fleet healthy post-reboot: 77 adopted / 2 disconnected, unchanged.) ### Phase 1 — 2.4 power-down to Low: COMPLETE (65 radios) Applied auto->low across Floors 1,2,3 + misc (Floor 4 was already done). Floor 3 by-zone (17); Floors 1/2 + misc PER-AP to skip mesh + offline + 128. Final ng state: low=65, auto=11 (correctly left: mesh parents 2nd Floor Atrium/206 U7 Pro, mesh children CC Bridge/salon, offline 108/108U7 Pro, EXCLUDED Floors 5/6 = 505/517/608/615/622), disabled=1 (128). 0 failures, no AP knocked offline. Initially missed 247 (Floor 2) and 4 Floor-1 stragglers (121/122/109/102) — see Problems; all fixed. ### Phase 2 — re-measure (15-min settle): the cumulative WIN Site-wide 2.4 averages (live-stats, ~74 radios): PRE power-down: cu_total 77% cu_interf 64% retry 12.2% clients 134 POST settle: cu_total 68% cu_interf 53% retry 13.8% clients 111 => cu_interf -11 pts, cu_total -9 pts site-wide (vs only ~4 pts from the Floor-4-only test earlier -> confirms the benefit is cumulative/site-wide). Retry +1.6 is midnight client-mix noise (fewer clients). ### Phase 3 — disables: 15 of 23 done, PAUSED before Floor 2 Refreshed coverage-thin (MINCOV=2, mesh-aware) -> conservative set = 23 (low-client <2 avg, >=2 active coverers, non-mesh, non-memcare, Floors 1-4). Disabled (apply-radio ng disable --ap, spaced): Floor 4: 445 442 407 450 Floor 3: 342 317 304 336 Floor 1: 139 115 102 145 127 122 116 REMAINING (Howard said "lets pause"): **Floor 2: 240 229 241 209 203 248 204 236** (8 APs). Fleet after disables: 77 adopted / 2 disconnected — disabling 2.4 just drops that radio; AP stays up on 5/6 GHz, every disabled radio's area keeps >=2 active 2.4 coverers. The force-provision hazard does NOT apply to `disable` (clean radio-off, unlike the 445 force-provision incident). ### Other this turn - apply-wlan.sh BUG FIXED: wlan_bands token was "6e" but THIS controller stores "6g" (verified live on Guest SSID) -> 6 GHz enablement would have failed. Fixed (5g6g/6g/all). Commit 5ad25d1. - Runbook: folded in Phase 5 (5 GHz: 80->40 width on 76 radios + channel plan with DFS decision flagged -- DFS empirically CLEAN here so including clean-DFS gives ~20 ch vs ~5 non-DFS-only) and Phase 6 (6 GHz: ROOT CAUSE 6 GHz empty = production SSID CSCNet not on 6 GHz [bands 2g,5g]; add 6g + bss-transition). - Floors 5 & 6 EXCLUDED from the whole plan (Howard). ### RESUME / next 1. Finish Phase 3: disable Floor 2 (240 229 241 209 203 248 204 236), spaced, validate. 2. Proper site-wide re-measure after settle (compare cu_interf again). 3. Phase 5 (5 GHz width+channel) + Phase 6 (6 GHz CSCNet) when ready. ROLLBACK: re-enable a radio = apply-radio cascades ng enable --ap "" --apply; restore power = apply-radio cascades ng power auto --zone "Floor X" --apply. (apply-radio rollback JSON is per-call in .claude/tmp, overwritten each call -> for power-down revert just set power auto.) ### Friction logged apply-radio per-AP --apply loop: each call re-logs-in to the controller; rapid succession throttles and the write silently skips (no [ok]). Fix: space calls (sleep ~4) or add multi-AP/one-login support. ## Update: 00:30 PT (2026-06-17) — Phase 3 disables COMPLETE (all 23 + 128); daytime re-check scheduled Finished Floor 2 disables (240 229 241 209 203 248 204 236). Final 2.4 state: **disabled=24, low=42, auto=11** (auto = mesh parents/children 2nd Floor Atrium/206/CC Bridge/salon + offline 108/108U7Pro + EXCLUDED Floors 5/6 = 505/517/608/615/622). Fleet 77 adopted / 2 disconnected — no AP offline. **Experienced improvement so far (midnight, low load — directional):** active-radio-only site-wide 2.4 cu_interf **64% -> 48%** (-16 pts, ~25% rel), cu_total 77% -> 66%. (NOTE: averaging ALL ng radios incl. the 24 disabled-reporting-0 misleadingly showed cu 45%/interf 33% -> always exclude disabled for a fair read.) Hourly time-of-day-controlled (same hr last night): retry 27.8->26.0, sat ~flat. Real client- experience delta needs BUSINESS-HOURS load -> daytime re-check scheduled. Daytime re-check: CronCreate one-shot 9:33 AM 6/17 (reported session-only — NOT durable) + coord todo (reliable backstop) filed for Howard. Remaining: Phase 5 (5GHz 80->40 + channel) + Phase 6 (6GHz: add 6g to CSCNet + bss-tm). apply-wlan 6e->6g fix already committed (5ad25d1). ## Update: 16:00 PT (2026-06-17) — daytime re-checks: interference DOWN but client experience WORSE (over-thinning) -> Low->Medium recommended. (Howard continuing in another session.) Two loaded-network re-checks (09:33 + 11:27, time-of-day controlled vs same hours yesterday at full power) both say the SAME thing: the 2.4 remediation cleaned too aggressive. metric (2.4 site-wide, hourly) yesterday FULL now POST-change interference cu_interf 65-66% 32% <- WIN (down ~half) total cu_total 79% 44% <- WIN retry% 17-18.5% 23-32% <- WORSE (roughly doubled) satisfaction 39 30-31 <- WORSE Active-radio-only snapshot: cu_interf ~48% (vs 64 pre), retry ~22%. Fleet: 77 adopted / 2 disconnected, NO AP offline. DIAGNOSIS: OVER-THINNING. Power to Low (~6 dBm) is a 17 dB cut from auto(~23); combined with 24 disabled radios, some edge 2.4 clients now reach a farther/weaker AP -> retries up, satisfaction down EVEN THOUGH the air is much cleaner. The interference goal is met; the lever overshot on coverage. >>> CURRENT DECISION POINT (next action, NOT yet done): 1. Bump the 42 KEPT 2.4 radios Low -> MEDIUM (~12-15 dBm): restores edge signal, cells still tighter than full, and the 24 disables STAY OFF so most of the interference win is kept. Command per floor (skip 5/6, skip mesh/offline like the power-down was): bash .claude/skills/unifi-wifi/scripts/apply-radio.sh cascades ng power medium --zone "Floor 3" --apply (Floor 1/2/misc per-AP, spaced ~4s, to skip mesh parents + 128/108 + login throttle) 2. Re-measure retry/satisfaction (same time-of-day) after the bump to confirm recovery. 3. If still high, re-enable a few of the 24 disabled radios in the weakest spots. STATE RECAP for the other session: Phase1 power-down DONE (was Low on 65; recommend ->Medium on the kept 42). Phase3 disables DONE (24 off: 102 115 116 122 127 128 139 145 203 204 209 229 236 240 241 248 304 317 336 342 407 442 445 450). Floors 5/6 + mesh (2nd Floor Atrium/206/CC Bridge/salon) + offline 108/108U7Pro UNTOUCHED. Phase 5 (5GHz 80->40 + channel) and Phase 6 (6GHz: add 6g to CSCNet + bss-tm; apply-wlan 6e->6g already fixed) still PENDING. Runbook: reports/2026-06-16-2.4ghz-remediation-runbook.md. Rollback: ng power auto / ng enable --ap. Friction logged: per-AP apply re-logins -> space calls.