claudetools/clients/cascades-tucson/docs/network/network-optimization-master-plan.md

# Cascades — Network Optimization Master Plan (all devices, holistic)

- **Created:** 2026-06-18 (Howard-Home / claude-main)
- **Status:** PLAN — for execution tonight (floors 1–4) per Howard. Floors 5 & 6 (MemCare) EXCLUDED this round.
- **Goal:** Fix the *system*, not one device at a time. Improve quality for **every** client (~587), not just
  the 31 voice devices, by sequencing AP + WLAN + QoS + firewall changes so we don't trade one problem for another.
- **Builds on:** `reports/2026-06-16-unifi-full-audit.md`, `reports/2026-06-16-2.4ghz-remediation-runbook.md`
  (RF mechanics + gated apply commands), `reports/2026-06-18-voice-quality-diagnostic.md`, and the live
  2026-06-18 fleet sample. All RF changes use the gated `unifi-wifi` scripts (per-zone, dry-run, rollback JSON).

---

## 1. Current state (what's actually true right now)

| Layer | State | Verdict |
|---|---|---|
| **2.4 GHz** | **OVER-THINNED.** Overnight 6/17: 24 radios disabled + 42 set Low (~6 dBm). Interference dropped (cu_interf 64→32–48%) BUT **retry rose 17→23.4%, satisfaction fell 39→30** (time-of-day-controlled). Edge clients now reach farther/weaker APs. Mesh + Floors 5/6 untouched (full 23 dBm). | **Regressed — must correct power floor** |
| **5 GHz** | 80 MHz width on ~76/77 (too wide for the density). 55/77 on DFS (empirically clean — 0 radar). Channels biased to busy upper (149/157). **AP 103 saturated: ch149, 75% airtime, ~25,900 retries, 12 clients** (and Lauren's phone now locked there). Dining/Rec Room high retry (810/1083). | **Constrain width + spread channels + relieve hotspots** |
| **6 GHz** | 75 radios live, **~1 client.** Root cause: **CSCNet not broadcasting 6 GHz** (`wlan_bands=[2g,5g]`). Cleanest untapped capacity. | **Open it — the relief valve** |
| **QoS** | **NONE.** Voice now isolated on VLAN 30 but not prioritized — voice packets compete with data under load → jitter/breaks. | **Add — guaranteed win, now possible** |
| **pfSense/WAN/DHCP/DNS** | Healthy; ruled out as a WiFi factor (2026-06-16). Dual-WAN stable, DHCP 53% pool, unbound up. | **Fine — add voice QoS shaping only** |
| **Switching / physical** | ~25 ports linked 100 M but gig-capable (caps some AP uplinks); 3 offline switches; AP 108 cable pending; p38 4% tx-drop. | **Physical work — not tonight, but tracked** |

---

## 2. Root-cause model (why "some devices" are bad)

Three compounding RF causes, plus a missing QoS layer:
1. **2.4 GHz contention** — extreme neighbor density (ch6 ~33k BSSIDs). Any client that lands/sticks on 2.4 GHz
   suffers. Made *worse* by the over-thinning (weaker signal → more retransmits).
2. **5 GHz over-width + hotspots** — 80 MHz halves the usable channel count → co-channel overlap → retries;
   a few APs (103) are simply overloaded.
3. **6 GHz unused** — the clean band that should absorb modern clients is dark, so everything piles onto 5 GHz.
4. **No voice prioritization** — even with perfect RF, voice breaks under data bursts without QoS.

**The trap we must avoid (the "whack-a-mole"):** narrowing 5 GHz to 40 MHz *without* first opening 6 GHz pushes
more clients onto fewer 5 GHz channels → congestion moves, not improves. And dropping 2.4 power further (it's
already too low) starves edge clients. **Sequence matters.**

---

## 3. The holistic sequence (open relief valves BEFORE constraining)

> Principle: **(A) add capacity/priority that can't hurt → (B) fix the regression → (C) then constrain/optimize
> → (D) fine-tune → validate at every gate.** Each step is reversible; gate on live metrics before the next.

### PHASE 0 — Pre-flight + baseline (always)
- VPN up; `live-stats.sh cascades | head -3` (expect 77 APs).
- Baseline (compare after, same time-of-day): `live-stats.sh cascades > .claude/tmp/opt-pre.txt`;
  `radio-usage.sh cascades ng 77 > .claude/tmp/usage-pre.txt`.
- Pick a watch AP per floor (`watch-ap.sh <ip>`).

### PHASE 1 — QoS for voice (orthogonal, lowest risk — but INSURANCE, not the everyday fix)
Voice VLAN 30 is isolated → mark + prioritize it end-to-end so calls beat data under load.
> **Reframe (measured 2026-06-18):** WAN1 fiber upload is **~522 Mbps** vs ~98 Mbps peak usage — huge
> headroom, so the WAN is **not** the day-to-day voice bottleneck (that's RF, Phases 2–4). QoS still earns its
> place as insurance for **WAN2 (coax) failover** and **rare WAN1 saturation** (you hit 680 Mbps down). Build
> it (cheap, correct), but don't expect it to fix the complaints — the RF work does. Full design:
> `docs/network/phase1-voice-qos-design.md`. **Phones confirmed marking DSCP EF** → rely on DSCP; subnet match is the net.
- **UniFi (WLAN/switch):** ensure WMM/QoS on; the AudioCodes/Poly tag voice DSCP — trust/honor it. On the
  USW, voice VLAN traffic should hit the high-priority queue.
- **pfSense:** add a traffic-shaper/limiter or floating QoS rule that puts `VOICE net (10.0.30.0/24)` DSCP EF
  (46) / RTP UDP into a priority queue on the WAN(s). Low risk — additive, voice-only.
- **Validate:** place test calls during a data-heavy moment; confirm no breakup. (No RF change here.)
- *Skill gap:* the `unifi-wifi` skill has no QoS verb — this is a pfSense + UniFi config task; consider a small
  `voice-qos` helper later.

### PHASE 2 — Open the relief valves (capacity + correct the regression)
**2a. Enable 6 GHz on CSCNet + steering** (creates the offload path BEFORE we narrow 5 GHz):
```
apply-wlan.sh cascades bands all --wlan CSCNet --apply     # -> [2g,5g,6g]
apply-wlan.sh cascades bsstm on --wlan CSCNet --apply       # 802.11v BSS-transition (assists up-band + roam)
```
Band-steering (`no2ghz_oui`) already ON. 6E/7 clients gravitate to clean 6 GHz, offloading 5 GHz. Validate:
client mix shifts toward 6g; no SSID-visibility loss for legacy (2.4/5 stay on).

**2b. Correct the 2.4 over-thinning — Low → MEDIUM on kept radios** (restores edge signal; keeps cells smaller
than full power). Per floor, dry-run then apply; regenerate the kept-radio list live:
```
for z in "Floor 1" "Floor 2" "Floor 3" "Floor 4"; do \
  apply-radio.sh cascades ng power medium --zone "$z" --apply; done   # ~12–15 dBm
```
Do NOT expand disables. If a specific area shows a dead zone/complaint, re-enable that one radio
(`ng enable --ap "<name>"`). **Gate:** re-measure retry%/satisfaction same time-of-day vs `opt-pre.txt` —
expect retry back down from ~23% and satisfaction recovering.

### PHASE 3 — Constrain + optimize 5 GHz (now that 6 GHz absorbs load)
**3a. Width 80 → 40 MHz** (doubles non-overlapping channels → spatial reuse):
```
for z in "Floor 3" "Floor 1" "Floor 2" "Floor 4"; do \
  apply-radio.sh cascades na width 40 --zone "$z" --apply; done   # rollback: na width 80
```
**3b. Channel plan — NON-DFS ONLY (decided 2026-06-18 after rigorous DFS verification).**
Use **UNII-1 (36–48) + UNII-3 (149–165) only**; do NOT use DFS channels (52–144) on this voice-critical
network. A precise radar-detection sweep (real `radar found`/`NOL` signatures, CAC/control housekeeping
excluded) found **ZERO genuine hits across all 53 DFS APs** — BUT the window was only ~21–23h (APs rebooted
~23h ago, the 6/17 outage). Near Davis-Monthan AFB + TUS (~10 mi), military radar is sporadic and a single
hit forces a 30-min channel vacate = **dropped calls** — unacceptable for voice. **Resilience > diversity.**
The lost 5 GHz channel count is covered by **6 GHz (Phase 2a) absorbing capacity** — this is WHY 6 GHz comes first.
```
SURVEY=.claude/tmp/cascades-survey.json; SURVEY_JSON=$SURVEY survey-collect.sh cascades
SURVEY_JSON=$SURVEY channel-plan.sh cascades na    # dry-run; CONSTRAIN to non-DFS (36-48,149-165); review; apply per zone
```
**Periodic DFS monitoring:** the ~1-day window isn't conclusive, so add a recurring precise `dfs-check.sh`
(fold into the network-logging plan). Staying on non-DFS means a future hit can't affect us; the monitor just
confirms the choice stays right.
**3c. Relieve AP 103 specifically** (it now carries Lauren + 11 others on a 75%-busy ch149): move it off 149 to
a clean channel from the plan, 40 MHz. Verify Lauren `.202` retry drops after.
**Gate:** 5 GHz retry down on the busy APs; AP 103 cu_total well under 50%; no client stranded.

### PHASE 4 — Fine-tune (after 1–3 settle)
- **2.4 channel plan 1/6/11** (graph-color; co-channel pairs 92→35) + **pin the 4 off-plan APs** (128/108/108U7/salon)
  to 1/6/11.
- **2.4 min-RSSI ON** for the 6 APs where it's OFF (615/608/505/517/622/salon) — *note 505/517/615/608/622 are
  Floors 5/6 → DEFER with the rest of 5/6*; do `salon` only this round.
- **Roaming for voice continuity:** confirm 802.11k/v on CSCNet (r optional — test; some phones dislike 802.11r).
  Keeps calls alive when staff walk between APs.
- **min-RSSI tuning:** only tighten where sticky-client far-AP behavior is proven; too aggressive blocks association.

### PHASE 5 — Physical (separate visit, not tonight — but it caps results)
- Re-terminate/replace the ~25 cables on ports stuck at 100 M (limits those APs' uplink throughput).
- Chase the 3 offline switches (2nd Floor #2, 4th Floor #2, USW Pro Max 16); finish AP 108 cable run.
- p38 (1st Floor USW) 4% tx-drop after the above.

---

## 4. Interdependency map (read before changing anything)
- **6 GHz BEFORE 5 GHz 40 MHz** — else 5 GHz congestion just relocates. (Phase 2a before 3a.)
- **2.4 power MEDIUM not LOW** — Low already over-thinned; going lower starves edge clients. (Phase 2b.)
- **AP-lock needs AP capacity** — Lauren locked to 103 ⇒ 103 must be relieved (Phase 3c) or she trades mesh for congestion.
- **QoS is independent** — do it first; it can't hurt RF and guarantees a voice win even before RF settles. (Phase 1.)
- **Disables + power-down compound** — never do both aggressively in the same area; we already saw the satisfaction hit.
- **min-RSSI + power interact** — raising min-RSSI while lowering power can orphan clients; tune one lever at a time.
- **Mesh-protected APs** (`2nd Floor Atrium, CC Bridge, salon, 206 U7 Pro, 108`) — never disable; power changes only with watch.

## 5. Data-driven decision framework — improve quality for ALL devices (measure → decide → adjust)

**Principle (Howard 2026-06-18): every choice is made FROM measured network data, not assumptions.** Each
change is a hypothesis; we gate it on fleet-wide metrics before keeping it or moving on. The goal is *all*
devices (CSCNet 427 + CSC ENT 131 + Guest 13), not just the 31 voice phones.

### 5.1 How each change affects the OTHER (non-voice) devices
Almost every change targets the **shared RF environment**, so it helps everyone — voice is just the most
sensitive canary:
| Change | Effect on non-voice devices |
|---|---|
| QoS (voice VLAN priority) | **Negligible** — voice is ~3 Mbps of 522; normally zero effect; ACK queue can make others snappier under load |
| Enable 6 GHz on CSCNet | **Positive** — 6E devices move to clean 6 GHz → faster for them + clears 5 GHz for everyone left |
| 2.4 Low→Medium power | **Positive for ALL 2.4 devices** — undoes the over-thinning regression (IoT/printers/2.4 DirecTV get signal back) |
| 5 GHz 80→40 MHz | **Net positive (reliability), small peak-speed cost** — density win; lone heavy transfer sees lower peak |
| AP 103 relief | **Positive for all 16 clients on 103**, not just Lauren |
| 2.4 1/6/11 channel plan | **Positive for all 2.4 devices** (less co-channel) |
| Phone-side (Vertical) | Phones only — **no effect on others** |

### 5.2 Trade-offs to WATCH in the data (don't help voice, hurt others)
1. **Non-DFS 5 GHz + 5 GHz-only devices** — the DirecTV fleet + older laptops **can't use 6 GHz**, so they
   stay on the fewer non-DFS channels. 6 GHz offloading the newer devices is what keeps this OK; **watch
   non-DFS 5 GHz cu_total** — if it climbs, that's the signal to rebalance.
2. **min-RSSI** affects every client on the AP — too aggressive orphans weak IoT/resident devices. Tune gently.
3. **40 MHz** trades single-user peak for fleet reliability — right in density, but it is a trade.

### 5.3 Fleet-wide metrics we pull at every gate (the data we decide on)
Same time-of-day comparison (load varies hourly). Capture before each change and ~15 min after:
- `live-stats.sh cascades` → per-band **avg retry%, cu_total, cu_interf, satisfaction (min/median), client counts**
- `radio-usage.sh cascades <band>` → per-AP outliers (saturated/high-retry APs)
- `/stat/sta` band split (2.4 / 5 / 6 distribution) + count of clients retry>15% (by band) + satisfaction<70 count
- Per-AP: any AP whose client count drops toward ~0 in a covered area (= coverage hole)

### 5.4 The GATE decision rule (per change, per zone)
- **KEEP + proceed** only if: the target metric improved **AND** fleet-wide **satisfaction did not fall**,
  **retry% did not rise**, band split moved the intended way, and **no AP lost its clients** (no hole).
- **HOLD** (stop, don't expand) if: target improved but a secondary metric regressed → investigate before more.
- **ROLLBACK that step** if: fleet-wide retry up / satisfaction down / a coverage hole / user complaint.
- Do **one lever per zone at a time** so cause/effect is attributable (the over-thinning happened because power-down
  + disables were stacked).

### 5.5 Rollback
Every `apply-radio`/`apply-wlan` writes a rollback JSON to `.claude/tmp/`; `device-control poe-cycle` for a hung
AP (NOT force-provision). Power-up / width-80 / re-enable / channel-revert are all safe reversals.

## 6. Out of scope tonight (explicit)
- **Floors 5 & 6 (MemCare)** — all RF + the MemCare voice phones (`.217/.218/.219/.220`) DEFERRED per Howard.
- Physical cabling / offline switches (Phase 5 — separate visit).
- The 6 straggler phones — Howard re-keying separately; they'll benefit from the RF work regardless.

## 7. Open decisions for Howard
1. ~~**5 GHz channel plan:** clean-DFS vs non-DFS-only~~ — **RESOLVED 2026-06-18: NON-DFS ONLY** (UNII-1 36–48 + UNII-3 149–165). DFS sweep was clean but only a ~1-day window near Davis-Monthan/TUS; a radar vacate = dropped calls, so resilience wins. 6 GHz covers the capacity gap. (See Phase 3b.)
2. **QoS depth:** UniFi WMM + DSCP-honor only, or also a pfSense WAN priority queue/limiter for RTP? Recommendation: both (additive).
3. **802.11r** on CSCNet: enable for seamless voice roaming, or k/v only (safer for mixed phones)? Recommendation: k/v now, test r on one phone first.
4. Tonight's stopping point: Phases 1–2 alone are a legitimate, lower-risk night; 3–4 can be a second night.
5. ~~**Dedicated voice SSID?**~~ — **RESOLVED 2026-06-18: NO — voice stays on the shared CSCNet PPSK.** UniFi
   3-SSID cap (sound RF hygiene — each SSID = beacon airtime overhead at 77 APs). The only retirement candidate,
   CSC ENT, still has **131 active clients** (staff PCs, printers, DirecTV fleet) → not retireable soon. And it's
   not needed: **QoS is VLAN/DSCP-based (SSID-independent)**, band preference is best done **phone-side** (Vertical),
   and roaming/power-save are phone+AP settings — all work on the shared SSID. A dedicated voice SSID would only
   add voice-specific WiFi *policy* (per-SSID DTIM/min-RSSI/airtime), a marginal gain not worth a slot. Revisit only
   if/when CSC ENT's 131 clients migrate off it.