sync: auto-sync from HOWARD-HOME at 2026-06-18 18:26:19
Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-18 18:26:19
This commit is contained in:
1
.fleet325.dev
Normal file
1
.fleet325.dev
Normal file
File diff suppressed because one or more lines are too long
@@ -0,0 +1,140 @@
|
||||
# Cascades — Network Optimization Master Plan (all devices, holistic)
|
||||
|
||||
- **Created:** 2026-06-18 (Howard-Home / claude-main)
|
||||
- **Status:** PLAN — for execution tonight (floors 1–4) per Howard. Floors 5 & 6 (MemCare) EXCLUDED this round.
|
||||
- **Goal:** Fix the *system*, not one device at a time. Improve quality for **every** client (~587), not just
|
||||
the 31 voice devices, by sequencing AP + WLAN + QoS + firewall changes so we don't trade one problem for another.
|
||||
- **Builds on:** `reports/2026-06-16-unifi-full-audit.md`, `reports/2026-06-16-2.4ghz-remediation-runbook.md`
|
||||
(RF mechanics + gated apply commands), `reports/2026-06-18-voice-quality-diagnostic.md`, and the live
|
||||
2026-06-18 fleet sample. All RF changes use the gated `unifi-wifi` scripts (per-zone, dry-run, rollback JSON).
|
||||
|
||||
---
|
||||
|
||||
## 1. Current state (what's actually true right now)
|
||||
|
||||
| Layer | State | Verdict |
|
||||
|---|---|---|
|
||||
| **2.4 GHz** | **OVER-THINNED.** Overnight 6/17: 24 radios disabled + 42 set Low (~6 dBm). Interference dropped (cu_interf 64→32–48%) BUT **retry rose 17→23.4%, satisfaction fell 39→30** (time-of-day-controlled). Edge clients now reach farther/weaker APs. Mesh + Floors 5/6 untouched (full 23 dBm). | **Regressed — must correct power floor** |
|
||||
| **5 GHz** | 80 MHz width on ~76/77 (too wide for the density). 55/77 on DFS (empirically clean — 0 radar). Channels biased to busy upper (149/157). **AP 103 saturated: ch149, 75% airtime, ~25,900 retries, 12 clients** (and Lauren's phone now locked there). Dining/Rec Room high retry (810/1083). | **Constrain width + spread channels + relieve hotspots** |
|
||||
| **6 GHz** | 75 radios live, **~1 client.** Root cause: **CSCNet not broadcasting 6 GHz** (`wlan_bands=[2g,5g]`). Cleanest untapped capacity. | **Open it — the relief valve** |
|
||||
| **QoS** | **NONE.** Voice now isolated on VLAN 30 but not prioritized — voice packets compete with data under load → jitter/breaks. | **Add — guaranteed win, now possible** |
|
||||
| **pfSense/WAN/DHCP/DNS** | Healthy; ruled out as a WiFi factor (2026-06-16). Dual-WAN stable, DHCP 53% pool, unbound up. | **Fine — add voice QoS shaping only** |
|
||||
| **Switching / physical** | ~25 ports linked 100 M but gig-capable (caps some AP uplinks); 3 offline switches; AP 108 cable pending; p38 4% tx-drop. | **Physical work — not tonight, but tracked** |
|
||||
|
||||
---
|
||||
|
||||
## 2. Root-cause model (why "some devices" are bad)
|
||||
|
||||
Three compounding RF causes, plus a missing QoS layer:
|
||||
1. **2.4 GHz contention** — extreme neighbor density (ch6 ~33k BSSIDs). Any client that lands/sticks on 2.4 GHz
|
||||
suffers. Made *worse* by the over-thinning (weaker signal → more retransmits).
|
||||
2. **5 GHz over-width + hotspots** — 80 MHz halves the usable channel count → co-channel overlap → retries;
|
||||
a few APs (103) are simply overloaded.
|
||||
3. **6 GHz unused** — the clean band that should absorb modern clients is dark, so everything piles onto 5 GHz.
|
||||
4. **No voice prioritization** — even with perfect RF, voice breaks under data bursts without QoS.
|
||||
|
||||
**The trap we must avoid (the "whack-a-mole"):** narrowing 5 GHz to 40 MHz *without* first opening 6 GHz pushes
|
||||
more clients onto fewer 5 GHz channels → congestion moves, not improves. And dropping 2.4 power further (it's
|
||||
already too low) starves edge clients. **Sequence matters.**
|
||||
|
||||
---
|
||||
|
||||
## 3. The holistic sequence (open relief valves BEFORE constraining)
|
||||
|
||||
> Principle: **(A) add capacity/priority that can't hurt → (B) fix the regression → (C) then constrain/optimize
|
||||
> → (D) fine-tune → validate at every gate.** Each step is reversible; gate on live metrics before the next.
|
||||
|
||||
### PHASE 0 — Pre-flight + baseline (always)
|
||||
- VPN up; `live-stats.sh cascades | head -3` (expect 77 APs).
|
||||
- Baseline (compare after, same time-of-day): `live-stats.sh cascades > .claude/tmp/opt-pre.txt`;
|
||||
`radio-usage.sh cascades ng 77 > .claude/tmp/usage-pre.txt`.
|
||||
- Pick a watch AP per floor (`watch-ap.sh <ip>`).
|
||||
|
||||
### PHASE 1 — QoS for voice (do FIRST; orthogonal, lowest risk, guaranteed win)
|
||||
Voice VLAN 30 is isolated → mark + prioritize it end-to-end so calls beat data under load.
|
||||
- **UniFi (WLAN/switch):** ensure WMM/QoS on; the AudioCodes/Poly tag voice DSCP — trust/honor it. On the
|
||||
USW, voice VLAN traffic should hit the high-priority queue.
|
||||
- **pfSense:** add a traffic-shaper/limiter or floating QoS rule that puts `VOICE net (10.0.30.0/24)` DSCP EF
|
||||
(46) / RTP UDP into a priority queue on the WAN(s). Low risk — additive, voice-only.
|
||||
- **Validate:** place test calls during a data-heavy moment; confirm no breakup. (No RF change here.)
|
||||
- *Skill gap:* the `unifi-wifi` skill has no QoS verb — this is a pfSense + UniFi config task; consider a small
|
||||
`voice-qos` helper later.
|
||||
|
||||
### PHASE 2 — Open the relief valves (capacity + correct the regression)
|
||||
**2a. Enable 6 GHz on CSCNet + steering** (creates the offload path BEFORE we narrow 5 GHz):
|
||||
```
|
||||
apply-wlan.sh cascades bands all --wlan CSCNet --apply # -> [2g,5g,6g]
|
||||
apply-wlan.sh cascades bsstm on --wlan CSCNet --apply # 802.11v BSS-transition (assists up-band + roam)
|
||||
```
|
||||
Band-steering (`no2ghz_oui`) already ON. 6E/7 clients gravitate to clean 6 GHz, offloading 5 GHz. Validate:
|
||||
client mix shifts toward 6g; no SSID-visibility loss for legacy (2.4/5 stay on).
|
||||
|
||||
**2b. Correct the 2.4 over-thinning — Low → MEDIUM on kept radios** (restores edge signal; keeps cells smaller
|
||||
than full power). Per floor, dry-run then apply; regenerate the kept-radio list live:
|
||||
```
|
||||
for z in "Floor 1" "Floor 2" "Floor 3" "Floor 4"; do \
|
||||
apply-radio.sh cascades ng power medium --zone "$z" --apply; done # ~12–15 dBm
|
||||
```
|
||||
Do NOT expand disables. If a specific area shows a dead zone/complaint, re-enable that one radio
|
||||
(`ng enable --ap "<name>"`). **Gate:** re-measure retry%/satisfaction same time-of-day vs `opt-pre.txt` —
|
||||
expect retry back down from ~23% and satisfaction recovering.
|
||||
|
||||
### PHASE 3 — Constrain + optimize 5 GHz (now that 6 GHz absorbs load)
|
||||
**3a. Width 80 → 40 MHz** (doubles non-overlapping channels → spatial reuse):
|
||||
```
|
||||
for z in "Floor 3" "Floor 1" "Floor 2" "Floor 4"; do \
|
||||
apply-radio.sh cascades na width 40 --zone "$z" --apply; done # rollback: na width 80
|
||||
```
|
||||
**3b. Channel plan** (decision: INCLUDE clean-DFS for diversity vs non-DFS-only for resilience — DFS is
|
||||
empirically clean here, 0 radar):
|
||||
```
|
||||
SURVEY=.claude/tmp/cascades-survey.json; SURVEY_JSON=$SURVEY survey-collect.sh cascades
|
||||
SURVEY_JSON=$SURVEY channel-plan.sh cascades na # dry-run; review; apply per zone w/ validation
|
||||
```
|
||||
**3c. Relieve AP 103 specifically** (it now carries Lauren + 11 others on a 75%-busy ch149): move it off 149 to
|
||||
a clean channel from the plan, 40 MHz. Verify Lauren `.202` retry drops after.
|
||||
**Gate:** 5 GHz retry down on the busy APs; AP 103 cu_total well under 50%; no client stranded.
|
||||
|
||||
### PHASE 4 — Fine-tune (after 1–3 settle)
|
||||
- **2.4 channel plan 1/6/11** (graph-color; co-channel pairs 92→35) + **pin the 4 off-plan APs** (128/108/108U7/salon)
|
||||
to 1/6/11.
|
||||
- **2.4 min-RSSI ON** for the 6 APs where it's OFF (615/608/505/517/622/salon) — *note 505/517/615/608/622 are
|
||||
Floors 5/6 → DEFER with the rest of 5/6*; do `salon` only this round.
|
||||
- **Roaming for voice continuity:** confirm 802.11k/v on CSCNet (r optional — test; some phones dislike 802.11r).
|
||||
Keeps calls alive when staff walk between APs.
|
||||
- **min-RSSI tuning:** only tighten where sticky-client far-AP behavior is proven; too aggressive blocks association.
|
||||
|
||||
### PHASE 5 — Physical (separate visit, not tonight — but it caps results)
|
||||
- Re-terminate/replace the ~25 cables on ports stuck at 100 M (limits those APs' uplink throughput).
|
||||
- Chase the 3 offline switches (2nd Floor #2, 4th Floor #2, USW Pro Max 16); finish AP 108 cable run.
|
||||
- p38 (1st Floor USW) 4% tx-drop after the above.
|
||||
|
||||
---
|
||||
|
||||
## 4. Interdependency map (read before changing anything)
|
||||
- **6 GHz BEFORE 5 GHz 40 MHz** — else 5 GHz congestion just relocates. (Phase 2a before 3a.)
|
||||
- **2.4 power MEDIUM not LOW** — Low already over-thinned; going lower starves edge clients. (Phase 2b.)
|
||||
- **AP-lock needs AP capacity** — Lauren locked to 103 ⇒ 103 must be relieved (Phase 3c) or she trades mesh for congestion.
|
||||
- **QoS is independent** — do it first; it can't hurt RF and guarantees a voice win even before RF settles. (Phase 1.)
|
||||
- **Disables + power-down compound** — never do both aggressively in the same area; we already saw the satisfaction hit.
|
||||
- **min-RSSI + power interact** — raising min-RSSI while lowering power can orphan clients; tune one lever at a time.
|
||||
- **Mesh-protected APs** (`2nd Floor Atrium, CC Bridge, salon, 206 U7 Pro, 108`) — never disable; power changes only with watch.
|
||||
|
||||
## 5. Validation methodology (so we KNOW it worked)
|
||||
- **Time-of-day controlled:** compare to the same hour on a prior day, not just before/after minutes (load varies).
|
||||
- Metrics: 2.4 retry% + cu_interf (down), 5 GHz retry on busy APs (down), satisfaction (up, recovering from 30),
|
||||
client band split (5g→6g shift), per-voice-phone retry (Lauren, Dining, rm204/206/210).
|
||||
- Per-phase **GATE**: client counts steady, no AP offline, no new dead zone/complaint. Else rollback that step.
|
||||
- **Rollback:** every `apply-radio`/`apply-wlan` has a rollback JSON in `.claude/tmp/`; `device-control poe-cycle`
|
||||
for a hung AP (NOT force-provision). Power-up/width-80/re-enable are all safe reversals.
|
||||
|
||||
## 6. Out of scope tonight (explicit)
|
||||
- **Floors 5 & 6 (MemCare)** — all RF + the MemCare voice phones (`.217/.218/.219/.220`) DEFERRED per Howard.
|
||||
- Physical cabling / offline switches (Phase 5 — separate visit).
|
||||
- The 6 straggler phones — Howard re-keying separately; they'll benefit from the RF work regardless.
|
||||
|
||||
## 7. Open decisions for Howard
|
||||
1. **5 GHz channel plan:** include clean-DFS (more diversity, tiny resilience risk) vs non-DFS-only (max resilience, more co-channel)? Recommendation: include clean-DFS.
|
||||
2. **QoS depth:** UniFi WMM + DSCP-honor only, or also a pfSense WAN priority queue/limiter for RTP? Recommendation: both (additive).
|
||||
3. **802.11r** on CSCNet: enable for seamless voice roaming, or k/v only (safer for mixed phones)? Recommendation: k/v now, test r on one phone first.
|
||||
4. Tonight's stopping point: Phases 1–2 alone are a legitimate, lower-risk night; 3–4 can be a second night.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Cascades — Voice Quality Diagnostic (post VLAN 30 cutover)
|
||||
|
||||
- **Date:** 2026-06-18 (Howard-Home / claude-main)
|
||||
- **Trigger:** All phones migrated to isolated VOICE VLAN 30 to improve call quality; users report
|
||||
**dropped calls, breaks in voice, reception issues.** This is the RF/quality assessment.
|
||||
- **Data source:** live UniFi controller `/stat/sta` + USW-16-PoE `port_table`, 2026-06-18.
|
||||
|
||||
## Cutover status — COMPLETE
|
||||
31 devices on VOICE (`10.0.30.0/24`): 8 AudioCodes (`.224-.231`), 22 Poly (`.202-.223`), Vertical
|
||||
desktop (`.201`). AudioCodes required a full power-off/on (external-powered; not PoE -> UniFi
|
||||
power-cycle is a no-op) before they re-DHCP'd.
|
||||
|
||||
## Headline finding
|
||||
**The VLAN move gives separation + sets up QoS, but it does NOT by itself fix call quality.** The
|
||||
dropped calls / voice breaks are an **RF problem on the WiFi (Poly) phones.** The wired AudioCodes
|
||||
are clean. Quality fixes are RF + QoS, below.
|
||||
|
||||
## Wired AudioCodes (8) — HEALTHY
|
||||
All USW-16-PoE ports 1-8: up, 100M full-duplex, **rx_err=0 tx_err=0 rx_drop=0.** No network-layer
|
||||
problem. 100M is fine for voice. With VLAN isolation + QoS these desk phones should be solid.
|
||||
|
||||
## WiFi Poly phones — RF problems (retry% = the call-quality killer)
|
||||
Thresholds: retry >10% = audible breaks; RSSI <-67 marginal, <-75 bad; voice wants 5 GHz.
|
||||
|
||||
**SEVERE — fix first:**
|
||||
| Phone (IP) | User/Loc | AP | Band | RSSI | Retry | Issue |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 10.0.30.202 | Lauren / Accounting | CC Bridge | **2.4** | -56 | **50%** | stuck on 2.4 GHz, half packets retransmit |
|
||||
| 10.0.30.218 | Shelby / MemCare Dir | MemCare Nurse Stn | **2.4** | -56 | **53%** | stuck on 2.4 GHz |
|
||||
| 10.0.30.220 | Christine / rm 515 | 517 | 5 | **-82** | 7% | coverage gap (signal near-unusable) |
|
||||
| 10.0.30.219 | Karen Rossini / rm 515 | 517 | 5 | -75 | 16% | weak + high retry |
|
||||
|
||||
**MODERATE:**
|
||||
| 10.0.30.212 | rm 204 | 204 | 5 | -74 | 13% | weak + retry |
|
||||
| 10.0.30.213 | Medtech rm 206 | 206 U7 Pro | 5 | -66 | 13% | 5 GHz congestion |
|
||||
| 10.0.30.214 / .215 | rm 210 | 210 | 5 | -72 | 7-9% | weak |
|
||||
| 10.0.30.206 | Dining Room | Dining Room | 5 | -70 | 9% | borderline |
|
||||
|
||||
**Healthy (reference):** .207/.209/.221/.222/.223 etc. — 5 GHz, RSSI -41 to -60, retry <3%.
|
||||
|
||||
### Three root causes
|
||||
1. **2.4 GHz with ~50% retry** (Lauren .202, Shelby .218) — the single worst issue; matches the
|
||||
documented Cascades 2.4 GHz saturation. **Must force these to 5 GHz.**
|
||||
2. **Coverage gaps** — rooms 515 (-82/-75), 210/204 (-72/-74): too far from the serving AP; weak
|
||||
signal drops calls when RF varies or people move.
|
||||
3. **5 GHz congestion** — several at 13-16% retry on 5 GHz (80 MHz width + channel overlap, per the
|
||||
2026-06-16 audit).
|
||||
|
||||
## Stragglers — 6 Poly phones NOT on VOICE
|
||||
Five on VLAN 20 (`10.0.20.64/.65/.66/.67/.195`) + one on `192.168.1.126`. `10.0.20.66` (Dining
|
||||
Room) is at **35% retry.** Missed in cutover or still on the old PPSK key -> migrate to the voice
|
||||
PPSK so all phones are isolated + benefit from voice QoS.
|
||||
|
||||
## Recommended fixes (prioritized; NONE applied — Cascades requires explicit per-change go)
|
||||
1. **QoS for the voice VLAN (NEW capability the move enables) — highest ROI, lowest risk.** Mark
|
||||
VLAN 30 voice traffic DSCP EF / priority on pfSense + UniFi so voice gets priority under load ->
|
||||
reduces jitter/breaks network-wide.
|
||||
2. **Force voice phones off 2.4 GHz** — on the CSCNet voice PPSK / the APs serving .202 & .218,
|
||||
disable 2.4 GHz association for voice (or band-steer to 5/6 GHz). Fixes Lauren + Shelby (the two
|
||||
worst) immediately.
|
||||
- **DONE 2026-06-18: Lauren `.202` locked to AP 103** (off the CC Bridge wireless-mesh AP -> wired AP). **INTERDEPENDENCY:** AP 103's 5 GHz is saturated (ch149, 75% airtime, ~25,900 retries, 12 clients) -> tonight's 5 GHz plan MUST relieve AP 103 (channel off 149 / 80->40 MHz / load-balance) or she trades a mesh problem for a congestion problem.
|
||||
- Shelby `.218` is floor 5/6 (MemCare) -> **out of scope tonight** per Howard.
|
||||
3. **Coverage** — rooms 515, 210, 204: check AP placement/power; consider a closer AP or raising the
|
||||
nearest AP's power; min-RSSI to push phones off far APs. (Ties into the staged coverage-thin /
|
||||
2.4 remediation runbooks.)
|
||||
4. **Migrate the 6 straggler phones** to the voice PPSK (VLAN 30).
|
||||
5. **5 GHz width/channel** — apply the staged audit recommendation (40 MHz width, non-DFS plan) to
|
||||
cut co-channel retry.
|
||||
|
||||
## Next
|
||||
Discuss + pick changes. QoS (#1) + 2.4 GHz force-off (#2) are the fastest wins for the complaints.
|
||||
Reference in New Issue
Block a user