claudetools/clients/cascades-tucson/session-logs/2026-06/2026-06-16-howard-vertical-voice-vlan-plan.md

User

• User: Howard Enos (howard)
• Machine: Howard-Home
• Role: tech

Session Summary

Vertical's VoIP tech (Richard Turner, RTurner@vertical.com) reported two problems at Cascades: phone IP addresses drift after reboots, and he cannot reach any phones from the Vertical remote-management desktop (192.168.2.180) to troubleshoot. The session diagnosed the cause, designed a fix, ran live controller + endpoint recon to validate it, produced a cutover runbook, and drafted/sent a vendor email.

Root cause confirmed from the wiki and the UOS controller: when the Cascades network was segmented into VLANs, the voice gear was left split. The wireless Poly phones (OUI 48:25:67) land on VLAN 20 "Internal" (10.0.20.0/24) via the CSCNet SSID, while the wired AudioCodes phones (OUI 00:90:8f, USW-16-PoE ports 1-8) and the Vertical desktop (USW-16-PoE port 16) stayed on the original Default/main LAN (192.168.0.0/22). pfSense blocks main-LAN -> VLAN 20, so the desktop has no path to the wireless phones.

The agreed fix (Mike's direction, refined with Howard) is a dedicated, isolated voice VLAN (VLAN 30, 10.0.30.0/24) holding all phones plus the Vertical desktop: voice gets internet egress but is firewalled off from VLAN 20 / main LAN / PHI, and Vertical's pfSense OpenVPN is scoped to the voice subnet only. A key constraint surfaced: the desktop is statically addressed and ACG has no login to it, so the NIC change to DHCP must be done by Vertical (or via temp access) at cutover.

Controller recon (uos-mongo.sh) revealed CSCNet is a shared PPSK SSID (~230 per-key->network mappings: resident room VLANs, Default, and one phone key -> VLAN 20; 1,190 historical clients). This means the SSID itself must NOT be repointed; phones move at the PPSK level (dedicated voice key recommended over remapping the existing key, since ~70 non-phone devices also appear on VLAN 20). Endpoint recon via GuruRMM (port + SIP probe from CS-SERVER) showed the desktop is RDP-only (not a PBX) and CS-QB (192.168.2.228, labeled "VoIP server") is SMB-only with no SIP response — strongly indicating the phones register to a cloud/hosted PBX, which means no on-prem firewall pinhole is needed.

Deliverables: a full cutover runbook saved to the client docs, and a vendor email (apology + plan + static-IP question) which Howard sent. Execution is pending Richard's confirmation (cloud PBX, desktop static, VPN cert CN) and a scheduled maintenance window.

Key Decisions

• Dedicated voice VLAN (VLAN 30) instead of a pfSense firewall exception from 192.168.2.180 -> 10.0.20.0/24. Rationale: puts the desktop on the same L2 as the phones (direct reach, no routing rule), and isolates vendor-accessible voice gear from PHI (HIPAA) in one move. Howard's framing.
• All voice gear consolidated, including the wired AudioCodes — moving the desktop to VLAN 30 while leaving AudioCodes on the main LAN would break the desktop's current reach to them.
• Move the WiFi Poly phones via a NEW dedicated voice PPSK (not by repointing CSCNet, and not by remapping the existing phone key) because CSCNet is shared by residents/staff and ~70 non-phone devices share VLAN 20.
• Desktop set to DHCP with a reservation (10.0.30.10) rather than a new static, since Vertical (not ACG) must make the in-Windows change and DHCP+reservation is simpler for them.
• Phone IP "locking" was deliberately NOT promised to the vendor (Mike's call) — emphasis is "all on one voice network, reachable from the desktop," since the desktop on-subnet can find a phone even if its IP shifts.
• Scope Vertical's VPN via an OpenVPN Client-Specific-Override (push only 10.0.30.0/24) + per-tunnel-IP firewall rules, rather than widening the shared server, so other VPN users are unaffected. His .ovpn needs no re-export (routes are server-pushed).
• Verify PBX location by asking the vendor AND by our own recon, rather than relying on either alone.

Problems Encountered

• Mike's original "set the desktop to DHCP" step assumed login access; Howard corrected that the desktop is static and ACG has no login. Resolved by making the NIC change a coordinated vendor step (or temp access) and adjusting the email/runbook.
• Initial assumption that the AudioCodes were legacy/out-of-scope was wrong — Richard's follow-up list (00:90:8f MACs) showed they are in scope and must move too. Corrected scope.
• uos-mongo.sh .forEach(printjson) chained after a Type "it" for more pager emitted a harmless SyntaxError tail on large result sets; output was complete and usable, ignored.
• TCP port scan could not see SIP (UDP); added a follow-up UDP SIP OPTIONS probe to close the blind spot. Both desktop and CS-QB returned no SIP reply.

Configuration Changes

No production changes were made this session (planning + read-only recon only). Files created in the repo:

• clients/cascades-tucson/docs/network/voice-vlan-cutover.md — full cutover runbook + recon.
• clients/cascades-tucson/session-logs/2026-06/2026-06-16-howard-vertical-voice-vlan-plan.md — this log.

Credentials & Secrets

• No new credentials created. CSCNet PPSK list (incl. the VLAN-20 phone key) was viewed in the controller config during recon; not exported here. When the voice PPSK is created at cutover, vault it under clients/cascades-tucson/. Existing CSCNet wifi entry: clients/cascades-tucson/wifi-cscnet.sops.yaml.
• Controller/RMM access used existing vaulted creds: infrastructure/uos-server-ssh-key, infrastructure/gururmm-server.sops.yaml.

Infrastructure & Servers

• pfSense 192.168.0.1 (igc1 trunk; routes + ALL DHCP). Native LAN/Default 192.168.0.0/22; VLAN 20 Internal 10.0.20.0/24 (igc1.20); Guest VLAN 50 10.0.50.0/24; OpenVPN Server 192.168.8.0/24.
• Planned: VLAN 30 VOICE 10.0.30.0/24 gw 10.0.30.1 (igc1.30); DHCP 10.0.30.100-.250; desktop reservation 10.0.30.10.
• UOS controller 172.16.3.29, Cascades site 685f39068e65331c46ef6dd2. CSCNet wlanconf 685f39078e65331c46ef7ee5. Networks: Default 685f39078e65331c46ef8ac4, Internal/VLAN20 69405ba36db796548c947130.
• CS-SERVER 192.168.2.254 (GuruRMM agent c39f1de7-d5b6-45ae-b132-e06977ab1713, online) — used as the recon vantage point.
• Vertical-Remote desktop 192.168.2.180, MAC e4:e7:49:52:3a:06, USW-16-PoE port 16 — RDP (3389) only; not a PBX.
• CS-QB 192.168.2.228 (cs-qb.cascades.local), MAC 00:15:5d:02:3b:02 — SMB (445) only, no SIP; labeled "VoIP server" but not a live SIP PBX.
• AudioCodes phones (8): USW-16-PoE ports 1-8, OUI 00:90:8f, currently on Default LAN. Poly phones (22): WiFi via CSCNet, OUI 48:25:67, currently VLAN 20. Full MAC inventory in the runbook appendix.

Commands & Outputs

• bash .claude/scripts/uos-mongo.sh — enumerated phones (user collection: is_wired, last_uplink_name/port, wlanconf_id, network), wlanconf (CSCNet PPSK), networkconf (VLAN list). Confirmed Poly=WiFi/CSCNet/VLAN20, AudioCodes=wired USW-16-PoE/Default.
• GuruRMM dispatch to CS-SERVER (cmd 50eac6c8): TCP probe -> 192.168.2.180 OPEN 3389; 192.168.2.228 OPEN 445; ARP confirmed both live.
• GuruRMM dispatch to CS-SERVER (cmd 37522673): UDP SIP OPTIONS -> SIP-NOREPLY from both 192.168.2.180 and 192.168.2.228.

Pending / Incomplete Tasks

• Awaiting Richard's reply: confirm phones are cloud/hosted PBX (recon says yes); confirm desktop is static + arrange NIC change or temp access; provide his VPN certificate CN for the scoped CSO; agree a maintenance window.
• At execution: build VLAN 30 on pfSense (interface/DHCP/reservations/firewall) + UniFi (network, ports 1-8 + 16 to VOICE, voice PPSK); confirm pfSense DHCP backend (ISC vs Kea); re-key WiFi phones; coordinated desktop move; validate isolation + reachability.
• If Richard reports an on-prem PBX after all, add the Part A step-5b SIP/RTP/provisioning pinhole.
• Note CS-QB "VoIP server" label looks stale (SMB-only) — revisit/clean the topology doc entry.

Reference Information

• Runbook: clients/cascades-tucson/docs/network/voice-vlan-cutover.md
• Vendor: Richard Turner, RTurner@vertical.com (Vertical Communications).
• Wiki article: wiki/clients/cascades-tucson.md. Topology/VLAN docs: clients/cascades-tucson/docs/network/{topology.md,vlans.md} — the "CSCNet = staff/VLAN 20" note is now incomplete (CSCNet is a shared PPSK SSID); flag for /wiki-compile.
• GuruRMM cmd IDs: 50eac6c8-e125-4bb7-b8fb-6d7f05a53c7f (TCP probe), 37522673-514c-43db-a4fc-ea7e52adfb33 (SIP probe).

Update: 19:39 PT — wiki recompile (cascades-tucson --full)

Ran /wiki-compile client:cascades-tucson --full to fold this session's work into the wiki. The skill's default Sonnet subagent path FAILED: the subagent ran ~54 minutes then crashed on the 32k output-token cap (it tried to emit the full ~490-line article despite being told to write-to-file and return only a summary), so it never wrote the staging file. Recovered by integrating directly via surgical Edits to the live article — lossless, preserving all existing Entra Architecture / Patterns / Security / History detail. Logged the friction to errorlog.md (--friction, skill=wiki-compile) so the skill can be fixed (prefer targeted Edit integration over a subagent rewrite for large existing articles).

Integrated into wiki/clients/cascades-tucson.md (+ wiki/index.md):

• Voice VLAN (VLAN 30) project + runbook reference; Active Work pending item; 2026-06-16 History entry.
• CSCNet correction: shared PPSK SSID (~230 per-key->network mappings, ~1,190 historical clients), NOT "staff/VLAN 20"; with the real device attach map (Poly=WiFi/PPSK->VLAN20, AudioCodes=wired USW-16-PoE ports 1-8, desktop=wired port 16/static/no-login).
• PBX recon: desktop RDP-only (not a PBX); CS-QB SMB-only/no SIP (label flagged stale); phones likely cloud-registered.
• CS-SERVER degraded RAID-1 upgraded to [CRITICAL] (data-loss risk) + cloud backup now installed/started (closes the §164.308(a)(7) HIPAA no-backup gap) + Datto agent-sprawl cleanup; 2026-06-15 History entry.
• Frontmatter sources + last_compiled 2026-06-16. Syncro fields unchanged (55.75 hrs, 0 open tickets, 29 assets).

Coord serialize lock claimed (478b2927) then released — note: coord.py lock release takes the lock ID, not the resource path (initial path-based release soft-failed; TTL would have auto-evicted in 1h regardless).

Commits this update: bc0ceea (wiki compile cascades-tucson full) + a follow-up errorlog friction commit. All pushed to origin/main.