sync: Add Yealink tools and session log for 2026-02-24/25
Session covering YMCS setup, Yealink phone scanner tool development, and Peaceful Spirit UCG Ultra speed diagnostics (ECM crash-loop, Cox plant issue). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
218
session-logs/2026-02-24-session.md
Normal file
218
session-logs/2026-02-24-session.md
Normal file
@@ -0,0 +1,218 @@
|
||||
# Session Log: 2026-02-24
|
||||
|
||||
## Session Summary
|
||||
|
||||
Two major topics covered this session:
|
||||
|
||||
### 1. Yealink YMCS Setup & Phone Scanner Tool
|
||||
Set up Yealink Management Cloud Service (YMCS) for managing phones across ACG clients. Built a PowerShell scanner tool to discover Yealink phones on client networks and extract serial numbers for RPS/YMCS registration.
|
||||
|
||||
### 2. Peaceful Spirit (Country Club) - UCG Ultra Speed Issues
|
||||
Diagnosed severe speed degradation on a Cox 300/30 circuit behind a Unifi Cloud Gateway Ultra. Root cause identified as ECM hardware offload engine crash-looping combined with Suricata IDS/IPS on High consuming excessive CPU.
|
||||
|
||||
---
|
||||
|
||||
## Topic 1: Yealink YMCS Setup
|
||||
|
||||
### What Was Accomplished
|
||||
- Reviewed YMCS dashboard structure: Arizona Computer Guru LLC org with sites VWP and GuruHQ
|
||||
- Confirmed YMCS pass-through/relay provisioning works - YMCS redirects phones to PacketDials for SIP config
|
||||
- Two phones already online in YMCS:
|
||||
- **ACG Test Phone**: MAC `805ec097dacf`, SIP-T46S, firmware 66.86.0.15, IP 172.16.1.58
|
||||
- **Winter**: MAC `805e0c08fefa`, SIP-T46S, firmware 66.86.0.15, IP 172.16.1.29
|
||||
- YMCS Site Configuration (GuruHQ) already has relay config to PacketDials:
|
||||
```
|
||||
auto_provision.pnp_enable=1
|
||||
auto_provision.power_on=1
|
||||
auto_provision.repeat.enable=1
|
||||
auto_provision.repeat.minutes=30
|
||||
auto_provision.server.password=********
|
||||
auto_provision.server.url=ftp://p.packetdials.net
|
||||
auto_provision.server.username=lrshwh
|
||||
firmware.url=ftp://p.packetdials.net
|
||||
static.zero_touch.enable=1
|
||||
```
|
||||
|
||||
### Migration Plan (wlcomm to OIT VoIP)
|
||||
- YMCS acts as relay/pass-through to provider's provisioning server
|
||||
- When ready: change `auto_provision.server.url` in YMCS site config from PacketDials to OIT
|
||||
- Push config, phones re-provision from OIT on next check-in (every 30 min) or reboot
|
||||
- Each client in PacketDials/Whitelabel has shared device password, username always `admin`
|
||||
|
||||
### Winter Phone SIP Details (for reference)
|
||||
- SIP Server: `computerguru.voip.packetdials.net`
|
||||
- Username: `5f54f3c8b216`
|
||||
- Password: `3eb7d67260efe017`
|
||||
- Transport: DNS NAPTR
|
||||
- Expires: 360
|
||||
- Assigned to: Winter Williams
|
||||
- E911: (520) 304-8300 - 7437 E 22...
|
||||
- Line Keys: Device (Winter), Park 1-4 (*31-*34), BLF Mike (7003), BLF Rob (7007), Speed Dial Mike-Cell (1-520-289-1912), Howard-Cell (1-520-585-1310), Rob-Cell (1-520-303-6791)
|
||||
|
||||
### Yealink Phone Scanner Tool
|
||||
Built `tools/Scan-YealinkPhones.ps1` - PowerShell script to scan subnets for Yealink phones.
|
||||
|
||||
**What works:**
|
||||
- Ping sweep using .NET SendPingAsync (parallel batches)
|
||||
- ARP table + Get-NetNeighbor parsing to find Yealink MACs
|
||||
- Yealink OUI prefixes: `80:5E:C0`, `80:5E:0C`, `80:5A:35`, `00:15:65`, `28:6D:97`, `24:4B:FE`
|
||||
- SSL certificate bypass for self-signed certs
|
||||
- Unsafe header parsing for Yealink's non-standard HTTP responses
|
||||
- CSV output with append capability
|
||||
|
||||
**What doesn't work (yet):**
|
||||
- Serial number extraction from web UI - Yealink T46S firmware 66.86.0.15 uses RSA+AES encrypted login
|
||||
- Login flow: AES-128-CBC encrypts password (with random prefix + JSESSIONID), RSA encrypts AES key/IV
|
||||
- Implemented the crypto in PowerShell but got error -3 (authentication format mismatch)
|
||||
- The JS crypto uses CryptoJS AES with ZeroPadding + custom RSA (pkcs1pad2)
|
||||
- Issue likely related to session/nonce handling
|
||||
|
||||
**Alternative approaches tried:**
|
||||
- SSDP/UPnP discovery: No response from Yealink phones
|
||||
- SNMP (community: public): No response
|
||||
- Digest auth on cgiServer.exx: 401 (auth not accepted)
|
||||
- Various API endpoints: All return login page or 403
|
||||
|
||||
**Backup tool created:** `tools/yealink-serial-scanner.html` - Browser-based scanner that uses the phone's own JavaScript crypto. Not yet tested.
|
||||
|
||||
**Recommended approach:** Yealink IP Discovery Tool (official tool, not publicly available - request from Yealink distributor or check YMCS Resources section)
|
||||
|
||||
### Files Created/Modified
|
||||
- `tools/Scan-YealinkPhones.ps1` - Main scanner script
|
||||
- `tools/test-yealink.ps1` - Debug/test script (can be deleted)
|
||||
- `tools/yealink-serial-scanner.html` - Browser-based scanner (backup approach)
|
||||
|
||||
### Credentials
|
||||
- GuruHQ Yealink phone web UI: admin / b4e765c3
|
||||
- PacketDials provisioning: username `lrshwh` (password masked in YMCS)
|
||||
- YMCS RPS example serial: `3146019091637071` (ACG Test Phone)
|
||||
|
||||
---
|
||||
|
||||
## Topic 2: Peaceful Spirit Country Club - UCG Ultra Speed Issues
|
||||
|
||||
### Problem
|
||||
Cox 300/30 Mbps circuit delivering 1 Mbps download with hardware acceleration ON + auto MSS clamping. Was working at full speed a few days prior.
|
||||
|
||||
### Equipment
|
||||
- **Gateway:** Unifi Cloud Gateway Ultra (UCG-PST-CC)
|
||||
- **Firmware:** UniFi OS 5.0.12, Network 10.1.85 (Official channel, auto-update ON)
|
||||
- **Kernel:** 5.4.213-ui-ipq5322 (aarch64)
|
||||
- **WAN:** eth4, 2500 Mbps full duplex to Cox modem
|
||||
- **VPN:** WireGuard site-to-site (wgsts1000, MTU 1420) + tun1 (Teleport)
|
||||
- **Cox IP:** 98.190.129.150 (wsip-98-190-129-150.ph.ph.cox.net)
|
||||
- **LAN:** 192.168.0.0/24
|
||||
- **Modem:** New, replaced day before session
|
||||
|
||||
### Test Results
|
||||
| Configuration | Download | Upload |
|
||||
|--------------|----------|--------|
|
||||
| HW accel ON + Auto MSS | ~1 Mbps | 29 Mbps |
|
||||
| HW accel ON + MSS 1300 | 28 Mbps | 29 Mbps |
|
||||
| HW accel OFF + Auto MSS | 28 Mbps | 22 Mbps |
|
||||
| HW accel ON + MSS 1452 | <1 Mbps | - |
|
||||
| HW accel ON + MSS disabled | <2 Mbps | - |
|
||||
| Later (no changes) | 150 Mbps | - |
|
||||
| Later (no changes) | 271 Mbps | - |
|
||||
|
||||
### Root Cause Analysis (via SSH)
|
||||
1. **Suricata IDS/IPS running on HIGH** - consuming 20.3% RAM (614MB), forcing all traffic through CPU
|
||||
2. **ECM hardware offload NOT loaded** - `lsmod | grep ecm` returned empty; ECM is disabled when IDS/IPS is active
|
||||
3. **ECM was crash-looping** in dmesg - repeated `ECM exit / ECM init` cycles
|
||||
4. **MSS clamping rules only apply to tun1 (VPN)**, NOT to WAN (eth4) - UI MSS setting had no effect on WAN traffic
|
||||
5. **QUIC reassembly failures** in dmesg: `[quic_sm_reassemble_func#1025]: failed to allocate reassemble cont.`
|
||||
6. **WAN link flapped** - eth4 went down/up during the session period
|
||||
|
||||
### Key Finding
|
||||
MSS clamping in the Unifi UI was a red herring - iptables showed MSS rules only on `tun1`, not `eth4`. The real issue was Suricata on High preventing hardware offload, combined with ECM instability.
|
||||
|
||||
### Resolution
|
||||
Speed recovered to 271 Mbps without making changes - likely ECM crash loop resolved itself. Monitoring recommended.
|
||||
|
||||
### Recommendations
|
||||
- Consider switching IDS/IPS from High to Medium/Low for better throughput
|
||||
- Monitor for ECM crash recurrence
|
||||
- If speeds drop again, reboot UCG Ultra to reset ECM state
|
||||
- Keep SSH key in place for future diagnostics
|
||||
|
||||
### SSH Access
|
||||
- **Host:** 192.168.0.10 (via VPN) or 98.190.129.150 (WAN)
|
||||
- **User:** root (also requires password via GUI-added key)
|
||||
- **Key:** `~/.ssh/ucg_peaceful_spirit` (ed25519)
|
||||
- **Public key:** `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBw+BK25MXpm91XBtDsSp7K0nTcKwFDLFZDx7tAO/N8 claude@claudetools`
|
||||
- **Note:** Key was added via Unifi GUI; SSH still prompts for password in addition to key
|
||||
|
||||
### Infrastructure
|
||||
- UCG Ultra hostname: UCG-PST-CC
|
||||
- WAN interface: eth4 (NOT eth0)
|
||||
- LAN interfaces: eth0-eth3 on switch0, br0
|
||||
- VPN: wgsts1000 (WireGuard site-to-site), tun1 (Teleport)
|
||||
|
||||
---
|
||||
|
||||
## MSS Clamping Reference (Cox Cable)
|
||||
- Cox uses standard DOCSIS, MTU 1500, no PPPoE
|
||||
- Standard MSS: 1460 (1500 - 20 IP - 20 TCP)
|
||||
- With IPsec VPN: ~1390-1400
|
||||
- With WireGuard: 1420
|
||||
- UCG Ultra max MSS input: 1452
|
||||
|
||||
---
|
||||
|
||||
## Pending/Incomplete Tasks
|
||||
|
||||
### Yealink YMCS
|
||||
- [ ] Get Yealink IP Discovery Tool from distributor (for serial number extraction)
|
||||
- [ ] Test browser-based scanner (`tools/yealink-serial-scanner.html`) as fallback
|
||||
- [ ] Onboard remaining phones across all client sites into YMCS
|
||||
- [ ] Build OIT VoIP config templates in YMCS when ready for migration
|
||||
- [ ] Clean up test files (`tools/test-yealink.ps1`)
|
||||
|
||||
### Peaceful Spirit
|
||||
- [ ] Monitor UCG Ultra speed stability over coming days
|
||||
- [ ] If speeds drop again, consider IDS/IPS High -> Medium/Low
|
||||
- [ ] Investigate why GUI-added SSH key still requires password
|
||||
- [ ] Consider disabling auto-update on UCG to prevent firmware regressions
|
||||
|
||||
---
|
||||
|
||||
## Update: 2026-02-25 Follow-up
|
||||
|
||||
### Peaceful Spirit - Continued Degradation
|
||||
|
||||
After initial recovery to 278 Mbps (HW accel ON, auto MSS), speeds dropped back to 1 Mbps within minutes. ECM confirmed crash-looping again via SSH dmesg — cycling every ~6 minutes (init -> run -> exit -> repeat).
|
||||
|
||||
### IDS/IPS Disabled
|
||||
- Switched IDS/IPS from High to disabled entirely
|
||||
- Speed still unstable: initial 200+ Mbps then **decays to ~70 Mbps under sustained load**
|
||||
- This speed decay pattern (burst then drop) indicates external plant issue, not gateway
|
||||
|
||||
### Conclusion: Cox Plant Issue
|
||||
- ECM crash-looping is a SYMPTOM, not the cause
|
||||
- Gateway offload engine crashing because it's receiving corrupted/incomplete frames from modem
|
||||
- Speed decay under sustained load consistent with:
|
||||
- Upstream noise/ingress causing CMTS power level adjustments
|
||||
- Overheating or failing amplifier in plant
|
||||
- Partial bonding failure (marginal channels dropping under load)
|
||||
- T3 timeouts accumulating as modem loses sync on noisy channels
|
||||
- **Cox tech dispatched** — needs line tech with meter at the tap
|
||||
|
||||
### Summary Provided to Cox Tech
|
||||
- 300/30 circuit delivering 70-200 Mbps (intermittent drops to <1 Mbps)
|
||||
- 50% packet loss at all packet sizes
|
||||
- New modem (replaced day prior), same issue
|
||||
- Speed starts 200+ then decays to 70 under sustained load
|
||||
- Download severely impacted, upload less affected = downstream RF/signal issue
|
||||
- Need tech to check: downstream SNR, power levels, uncorrectable codewords, T3/T4 timeouts, physical plant, RF ingress
|
||||
|
||||
---
|
||||
|
||||
## Files Reference
|
||||
- `tools/Scan-YealinkPhones.ps1` - Yealink phone subnet scanner
|
||||
- `tools/test-yealink.ps1` - Debug script (temporary)
|
||||
- `tools/yealink-serial-scanner.html` - Browser-based serial scanner
|
||||
- `~/.ssh/ucg_peaceful_spirit` - SSH key for Peaceful Spirit UCG Ultra
|
||||
- `C:\temp\phones.csv` - Scanner output (test data)
|
||||
- `C:\temp\yealink_common.js` - Yealink phone JS (for crypto analysis)
|
||||
- `C:\temp\yealink_login.js` - Yealink login JS
|
||||
- `C:\temp\yealink_loginform.txt` - Login form response dump
|
||||
Reference in New Issue
Block a user