claudetools/session-logs/2026-02-24-session.md

# Session Log: 2026-02-24

## Session Summary

Two major topics covered this session:

### 1. Yealink YMCS Setup & Phone Scanner Tool
Set up Yealink Management Cloud Service (YMCS) for managing phones across ACG clients. Built a PowerShell scanner tool to discover Yealink phones on client networks and extract serial numbers for RPS/YMCS registration.

### 2. Peaceful Spirit (Country Club) - UCG Ultra Speed Issues
Diagnosed severe speed degradation on a Cox 300/30 circuit behind a Unifi Cloud Gateway Ultra. Root cause identified as ECM hardware offload engine crash-looping combined with Suricata IDS/IPS on High consuming excessive CPU.

---

## Topic 1: Yealink YMCS Setup

### What Was Accomplished
- Reviewed YMCS dashboard structure: Arizona Computer Guru LLC org with sites VWP and GuruHQ
- Confirmed YMCS pass-through/relay provisioning works - YMCS redirects phones to PacketDials for SIP config
- Two phones already online in YMCS:
  - **ACG Test Phone**: MAC `805ec097dacf`, SIP-T46S, firmware 66.86.0.15, IP 172.16.1.58
  - **Winter**: MAC `805e0c08fefa`, SIP-T46S, firmware 66.86.0.15, IP 172.16.1.29
- YMCS Site Configuration (GuruHQ) already has relay config to PacketDials:
  ```
  auto_provision.pnp_enable=1
  auto_provision.power_on=1
  auto_provision.repeat.enable=1
  auto_provision.repeat.minutes=30
  auto_provision.server.password=********
  auto_provision.server.url=ftp://p.packetdials.net
  auto_provision.server.username=lrshwh
  firmware.url=ftp://p.packetdials.net
  static.zero_touch.enable=1
  ```

### Migration Plan (wlcomm to OIT VoIP)
- YMCS acts as relay/pass-through to provider's provisioning server
- When ready: change `auto_provision.server.url` in YMCS site config from PacketDials to OIT
- Push config, phones re-provision from OIT on next check-in (every 30 min) or reboot
- Each client in PacketDials/Whitelabel has shared device password, username always `admin`

### Winter Phone SIP Details (for reference)
- SIP Server: `computerguru.voip.packetdials.net`
- Username: `5f54f3c8b216`
- Password: `3eb7d67260efe017`
- Transport: DNS NAPTR
- Expires: 360
- Assigned to: Winter Williams
- E911: (520) 304-8300 - 7437 E 22...
- Line Keys: Device (Winter), Park 1-4 (*31-*34), BLF Mike (7003), BLF Rob (7007), Speed Dial Mike-Cell (1-520-289-1912), Howard-Cell (1-520-585-1310), Rob-Cell (1-520-303-6791)

### Yealink Phone Scanner Tool
Built `tools/Scan-YealinkPhones.ps1` - PowerShell script to scan subnets for Yealink phones.

**What works:**
- Ping sweep using .NET SendPingAsync (parallel batches)
- ARP table + Get-NetNeighbor parsing to find Yealink MACs
- Yealink OUI prefixes: `80:5E:C0`, `80:5E:0C`, `80:5A:35`, `00:15:65`, `28:6D:97`, `24:4B:FE`
- SSL certificate bypass for self-signed certs
- Unsafe header parsing for Yealink's non-standard HTTP responses
- CSV output with append capability

**What doesn't work (yet):**
- Serial number extraction from web UI - Yealink T46S firmware 66.86.0.15 uses RSA+AES encrypted login
  - Login flow: AES-128-CBC encrypts password (with random prefix + JSESSIONID), RSA encrypts AES key/IV
  - Implemented the crypto in PowerShell but got error -3 (authentication format mismatch)
  - The JS crypto uses CryptoJS AES with ZeroPadding + custom RSA (pkcs1pad2)
  - Issue likely related to session/nonce handling

**Alternative approaches tried:**
- SSDP/UPnP discovery: No response from Yealink phones
- SNMP (community: public): No response
- Digest auth on cgiServer.exx: 401 (auth not accepted)
- Various API endpoints: All return login page or 403

**Backup tool created:** `tools/yealink-serial-scanner.html` - Browser-based scanner that uses the phone's own JavaScript crypto. Not yet tested.

**Recommended approach:** Yealink IP Discovery Tool (official tool, not publicly available - request from Yealink distributor or check YMCS Resources section)

### Files Created/Modified
- `tools/Scan-YealinkPhones.ps1` - Main scanner script
- `tools/test-yealink.ps1` - Debug/test script (can be deleted)
- `tools/yealink-serial-scanner.html` - Browser-based scanner (backup approach)

### Credentials
- GuruHQ Yealink phone web UI: admin / b4e765c3
- PacketDials provisioning: username `lrshwh` (password masked in YMCS)
- YMCS RPS example serial: `3146019091637071` (ACG Test Phone)

---

## Topic 2: Peaceful Spirit Country Club - UCG Ultra Speed Issues

### Problem
Cox 300/30 Mbps circuit delivering 1 Mbps download with hardware acceleration ON + auto MSS clamping. Was working at full speed a few days prior.

### Equipment
- **Gateway:** Unifi Cloud Gateway Ultra (UCG-PST-CC)
- **Firmware:** UniFi OS 5.0.12, Network 10.1.85 (Official channel, auto-update ON)
- **Kernel:** 5.4.213-ui-ipq5322 (aarch64)
- **WAN:** eth4, 2500 Mbps full duplex to Cox modem
- **VPN:** WireGuard site-to-site (wgsts1000, MTU 1420) + tun1 (Teleport)
- **Cox IP:** 98.190.129.150 (wsip-98-190-129-150.ph.ph.cox.net)
- **LAN:** 192.168.0.0/24
- **Modem:** New, replaced day before session

### Test Results
| Configuration | Download | Upload |
|--------------|----------|--------|
| HW accel ON + Auto MSS | ~1 Mbps | 29 Mbps |
| HW accel ON + MSS 1300 | 28 Mbps | 29 Mbps |
| HW accel OFF + Auto MSS | 28 Mbps | 22 Mbps |
| HW accel ON + MSS 1452 | <1 Mbps | - |
| HW accel ON + MSS disabled | <2 Mbps | - |
| Later (no changes) | 150 Mbps | - |
| Later (no changes) | 271 Mbps | - |

### Root Cause Analysis (via SSH)
1. **Suricata IDS/IPS running on HIGH** - consuming 20.3% RAM (614MB), forcing all traffic through CPU
2. **ECM hardware offload NOT loaded** - `lsmod | grep ecm` returned empty; ECM is disabled when IDS/IPS is active
3. **ECM was crash-looping** in dmesg - repeated `ECM exit / ECM init` cycles
4. **MSS clamping rules only apply to tun1 (VPN)**, NOT to WAN (eth4) - UI MSS setting had no effect on WAN traffic
5. **QUIC reassembly failures** in dmesg: `[quic_sm_reassemble_func#1025]: failed to allocate reassemble cont.`
6. **WAN link flapped** - eth4 went down/up during the session period

### Key Finding
MSS clamping in the Unifi UI was a red herring - iptables showed MSS rules only on `tun1`, not `eth4`. The real issue was Suricata on High preventing hardware offload, combined with ECM instability.

### Resolution
Speed recovered to 271 Mbps without making changes - likely ECM crash loop resolved itself. Monitoring recommended.

### Recommendations
- Consider switching IDS/IPS from High to Medium/Low for better throughput
- Monitor for ECM crash recurrence
- If speeds drop again, reboot UCG Ultra to reset ECM state
- Keep SSH key in place for future diagnostics

### SSH Access
- **Host:** 192.168.0.10 (via VPN) or 98.190.129.150 (WAN)
- **User:** root (also requires password via GUI-added key)
- **Key:** `~/.ssh/ucg_peaceful_spirit` (ed25519)
- **Public key:** `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBw+BK25MXpm91XBtDsSp7K0nTcKwFDLFZDx7tAO/N8 claude@claudetools`
- **Note:** Key was added via Unifi GUI; SSH still prompts for password in addition to key

### Infrastructure
- UCG Ultra hostname: UCG-PST-CC
- WAN interface: eth4 (NOT eth0)
- LAN interfaces: eth0-eth3 on switch0, br0
- VPN: wgsts1000 (WireGuard site-to-site), tun1 (Teleport)

---

## MSS Clamping Reference (Cox Cable)
- Cox uses standard DOCSIS, MTU 1500, no PPPoE
- Standard MSS: 1460 (1500 - 20 IP - 20 TCP)
- With IPsec VPN: ~1390-1400
- With WireGuard: 1420
- UCG Ultra max MSS input: 1452

---

## Pending/Incomplete Tasks

### Yealink YMCS
- [ ] Get Yealink IP Discovery Tool from distributor (for serial number extraction)
- [ ] Test browser-based scanner (`tools/yealink-serial-scanner.html`) as fallback
- [ ] Onboard remaining phones across all client sites into YMCS
- [ ] Build OIT VoIP config templates in YMCS when ready for migration
- [ ] Clean up test files (`tools/test-yealink.ps1`)

### Peaceful Spirit
- [ ] Monitor UCG Ultra speed stability over coming days
- [ ] If speeds drop again, consider IDS/IPS High -> Medium/Low
- [ ] Investigate why GUI-added SSH key still requires password
- [ ] Consider disabling auto-update on UCG to prevent firmware regressions

---

## Update: 2026-02-25 Follow-up

### Peaceful Spirit - Continued Degradation

After initial recovery to 278 Mbps (HW accel ON, auto MSS), speeds dropped back to 1 Mbps within minutes. ECM confirmed crash-looping again via SSH dmesg — cycling every ~6 minutes (init -> run -> exit -> repeat).

### IDS/IPS Disabled
- Switched IDS/IPS from High to disabled entirely
- Speed still unstable: initial 200+ Mbps then **decays to ~70 Mbps under sustained load**
- This speed decay pattern (burst then drop) indicates external plant issue, not gateway

### Conclusion: Cox Plant Issue
- ECM crash-looping is a SYMPTOM, not the cause
- Gateway offload engine crashing because it's receiving corrupted/incomplete frames from modem
- Speed decay under sustained load consistent with:
  - Upstream noise/ingress causing CMTS power level adjustments
  - Overheating or failing amplifier in plant
  - Partial bonding failure (marginal channels dropping under load)
  - T3 timeouts accumulating as modem loses sync on noisy channels
- **Cox tech dispatched** — needs line tech with meter at the tap

### Summary Provided to Cox Tech
- 300/30 circuit delivering 70-200 Mbps (intermittent drops to <1 Mbps)
- 50% packet loss at all packet sizes
- New modem (replaced day prior), same issue
- Speed starts 200+ then decays to 70 under sustained load
- Download severely impacted, upload less affected = downstream RF/signal issue
- Need tech to check: downstream SNR, power levels, uncorrectable codewords, T3/T4 timeouts, physical plant, RF ingress

---

## Files Reference
- `tools/Scan-YealinkPhones.ps1` - Yealink phone subnet scanner
- `tools/test-yealink.ps1` - Debug script (temporary)
- `tools/yealink-serial-scanner.html` - Browser-based serial scanner
- `~/.ssh/ucg_peaceful_spirit` - SSH key for Peaceful Spirit UCG Ultra
- `C:\temp\phones.csv` - Scanner output (test data)
- `C:\temp\yealink_common.js` - Yealink phone JS (for crypto analysis)
- `C:\temp\yealink_login.js` - Yealink login JS
- `C:\temp\yealink_loginform.txt` - Login form response dump