## User
- **Executed by:** ClaudeTools Discord Bot (GURU-BEAST-ROG)
- **Requested by:** Howard Enos (@howard9645, via Discord) - tech
- **Role:** automation (acting on the requester's behalf)

## Session Summary

Howard reported (Discord, #tech-department) that calls to three Dataforth DIDs — Otto 520-917-2267, Dan 520-917-2269, Georg 520-917-2246 — "did not ring on his end." Loaded Dataforth context (wiki + prior session logs): this matched the known recurring failure where the UDM-Pro's SIP 5060 DNAT (FirstDigital SBC -> PBX 192.168.100.2) exists only in `/data/on_boot.d/30-freepbx-sip-forward.sh` and gets flushed by any UniFi provision (prior occurrences 2026-06-08 and 2026-06-23).

Reached the PBX from BEAST via the Tailscale subnet router d2testnas (100.85.152.90 advertises 192.168.100.0/24; 192.168.0.0/24 is NOT advertised). OpenSSH key auth and plink password auth both failed from BEAST (plink hangs headless); used paramiko with the vaulted sangoma password. Confirmed Asterisk running (PBX rebooted Jul 13 06:54), trunk endpoint up, an active outbound call in progress — outbound and internal calling unaffected. CDR showed the last inbound trunk call was Jul 13 11:04 AM; zero inbound since, and zero CDR entries from Howard's cell (520-334-5551) that morning — his calls never reached the PBX. A loopback test (originate out the trunk to DID 2267 while tcpdump captured 5060) showed FD returning 480 Temporarily Unavailable with no inbound INVITE ever arriving — FD could not deliver to the WAN.

Accessed the UDM (192.168.0.254) via paramiko double-hop through d2testnas using keyboard-interactive auth with the previously-"stale"-flagged root password `Paper123!@#-unifi` (it works; vault note was wrong). Confirmed `NO_SIP_DNAT`, re-ran the intact on_boot.d script at ~09:39 AM AZ, verified DNAT + forward-accept rules restored. Real-world verification followed immediately: 09:39:24 outside caller answered on DID 2246; 09:42 caller to ring group 600; Howard's test call 09:47:16 to ext 246 ANSWERED 40s.

Durable fix: walked Howard through creating a native UniFi UI Port Forwarding policy (FreePBX-SIP, WAN1 67.206.163.122 udp/5060 -> 192.168.100.2:5060, source-Limited). Reviewed his Create Policy screenshot and caught one error before save — source was `66.7.123.0` (single IP / network address, would never match) instead of `66.7.123.0/24`. He corrected and saved. Verified on the UDM: UI-generated DNAT + WAN_LAN accept rules present and correct; notably the provision from saving the UI rule wiped the script-applied rules again, re-proving the failure mode. The UI rule is provision-persistent, closing the recurrence (coord to-do 45572ee1's intent). Howard test-called 2246 at 10:28:33 — ANSWERED through the new rule. Outage window: Jul 13 ~11:05 AM to Jul 14 09:39 AM AZ (inbound only).

Also corrected two vault entries via the vault skill and pushed: UDM root SSH password documented as working (keyboard-interactive only), d2testnas root password de-corrupted.

## Key Decisions

- **Diagnosed before touching anything:** loopback originate + tcpdump on the PBX proved the inbound leg died upstream of the PBX (no INVITE arriving), isolating the UDM DNAT rather than FreePBX/trunk config.
- **Did NOT forward RTP 10000-20000** — standing hard rule from 2026-06-09 (breaks outbound audio via conntrack collision). SIP 5060 only, source-locked 66.7.123.0/24.
- **Durable fix as a native UniFi UI port-forward rule** (survives provisions) instead of the proposed cron watcher — Howard created it in the UI; simpler, no custom moving parts. on_boot.d script retained as harmless idempotent backup.
- **Caught `66.7.123.0` vs `66.7.123.0/24`** in Howard's draft policy before save — a single-IP source (the network address) would have silently never matched and reintroduced the outage at the next flush.
- **paramiko over plink/OpenSSH for password SSH from the headless bot** — plink hangs waiting on console prompts in this environment; paramiko handles password + keyboard-interactive + direct-tcpip jump cleanly (logged to errorlog as friction).
- **Tested the vault-flagged-stale UDM password before escalating** — it authenticates via keyboard-interactive; avoided needing Mike's GURU-5070 key or a 2FA web login.

## Problems Encountered

- **BEAST cannot reach 192.168.0.0/24** (only VLAN 100 is Tailscale-advertised by d2testnas) — solved with a paramiko direct-tcpip double-hop through d2testnas to the UDM.
- **plink.exe hangs indefinitely headless** (hostkey/password prompts read from console); OpenSSH BatchMode has no password path; BEAST's SSH key not authorized on PBX/d2testnas/UDM. paramiko was the working path.
- **Vault data wrong twice:** UDM entry claimed `Paper123!@#-unifi` was stale (it works, keyboard-interactive only); d2testnas password stored backslash-corrupted (`Paper123\!@#` vs actual `Paper123!@#`). Both corrected via vault skill and pushed.
- **`asterisk -rx` denied for sangoma** (socket owner-writable only) and the "sangoma sudo not authorized" note from 2026-06-23 proved wrong — `sudo -S` with the password works; the prior failure was shell-quoting mangling.
- **Loopback test returned 480 even after the fix** — FD's SBC loop-detects calls to one's own DID from the same trunk; real external calls (CDR-verified) were the valid proof.

## Configuration Changes

- **Dataforth UDM 192.168.0.254:** re-applied `/data/on_boot.d/30-freepbx-sip-forward.sh` (~09:39 AZ) restoring SIP 5060 DNAT + forward-accept. Later (~10:15 AZ) Howard created native UI Port Forwarding policy "FreePBX-SIP": WAN1 (67.206.163.122) udp/5060, From Limited 66.7.123.0/24 -> 192.168.100.2:5060. Verified in iptables (UBIOS_PREROUTING_USER_HOOK DNAT via ipset UBIOS_KEY_ADDRv4_eth8; UBIOS_WAN_LAN_USER ACCEPT). No file changes on the UDM.
- **Vault `clients/dataforth/udm.sops.yaml`:** added `credentials.ssh_root_password` = Paper123!@#-unifi and `credentials.ssh_root_password_usage` (keyboard-interactive only; prior stale claim wrong).
- **Vault `clients/dataforth/d2testnas.sops.yaml`:** corrected `credentials.password` to `Paper123!@#` (was backslash-corrupted).
- **errorlog.md:** friction entry — plink hangs headless; use paramiko.
- **Session log:** this file. No PBX config changes; no repo code changes.

## Credentials & Secrets

- **Dataforth PBX:** sangoma@192.168.100.2, password in `clients/dataforth/pbx.sops.yaml` (verified working; `sudo -S` with same password grants root despite older note).
- **Dataforth UDM root SSH:** `Paper123!@#-unifi` — WORKS via keyboard-interactive only (plink -pw and paramiko `connect(password=)` are refused; use paramiko `auth_interactive`). Vaulted at `clients/dataforth/udm.sops.yaml` `credentials.ssh_root_password`. GURU-5070 id_ed25519 key also authorized.
- **d2testnas:** root@100.85.152.90 (Tailscale) / 192.168.0.9, password `Paper123!@#` (vault corrected). BEAST's SSH key NOT authorized on any of these hosts.
- Scratch paramiko scripts containing these passwords were created under the repo root during the session and deleted afterward.

## Infrastructure & Servers

- **PBX:** 192.168.100.2 (+ .196 egress IP on pbxeth0), Sangoma FreePBX 17 / Asterisk, Debian 12, VLAN 100. Rebooted 2026-07-13 06:54.
- **Trunk:** FirstDigital SBC 66.7.123.215:5060 (Sonus), IP-auth, match 66.7.123.0/24. qualify_frequency=0 (do not change).
- **UDM-Pro:** 192.168.0.254, WAN 67.206.163.122 (eth8). on_boot.d: 10-neptune-snat.sh, 20-acg-network-isolation.sh, 30-freepbx-sip-forward.sh.
- **Access path from BEAST:** Tailscale -> d2testnas (100.85.152.90, subnet router for 192.168.100.0/24 ONLY) -> direct-tcpip channel -> 192.168.0.254:22. GURU-5070 (100.81.65.103) has no sshd.
- **DID -> extension observed:** 520-917-2246 -> ext 246 (Georg). Main ring group 600.

## Commands & Outputs

- CDR last-inbound check: `mysql -N asteriskcdrdb -e "SELECT ... FROM cdr WHERE channel LIKE 'PJSIP/FirstDigital%' ORDER BY calldate DESC"` -> last inbound 2026-07-13 11:04:52 before fix.
- Loopback test: `(timeout 30 tcpdump -i any -n 'udp port 5060 and host 66.7.123.215' -w /tmp/siptest.pcap &); asterisk -rx 'channel originate PJSIP/5209172267@FirstDigital application Wait 3'` -> 100/183/480, NO inbound INVITE = broken; FD loop-detects own-DID calls so 480 persists even when healthy (use real external calls to verify).
- UDM check/fix: `iptables -t nat -S UBIOS_PREROUTING_USER_HOOK | grep 5060 || echo NO_SIP_DNAT` -> NO_SIP_DNAT; `sh /data/on_boot.d/30-freepbx-sip-forward.sh` -> restored.
- sangoma root: `echo '<pw>' | sudo -S -p '' whoami` -> root.
- paramiko jump pattern: nas.get_transport().open_channel('direct-tcpip',(UDM,22),...) + Transport.auth_interactive(user, handler).

## Pending / Incomplete Tasks

- **Syncro ticket/billing: UNRESOLVED** — asked Howard multiple times (standard vs emergency, ~30-45 min remote, Dataforth prepaid block); no answer by session end. Follow up with Howard/Winter. Prior comparable: #32450 (emergency x1.5, product 26184).
- **Coord to-do 45572ee1 (durable UDM SIP-forward fix):** effectively DONE via the UI rule — close it in coord after confirming stability.
- **Wiki refresh:** run `/wiki-compile client:dataforth --full` to fold in: UI port-forward rule now primary, UDM root password correction, sangoma sudo-works correction, d2testnas Tailscale-advertises-VLAN100-only.

## Reference Information

- Discord thread: 1526624583757861049 (#tech-department, Arizona Computer Guru). Requester Howard (@howard9645); Mike supplied Howard's cell (520-334-5551) mid-thread.
- Outage window: 2026-07-13 ~11:05 AM -> 2026-07-14 09:39 AM AZ (inbound only).
- Verification calls: 09:39:24 (725-350-1173 -> 2246, ANSWERED 31s), 09:47:16 (Howard -> 246, ANSWERED 40s), 10:28:33 (Howard -> 246 via UI rule, ANSWERED 17s).
- Prior incidents: session-logs 2026-06-09 (#32392), 2026-06-23 (#32450) in clients/dataforth/session-logs/.
- Vault: clients/dataforth/pbx.sops.yaml, udm.sops.yaml, d2testnas.sops.yaml.

## Update: 10:45 AZ — Syncro billing completed

Howard confirmed 30 min standard remote labor. Created ticket #32553 (id 113827474, Dataforth Corp 578095, Phone/VOIP, assigned Howard 1750, attributed via Howard's API key): Initial Issue comment 423681603 (do_not_email), customer-visible Resolution comment 423681605. Line item 43262689 (Labor - Remote Business 1190473, 0.5 hr @ $150, taxable false). Invoice 68044 (id 1651043609) — $0.00, applied 0.5 prepay hrs; block 25.75 -> 24.75 (the other 0.5 debit today was invoice 68043 / ticket 113826861, unrelated work). Invoice note "Block hours remaining: 24.75." Ticket marked Invoiced. Bot alert posted (message 1526653277687648429). Pending-tasks item "Syncro ticket/billing" is now RESOLVED.
