claudetools/clients/dataforth/session-logs/2026-07/2026-07-14-howard-pbx-inbound-dnat-flush-fix.md

User

• Executed by: ClaudeTools Discord Bot (GURU-BEAST-ROG)
• Requested by: Howard Enos (@howard9645, via Discord) - tech
• Role: automation (acting on the requester's behalf)

Session Summary

Howard reported (Discord, #tech-department) that calls to three Dataforth DIDs — Otto 520-917-2267, Dan 520-917-2269, Georg 520-917-2246 — "did not ring on his end." Loaded Dataforth context (wiki + prior session logs): this matched the known recurring failure where the UDM-Pro's SIP 5060 DNAT (FirstDigital SBC -> PBX 192.168.100.2) exists only in /data/on_boot.d/30-freepbx-sip-forward.sh and gets flushed by any UniFi provision (prior occurrences 2026-06-08 and 2026-06-23).

Reached the PBX from BEAST via the Tailscale subnet router d2testnas (100.85.152.90 advertises 192.168.100.0/24; 192.168.0.0/24 is NOT advertised). OpenSSH key auth and plink password auth both failed from BEAST (plink hangs headless); used paramiko with the vaulted sangoma password. Confirmed Asterisk running (PBX rebooted Jul 13 06:54), trunk endpoint up, an active outbound call in progress — outbound and internal calling unaffected. CDR showed the last inbound trunk call was Jul 13 11:04 AM; zero inbound since, and zero CDR entries from Howard's cell (520-334-5551) that morning — his calls never reached the PBX. A loopback test (originate out the trunk to DID 2267 while tcpdump captured 5060) showed FD returning 480 Temporarily Unavailable with no inbound INVITE ever arriving — FD could not deliver to the WAN.

Accessed the UDM (192.168.0.254) via paramiko double-hop through d2testnas using keyboard-interactive auth with the previously-"stale"-flagged root password Paper123!@#-unifi (it works; vault note was wrong). Confirmed NO_SIP_DNAT, re-ran the intact on_boot.d script at ~09:39 AM AZ, verified DNAT + forward-accept rules restored. Real-world verification followed immediately: 09:39:24 outside caller answered on DID 2246; 09:42 caller to ring group 600; Howard's test call 09:47:16 to ext 246 ANSWERED 40s.

Durable fix: walked Howard through creating a native UniFi UI Port Forwarding policy (FreePBX-SIP, WAN1 67.206.163.122 udp/5060 -> 192.168.100.2:5060, source-Limited). Reviewed his Create Policy screenshot and caught one error before save — source was 66.7.123.0 (single IP / network address, would never match) instead of 66.7.123.0/24. He corrected and saved. Verified on the UDM: UI-generated DNAT + WAN_LAN accept rules present and correct; notably the provision from saving the UI rule wiped the script-applied rules again, re-proving the failure mode. The UI rule is provision-persistent, closing the recurrence (coord to-do 45572ee1's intent). Howard test-called 2246 at 10:28:33 — ANSWERED through the new rule. Outage window: Jul 13 ~11:05 AM to Jul 14 09:39 AM AZ (inbound only).

Also corrected two vault entries via the vault skill and pushed: UDM root SSH password documented as working (keyboard-interactive only), d2testnas root password de-corrupted.

Key Decisions

• Diagnosed before touching anything: loopback originate + tcpdump on the PBX proved the inbound leg died upstream of the PBX (no INVITE arriving), isolating the UDM DNAT rather than FreePBX/trunk config.
• Did NOT forward RTP 10000-20000 — standing hard rule from 2026-06-09 (breaks outbound audio via conntrack collision). SIP 5060 only, source-locked 66.7.123.0/24.
• Durable fix as a native UniFi UI port-forward rule (survives provisions) instead of the proposed cron watcher — Howard created it in the UI; simpler, no custom moving parts. on_boot.d script retained as harmless idempotent backup.
• Caught 66.7.123.0 vs 66.7.123.0/24 in Howard's draft policy before save — a single-IP source (the network address) would have silently never matched and reintroduced the outage at the next flush.
• paramiko over plink/OpenSSH for password SSH from the headless bot — plink hangs waiting on console prompts in this environment; paramiko handles password + keyboard-interactive + direct-tcpip jump cleanly (logged to errorlog as friction).
• Tested the vault-flagged-stale UDM password before escalating — it authenticates via keyboard-interactive; avoided needing Mike's GURU-5070 key or a 2FA web login.

Problems Encountered

• BEAST cannot reach 192.168.0.0/24 (only VLAN 100 is Tailscale-advertised by d2testnas) — solved with a paramiko direct-tcpip double-hop through d2testnas to the UDM.
• plink.exe hangs indefinitely headless (hostkey/password prompts read from console); OpenSSH BatchMode has no password path; BEAST's SSH key not authorized on PBX/d2testnas/UDM. paramiko was the working path.
• Vault data wrong twice: UDM entry claimed Paper123!@#-unifi was stale (it works, keyboard-interactive only); d2testnas password stored backslash-corrupted (Paper123\!@# vs actual Paper123!@#). Both corrected via vault skill and pushed.
• asterisk -rx denied for sangoma (socket owner-writable only) and the "sangoma sudo not authorized" note from 2026-06-23 proved wrong — sudo -S with the password works; the prior failure was shell-quoting mangling.
• Loopback test returned 480 even after the fix — FD's SBC loop-detects calls to one's own DID from the same trunk; real external calls (CDR-verified) were the valid proof.

Configuration Changes

• Dataforth UDM 192.168.0.254: re-applied /data/on_boot.d/30-freepbx-sip-forward.sh (~09:39 AZ) restoring SIP 5060 DNAT + forward-accept. Later (~10:15 AZ) Howard created native UI Port Forwarding policy "FreePBX-SIP": WAN1 (67.206.163.122) udp/5060, From Limited 66.7.123.0/24 -> 192.168.100.2:5060. Verified in iptables (UBIOS_PREROUTING_USER_HOOK DNAT via ipset UBIOS_KEY_ADDRv4_eth8; UBIOS_WAN_LAN_USER ACCEPT). No file changes on the UDM.
• Vault clients/dataforth/udm.sops.yaml: added credentials.ssh_root_password = Paper123!@#-unifi and credentials.ssh_root_password_usage (keyboard-interactive only; prior stale claim wrong).
• Vault clients/dataforth/d2testnas.sops.yaml: corrected credentials.password to Paper123!@# (was backslash-corrupted).
• errorlog.md: friction entry — plink hangs headless; use paramiko.
• Session log: this file. No PBX config changes; no repo code changes.

Credentials & Secrets

• Dataforth PBX: sangoma@192.168.100.2, password in clients/dataforth/pbx.sops.yaml (verified working; sudo -S with same password grants root despite older note).
• Dataforth UDM root SSH: Paper123!@#-unifi — WORKS via keyboard-interactive only (plink -pw and paramiko connect(password=) are refused; use paramiko auth_interactive). Vaulted at clients/dataforth/udm.sops.yaml credentials.ssh_root_password. GURU-5070 id_ed25519 key also authorized.
• d2testnas: root@100.85.152.90 (Tailscale) / 192.168.0.9, password Paper123!@# (vault corrected). BEAST's SSH key NOT authorized on any of these hosts.
• Scratch paramiko scripts containing these passwords were created under the repo root during the session and deleted afterward.

Infrastructure & Servers

• PBX: 192.168.100.2 (+ .196 egress IP on pbxeth0), Sangoma FreePBX 17 / Asterisk, Debian 12, VLAN 100. Rebooted 2026-07-13 06:54.
• Trunk: FirstDigital SBC 66.7.123.215:5060 (Sonus), IP-auth, match 66.7.123.0/24. qualify_frequency=0 (do not change).
• UDM-Pro: 192.168.0.254, WAN 67.206.163.122 (eth8). on_boot.d: 10-neptune-snat.sh, 20-acg-network-isolation.sh, 30-freepbx-sip-forward.sh.
• Access path from BEAST: Tailscale -> d2testnas (100.85.152.90, subnet router for 192.168.100.0/24 ONLY) -> direct-tcpip channel -> 192.168.0.254:22. GURU-5070 (100.81.65.103) has no sshd.
• DID -> extension observed: 520-917-2246 -> ext 246 (Georg). Main ring group 600.

Commands & Outputs

• CDR last-inbound check: mysql -N asteriskcdrdb -e "SELECT ... FROM cdr WHERE channel LIKE 'PJSIP/FirstDigital%' ORDER BY calldate DESC" -> last inbound 2026-07-13 11:04:52 before fix.
• Loopback test: (timeout 30 tcpdump -i any -n 'udp port 5060 and host 66.7.123.215' -w /tmp/siptest.pcap &); asterisk -rx 'channel originate PJSIP/5209172267@FirstDigital application Wait 3' -> 100/183/480, NO inbound INVITE = broken; FD loop-detects own-DID calls so 480 persists even when healthy (use real external calls to verify).
• UDM check/fix: iptables -t nat -S UBIOS_PREROUTING_USER_HOOK | grep 5060 || echo NO_SIP_DNAT -> NO_SIP_DNAT; sh /data/on_boot.d/30-freepbx-sip-forward.sh -> restored.
• sangoma root: echo '<pw>' | sudo -S -p '' whoami -> root.
• paramiko jump pattern: nas.get_transport().open_channel('direct-tcpip',(UDM,22),...) + Transport.auth_interactive(user, handler).

Pending / Incomplete Tasks

• Syncro ticket/billing: UNRESOLVED — asked Howard multiple times (standard vs emergency, ~30-45 min remote, Dataforth prepaid block); no answer by session end. Follow up with Howard/Winter. Prior comparable: #32450 (emergency x1.5, product 26184).
• Coord to-do 45572ee1 (durable UDM SIP-forward fix): effectively DONE via the UI rule — close it in coord after confirming stability.
• Wiki refresh: run /wiki-compile client:dataforth --full to fold in: UI port-forward rule now primary, UDM root password correction, sangoma sudo-works correction, d2testnas Tailscale-advertises-VLAN100-only.

Reference Information

• Discord thread: 1526624583757861049 (#tech-department, Arizona Computer Guru). Requester Howard (@howard9645); Mike supplied Howard's cell (520-334-5551) mid-thread.
• Outage window: 2026-07-13 ~11:05 AM -> 2026-07-14 09:39 AM AZ (inbound only).
• Verification calls: 09:39:24 (725-350-1173 -> 2246, ANSWERED 31s), 09:47:16 (Howard -> 246, ANSWERED 40s), 10:28:33 (Howard -> 246 via UI rule, ANSWERED 17s).
• Prior incidents: session-logs 2026-06-09 (#32392), 2026-06-23 (#32450) in clients/dataforth/session-logs/.
• Vault: clients/dataforth/pbx.sops.yaml, udm.sops.yaml, d2testnas.sops.yaml.

Update: 10:45 AZ — Syncro billing completed

Howard confirmed 30 min standard remote labor. Created ticket #32553 (id 113827474, Dataforth Corp 578095, Phone/VOIP, assigned Howard 1750, attributed via Howard's API key): Initial Issue comment 423681603 (do_not_email), customer-visible Resolution comment 423681605. Line item 43262689 (Labor - Remote Business 1190473, 0.5 hr @ $150, taxable false). Invoice 68044 (id 1651043609) — $0.00, applied 0.5 prepay hrs; block 25.75 -> 24.75 (the other 0.5 debit today was invoice 68043 / ticket 113826861, unrelated work). Invoice note "Block hours remaining: 24.75." Ticket marked Invoiced. Bot alert posted (message 1526653277687648429). Pending-tasks item "Syncro ticket/billing" is now RESOLVED.

Update: 13:01 AZ — Session closed

Howard confirmed nothing else. Ticket #32553 status set to Resolved (via Howard's API key); bot alert posted (message 1526679628931141783). Coord to-do 45572ee1 (durable UDM SIP-forward fix) marked done (completed_by howard, 2026-07-14T20:01:30 UTC) — satisfied by the provision-persistent UniFi UI rule. Coord API note: todo completion is PUT /api/coord/todos/<full-uuid> with {"status":"done"} — short 8-char ids and status "completed" are rejected. Remaining optional item: /wiki-compile client:dataforth --full (operator-run, deliberate).