azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from GURU-5070 at 2026-06-23 12:10:19

Browse Source 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-23 12:10:19
This commit is contained in:
Mike Swanson
2026-06-23 12:11:12 -07:00
parent a28b52da9a
commit 0171107d41
2 changed files with 152 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
clients/dataforth/session-logs/2026-06/2026-06-23-mike-pbx-no-inbound-calls-fix.md Normal file
View File

@@ -0,0 +1,74 @@
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
## Session Summary
Dataforth reported no inbound calls to their main line (emergency). Loaded context: Dataforth runs Sangoma/FreePBX 17 (Asterisk 22.5.2) at 192.168.100.2 on VLAN 100, inbound via a FirstDigital SIP trunk (SBC 66.7.123.215, IP-auth, match 66.7.123.0/24, no registration). Prior work (2026-06-09, #32392) established that inbound depends on a UDM-Pro port-forward (WAN 67.206.163.122 UDP 5060 -> 192.168.100.2:5060, source-locked to FD) persisted only in /data/on_boot.d/30-freepbx-sip-forward.sh and never added as a UI rule.
Confirmed connectivity over the Dataforth OpenVPN (all targets reachable; VPN MTU correctly 1400). Root SSH key worked directly to the UDM (192.168.0.254). Found the smoking gun: the UDM had NO SIP 5060 DNAT rule at all (NO_SIP_DNAT) — flushed by a UniFi provision/update because the rule was never a UI/provision-surviving rule and on_boot.d only re-applies on boot. Verified the persisted script was intact and the correct SIP-only version (RTP forward removed per the 2026-06-09 outbound-audio fix), then re-ran it; verified the DNAT and forward-accept are present. Confirmed the PBX is up and listening on UDP 5060, restoring the full inbound path. Mike later confirmed inbound calls tested OK.
Updated Syncro ticket #32450 (Dataforth Corp): posted a customer-visible resolution comment, added a 1.5h emergency-remote line item (product 26184; 1h x 1.5 emergency premium in the quantity because Dataforth is prepaid with 31.5 block hours), created the invoice (nets $0.00, debits 1.5 -> block 30.0), set the block-hours invoice note, marked the ticket Invoiced, and posted to #bot-alerts.
Cleanup/credential work: investigated the PBX SSH password. The ACG SSH key authenticates as sangoma@192.168.100.2 (operational access fine). I initially mis-diagnosed the vaulted password as "stale" off a sudo test that was mangled by shell-quoting of the special chars, then a de-escape that silently no-op'd; Mike corrected that the password is not stale. Confirmed the correct password is Gptf*77ttb!@#!@# and the vault had a backslash-corrupted copy (Gptf*77ttb\!@#\!@#). Corrected the vault entry via sops set (byte-verified, still encrypted) and pushed. Filed coord to-dos for the durable UDM fix.
## Key Decisions
- **Re-applied the existing on_boot.d SIP-forward script rather than hand-crafting iptables** — it's idempotent, source-locked, and already the corrected SIP-only version; avoids re-introducing the RTP-forward bug that breaks outbound audio.
- **Did NOT forward the RTP range** — hard rule from 2026-06-09: a static RTP DNAT collides with the PBX outbound RTP conntrack and kills outbound-call audio. SIP 5060 only.
- **Emergency billing for a prepaid customer: product 26184 at qty = actual hours x 1.5** (premium in the quantity; invoice nets $0 and debits the block by quantity) per the 2026-05-27 rule.
- **Did not overwrite the vault with a guessed/unverified password** when I (wrongly) thought it was stale — waited for Mike's authoritative value, then set exactly Gptf*77ttb!@#!@#.
- **Filed the durable fix as a coord to-do tagged to clients/dataforth** so it's reliably findable (an earlier untagged to-do wasn't surfacing in the 100-cap list).
## Problems Encountered
- **No inbound calls (root cause):** UDM SIP 5060 DNAT missing — flushed by a UniFi provision; on_boot.d only re-applies on boot. Fixed by re-running the persist script; verified rules present.
- **plink password auth failed to UDM and PBX:** UDM uses keyboard-interactive and the azcomputerguru SSH password path was denied; PBX password rejected. Resolved by using the ACG root SSH key for the UDM and the sangoma SSH key for the PBX.
- **SSH to IX (earlier in day) and rapid retries:** not this incident, but fail2ban/cPHulk sensitivity noted; used pinned host keys / key auth.
- **Mis-diagnosed PBX password as stale:** sudo -S test piped a password full of !@#* through two shell layers (mangled) -> false negative; a base64 "clean" retry also failed, likely because sangoma is in the `sudo` group but not authorized in sudoers (authorization failure, not auth). Mike corrected; the password is correct. Lesson: don't infer credential validity from a sudo test that can fail on authorization or quoting.
- **Vault password backslash-corruption:** stored as Gptf*77ttb\!@#\!@# (shell-escaping leaked into storage). bash `${VAR//\\/}` de-escape silently no-op'd in one attempt; resolved by building the JSON value with jq and `sops set` from inside the vault dir.
- **sops set from wrong dir earlier:** "no creation rules"; must cd into vault root (or use --config). Already logged as friction in a prior session.
## Configuration Changes
- **Dataforth UDM 192.168.0.254:** re-applied `/data/on_boot.d/30-freepbx-sip-forward.sh` — restored `UBIOS_PREROUTING_USER_HOOK` SIP 5060 DNAT (WAN 67.206.163.122, src 66.7.123.0/24 -> 192.168.100.2:5060) + `UBIOS_FORWARD_IN_USER` accept for udp/5060 to the PBX. No file edits; rule re-instantiation only.
- **Vault `clients/dataforth/pbx.sops.yaml`:** corrected `credentials.password` from `Gptf*77ttb\!@#\!@#` to `Gptf*77ttb!@#!@#` (sops set; committed vault 0f503d4).
- **Syncro ticket #32450:** comment 420355206, line item 42980800 (26184, qty 1.5), invoice 1650772809 ($0.00), status Invoiced, invoice note "Block hours remaining: 30."
- **Coord to-dos:** created 5cfa4263 (cryoweave reCAPTCHA, prior task), 1a362d8a (closed — carried wrong "stale credential" framing), 45572ee1 (durable UDM SIP-forward fix, clients/dataforth). (8536155f was created but not findable in the 100-cap list; 45572ee1 supersedes it.)
- **Session log:** this file.
## Credentials & Secrets
- **Dataforth PBX (FreePBX):** `sangoma@192.168.100.2`, password **Gptf*77ttb!@#!@#** (corrected in vault `clients/dataforth/pbx.sops.yaml`). ACG SSH key (`~/.ssh/id_ed25519`) also authenticates as sangoma. NOTE: sangoma is in the `sudo` group but sudo authorization appears not granted on this distro — privileged ops need verification.
- **Dataforth UDM-Pro 192.168.0.254:** root via ACG SSH key (`~/.ssh/id_ed25519`) works directly over the OpenVPN. Vaulted `azcomputerguru` SSH password was rejected by plink (keyboard-interactive) — key is the reliable path. Vault: `clients/dataforth/udm.sops.yaml`.
- No new secrets created.
## Infrastructure & Servers
- **FreePBX:** 192.168.100.2 (pbx.intranet.dataforth.com), Sangoma FreePBX 17 / Asterisk 22.5.2, Debian 12, VLAN 100. Listening UDP 5060. Extensions 201-343.
- **SIP trunk:** FirstDigital, SBC 66.7.123.215:5060 (Sonus), match 66.7.123.0/24, IP-auth (no registration), ignores OPTIONS but answers INVITEs.
- **UDM-Pro:** 192.168.0.254 / 192.168.0.1, UniFi OS. WAN 67.206.163.122. SIP forward persisted at `/data/on_boot.d/30-freepbx-sip-forward.sh` (boot-only re-apply). OpenVPN server, client subnet 192.168.6.0/24 (GURU-5070 = 192.168.6.2).
- **Access path:** GURU-5070 -> Dataforth OpenVPN (adapter MTU pinned 1400) -> 192.168.0.0/24 + 192.168.100.0/24 via 192.168.6.1.
- **D2TESTNAS jump:** 192.168.0.9 (root / Paper123!@#) — not needed this session (direct VPN worked).
## Commands & Outputs
- Confirm missing DNAT: `ssh -i ~/.ssh/id_ed25519 root@192.168.0.254 "iptables -t nat -S | grep -iE '5060|192.168.100.2'"` -> NO_SIP_DNAT.
- Re-apply: `ssh ... root@192.168.0.254 "sh /data/on_boot.d/30-freepbx-sip-forward.sh"`; verify `iptables -t nat -S UBIOS_PREROUTING_USER_HOOK | grep 5060` and `iptables -S UBIOS_FORWARD_IN_USER | grep 192.168.100.2`.
- PBX listening check: `ssh -i ~/.ssh/id_ed25519 sangoma@192.168.100.2 "ss -lun | grep 5060"` -> `0.0.0.0:5060`.
- Syncro emergency prepaid: product 26184, qty = hours x 1.5; invoice nets $0, block debits by qty.
- Vault fix: `cd /d/vault; VAL=$(jq -nc --arg p 'Gptf*77ttb!@#!@#' '$p'); sops set clients/dataforth/pbx.sops.yaml '["credentials"]["password"]' "$VAL"` (SOPS_AGE_KEY_FILE=/c/Users/guru/.config/sops/age/keys.txt).
## Pending / Incomplete Tasks
- **Durable UDM fix (coord 45572ee1):** make the SIP 5060 forward provision-surviving (UI port-forward or a cron/watcher re-running the idempotent script). Needs a maintenance window. SIP 5060 only — never RTP.
- **PBX sudo authorization:** sangoma is in `sudo` group but sudo appears unauthorized — verify how privileged FreePBX ops are intended to run (root console, asterisk group, etc.).
- **cryoweave reCAPTCHA (coord 5cfa4263):** still parked, awaiting Google reCAPTCHA v3 keys (separate task).
## Reference Information
- Syncro ticket: #32450 id 112979216, customer Dataforth Corp 578095. Invoice 1650772809. https://computerguru.syncromsp.com/tickets/112979216
- Coord to-dos: 45572ee1 (durable UDM fix), 5cfa4263 (cryoweave reCAPTCHA).
- Vault: clients/dataforth/pbx.sops.yaml, clients/dataforth/udm.sops.yaml. Vault commit 0f503d4.
- Prior context: clients/dataforth wiki; session-logs/2026-06/2026-06-09-mike-dataforth-freepbx-safesite-forensics.md; ticket #32392.