sync: auto-sync from GURU-BEAST-ROG at 2026-07-14 10:31:06
Author: Mike Swanson Machine: GURU-BEAST-ROG Timestamp: 2026-07-14 10:31:06
This commit is contained in:
ClaudeTools Bot
@@ -0,0 +1,78 @@
|
||||
## User
|
||||
- **Executed by:** ClaudeTools Discord Bot (GURU-BEAST-ROG)
|
||||
- **Requested by:** Howard Enos (@howard9645, via Discord) - tech
|
||||
- **Role:** automation (acting on the requester's behalf)
|
||||
|
||||
## Session Summary
|
||||
|
||||
Howard reported (Discord, #tech-department) that calls to three Dataforth DIDs — Otto 520-917-2267, Dan 520-917-2269, Georg 520-917-2246 — "did not ring on his end." Loaded Dataforth context (wiki + prior session logs): this matched the known recurring failure where the UDM-Pro's SIP 5060 DNAT (FirstDigital SBC -> PBX 192.168.100.2) exists only in `/data/on_boot.d/30-freepbx-sip-forward.sh` and gets flushed by any UniFi provision (prior occurrences 2026-06-08 and 2026-06-23).
|
||||
|
||||
Reached the PBX from BEAST via the Tailscale subnet router d2testnas (100.85.152.90 advertises 192.168.100.0/24; 192.168.0.0/24 is NOT advertised). OpenSSH key auth and plink password auth both failed from BEAST (plink hangs headless); used paramiko with the vaulted sangoma password. Confirmed Asterisk running (PBX rebooted Jul 13 06:54), trunk endpoint up, an active outbound call in progress — outbound and internal calling unaffected. CDR showed the last inbound trunk call was Jul 13 11:04 AM; zero inbound since, and zero CDR entries from Howard's cell (520-334-5551) that morning — his calls never reached the PBX. A loopback test (originate out the trunk to DID 2267 while tcpdump captured 5060) showed FD returning 480 Temporarily Unavailable with no inbound INVITE ever arriving — FD could not deliver to the WAN.
|
||||
|
||||
Accessed the UDM (192.168.0.254) via paramiko double-hop through d2testnas using keyboard-interactive auth with the previously-"stale"-flagged root password `Paper123!@#-unifi` (it works; vault note was wrong). Confirmed `NO_SIP_DNAT`, re-ran the intact on_boot.d script at ~09:39 AM AZ, verified DNAT + forward-accept rules restored. Real-world verification followed immediately: 09:39:24 outside caller answered on DID 2246; 09:42 caller to ring group 600; Howard's test call 09:47:16 to ext 246 ANSWERED 40s.
|
||||
|
||||
Durable fix: walked Howard through creating a native UniFi UI Port Forwarding policy (FreePBX-SIP, WAN1 67.206.163.122 udp/5060 -> 192.168.100.2:5060, source-Limited). Reviewed his Create Policy screenshot and caught one error before save — source was `66.7.123.0` (single IP / network address, would never match) instead of `66.7.123.0/24`. He corrected and saved. Verified on the UDM: UI-generated DNAT + WAN_LAN accept rules present and correct; notably the provision from saving the UI rule wiped the script-applied rules again, re-proving the failure mode. The UI rule is provision-persistent, closing the recurrence (coord to-do 45572ee1's intent). Howard test-called 2246 at 10:28:33 — ANSWERED through the new rule. Outage window: Jul 13 ~11:05 AM to Jul 14 09:39 AM AZ (inbound only).
|
||||
|
||||
Also corrected two vault entries via the vault skill and pushed: UDM root SSH password documented as working (keyboard-interactive only), d2testnas root password de-corrupted.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Diagnosed before touching anything:** loopback originate + tcpdump on the PBX proved the inbound leg died upstream of the PBX (no INVITE arriving), isolating the UDM DNAT rather than FreePBX/trunk config.
|
||||
- **Did NOT forward RTP 10000-20000** — standing hard rule from 2026-06-09 (breaks outbound audio via conntrack collision). SIP 5060 only, source-locked 66.7.123.0/24.
|
||||
- **Durable fix as a native UniFi UI port-forward rule** (survives provisions) instead of the proposed cron watcher — Howard created it in the UI; simpler, no custom moving parts. on_boot.d script retained as harmless idempotent backup.
|
||||
- **Caught `66.7.123.0` vs `66.7.123.0/24`** in Howard's draft policy before save — a single-IP source (the network address) would have silently never matched and reintroduced the outage at the next flush.
|
||||
- **paramiko over plink/OpenSSH for password SSH from the headless bot** — plink hangs waiting on console prompts in this environment; paramiko handles password + keyboard-interactive + direct-tcpip jump cleanly (logged to errorlog as friction).
|
||||
- **Tested the vault-flagged-stale UDM password before escalating** — it authenticates via keyboard-interactive; avoided needing Mike's GURU-5070 key or a 2FA web login.
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **BEAST cannot reach 192.168.0.0/24** (only VLAN 100 is Tailscale-advertised by d2testnas) — solved with a paramiko direct-tcpip double-hop through d2testnas to the UDM.
|
||||
- **plink.exe hangs indefinitely headless** (hostkey/password prompts read from console); OpenSSH BatchMode has no password path; BEAST's SSH key not authorized on PBX/d2testnas/UDM. paramiko was the working path.
|
||||
- **Vault data wrong twice:** UDM entry claimed `Paper123!@#-unifi` was stale (it works, keyboard-interactive only); d2testnas password stored backslash-corrupted (`Paper123\!@#` vs actual `Paper123!@#`). Both corrected via vault skill and pushed.
|
||||
- **`asterisk -rx` denied for sangoma** (socket owner-writable only) and the "sangoma sudo not authorized" note from 2026-06-23 proved wrong — `sudo -S` with the password works; the prior failure was shell-quoting mangling.
|
||||
- **Loopback test returned 480 even after the fix** — FD's SBC loop-detects calls to one's own DID from the same trunk; real external calls (CDR-verified) were the valid proof.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
- **Dataforth UDM 192.168.0.254:** re-applied `/data/on_boot.d/30-freepbx-sip-forward.sh` (~09:39 AZ) restoring SIP 5060 DNAT + forward-accept. Later (~10:15 AZ) Howard created native UI Port Forwarding policy "FreePBX-SIP": WAN1 (67.206.163.122) udp/5060, From Limited 66.7.123.0/24 -> 192.168.100.2:5060. Verified in iptables (UBIOS_PREROUTING_USER_HOOK DNAT via ipset UBIOS_KEY_ADDRv4_eth8; UBIOS_WAN_LAN_USER ACCEPT). No file changes on the UDM.
|
||||
- **Vault `clients/dataforth/udm.sops.yaml`:** added `credentials.ssh_root_password` = Paper123!@#-unifi and `credentials.ssh_root_password_usage` (keyboard-interactive only; prior stale claim wrong).
|
||||
- **Vault `clients/dataforth/d2testnas.sops.yaml`:** corrected `credentials.password` to `Paper123!@#` (was backslash-corrupted).
|
||||
- **errorlog.md:** friction entry — plink hangs headless; use paramiko.
|
||||
- **Session log:** this file. No PBX config changes; no repo code changes.
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- **Dataforth PBX:** sangoma@192.168.100.2, password in `clients/dataforth/pbx.sops.yaml` (verified working; `sudo -S` with same password grants root despite older note).
|
||||
- **Dataforth UDM root SSH:** `Paper123!@#-unifi` — WORKS via keyboard-interactive only (plink -pw and paramiko `connect(password=)` are refused; use paramiko `auth_interactive`). Vaulted at `clients/dataforth/udm.sops.yaml` `credentials.ssh_root_password`. GURU-5070 id_ed25519 key also authorized.
|
||||
- **d2testnas:** root@100.85.152.90 (Tailscale) / 192.168.0.9, password `Paper123!@#` (vault corrected). BEAST's SSH key NOT authorized on any of these hosts.
|
||||
- Scratch paramiko scripts containing these passwords were created under the repo root during the session and deleted afterward.
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **PBX:** 192.168.100.2 (+ .196 egress IP on pbxeth0), Sangoma FreePBX 17 / Asterisk, Debian 12, VLAN 100. Rebooted 2026-07-13 06:54.
|
||||
- **Trunk:** FirstDigital SBC 66.7.123.215:5060 (Sonus), IP-auth, match 66.7.123.0/24. qualify_frequency=0 (do not change).
|
||||
- **UDM-Pro:** 192.168.0.254, WAN 67.206.163.122 (eth8). on_boot.d: 10-neptune-snat.sh, 20-acg-network-isolation.sh, 30-freepbx-sip-forward.sh.
|
||||
- **Access path from BEAST:** Tailscale -> d2testnas (100.85.152.90, subnet router for 192.168.100.0/24 ONLY) -> direct-tcpip channel -> 192.168.0.254:22. GURU-5070 (100.81.65.103) has no sshd.
|
||||
- **DID -> extension observed:** 520-917-2246 -> ext 246 (Georg). Main ring group 600.
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
- CDR last-inbound check: `mysql -N asteriskcdrdb -e "SELECT ... FROM cdr WHERE channel LIKE 'PJSIP/FirstDigital%' ORDER BY calldate DESC"` -> last inbound 2026-07-13 11:04:52 before fix.
|
||||
- Loopback test: `(timeout 30 tcpdump -i any -n 'udp port 5060 and host 66.7.123.215' -w /tmp/siptest.pcap &); asterisk -rx 'channel originate PJSIP/5209172267@FirstDigital application Wait 3'` -> 100/183/480, NO inbound INVITE = broken; FD loop-detects own-DID calls so 480 persists even when healthy (use real external calls to verify).
|
||||
- UDM check/fix: `iptables -t nat -S UBIOS_PREROUTING_USER_HOOK | grep 5060 || echo NO_SIP_DNAT` -> NO_SIP_DNAT; `sh /data/on_boot.d/30-freepbx-sip-forward.sh` -> restored.
|
||||
- sangoma root: `echo '<pw>' | sudo -S -p '' whoami` -> root.
|
||||
- paramiko jump pattern: nas.get_transport().open_channel('direct-tcpip',(UDM,22),...) + Transport.auth_interactive(user, handler).
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **Syncro ticket/billing: UNRESOLVED** — asked Howard multiple times (standard vs emergency, ~30-45 min remote, Dataforth prepaid block); no answer by session end. Follow up with Howard/Winter. Prior comparable: #32450 (emergency x1.5, product 26184).
|
||||
- **Coord to-do 45572ee1 (durable UDM SIP-forward fix):** effectively DONE via the UI rule — close it in coord after confirming stability.
|
||||
- **Wiki refresh:** run `/wiki-compile client:dataforth --full` to fold in: UI port-forward rule now primary, UDM root password correction, sangoma sudo-works correction, d2testnas Tailscale-advertises-VLAN100-only.
|
||||
|
||||
## Reference Information
|
||||
|
||||
- Discord thread: 1526624583757861049 (#tech-department, Arizona Computer Guru). Requester Howard (@howard9645); Mike supplied Howard's cell (520-334-5551) mid-thread.
|
||||
- Outage window: 2026-07-13 ~11:05 AM -> 2026-07-14 09:39 AM AZ (inbound only).
|
||||
- Verification calls: 09:39:24 (725-350-1173 -> 2246, ANSWERED 31s), 09:47:16 (Howard -> 246, ANSWERED 40s), 10:28:33 (Howard -> 246 via UI rule, ANSWERED 17s).
|
||||
- Prior incidents: session-logs 2026-06-09 (#32392), 2026-06-23 (#32450) in clients/dataforth/session-logs/.
|
||||
- Vault: clients/dataforth/pbx.sops.yaml, udm.sops.yaml, d2testnas.sops.yaml.
|
||||
@@ -19,6 +19,8 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
|
||||
|
||||
<!-- Append entries below this line -->
|
||||
|
||||
2026-07-14 | GURU-BEAST-ROG | bash/ssh-tooling | [friction] plink.exe hangs indefinitely in headless bot bash env (both hostkey and password prompts read from console, echo y| insufficient); OpenSSH BatchMode fails fast but no password support. Fix that worked: paramiko (py) for password + keyboard-interactive SSH, incl. jump via direct-tcpip channel [ctx: host=BEAST target=dataforth-pbx/udm]
|
||||
|
||||
2026-07-13 | Howard-Home | bash/env | [friction] local Windows Defender killed /mingw64/bin/curl (Permission denied) mid-session on HOWARD-HOME; user disabled Defender to continue - RMM dispatch loop needs curl; consider Defender exclusion for Git-for-Windows mingw64 bin
|
||||
|
||||
2026-07-13 | Howard-Home | mailbox/link-extraction | picked wrong INKY-wrapped link from Anthropic sign-in email - selected the url8792.mail.anthropic.com tracker (the Contact Support link, lands on support.claude.com) instead of the domain=claude.ai magic link; fix = decode ALL inky blobs and select the claude.ai/magic-link URL [ctx: email=secure-link-login mailbox=mike@azcomputerguru.com]
|
||||
|
||||
Reference in New Issue
Block a user