82 lines
11 KiB
Markdown
82 lines
11 KiB
Markdown
# Dataforth FreePBX restore · Birth Biologic admin reset tooling · Safesite PDF forensics
|
||
|
||
## User
|
||
- **User:** Mike Swanson (mike)
|
||
- **Machine:** GURU-5070
|
||
- **Role:** admin
|
||
|
||
## Session Summary
|
||
|
||
Three threads across two days (2026-06-08 → 06-09). The largest was a **total phone outage at Dataforth** (Sangoma FreePBX 17 / Asterisk 22.5.2 at 192.168.100.2, FirstDigital PJSIP trunk). Outbound was failing with `Could not create dialog to invalid URI 'FirstDigital'` — the trunk contact had gone *Unavailable* because FirstDigital's Sonus SBC stopped answering SIP `OPTIONS` (measured 0/5), and Asterisk 22 refuses to build a call to an Unavailable contact, so no INVITE ever left the box. Fix: set the trunk `qualify_frequency=0` (DB `pjsip` id=1) and re-applied the recurring `PJSip.class.php` line-504 patch (wiped again by the Oct FreePBX update, which had broken `fwconsole reload`). After reload, a test INVITE got `100/183/200` from FirstDigital — outbound restored. Then **inbound** was reported dead too; packet captures proved FD's INVITEs weren't reaching the PBX at all. SSH'd into the Dataforth UDM-Pro (via the D2TESTNAS jump + a root key Mike authorized) and found the root cause: **there was never an inbound SIP port-forward** — inbound had only ever survived on NAT pinholes punched by the qualify-OPTIONS keepalive, which the `qualify=0` fix removed. Added a source-locked (66.7.123.0/24) WAN UDP 5060 + RTP 10000-20000 → 192.168.100.2 DNAT + forward-accept, persisted in `/data/on_boot.d/30-freepbx-sip-forward.sh`. Inbound test calls answered. Ticket #32392 resolved, 1.0 hr emergency remote billed (prepaid ×1.5).
|
||
|
||
Second thread: **Birth Biologic** `operations@` M365 password reset. The plain Graph `passwordProfile` PATCH 403'd because operations@ holds SharePoint+Teams Admin roles (Microsoft protects admin accounts — needs Global/Privileged Authentication Administrator). Mike reset it via the portal. To make admin-account resets programmatic going forward, built `scripts/reset-password.sh` in the remediation-tool skill: JIT-assigns the Tenant Admin SP the Privileged Authentication Administrator role (the app holds `RoleManagement.ReadWrite.Directory`), resets, then de-elevates. Committed + the vaulted UDM creds correction synced to the fleet. Also confirmed operations@ already has all-SharePoint access via its SharePoint Admin role.
|
||
|
||
Third thread: **Safesite (Safe Site Utility Services)** forensic review of a recalled phishing email (`SSUS 06122026.PDF` from m.paris@nexsitepartners.com to 9 recipients). Mail side: all 9 mailboxes clean (recall succeeded), UAL 0 hits, message-trace confirmed delivery-then-recall. Endpoint side: a GuruRMM forensic sweep (per-user Downloads/Outlook-cache/Recent/OneDrive search for the file + Zone.Identifier MotW + browser DL history). First dispatch mostly timed out (Safesite agents are WS-disconnected); re-dispatched with `timeout_seconds=1800`. **7 of 9 machines swept CLEAN; no evidence the PDF was downloaded/opened anywhere.** 2 machines (lennyg DESKTOP-3USU20B, jeremiahw DESKTOP-LOPKB4G) stayed offline → deferred. Opened Syncro #32395 (contact Jonathan Byrd), emailed an initial update, billed 1.5 hr remote (prepaid → 0), and set up a coord auto-followup (todo + fleet broadcast) so any free session completes the last 2 machines and closes out with Jonathan.
|
||
|
||
## Key Decisions
|
||
|
||
- **Dataforth: do NOT revert `qualify=0`.** Reverting would re-break outbound (FD ignores OPTIONS). The inbound problem was the missing UDM port-forward, not our change — proven by timeline (inbound worked 2.5 h after the AM change, then died with the pinhole).
|
||
- **Source-locked the UDM SIP forward to 66.7.123.0/24** (FD's subnet) to keep internet SIP scanners off the PBX.
|
||
- **Persisted UDM rules via `/data/on_boot.d/`** (matching the existing Neptune SNAT pattern) rather than the UI, for reboot survival; recommended Mike add a UI rule afterhours for provision-safe persistence.
|
||
- **Built JIT-elevation password reset** rather than granting the Tenant Admin app a standing Privileged Auth Admin role — minimizes blast radius; the app could already self-elevate via RoleManagement.ReadWrite.Directory, so no new exposure.
|
||
- **Safesite: long timeouts (1800s) for this fleet** — the agents are alive but WS-disconnected (recent last_seen, is_connected=false), so commands must survive until the next reconnect.
|
||
- **Safesite followup via coord todo + fleet broadcast, not a cloud routine** — the work needs internal-network access (GuruRMM 172.16.3.30, vault) that a cloud-scheduled agent can't reach, so a fleet workstation session must run it.
|
||
|
||
## Problems Encountered
|
||
|
||
- **paramiko quoting through nested `sudo bash -c "..."`** truncated Asterisk CLI commands (`asterisk -rx core` → "No such command"). Fixed by uploading scripts via SFTP and running `sudo bash <file>`, or single-quoting inner commands.
|
||
- **UDM SSH auth failures** — vaulted password `Paper123!@#-unifi` was stale; the device SSH wanted `azcomputerguru`/`r3tr0gradE99#` and 2FA. Resolved by tunneling through D2TESTNAS and having Mike add a root SSH key. His first add-key command lost the `>>` redirect (echoed the key instead of writing it); re-issued with `tee -a`.
|
||
- **GuruRMM forensic timeouts** — first sweep failed with "Command timeout" on WS-disconnected agents; fixed with `timeout_seconds=1800` + re-dispatch.
|
||
- **Syncro ticket POST returned empty once** (both #32392 and the recall work) — per skill rule, GET-verified no duplicate before retrying.
|
||
- **Coord todo via raw API returned null** — switched to the `coord` skill's `coord.py` which created it cleanly.
|
||
|
||
## Configuration Changes
|
||
|
||
- `.claude/skills/remediation-tool/scripts/reset-password.sh` — NEW (JIT admin password reset). Mirrored to repo `.claude/skills/...`.
|
||
- `.claude/commands/remediation-tool.md` — documented the JIT password-reset pattern + admin-target caveat.
|
||
- Dataforth PBX `192.168.100.2`: `pjsip` DB id=1 `qualify_frequency` 60→0; `PJSip.class.php` line 504 re-patched (backup `.bak.20260608083954`).
|
||
- Dataforth UDM `192.168.0.254`: `/data/on_boot.d/30-freepbx-sip-forward.sh` — NEW (SIP/RTP DNAT + forward-accept); root key added to `/root/.ssh/authorized_keys`.
|
||
- Vault `clients/dataforth/udm.sops.yaml` — corrected creds (azcomputerguru/r3tr0gradE99#), added console_ssh_user + notes (committed 880761d).
|
||
- `clients/safesite/session-logs/2026-06-08-safesite-nexsite-pdf-forensics.md` — NEW (Safesite forensic log).
|
||
|
||
## Credentials & Secrets
|
||
|
||
- **Dataforth UDM** `192.168.0.254`: SSH `azcomputerguru` / `r3tr0gradE99#`; console user `root` (ACG via root SSH key over D2TESTNAS jump); web `azcomputerguru` / `r3tr0gradE99#`. 2FA push. Vault: `clients/dataforth/udm.sops.yaml`.
|
||
- **Dataforth PBX** `192.168.100.2`: `sangoma` / `Gptf*77ttb!@#!@#`. Vault: `clients/dataforth/pbx.sops.yaml`.
|
||
- **D2TESTNAS** `192.168.0.9`: `root` / `Paper123!@#` (jump host). Vault: `clients/dataforth/d2testnas.sops.yaml`.
|
||
- **Birth Biologic** tenant `birthbiologic.com` (19a568e8-9e88-413b-9341-cbc224b39145 via openid; tenant-admin app 709e6eed). operations@ id `d9a0a1af-d216-4cc0-929a-3170573f7dd5`, new password set by Mike in portal (C@lmOp$26).
|
||
|
||
## Infrastructure & Servers
|
||
|
||
- **Dataforth FreePBX:** 192.168.100.2 (Sangoma FreePBX 17 / Asterisk 22.5.2). Trunk FirstDigital, SBC 66.7.123.215:5060 (Sonus), match 66.7.123.0/24, IP-auth (no registration). Public IP 67.206.163.122 (eth8 on UDM). FD ignores OPTIONS but answers INVITEs.
|
||
- **Dataforth UDM-Pro:** 192.168.0.254 / 192.168.0.1, UniFi OS 5.1.15. WAN eth8 67.206.163.122/29 + 67.206.163.124/32 (Neptune). Port-forwards in mongo `ace.portforward` (Exchange→172.16.3.11; new SIP via on_boot.d).
|
||
- **Safesite:** M365 `safesitellc.com` (71b4e637-c802-4137-a812-ae50dbc839e3). GuruRMM client `fe17552f-736b-42ec-86a2-0e6f107f2397` (sites Bell/Glendale/Unknown), ~28 agents, all WS-disconnected. GuruRMM API `http://172.16.3.30:3001`.
|
||
|
||
## Commands & Outputs
|
||
|
||
- UDM jump+key SSH: paramiko Transport over D2TESTNAS `direct-tcpip` channel to 192.168.0.254:22, `auth_publickey('root', ~/.ssh/id_ed25519)`.
|
||
- UDM SIP forward: `iptables -t nat -A UBIOS_PREROUTING_USER_HOOK -d 67.206.163.122/32 -s 66.7.123.0/24 -p udp --dport 5060 -j DNAT --to-destination 192.168.100.2:5060` (+ RTP 10000:20000, + `UBIOS_FORWARD_IN_USER` ACCEPTs).
|
||
- Safesite forensic sweep: PowerShell searching `C:\Users\*\{Downloads,Desktop,Documents,AppData\Local\Microsoft\Windows\INetCache\Content.Outlook,Temp,Recent,OneDrive*}` for `*06122026*`, reads `-Stream Zone.Identifier`, scans browser `History` DBs; emits `{host,hitCount,hits}`. Result: 7/9 hitCount=0.
|
||
|
||
## Pending / Incomplete Tasks
|
||
|
||
- **Safesite #32395:** sweep DESKTOP-3USU20B (lennyg, enrolled, cmd 1cf8dfea queued) and DESKTOP-LOPKB4G (jeremiahw, NOT enrolled) when online; then email Jonathan final findings + mark coord todo `5766a59f` done. (Auto-followup coordinated: todo 5766a59f + broadcast faaec0ce.)
|
||
- **Dataforth #32392:** Mike to add the UI port-forward afterhours (on_boot.d covers reboots meanwhile). Re-apply `PJSip.class.php` patch after any future `fwconsole ma updateall`.
|
||
- **Birth Biologic:** validate `reset-password.sh` end-to-end on next real admin reset.
|
||
|
||
## Reference Information
|
||
|
||
- **Syncro tickets:** #32392 (Dataforth FreePBX, Resolved, 1.0h emergency remote, prepaid 34.5→33.0). #32395 (Safesite forensics, In Progress, 1.5h remote, prepaid 1.5→0; contact Jonathan Byrd 3458770; invoice 1650620815).
|
||
- **Commits:** reset-password.sh + doc (31e5cbd3); vault UDM creds (880761d).
|
||
- **Coord:** todo `5766a59f-0ddf-43d8-b16b-1c60024a3c04`; broadcast `faaec0ce-ed5f-4e0f-8693-904a3d000c38`.
|
||
- **Artifacts on GURU-5070:** `~/Downloads/safesite-recall-proof.json`, `~/Downloads/safesite-forensic-results.txt`.
|
||
- **Forensic cmd ids (Safesite re-dispatch):** 86340d9b, 8d3e6530, 9aa25e67, 1cf8dfea, 3322e787, 16b2a2b1.
|
||
|
||
## Update — Dataforth outbound no-audio (RTP forward removed)
|
||
|
||
After the inbound fix, outbound calls connected but had **no audio** (FirstDigital confirmed they saw no audio sent from us). Packet capture: FD→PBX RTP flowing, **PBX→FD RTP = 0**. Root cause: the static **RTP port-forward** I'd added on the UDM (WAN UDP 10000-20000 → 192.168.100.2) created an inbound-initiated conntrack that **collided with the PBX's outbound RTP** to the same ports — inbound RTP arrived via the DNAT (one-way audio), outbound RTP was dropped.
|
||
|
||
**Fix:** removed the RTP DNAT + forward-accept from the UDM; kept **only** the SIP 5060 forward. Media now flows both ways via **symmetric RTP pinholes** (standard FreePBX-behind-NAT). Verified: outbound call answered **59s with two-way audio**. `/data/on_boot.d/30-freepbx-sip-forward.sh` rewritten to SIP-only with a warning comment.
|
||
|
||
**RULE: do NOT port-forward the RTP range for this trunk — it breaks outbound audio. SIP 5060 forward only.** (Supersedes the RTP-forward line in Configuration Changes above.) Logged on #32392.
|