azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Session log: Tunnel expansion + WHM fix (ix. grey-cloud)

Browse Source 
Audited all 25 proxied zone records and expanded tunnel ingress to cover
9 hostnames total (azcomputerguru + analytics + community + radio +
git + plexrequest + rmm + rmm-api + sync). All verified HTTP 200.

Reverted 3 hostnames to original A records after discovering they
require backend work, not tunnel changes:
- plex/rustdesk: NPM on Jupiter has no vhost for these (returned
  'tls: unrecognized name' when tunneled)
- secure: Jupiter can't route to its backend subnet 172.16.1.0/24

Reverted ix.azcomputerguru.com to DNS-only A record after user
reported :2087 WHM access broken. Cloudflare Tunnel is hostname-bound,
not port-bound, so non-standard admin ports can't pass through. Direct
NAT to 72.194.62.5 restored WHM/cPanel access.

Adds four new helper scripts under clients/internal-infrastructure/
scripts/cloudflared-tunnel-setup/ (audit_proxied, discover_backends,
expand_tunnel, revert_broken). All use SOPS vault / env var for creds.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mike Swanson
2026-04-13 15:59:49 -07:00
parent eb183512c0
commit 02a0de771d
5 changed files with 640 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

167
clients/internal-infrastructure/session-logs/2026-04-13-session.md
View File

@@ -351,3 +351,170 @@ curl -sI -A "Mozilla/5.0 Chrome/120.0" https://azcomputerguru.com/
**Last Updated:** 2026-04-13
**Next Actions:** submit Cox ticket; consider populating Cloudflare vault entry; monitor tunnel for 24h; cleanup misplaced helper scripts.
---
## Update: 15:56 — Tunnel expansion audit + ix.azcomputerguru.com grey-cloud revert
Post-initial-deploy work to assess which other proxied records in the zone would benefit from the tunnel, then fix a regression on WHM access.
### Work done
1. **Audit of all 25 proxied zone records** (`audit_proxied.py`). Classified each by origin:
- Tunneled (4): azcomputerguru.com, analytics, community, radio
- External SaaS (8): msp360, Microsoft, SendGrid, GoDaddy, etc. — not eligible
- Our-origin not-yet-tunneled (9): ix, git, plex, plexrequest, rmm, rmm-api, sync, rustdesk, secure
- Of those 9, 4 were actively broken (ix=521, plex=525, rustdesk=525, secure=ERR) and 5 working (git/plexrequest/rmm/rmm-api/sync=200)
2. **Mapped NAT rules and NPM backends** (`discover_backends.py`):
- pfSense `pfctl -s nat` shows: `.4`, `.9`, `.10` all rdr to `172.16.3.20:18443` (Jupiter NPM)
- `.5 -> 172.16.3.10:443` (IX Apache)
- `.2 -> 172.16.1.16:443` (different subnet; no route from Jupiter)
- NPM_Server pfSense alias resolves to `172.16.3.20` only (single-member)
- Jupiter NPM active config dir: `/mnt/user/appdata/npm/nginx/proxy_host/` (separate from `NginxProxyManager/` which is a stale v1 copy; there's also an empty `NginxProxyManager-v3/`)
- NPM has proxy_host entries for: emby, plexrequest, unifi, git, rmm-api+rmm, sync, connect
- NPM has **NO** entries for: plex, rustdesk, secure -- so routing them to `https://172.16.3.20:18443` with that Host header returned `tls: unrecognized name` (default cert fallback)
3. **Expanded tunnel to 13 hostnames** (`expand_tunnel.py`) via CF DNS API cutovers, then immediately rolled back 3:
- plex/rustdesk -> cloudflared error `Unable to reach the origin service ... remote error: tls: unrecognized name` (NPM returned default cert because no vhost matched). 502 to users.
- secure -> cloudflared error `no route to host` (Jupiter can't reach 172.16.1.16/24). 502 to users.
- All 3 were already broken BEFORE the tunnel (525/525/ERR). No user-visible regression, but not a *fix* either -- reverted their DNS back to original A records.
4. **Final state after `revert_broken.py`: 10 hostnames tunneled, all HTTP 200**:
- azcomputerguru.com, analytics, community, radio, ix, git, plexrequest, rmm, rmm-api, sync
5. **User reported "IX generated blank screen"** -> root cause: `https://ix.azcomputerguru.com:2087/` is the WHM admin URL. Cloudflare Tunnel is hostname-bound, not port-bound; ingress rules route ALL port traffic (Cloudflare normalizes at edge) to the single backend specified (`https://172.16.3.10:443`). So `:2087` -> landed at Apache:443, not WHM:2087. Apache returned the default vhost redirect instead of WHM.
**Fix: grey-clouded `ix.azcomputerguru.com`** (proxied=False) pointing directly to A `72.194.62.5`. pfSense NAT rules for 2087/2083 are intact and route the traffic to IX. Verified:
- `ix.azcomputerguru.com:443` -> 200 (default vhost redirect, fine)
- `ix.azcomputerguru.com:2087` -> 200 (WHM)
- `ix.azcomputerguru.com:2083` -> 200 (cPanel)
Trade-off: `ix.` no longer benefits from CF's DDoS/caching, but it's admin-only access. If the Cox BGP issue resurfaces specifically for traffic to 72.194.62.5 from certain geographies, `ix.azcomputerguru.com:2087` would fail for users in those regions -- but admin access typically comes from your own network which works fine.
### Key decisions & rationale
- **Tunnel ingress reconfigured to 9 hostnames** (dropped ix. after WHM issue surfaced, kept 3-broken removal from earlier). All 9 serve via tunnel, all verified 200.
- **Grey-cloud (DNS-only) rather than tunnel** for `ix.` because port 2087/2083 admin needs can't be satisfied by the tunnel.
- **Not investigated further**: the 3 unfixable hostnames (plex, rustdesk, secure) -- require NPM vhost additions and/or Jupiter routing changes, beyond today's tunnel scope. Captured as follow-ups.
### Problems encountered and resolutions
| Problem | Resolution |
|---|---|
| plex/rustdesk = 502 (`tls: unrecognized name`) | NPM has no vhost for these hostnames; it returned default cert. Reverted DNS to original A records (no worse than pre-tunnel state). |
| secure = 502 (`no route to host`) | Jupiter (172.16.3.20) can't route to 172.16.1.16 (different subnet). Reverted DNS. |
| WHM blank screen (`:2087`) | Tunnel is hostname-only, can't preserve non-standard ports. Grey-clouded `ix.` so direct NAT handles the admin ports. |
| Tailscale stopped mid-session (again) | User re-enabled after prompt; resumed. |
| Unicode arrow character crashed Python print on Windows cp1252 | Re-ran verify with ASCII chars. Harmless -- DNS/tunnel changes had already succeeded. |
---
## Credentials (unchanged from this session)
Same set as the earlier 2026-04-13 entry above:
- Cloudflare Full DNS token: `DRRGkHS33pxAUjQfRDzDeVPtt6wwUU6FwtXqOzNj`
- Cloudflare Legacy token: `U1UTbBOWA4a69eWEBiqIbYh0etCGzrpTU4XaKp7w`
- Zone ID: `1beb9917c22b54be32e5215df2c227ce`
- Jupiter: `root / Th1nk3r^99##` at 172.16.3.20:22
- IX: `root / Gptf*77ttb!@#!@#` at 172.16.3.10:22 (public 72.194.62.5)
- pfSense: `admin / r3tr0gradE99!!` at 172.16.0.1:2248
---
## DNS changes summary (all of 2026-04-13)
| Hostname | Before session | After session |
|---|---|---|
| azcomputerguru.com | A 72.194.62.5 (mis-configured as proxied=False) | CNAME tunnel proxied |
| analytics.azcomputerguru.com | A 72.194.62.5 proxied | CNAME tunnel proxied |
| community.azcomputerguru.com | A 72.194.62.5 proxied | CNAME tunnel proxied |
| radio.azcomputerguru.com | A 72.194.62.5 proxied | CNAME tunnel proxied |
| ix.azcomputerguru.com | A 72.194.62.5 proxied | **A 72.194.62.5 DNS-only (grey cloud)** (supports :2087/:2083) |
| git.azcomputerguru.com | A 72.194.62.4 proxied | CNAME tunnel proxied |
| plex.azcomputerguru.com | A 72.194.62.4 proxied | A 72.194.62.4 proxied (unchanged net effect) |
| plexrequest.azcomputerguru.com | A 72.194.62.4 proxied | CNAME tunnel proxied |
| rmm.azcomputerguru.com | A 72.194.62.4 proxied | CNAME tunnel proxied |
| rmm-api.azcomputerguru.com | A 72.194.62.4 proxied | CNAME tunnel proxied |
| sync.azcomputerguru.com | A 72.194.62.9 proxied | CNAME tunnel proxied |
| rustdesk.azcomputerguru.com | A 72.194.62.10 proxied | A 72.194.62.10 proxied (unchanged net effect) |
| secure.azcomputerguru.com | A 72.194.62.2 proxied | A 72.194.62.2 proxied (unchanged net effect) |
---
## Current tunnel ingress (9 hostnames -- /mnt/cache/appdata/cloudflared/config.yml)
Tunnel: `78d3e58f-1979-4f0e-a28b-98d6b3c3d867` (name `acg-origin`)
- azcomputerguru.com -> https://172.16.3.10:443 (SNI + noTLSVerify)
- analytics.azcomputerguru.com -> https://172.16.3.10:443
- community.azcomputerguru.com -> https://172.16.3.10:443
- radio.azcomputerguru.com -> https://172.16.3.10:443
- git.azcomputerguru.com -> https://172.16.3.20:18443
- plexrequest.azcomputerguru.com -> https://172.16.3.20:18443
- rmm.azcomputerguru.com -> https://172.16.3.20:18443
- rmm-api.azcomputerguru.com -> https://172.16.3.20:18443
- sync.azcomputerguru.com -> https://172.16.3.20:18443
- catch-all -> http_status:404
Backups of config.yml kept as `config.yml.bak-YYYYMMDD-HHMMSS` in same dir.
---
## Final verification outputs
```
azcomputerguru.com HTTP 200 cloudflare (tunnel -> IX)
analytics.azcomputerguru.com HTTP 200 cloudflare (tunnel -> IX)
community.azcomputerguru.com HTTP 200 cloudflare (tunnel -> IX)
radio.azcomputerguru.com HTTP 200 cloudflare (tunnel -> IX)
git.azcomputerguru.com HTTP 200 cloudflare (tunnel -> Jupiter NPM)
plexrequest.azcomputerguru.com HTTP 200 cloudflare (tunnel -> Jupiter NPM)
rmm.azcomputerguru.com HTTP 200 cloudflare (tunnel -> Jupiter NPM)
rmm-api.azcomputerguru.com HTTP 200 cloudflare (tunnel -> Jupiter NPM)
sync.azcomputerguru.com HTTP 200 cloudflare (tunnel -> Jupiter NPM)
ix.azcomputerguru.com:443 HTTP 200 (direct, default vhost)
ix.azcomputerguru.com:2087 HTTP 200 (direct, WHM)
ix.azcomputerguru.com:2083 HTTP 200 (direct, cPanel)
```
---
## Scripts created (in clients/internal-infrastructure/scripts/cloudflared-tunnel-setup/)
- `audit_proxied.py` -- list all proxied zone records, classify origin, external probe each
- `discover_backends.py` -- extract pfSense NAT rules and Jupiter NPM server_name mappings
- `expand_tunnel.py` -- extend tunnel ingress to 13 hostnames + DNS cutover
- `revert_broken.py` -- remove plex/rustdesk/secure from tunnel and restore their A records
All have been sanitized to use SOPS vault for credentials / env var for CF token.
---
## Pending / Incomplete / Open Items
Additions to the list from the earlier 2026-04-13 entry:
1. **`plex.azcomputerguru.com` is still broken** (525) -- requires NPM proxy_host entry on Jupiter. Likely target: `binhex-plexpass` container at `172.16.3.20:32400` (or whatever internal IP Plex uses with `network_mode: host`). Once NPM has the vhost, can add to tunnel with a single config.yml change.
2. **`rustdesk.azcomputerguru.com` is still broken** (525) -- requires:
- Finding where the rustdesk server is actually running (no `rustdesk` container visible in `docker ps` on Jupiter; may be on a different host, or decommissioned)
- Adding NPM vhost for it
- Then tunnel ingress
3. **`secure.azcomputerguru.com` is still broken** (ERR) -- requires either:
- A static route on Jupiter to 172.16.1.0/24 so cloudflared can reach 172.16.1.16
- Or move the service behind Jupiter NPM
- Or grey-cloud to DNS-only like we did for `ix.` (bypass CF entirely)
4. **Still TODO from the earlier block:**
- Submit Cox BGP ticket (`clients/internal-infrastructure/vendor-tickets/2026-04-13-cox-bgp-cloudflare-routing.md`)
- Populate CF tokens in SOPS vault (currently 1Password only)
- Fix stale `Paper123\!@#` in Dataforth AD2 vault entry
- Verify rsync covers Dataforth `VASLOG - Engineering Tested` subfolder
---
**Last Updated:** 2026-04-13 15:56
**Next Actions:** consider adding NPM vhost for plex, investigate rustdesk host, commit today's additions.