Mike Swanson
a78fb96f95
Diagnosed azcomputerguru.com 521 errors: Cox's BGP route to specific Cloudflare origin-pull prefixes (162.158.0.0/16, 172.64.0.0/13, 173.245.48.0/20, 141.101.64.0/18) is broken from 72.194.62.0/29. Confirmed by TCP probe matrix from pfSense WAN, traceroute latency comparison, and state-table showing 0 inbound CF connections while direct-internet traffic still reached origin. Deployed Cloudflare Tunnel 'acg-origin' on Jupiter Unraid as a Docker container. Routes 4 proxied hostnames (azcomputerguru.com, analytics., community., radio.) through the tunnel with HTTPS backend to IX 172.16.3.10:443 with per-ingress SNI matching. All 4 hostnames return 200 OK through CF edge after the cutover. Repo hygiene: - Merged clients/ix-server/ into clients/internal-infrastructure/ (IX is internal infra, not a paying-client account). Git detected the session-log files as renames so history is preserved. Updated 4 stale path references in 2 files. - Moved cox-bgp ticket draft out of projects/dataforth-dos/ (wrong project) to clients/internal-infrastructure/vendor-tickets/. - Relocated tunnel-setup helper scripts from projects/dataforth-dos/datasheet-pipeline/implementation/ to clients/internal-infrastructure/scripts/cloudflared-tunnel-setup/. Deleted superseded/abandoned login attempts. Sanitized hardcoded Jupiter/pfSense SSH passwords to pull from SOPS vault at runtime; Cloudflare token reads from env var (tokens still in 1Password, vault entry is metadata-only). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4.3 KiB
Cox Business BGP / Routing Escalation Ticket — Draft
Account / Service: Mike Swanson, AZ Computer Guru — business static-IP block 72.194.62.0/29 WAN / upstream: Cox Business, Tucson AZ (or wherever applicable) Circuit public IP (pfSense WAN): 98.181.90.163 Destination affected public IPs: 72.194.62.2, .3, .4, .5, .8, .9, .10
Subject
Asymmetric/unreachable routing from Cox customer block 72.194.62.0/29 to specific Cloudflare /16 and /18 IP prefixes
Summary
Cloudflare PoP in Phoenix (PHX) cannot successfully establish TCP connections to our public IPs (72.194.62.2-.10) for origin-pull requests. HTTP requests from public clients reaching Cloudflare get a 521 "web server is down" response, because Cloudflare's origin-pull source prefixes cannot complete TCP handshakes to our netblock.
Evidence
1. Our WAN firewall can reach ~half of Cloudflare's IP ranges, not the others
From our pfSense firewall (FreeBSD, 2.8.1), TCP connect test to port 443 on representative IPs in each Cloudflare-advertised prefix:
| Cloudflare Prefix | Sample IP | TCP:443 connect |
|---|---|---|
| 104.16.0.0/13 | 104.16.0.1 | succeeds |
| 104.16.0.0/13 | 104.17.0.1 | succeeds |
| 104.24.0.0/14 | 104.26.0.1 | succeeds |
| 162.158.0.0/16 | 162.158.0.1 | timeout |
| 162.158.0.0/16 | 162.158.100.1 | timeout |
| 172.64.0.0/13 | 172.64.0.1 | timeout |
| 172.64.0.0/13 | 172.67.0.1 | timeout |
| 173.245.48.0/20 | 173.245.48.1 | timeout |
| 141.101.64.0/18 | 141.101.64.1 | timeout |
Reference list Cloudflare publishes at https://www.cloudflare.com/ips-v4
2. ICMP traceroute to failing Cloudflare prefixes reveals an unusually indirect path
Traceroute from pfSense WAN (98.181.90.163) to 162.158.0.1 — 8 hops, ~173 ms (suggests routing via a distant peering point):
1 * * *
2 100.120.164.200 3.236 ms
3 68.1.0.191 4.180 ms
4 184.183.131.9 23.671 ms
5 198.41.140.124 14.635 ms
6 198.41.140.244 161.626 ms <- huge latency jump (likely cross-country)
7 108.162.247.54 163.073 ms
8 162.158.0.1 173.018 ms
Compare to traceroute to the working prefix 104.26.8.237 — 6 hops, ~3.6 ms:
1 * * *
2 100.120.164.200 3.022 ms
3 68.1.0.191 3.799 ms
4 184.183.131.9 8.973 ms
5 162.158.140.21 3.909 ms <- nearby Cloudflare peering
6 104.26.8.237 3.445 ms
The ~170 ms added round-trip to 162.158.0.0/16 vs ~3.5 ms to 104.x suggests routes for 162.158, 172.64, 173.245, 141.101 are being withdrawn from the local peering and defaulting to a distant one (Ashburn or similar), with packet loss or asymmetric return on that path.
3. Direct-internet users reach our origin fine; only Cloudflare-proxied traffic fails
Our state table currently shows 285 active inbound :443 connections to our origin server from various non-Cloudflare IPs (Philippines, Russia, India, Pakistan users — direct clients). Zero inbound connections from any Cloudflare prefix. Origin is healthy; the problem is specifically the return path to Cloudflare's origin-pull source IPs.
4. Third-party test confirms routing is not symmetric
From an external network (different ISP egress), connecting to our public IP 72.194.62.5 on port 443 with correct SNI succeeds with HTTP 200.
Ask
Please have network engineering check the BGP advertisements and/or routing policy for:
- Cloudflare prefixes 162.158.0.0/16, 172.64.0.0/13, 173.245.48.0/20, 141.101.64.0/18
- Return path from our block 72.194.62.0/29 to those Cloudflare prefixes
It appears these prefixes are being routed through a distant Cox peering point rather than the nearby Cloudflare peering (visible at hop 5 on the working route), and the return path is either black-holed or lossy enough to drop TCP handshakes.
Contact: Mike Swanson, AZ Computer Guru Timeline: urgent — hosted sites (azcomputerguru.com, analytics., community., radio.) are intermittently unreachable to any visitor whose nearest Cloudflare PoP chooses an origin-pull source in one of the affected prefixes.
Workaround in place
We are setting up a Cloudflare Tunnel from inside our network outbound to Cloudflare (initiated from our side using working prefixes), so customer-visible outage is mitigated. Resolution of the underlying BGP issue is still required for any direct-proxied traffic and general Cox–Cloudflare connectivity health.