Session log: Cloudflare Tunnel for azcomputerguru + Cox BGP diagnosis
Diagnosed azcomputerguru.com 521 errors: Cox's BGP route to specific Cloudflare origin-pull prefixes (162.158.0.0/16, 172.64.0.0/13, 173.245.48.0/20, 141.101.64.0/18) is broken from 72.194.62.0/29. Confirmed by TCP probe matrix from pfSense WAN, traceroute latency comparison, and state-table showing 0 inbound CF connections while direct-internet traffic still reached origin. Deployed Cloudflare Tunnel 'acg-origin' on Jupiter Unraid as a Docker container. Routes 4 proxied hostnames (azcomputerguru.com, analytics., community., radio.) through the tunnel with HTTPS backend to IX 172.16.3.10:443 with per-ingress SNI matching. All 4 hostnames return 200 OK through CF edge after the cutover. Repo hygiene: - Merged clients/ix-server/ into clients/internal-infrastructure/ (IX is internal infra, not a paying-client account). Git detected the session-log files as renames so history is preserved. Updated 4 stale path references in 2 files. - Moved cox-bgp ticket draft out of projects/dataforth-dos/ (wrong project) to clients/internal-infrastructure/vendor-tickets/. - Relocated tunnel-setup helper scripts from projects/dataforth-dos/datasheet-pipeline/implementation/ to clients/internal-infrastructure/scripts/cloudflared-tunnel-setup/. Deleted superseded/abandoned login attempts. Sanitized hardcoded Jupiter/pfSense SSH passwords to pull from SOPS vault at runtime; Cloudflare token reads from env var (tokens still in 1Password, vault entry is metadata-only). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,88 @@
|
||||
# Cox Business BGP / Routing Escalation Ticket — Draft
|
||||
|
||||
**Account / Service:** Mike Swanson, AZ Computer Guru — business static-IP block 72.194.62.0/29
|
||||
**WAN / upstream:** Cox Business, Tucson AZ (or wherever applicable)
|
||||
**Circuit public IP (pfSense WAN):** 98.181.90.163
|
||||
**Destination affected public IPs:** 72.194.62.2, .3, .4, .5, .8, .9, .10
|
||||
|
||||
---
|
||||
|
||||
## Subject
|
||||
|
||||
Asymmetric/unreachable routing from Cox customer block 72.194.62.0/29 to specific Cloudflare /16 and /18 IP prefixes
|
||||
|
||||
## Summary
|
||||
|
||||
Cloudflare PoP in Phoenix (PHX) cannot successfully establish TCP connections to our public IPs (72.194.62.2-.10) for origin-pull requests. HTTP requests from public clients reaching Cloudflare get a 521 "web server is down" response, because Cloudflare's origin-pull source prefixes cannot complete TCP handshakes to our netblock.
|
||||
|
||||
## Evidence
|
||||
|
||||
### 1. Our WAN firewall can reach ~half of Cloudflare's IP ranges, not the others
|
||||
|
||||
From our pfSense firewall (FreeBSD, 2.8.1), TCP connect test to port 443 on representative IPs in each Cloudflare-advertised prefix:
|
||||
|
||||
| Cloudflare Prefix | Sample IP | TCP:443 connect |
|
||||
|---|---|---|
|
||||
| 104.16.0.0/13 | 104.16.0.1 | succeeds |
|
||||
| 104.16.0.0/13 | 104.17.0.1 | succeeds |
|
||||
| 104.24.0.0/14 | 104.26.0.1 | succeeds |
|
||||
| 162.158.0.0/16 | 162.158.0.1 | **timeout** |
|
||||
| 162.158.0.0/16 | 162.158.100.1 | **timeout** |
|
||||
| 172.64.0.0/13 | 172.64.0.1 | **timeout** |
|
||||
| 172.64.0.0/13 | 172.67.0.1 | **timeout** |
|
||||
| 173.245.48.0/20 | 173.245.48.1 | **timeout** |
|
||||
| 141.101.64.0/18 | 141.101.64.1 | **timeout** |
|
||||
|
||||
Reference list Cloudflare publishes at https://www.cloudflare.com/ips-v4
|
||||
|
||||
### 2. ICMP traceroute to failing Cloudflare prefixes reveals an unusually indirect path
|
||||
|
||||
Traceroute from pfSense WAN (98.181.90.163) to 162.158.0.1 — 8 hops, ~173 ms (suggests routing via a distant peering point):
|
||||
|
||||
```
|
||||
1 * * *
|
||||
2 100.120.164.200 3.236 ms
|
||||
3 68.1.0.191 4.180 ms
|
||||
4 184.183.131.9 23.671 ms
|
||||
5 198.41.140.124 14.635 ms
|
||||
6 198.41.140.244 161.626 ms <- huge latency jump (likely cross-country)
|
||||
7 108.162.247.54 163.073 ms
|
||||
8 162.158.0.1 173.018 ms
|
||||
```
|
||||
|
||||
Compare to traceroute to the working prefix 104.26.8.237 — 6 hops, ~3.6 ms:
|
||||
|
||||
```
|
||||
1 * * *
|
||||
2 100.120.164.200 3.022 ms
|
||||
3 68.1.0.191 3.799 ms
|
||||
4 184.183.131.9 8.973 ms
|
||||
5 162.158.140.21 3.909 ms <- nearby Cloudflare peering
|
||||
6 104.26.8.237 3.445 ms
|
||||
```
|
||||
|
||||
The ~170 ms added round-trip to 162.158.0.0/16 vs ~3.5 ms to 104.x suggests routes for 162.158, 172.64, 173.245, 141.101 are being withdrawn from the local peering and defaulting to a distant one (Ashburn or similar), with packet loss or asymmetric return on that path.
|
||||
|
||||
### 3. Direct-internet users reach our origin fine; only Cloudflare-proxied traffic fails
|
||||
|
||||
Our state table currently shows 285 active inbound :443 connections to our origin server from various non-Cloudflare IPs (Philippines, Russia, India, Pakistan users — direct clients). Zero inbound connections from any Cloudflare prefix. Origin is healthy; the problem is specifically the return path to Cloudflare's origin-pull source IPs.
|
||||
|
||||
### 4. Third-party test confirms routing is not symmetric
|
||||
|
||||
From an external network (different ISP egress), connecting to our public IP 72.194.62.5 on port 443 with correct SNI succeeds with HTTP 200.
|
||||
|
||||
## Ask
|
||||
|
||||
Please have network engineering check the BGP advertisements and/or routing policy for:
|
||||
|
||||
- Cloudflare prefixes **162.158.0.0/16**, **172.64.0.0/13**, **173.245.48.0/20**, **141.101.64.0/18**
|
||||
- Return path from our block **72.194.62.0/29** to those Cloudflare prefixes
|
||||
|
||||
It appears these prefixes are being routed through a distant Cox peering point rather than the nearby Cloudflare peering (visible at hop 5 on the working route), and the return path is either black-holed or lossy enough to drop TCP handshakes.
|
||||
|
||||
Contact: Mike Swanson, AZ Computer Guru
|
||||
Timeline: urgent — hosted sites (azcomputerguru.com, analytics., community., radio.) are intermittently unreachable to any visitor whose nearest Cloudflare PoP chooses an origin-pull source in one of the affected prefixes.
|
||||
|
||||
## Workaround in place
|
||||
|
||||
We are setting up a Cloudflare Tunnel from inside our network outbound to Cloudflare (initiated from our side using working prefixes), so customer-visible outage is mitigated. Resolution of the underlying BGP issue is still required for any direct-proxied traffic and general Cox–Cloudflare connectivity health.
|
||||
Reference in New Issue
Block a user