sync: auto-sync from GURU-5070 at 2026-06-18 12:49:38

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-18 12:49:38

clients/darrell-delphen/session-logs/2026-06/2026-06-18-mike-email-link-sni-block.md

@@ -0,0 +1,119 @@
+# 2026-06-18 — Darrell Delphen — Outlook email links failing (ISP SNI block)
+
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Darrell Delphen reported that links in Outlook email would not open on one workstation
+(DDDOffice072023), failing with "can't reach this page" / `ERR_CONNECTION_ABORTED` against a
+`url.emailprotection.link` URL, while links opened from Gmail worked fine. The failing host is
+Intermedia's Email Protection "Safe Link" rewriter — every link in Intermedia-protected mail is
+rewritten to `https://url.emailprotection.link/...`, so all Outlook-delivered links routed through it.
+
+Diagnosis was done entirely over GuruRMM against agent `000ed57d-fd05-4001-871c-244f43155c16`. DNS
+resolved correctly and TCP 443 connected, but the TLS handshake died with SChannel `0x80090326`
+(SEC_E_ILLEGAL_MESSAGE — "message received was unexpected or badly formatted"). The endpoint's TLS
+stack was clean: FIPS off, no SCHANNEL protocol/cipher overrides, no cipher-suite GPO, only Windows
+Defender (no third-party AV/proxy/VPN/LSP/WFP callout). The same node `199.193.205.140` handshook
+successfully with the real SNI from GURU-5070 on a different network, proving the origin was healthy
+and the interference was on the endpoint's path. A blast-radius sweep showed only
+`url.emailprotection.link` failed while `google.com`, `microsoft.com`, `outlook.office365.com`,
+`cloudflare.com`, `badssl.com`, and even `login.serverdata.net` (same Intermedia infra) all succeeded;
+MTU was fine. An SNI-varied handshake to the same IP isolated it conclusively: `example.com` and
+`login.serverdata.net` SNIs succeeded 12/12 while `url.emailprotection.link` failed 12/12, across
+interleaved source ports — deterministic, SNI-keyed, not flow-hash/LAG. Root cause: a network device
+on the path performing SNI-based content inspection that corrupted the handshake for that one hostname.
+
+The gateway turned out to be an ISP-provided Extreme **EXOS** device the client had no login for. The
+fix path was therefore ISP escalation. An escalation packet was drafted, and Cloudflare WARP was
+installed on the workstation as an interim workaround — tunneling past the SNI block. With WARP
+connected, egress moved to Cloudflare (104.28.152.216) and the real-SNI handshake succeeded (TLS 1.2,
+HTTP 200). The ISP then disabled a "NetIQ" web/URL-filtering feature on the gateway, which cleared the
+block at the source. After WARP was disconnected the native ISP path was verified working (5/5
+handshakes + HTTP 200, egress back to 167.89.210.225), so WARP was uninstalled and the machine
+returned to normal. Work was documented and billed on Syncro ticket #32437 — private technical note,
+customer-facing/emailed summary, and 1.0h remote labor ($150.00, invoice #1650728058).
+
+## Key Decisions
+
+- Diagnosed exclusively via repeated GuruRMM `SslStream`/`Test-NetConnection` probes rather than
+  asking the client to run tools — faster and reproducible.
+- Used an SNI-varied handshake to the *same fixed IP* as the decisive test. It separated
+  destination/server problems from path interference and proved the block was keyed on the SNI string.
+- Ran a 12x repeatability test per SNI to rule out a faulty LAG/ECMP member (intermittent, 5-tuple
+  keyed) vs deliberate content matching (deterministic). Result was 0/12 vs 12/12 — deterministic.
+- Chose Cloudflare WARP as the interim workaround because the block is SNI-based; any tunnel that
+  encrypts egress past the EXOS hides the SNI. WARP is the lightest deploy.
+- Installed/connected WARP in stages (install, then connect, then verify) so each step could be
+  confirmed before flipping the tunnel, given the agent bounces on network-stack changes.
+- Emailed the customer a plain-language summary (do_not_email:false) and kept the technical detail in
+  a hidden note.
+
+## Problems Encountered
+
+- **Shell state not persisting between Bash calls** — `$TOKEN`/`$RMM` from `rmm-auth.sh` were gone on
+  the next call (first dispatch produced no output). Fixed by `eval "$(rmm-auth.sh)"` inside every Bash
+  invocation.
+- **PowerShell single-quotes collided with bash single-quoted `SCRIPT='...'`** — embedded
+  `'C:\Program Files\...'` terminated the bash string (`FilesCloudflareCloudflare: command not found`).
+  Fixed by defining the script via a quoted heredoc `SCRIPT=$(cat <<'PS' ... PS)`.
+- **WARP install/connect bounced the RMM agent** — commands returned `interrupted` ("Agent restarted
+  during execution") because installing/connecting WARP resets the network stack/WFP. The agent
+  auto-reconnected (through WARP after connect); verified state with follow-up commands.
+- **First WARP uninstall found nothing** — the product registers as **"Cloudflare One Client"**, not
+  "Cloudflare WARP", so the DisplayName filter missed it. Found the GUID
+  `{9E49837E-2971-413F-9587-119FA819E572}` and removed via `msiexec /x`.
+
+## Configuration Changes
+
+- **Endpoint DDDOffice072023:** Cloudflare WARP (Cloudflare One Client 2026.4.1390.0) installed,
+  registered, connected, then later disconnected and **fully uninstalled**. Net change to the machine: none.
+- **ISP gateway (not us):** ISP disabled the "NetIQ" web/URL-filtering feature on the Extreme EXOS device.
+- No changes to the ClaudeTools repo code.
+
+## Credentials & Secrets
+
+None discovered, created, or rotated this session.
+
+## Infrastructure & Servers
+
+- **Endpoint:** DDDOffice072023 — Windows, GuruRMM agent `000ed57d-fd05-4001-871c-244f43155c16`
+  (v0.6.66), RMM client "AZ Computer Guru" / site "Discovery test site". LAN gateway 192.168.1.1,
+  ISP egress 167.89.210.225, WARP egress (while active) 104.28.152.216.
+- **ISP gateway:** Extreme Networks **EXOS** device, ISP-provided/managed (client has no login). Was
+  running a "NetIQ" URL-filtering feature doing SNI inspection.
+- **Blocked destination:** `url.emailprotection.link` → CNAME `urlrs.gslb.serverdata.net` → A
+  199.193.205.140 (Intermedia Email Protection link-rewriter). GSLB pool also advertises
+  199.193.200.65 / 64.78.20.65 / 162.244.196.65, all TCP-dead from any network (not an ISP block).
+- **GuruRMM API:** http://172.16.3.30:3001 (JWT via `rmm-auth.sh`).
+
+## Commands & Outputs
+
+- Decisive SNI test (same IP, varied SNI), via RMM PowerShell:
+  - SNI `url.emailprotection.link` → `0x80090326` SEC_E_ILLEGAL_MESSAGE (0/12 ok)
+  - SNI `example.com` / `login.serverdata.net` → TLS 1.2 AES256 (12/12 ok)
+- Off-network control (GURU-5070) to 199.193.205.140 with real SNI → OK TLS 1.2.
+- Post-ISP-fix native verify: real-SNI handshake 5/5 OK + `Invoke-WebRequest` HEAD → HTTP 200,
+  egress 167.89.210.225.
+- WARP removal: `msiexec /x {9E49837E-2971-413F-9587-119FA819E572} /qn /norestart` → exit 0;
+  warp-svc/warp-cli/install-dir/uninstall-entry all gone.
+
+## Pending / Incomplete Tasks
+
+- None functional. Block is resolved at the ISP. If the issue recurs, suspect the EXOS "NetIQ"
+  feature being re-enabled — re-run the SNI-varied handshake test to confirm.
+- Optional: if WARP is ever rolled to more machines as a workaround, harden it (force auto-connect,
+  lock client) and note egress moves to Cloudflare (breaks office-static-IP allowlisting).
+
+## Reference Information
+
+- **Syncro ticket:** #32437 (id 112766479) — https://computerguru.syncromsp.com/tickets/112766479
+  - Private note id 419714810; public/emailed summary id 419714813
+  - Line item id 42925426 (Labor - Remote Business 1.0h @ $150)
+  - Invoice #1650728058, total $150.00; status Invoiced
+- **Customer:** Darrell Delphen (Syncro customer_id 35996725), no prepaid block.
+- **SChannel error:** 0x80090326 = SEC_E_ILLEGAL_MESSAGE (handshake message malformed/unexpected) —
+  signature of in-path TLS/SNI tampering when paired with same-IP success off-network.