From 57d03c609771d6b94edc34af8f650f662f79f23a Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Thu, 28 May 2026 12:26:29 -0700 Subject: [PATCH] sync: auto-sync from GURU-5070 at 2026-05-28 12:26:25 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-05-28 12:26:25 --- .../session-logs/2026-05-28-session.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 clients/glaztech/session-logs/2026-05-28-session.md diff --git a/clients/glaztech/session-logs/2026-05-28-session.md b/clients/glaztech/session-logs/2026-05-28-session.md new file mode 100644 index 0000000..13a6252 --- /dev/null +++ b/clients/glaztech/session-logs/2026-05-28-session.md @@ -0,0 +1,113 @@ +# Glaztech — Email Delivery Investigation & Transport Rules + +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +--- + +## Session Summary + +Steve Eastman (Glaztech) reported that emails from two vendor contacts — `hartsglass@centurytel.net` and `olemons@eastexglass.com` — were showing "Deliver" in MailProtector but not arriving at `SHVSALES@glaztech.com`. The emails were not appearing in M365 Quarantine either, which was the key diagnostic clue. + +Investigation via Exchange Online message trace and connector inspection revealed the root cause: both sender domains (`centurytel.net` and `eastexglass.com`) publish DMARC policy `p=reject`. Glaztech's inbound connector ("Inbound Spam Filter") has Enhanced Filtering enabled with `EFSkipIPs` set to the three MailProtector IPs (162.248.93.233, 162.248.93.81, 65.113.52.82), but `SCLMinusOne` is `null` — meaning EOP does not trust MailProtector's verdict and re-evaluates all mail. With Enhanced Filtering, EOP looks past the MailProtector relay to the original sender's authentication. Both sender domains fail DMARC, producing hard `550 5.7.509` NDR rejections in Exchange Online before messages ever reach the mailbox or quarantine. + +Additional rejected senders were identified from the email thread Steve forwarded: `SSales@arkglass.com`, `bossier@glassservices.com`, and an unnamed contact at `aaaglassinc.com` (Donald Arnold). DNS inspection of `glassservices.com` found a broken SPF record (`v=spf1 -all` — no authorized senders), meaning their emails will be rejected by all mail providers regardless of any Glaztech-side fix. + +The fix applied was two SCL=-1 transport rules in the Glaztech Exchange Online tenant. Transport rules evaluate before the anti-spam pipeline (including DMARC enforcement) in EOP, so setting SCL=-1 bypasses the DMARC rejection for these specific senders. Rules were created via the Exchange Operator service principal using `POST /adminapi/beta/{tenant}/InvokeCommand` with `New-TransportRule` cmdlet (direct REST resource endpoints 404 in this tenant). + +--- + +## Key Decisions + +- **SCL=-1 transport rule over connector trust** — Setting `SCLMinusOne: true` on the connector would trust MailProtector's verdict for ALL inbound mail, which is too broad. Targeted transport rules scoped to specific senders are the more surgical fix. +- **Two rules instead of one** — The aaaglassinc.com sender was handled by domain (`SenderDomainIs`) rather than specific address since the contact email was not known. The other four were handled by exact address (`From`). Separate rules keep the logic clean. +- **glassservices.com SPF not fixable on our side** — Their `v=spf1 -all` will cause rejection everywhere. The appropriate action is notifying Steve so he can tell the glassservices.com vendor their IT needs to fix the SPF record. Our SCL=-1 rule still covers `bossier@glassservices.com` as a workaround. +- **No Syncro ticket created** — This was handled reactively via email thread; no ticket was opened during the session. Consider creating one retroactively if billing for this work. + +--- + +## Problems Encountered + +- **Direct REST resource endpoints 404**: `/adminapi/beta/{tenant}/TransportRule` returns 404. Fixed by using the `InvokeCommand` pattern: `POST /adminapi/beta/{tenant}/InvokeCommand` with `{"CmdletInput": {"CmdletName": "New-TransportRule", "Parameters": {...}}}`. +- **Transport rule name 66-char limit exceeded**: Initial rule name "SCL Bypass - hartsglass@centurytel.net and olemons@eastexglass.com" was 66 chars; EXO enforces a 64-char max. Fixed by shortening to "SCL Bypass - hartsglass + olemons (SHVSALES)" (44 chars). +- **Initial misdiagnosis**: First assumed emails were landing in Junk/spam folder (SCL scoring). Steve's follow-up with full message trace revealed actual cause was hard DMARC `550 5.7.509` NDR — the SCL=-1 fix still resolves this correctly since transport rules run before DMARC enforcement in the EXO pipeline. + +--- + +## Configuration Changes + +Two transport rules created in Glaztech M365 tenant (82931e3c-de7a-4f74-87f7-fe714be1f160): + +| Rule GUID | Name | Condition | Action | +|-----------|------|-----------|--------| +| 482c714a-8780-4c62-ae0a-0b6da9ca9d52 | SCL Bypass - hartsglass + olemons (SHVSALES) | From: hartsglass@centurytel.net, olemons@eastexglass.com, SSales@arkglass.com, bossier@glassservices.com | SetSCL: -1 | +| 7e0c01a8-ec22-43fe-b600-796c0f295aa5 | SCL Bypass - aaaglassinc.com (SHVSALES) | SenderDomainIs: aaaglassinc.com | SetSCL: -1 | + +Both rules: State = Enabled. + +--- + +## Credentials & Secrets + +- **Exchange Operator Service Principal** + - App ID: b43e7342-5b4b-492f-890f-bb5a4f7f40e9 + - Secret: Ct28Q~fKYUu.RvkMaGNAV1YeK6h-HBewCTPnwa.Y + - Vault: `msp-tools/computerguru-exchange-operator.sops.yaml` +- **Glaztech Tenant ID**: 82931e3c-de7a-4f74-87f7-fe714be1f160 + +--- + +## Infrastructure & Servers + +- **Glaztech M365 tenant**: 82931e3c-de7a-4f74-87f7-fe714be1f160 +- **MailProtector IPs (EFSkipIPs on inbound connector)**: 162.248.93.233, 162.248.93.81, 65.113.52.82 +- **EXO InvokeCommand endpoint**: `https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand` +- **Token endpoint**: `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token` + +--- + +## Commands & Outputs + +```bash +# Auth token +POST https://login.microsoftonline.com/82931e3c-.../oauth2/v2.0/token +scope=https://outlook.office365.com/.default + +# Check connector Enhanced Filtering state +POST /adminapi/beta/{tenant}/InvokeCommand +{"CmdletInput":{"CmdletName":"Get-InboundConnector","Parameters":{"Identity":"Inbound Spam Filter"}}} +# → EFSkipIPs: [162.248.93.233, 162.248.93.81, 65.113.52.82], SCLMinusOne: null + +# Create rule 1 (specific addresses) +{"CmdletInput":{"CmdletName":"New-TransportRule","Parameters":{ + "Name":"SCL Bypass - hartsglass + olemons (SHVSALES)", + "From":["hartsglass@centurytel.net","olemons@eastexglass.com","SSales@arkglass.com","bossier@glassservices.com"], + "SetSCL":-1,"Enabled":true}}} +# → GUID: 482c714a-8780-4c62-ae0a-0b6da9ca9d52 + +# Create rule 2 (domain) +{"CmdletInput":{"CmdletName":"New-TransportRule","Parameters":{ + "Name":"SCL Bypass - aaaglassinc.com (SHVSALES)", + "SenderDomainIs":["aaaglassinc.com"], + "SetSCL":-1,"Enabled":true}}} +# → GUID: 7e0c01a8-ec22-43fe-b600-796c0f295aa5 +``` + +--- + +## Pending / Incomplete Tasks + +- **Notify Steve about glassservices.com SPF**: `bossier@glassservices.com` will be rejected by all mail providers (SPF `v=spf1 -all`). Steve should tell the vendor their IT needs to fix the SPF record. Our SCL bypass is a workaround only. +- **Harts Glass resend**: The original rejected emails need to be resent by the sender. Our rule is now live but previously rejected messages are NDRed and will not auto-retry. +- **Syncro ticket**: No ticket opened for this work. Create retroactively if billing. + +--- + +## Reference Information + +- **DMARC root cause**: `centurytel.net` → `p=reject`; `eastexglass.com` → `p=reject` +- **NDR code**: `550 5.7.509` (DMARC failure under Enhanced Filtering) +- **Transport rule GUIDs**: 482c714a (addresses), 7e0c01a8 (aaaglassinc.com domain) +- **EXO InvokeCommand pattern**: Required because direct `/TransportRule` REST endpoint 404s in this tenant