# Glaztech — Email Delivery Investigation & Transport Rules

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

---

## Session Summary

Steve Eastman (Glaztech) reported that emails from two vendor contacts — `hartsglass@centurytel.net` and `olemons@eastexglass.com` — were showing "Deliver" in MailProtector but not arriving at `SHVSALES@glaztech.com`. The emails were not appearing in M365 Quarantine either, which was the key diagnostic clue.

Investigation via Exchange Online message trace and connector inspection revealed the root cause: both sender domains (`centurytel.net` and `eastexglass.com`) publish DMARC policy `p=reject`. Glaztech's inbound connector ("Inbound Spam Filter") has Enhanced Filtering enabled with `EFSkipIPs` set to the three MailProtector IPs (162.248.93.233, 162.248.93.81, 65.113.52.82), but `SCLMinusOne` is `null` — meaning EOP does not trust MailProtector's verdict and re-evaluates all mail. With Enhanced Filtering, EOP looks past the MailProtector relay to the original sender's authentication. Both sender domains fail DMARC, producing hard `550 5.7.509` NDR rejections in Exchange Online before messages ever reach the mailbox or quarantine.

Additional rejected senders were identified from the email thread Steve forwarded: `SSales@arkglass.com`, `bossier@glassservices.com`, and an unnamed contact at `aaaglassinc.com` (Donald Arnold). DNS inspection of `glassservices.com` found a broken SPF record (`v=spf1 -all` — no authorized senders), meaning their emails will be rejected by all mail providers regardless of any Glaztech-side fix.

The fix applied was two SCL=-1 transport rules in the Glaztech Exchange Online tenant. Transport rules evaluate before the anti-spam pipeline (including DMARC enforcement) in EOP, so setting SCL=-1 bypasses the DMARC rejection for these specific senders. Rules were created via the Exchange Operator service principal using `POST /adminapi/beta/{tenant}/InvokeCommand` with `New-TransportRule` cmdlet (direct REST resource endpoints 404 in this tenant).

---

## Key Decisions

- **SCL=-1 transport rule over connector trust** — Setting `SCLMinusOne: true` on the connector would trust MailProtector's verdict for ALL inbound mail, which is too broad. Targeted transport rules scoped to specific senders are the more surgical fix.
- **Two rules instead of one** — The aaaglassinc.com sender was handled by domain (`SenderDomainIs`) rather than specific address since the contact email was not known. The other four were handled by exact address (`From`). Separate rules keep the logic clean.
- **glassservices.com SPF not fixable on our side** — Their `v=spf1 -all` will cause rejection everywhere. The appropriate action is notifying Steve so he can tell the glassservices.com vendor their IT needs to fix the SPF record. Our SCL=-1 rule still covers `bossier@glassservices.com` as a workaround.
- **No Syncro ticket created** — This was handled reactively via email thread; no ticket was opened during the session. Consider creating one retroactively if billing for this work.

---

## Problems Encountered

- **Direct REST resource endpoints 404**: `/adminapi/beta/{tenant}/TransportRule` returns 404. Fixed by using the `InvokeCommand` pattern: `POST /adminapi/beta/{tenant}/InvokeCommand` with `{"CmdletInput": {"CmdletName": "New-TransportRule", "Parameters": {...}}}`.
- **Transport rule name 66-char limit exceeded**: Initial rule name "SCL Bypass - hartsglass@centurytel.net and olemons@eastexglass.com" was 66 chars; EXO enforces a 64-char max. Fixed by shortening to "SCL Bypass - hartsglass + olemons (SHVSALES)" (44 chars).
- **Initial misdiagnosis**: First assumed emails were landing in Junk/spam folder (SCL scoring). Steve's follow-up with full message trace revealed actual cause was hard DMARC `550 5.7.509` NDR — the SCL=-1 fix still resolves this correctly since transport rules run before DMARC enforcement in the EXO pipeline.

---

## Configuration Changes

Two transport rules created in Glaztech M365 tenant (82931e3c-de7a-4f74-87f7-fe714be1f160):

| Rule GUID | Name | Condition | Action |
|-----------|------|-----------|--------|
| 482c714a-8780-4c62-ae0a-0b6da9ca9d52 | SCL Bypass - hartsglass + olemons (SHVSALES) | From: hartsglass@centurytel.net, olemons@eastexglass.com, SSales@arkglass.com, bossier@glassservices.com | SetSCL: -1 |
| 7e0c01a8-ec22-43fe-b600-796c0f295aa5 | SCL Bypass - aaaglassinc.com (SHVSALES) | SenderDomainIs: aaaglassinc.com | SetSCL: -1 |

Both rules: State = Enabled.

---

## Credentials & Secrets

- **Exchange Operator Service Principal**
  - App ID: b43e7342-5b4b-492f-890f-bb5a4f7f40e9
  - Secret: Ct28Q~fKYUu.RvkMaGNAV1YeK6h-HBewCTPnwa.Y
  - Vault: `msp-tools/computerguru-exchange-operator.sops.yaml`
- **Glaztech Tenant ID**: 82931e3c-de7a-4f74-87f7-fe714be1f160

---

## Infrastructure & Servers

- **Glaztech M365 tenant**: 82931e3c-de7a-4f74-87f7-fe714be1f160
- **MailProtector IPs (EFSkipIPs on inbound connector)**: 162.248.93.233, 162.248.93.81, 65.113.52.82
- **EXO InvokeCommand endpoint**: `https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand`
- **Token endpoint**: `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token`

---

## Commands & Outputs

```bash
# Auth token
POST https://login.microsoftonline.com/82931e3c-.../oauth2/v2.0/token
scope=https://outlook.office365.com/.default

# Check connector Enhanced Filtering state
POST /adminapi/beta/{tenant}/InvokeCommand
{"CmdletInput":{"CmdletName":"Get-InboundConnector","Parameters":{"Identity":"Inbound Spam Filter"}}}
# → EFSkipIPs: [162.248.93.233, 162.248.93.81, 65.113.52.82], SCLMinusOne: null

# Create rule 1 (specific addresses)
{"CmdletInput":{"CmdletName":"New-TransportRule","Parameters":{
  "Name":"SCL Bypass - hartsglass + olemons (SHVSALES)",
  "From":["hartsglass@centurytel.net","olemons@eastexglass.com","SSales@arkglass.com","bossier@glassservices.com"],
  "SetSCL":-1,"Enabled":true}}}
# → GUID: 482c714a-8780-4c62-ae0a-0b6da9ca9d52

# Create rule 2 (domain)
{"CmdletInput":{"CmdletName":"New-TransportRule","Parameters":{
  "Name":"SCL Bypass - aaaglassinc.com (SHVSALES)",
  "SenderDomainIs":["aaaglassinc.com"],
  "SetSCL":-1,"Enabled":true}}}
# → GUID: 7e0c01a8-ec22-43fe-b600-796c0f295aa5
```

---

## Pending / Incomplete Tasks

- **Notify Steve about glassservices.com SPF**: `bossier@glassservices.com` will be rejected by all mail providers (SPF `v=spf1 -all`). Steve should tell the vendor their IT needs to fix the SPF record. Our SCL bypass is a workaround only.
- **Harts Glass resend**: The original rejected emails need to be resent by the sender. Our rule is now live but previously rejected messages are NDRed and will not auto-retry.
- **Syncro ticket**: No ticket opened for this work. Create retroactively if billing.

---

## Reference Information

- **DMARC root cause**: `centurytel.net` → `p=reject`; `eastexglass.com` → `p=reject`
- **NDR code**: `550 5.7.509` (DMARC failure under Enhanced Filtering)
- **Transport rule GUIDs**: 482c714a (addresses), 7e0c01a8 (aaaglassinc.com domain)
- **EXO InvokeCommand pattern**: Required because direct `/TransportRule` REST endpoint 404s in this tenant