diff --git a/clients/glaztech/session-logs/2026-07/2026-07-20-mike-anchordenver-cloudfilter-quarantine.md b/clients/glaztech/session-logs/2026-07/2026-07-20-mike-anchordenver-cloudfilter-quarantine.md new file mode 100644 index 00000000..7fd8244b --- /dev/null +++ b/clients/glaztech/session-logs/2026-07/2026-07-20-mike-anchordenver-cloudfilter-quarantine.md @@ -0,0 +1,139 @@ +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-BEAST-ROG +- **Role:** admin + +## Session Summary + +Robert Carnal (robert@glaztech.com, cc Steven Eastman) forwarded a request from Glaztech's +Colorado (Denver) office manager Tami Weisbrod: the Colorado location was not receiving email +from `zach@anchordenver.com` (Anchor Glass). Tami reported a specific recent failure — Zach's +mail to Jimmie Hitzke (jhitzke@glaztech.com) was not delivering — and that a prior round of the +same problem hitting Dave Hill (dhill@glaztech.com) had been fixed by someone she couldn't +identify ("I thought this problem was behind us"). Steve Eastman was out of office. + +Diagnosis proceeded outside-in. A public DNS check of anchordenver.com showed a Google Workspace +domain (Google MX) with NO SPF record (only a google-site-verification TXT) and NO DMARC record, +sending via an AWS relay (52.0.70.91) — i.e. effectively unauthenticated mail. An M365 message +trace (Get-MessageTraceV2 through the Exchange Operator app via the InvokeCommand REST endpoint) +proved the mail never reached Microsoft 365: zero messages zach->jhitzke in 10 days, while +jhitzke received 90 other inbound messages in ~2 days (mailbox healthy) and a single +zach->dhill message on 2026-07-14 was Delivered. That located the block upstream of Exchange, at +the MailProtector CloudFilter inbound gateway (Glaztech's sole MX). + +The mailprotector skill's CloudFilter API confirmed the block at the source: jhitzke's held queue +contained two messages from zach@anchordenver.com, both `quarantine_spam`, score 400, reason +"SPF None". The root cause of the recurrence was found in the CloudFilter allow/block rules — +dhill had a per-user Allow rule (`33590415` = Allow zach@anchordenver.com) added during the prior +incident, while jhitzke had no such rule. The earlier fix was scoped to one user and never +generalized, so it recurred for the next Anchor contact. + +Mike chose a domain-wide fix over the per-user precedent. Applied two gated CloudFilter writes: +(1) a domain-level Allow rule for the whole `anchordenver.com` domain on the glaztech.com domain +entity (27635), and (2) release of jhitzke's two held messages. Both verified. The domain rule +supersedes the per-user dhill patch and prevents recurrence for all Glaztech recipients at every +location. Sender-side, the durable fix belongs to Anchor Glass (publish SPF/DKIM/DMARC); flagged +for relay to Robert. + +## Key Decisions + +- **Domain-wide allow for the entire anchordenver.com domain** (not per-user, not address-scoped). + Rationale: the per-user precedent (dhill) caused the recurrence; Anchor is a known legitimate + vendor emailing multiple Glaztech staff across locations. Mike accepted the tradeoff that a + whole-domain allow on an SPF-less sender also permits spoofed anchordenver.com mail through. +- **Fix at CloudFilter, not Exchange.** An EOP/Exchange transport rule (the usual Glaztech + DMARC-reject workaround) would not help because the mail never reaches Exchange — it is + quarantined at MailProtector first. The only effective lever was a CloudFilter allow rule. +- **Used the mailprotector skill's CloudFilter API** rather than the manual partner-portal login + the Glaztech wiki describes. The API key in the vault (`msp-tools/mailprotector.sops.yaml`) + reaches Glaztech's domain, held mail, and rules — the "manual portal only" wiki note is + superseded for these operations. + +## Problems Encountered + +- **EOP message trace returned HTTP 401 `invalid_token` on the investigator-exo tier.** In the + Glaztech tenant the Exchange Administrator role / `Exchange.ManageAsApp` is held only by the + Exchange Operator SP (`b43e7342`), not the Security Investigator SP (`bfbc12a4`). Re-acquired the + token on the `exchange-op` tier and the InvokeCommand/Get-MessageTraceV2 call succeeded. This is + already documented in the Glaztech wiki access notes. +- **CloudFilter `logs` endpoint returned empty** for the sender filter (likely time-boxed/index + behavior), and the `messages --sender` filter did not reliably filter server-side. Worked around + by pulling the full held-message queue for the user and identifying the anchordenver entries by + eye. + +## Configuration Changes + +- **Created** CloudFilter allow rule `33661305`: value `anchordenver.com`, type Allow, on domain + entity `glaztech.com` (27635). Verified present in `rules domains 27635`. +- **Released** two held CloudFilter messages to jhitzke@glaztech.com: `4580147186`, `4583375982` + (both were subject "Test" from zach@anchordenver.com; API returned them in `delivered_messages`). +- No Exchange/M365 changes made. No repo files changed beyond this session log. +- Pre-existing (not changed): per-user Allow rule `33590415` (Allow zach@anchordenver.com) on + dhill — now redundant given the domain rule; left in place (harmless). + +## Credentials & Secrets + +- No credentials created or newly discovered this session. +- MailProtector CloudFilter API key: vault `msp-tools/mailprotector.sops.yaml`, field + `credentials.api_key` (used by the mailprotector skill; base URL https://emailservice.io/api/v1). +- Exchange Operator app token acquired via + `bash .claude/skills/remediation-tool/scripts/get-token.sh exchange-op` (cert auth; + SP `b43e7342-5b4b-492f-890f-bb5a4f7f40e9`, roles full_access_as_app + Exchange.ManageAsApp). + +## Infrastructure & Servers + +- **Glaztech M365 tenant:** glaztechindustries.onmicrosoft.com, tenant ID + `82931e3c-de7a-4f74-87f7-fe714be1f160`, primary domain glaztech.com. +- **MailProtector CloudFilter (emailservice.io):** glaztech.com domain entity `27635`; account + "GlazTech" id `16175`. Inbound host `glaztech-com.inbound.emailservice.io` (sole MX, priority 5). +- **CloudFilter users:** jhitzke@glaztech.com = user `1347467` (Jimmy Hitzke); dhill@glaztech.com + = user `1837854` (Dave Hill). +- **Sender — Anchor Glass (anchordenver.com):** Google Workspace (MX aspmx.l.google.com et al.); + no SPF record (only `google-site-verification=YGN-6dSjhfxSmRfGxiXm5I_daxlM8go0u0GnR2ZpkCo`); no + DMARC (`_dmarc.anchordenver.com` NXDOMAIN); observed sending IP 52.0.70.91 (AWS). + +## Commands & Outputs + +- DNS: `Resolve-DnsName -Type TXT/MX anchordenver.com` and `_dmarc.anchordenver.com` — + confirmed no SPF, no DMARC, Google MX. +- EOP trace (exchange-op token) via + `POST https://outlook.office365.com/adminapi/beta//InvokeCommand` with + `{"CmdletInput":{"CmdletName":"Get-MessageTraceV2","Parameters":{...}}}`: + - sender zach@anchordenver.com, 2026-07-06..16: 1 result -> dhill, 2026-07-14 15:08Z, Delivered, FromIP 52.0.70.91. + - recipient jhitzke + sender zach: count 0. + - recipient jhitzke, any sender, 2 days: 90 Delivered (mailbox healthy). +- CloudFilter (mailprotector skill, run from `.claude/skills/mailprotector/scripts`, + `CLAUDETOOLS_ROOT=c:/Users/guru/ClaudeTools`): + - `mp.py find-user jhitzke@glaztech.com` -> id 1347467, domain 27635. + - `mp.py messages users 1347467` -> held queue; zach@anchordenver.com x2 (ids 4583375982, + 4580147186), decision quarantine_spam, score 400, result title "SPF None". + - `mp.py rules users 1837854` -> included `33590415 Allow zach@anchordenver.com` (prior fix). + - `mp.py add-rule domains 27635 --value anchordenver.com --type allow --confirm` -> rule 33661305. + - `mp.py release-many users 1347467 --ids 4583375982,4580147186 --confirm` -> delivered both. + +## Pending / Incomplete Tasks + +- **Reply to Robert Carnal** (robert@glaztech.com, cc seastman@glaztech.com) — he asked "let me + know if you can or not." Draft prepared in-session (explains the SPF root cause, the allow rule, + the released mail, and the SPF tip for Anchor). Not yet sent — Mike to choose send-as-him via + /mailbox vs. hand off text. +- **Syncro ticket** — billable work not yet logged. Glaztech Syncro customer ID 143932. Create + via /syncro. +- **Wiki update** — add the CloudFilter-layer pattern (vendor with broken SPF -> CloudFilter + quarantine -> domain allow rule, done via mailprotector API not manual portal) to + wiki/clients/glaztech.md. Run /wiki-compile separately. +- **Sender-side recommendation to relay to Anchor Glass:** publish an SPF record for + anchordenver.com (ideally SPF + DKIM + DMARC) — the real root cause; fixes their deliverability + to all recipients, not just Glaztech. +- Optional cleanup: remove the now-redundant per-user dhill allow rule `33590415` (low priority). + +## Reference Information + +- Original request thread subject: "FW: Email Delivery- Anchor Glass" (Robert Carnal -> + Mike, 2026-07-14 17:52). +- MailProtector API base: https://emailservice.io/api/v1 ; skill: `.claude/skills/mailprotector/`. +- Remediation tool token flow: `.claude/skills/remediation-tool/scripts/get-token.sh exchange-op`. +- Glaztech tenant ID: 82931e3c-de7a-4f74-87f7-fe714be1f160. +- CloudFilter: glaztech domain 27635, account 16175; users jhitzke 1347467, dhill 1837854. +- New CloudFilter allow rule: 33661305 (anchordenver.com, Allow, domain 27635). +- Released message IDs: 4580147186, 4583375982.