sync: auto-sync from GURU-BEAST-ROG at 2026-07-20 10:25:51

Author: Mike Swanson
Machine: GURU-BEAST-ROG
Timestamp: 2026-07-20 10:25:51

clients/glaztech/session-logs/2026-07/2026-07-20-mike-anchordenver-cloudfilter-quarantine.md

@@ -0,0 +1,139 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-BEAST-ROG
+- **Role:** admin
+
+## Session Summary
+
+Robert Carnal (robert@glaztech.com, cc Steven Eastman) forwarded a request from Glaztech's
+Colorado (Denver) office manager Tami Weisbrod: the Colorado location was not receiving email
+from `zach@anchordenver.com` (Anchor Glass). Tami reported a specific recent failure — Zach's
+mail to Jimmie Hitzke (jhitzke@glaztech.com) was not delivering — and that a prior round of the
+same problem hitting Dave Hill (dhill@glaztech.com) had been fixed by someone she couldn't
+identify ("I thought this problem was behind us"). Steve Eastman was out of office.
+
+Diagnosis proceeded outside-in. A public DNS check of anchordenver.com showed a Google Workspace
+domain (Google MX) with NO SPF record (only a google-site-verification TXT) and NO DMARC record,
+sending via an AWS relay (52.0.70.91) — i.e. effectively unauthenticated mail. An M365 message
+trace (Get-MessageTraceV2 through the Exchange Operator app via the InvokeCommand REST endpoint)
+proved the mail never reached Microsoft 365: zero messages zach->jhitzke in 10 days, while
+jhitzke received 90 other inbound messages in ~2 days (mailbox healthy) and a single
+zach->dhill message on 2026-07-14 was Delivered. That located the block upstream of Exchange, at
+the MailProtector CloudFilter inbound gateway (Glaztech's sole MX).
+
+The mailprotector skill's CloudFilter API confirmed the block at the source: jhitzke's held queue
+contained two messages from zach@anchordenver.com, both `quarantine_spam`, score 400, reason
+"SPF None". The root cause of the recurrence was found in the CloudFilter allow/block rules —
+dhill had a per-user Allow rule (`33590415` = Allow zach@anchordenver.com) added during the prior
+incident, while jhitzke had no such rule. The earlier fix was scoped to one user and never
+generalized, so it recurred for the next Anchor contact.
+
+Mike chose a domain-wide fix over the per-user precedent. Applied two gated CloudFilter writes:
+(1) a domain-level Allow rule for the whole `anchordenver.com` domain on the glaztech.com domain
+entity (27635), and (2) release of jhitzke's two held messages. Both verified. The domain rule
+supersedes the per-user dhill patch and prevents recurrence for all Glaztech recipients at every
+location. Sender-side, the durable fix belongs to Anchor Glass (publish SPF/DKIM/DMARC); flagged
+for relay to Robert.
+
+## Key Decisions
+
+- **Domain-wide allow for the entire anchordenver.com domain** (not per-user, not address-scoped).
+  Rationale: the per-user precedent (dhill) caused the recurrence; Anchor is a known legitimate
+  vendor emailing multiple Glaztech staff across locations. Mike accepted the tradeoff that a
+  whole-domain allow on an SPF-less sender also permits spoofed anchordenver.com mail through.
+- **Fix at CloudFilter, not Exchange.** An EOP/Exchange transport rule (the usual Glaztech
+  DMARC-reject workaround) would not help because the mail never reaches Exchange — it is
+  quarantined at MailProtector first. The only effective lever was a CloudFilter allow rule.
+- **Used the mailprotector skill's CloudFilter API** rather than the manual partner-portal login
+  the Glaztech wiki describes. The API key in the vault (`msp-tools/mailprotector.sops.yaml`)
+  reaches Glaztech's domain, held mail, and rules — the "manual portal only" wiki note is
+  superseded for these operations.
+
+## Problems Encountered
+
+- **EOP message trace returned HTTP 401 `invalid_token` on the investigator-exo tier.** In the
+  Glaztech tenant the Exchange Administrator role / `Exchange.ManageAsApp` is held only by the
+  Exchange Operator SP (`b43e7342`), not the Security Investigator SP (`bfbc12a4`). Re-acquired the
+  token on the `exchange-op` tier and the InvokeCommand/Get-MessageTraceV2 call succeeded. This is
+  already documented in the Glaztech wiki access notes.
+- **CloudFilter `logs` endpoint returned empty** for the sender filter (likely time-boxed/index
+  behavior), and the `messages --sender` filter did not reliably filter server-side. Worked around
+  by pulling the full held-message queue for the user and identifying the anchordenver entries by
+  eye.
+
+## Configuration Changes
+
+- **Created** CloudFilter allow rule `33661305`: value `anchordenver.com`, type Allow, on domain
+  entity `glaztech.com` (27635). Verified present in `rules domains 27635`.
+- **Released** two held CloudFilter messages to jhitzke@glaztech.com: `4580147186`, `4583375982`
+  (both were subject "Test" from zach@anchordenver.com; API returned them in `delivered_messages`).
+- No Exchange/M365 changes made. No repo files changed beyond this session log.
+- Pre-existing (not changed): per-user Allow rule `33590415` (Allow zach@anchordenver.com) on
+  dhill — now redundant given the domain rule; left in place (harmless).
+
+## Credentials & Secrets
+
+- No credentials created or newly discovered this session.
+- MailProtector CloudFilter API key: vault `msp-tools/mailprotector.sops.yaml`, field
+  `credentials.api_key` (used by the mailprotector skill; base URL https://emailservice.io/api/v1).
+- Exchange Operator app token acquired via
+  `bash .claude/skills/remediation-tool/scripts/get-token.sh <tenant> exchange-op` (cert auth;
+  SP `b43e7342-5b4b-492f-890f-bb5a4f7f40e9`, roles full_access_as_app + Exchange.ManageAsApp).
+
+## Infrastructure & Servers
+
+- **Glaztech M365 tenant:** glaztechindustries.onmicrosoft.com, tenant ID
+  `82931e3c-de7a-4f74-87f7-fe714be1f160`, primary domain glaztech.com.
+- **MailProtector CloudFilter (emailservice.io):** glaztech.com domain entity `27635`; account
+  "GlazTech" id `16175`. Inbound host `glaztech-com.inbound.emailservice.io` (sole MX, priority 5).
+- **CloudFilter users:** jhitzke@glaztech.com = user `1347467` (Jimmy Hitzke); dhill@glaztech.com
+  = user `1837854` (Dave Hill).
+- **Sender — Anchor Glass (anchordenver.com):** Google Workspace (MX aspmx.l.google.com et al.);
+  no SPF record (only `google-site-verification=YGN-6dSjhfxSmRfGxiXm5I_daxlM8go0u0GnR2ZpkCo`); no
+  DMARC (`_dmarc.anchordenver.com` NXDOMAIN); observed sending IP 52.0.70.91 (AWS).
+
+## Commands & Outputs
+
+- DNS: `Resolve-DnsName -Type TXT/MX anchordenver.com` and `_dmarc.anchordenver.com` —
+  confirmed no SPF, no DMARC, Google MX.
+- EOP trace (exchange-op token) via
+  `POST https://outlook.office365.com/adminapi/beta/<tenant>/InvokeCommand` with
+  `{"CmdletInput":{"CmdletName":"Get-MessageTraceV2","Parameters":{...}}}`:
+  - sender zach@anchordenver.com, 2026-07-06..16: 1 result -> dhill, 2026-07-14 15:08Z, Delivered, FromIP 52.0.70.91.
+  - recipient jhitzke + sender zach: count 0.
+  - recipient jhitzke, any sender, 2 days: 90 Delivered (mailbox healthy).
+- CloudFilter (mailprotector skill, run from `.claude/skills/mailprotector/scripts`,
+  `CLAUDETOOLS_ROOT=c:/Users/guru/ClaudeTools`):
+  - `mp.py find-user jhitzke@glaztech.com` -> id 1347467, domain 27635.
+  - `mp.py messages users 1347467` -> held queue; zach@anchordenver.com x2 (ids 4583375982,
+    4580147186), decision quarantine_spam, score 400, result title "SPF None".
+  - `mp.py rules users 1837854` -> included `33590415 Allow zach@anchordenver.com` (prior fix).
+  - `mp.py add-rule domains 27635 --value anchordenver.com --type allow --confirm` -> rule 33661305.
+  - `mp.py release-many users 1347467 --ids 4583375982,4580147186 --confirm` -> delivered both.
+
+## Pending / Incomplete Tasks
+
+- **Reply to Robert Carnal** (robert@glaztech.com, cc seastman@glaztech.com) — he asked "let me
+  know if you can or not." Draft prepared in-session (explains the SPF root cause, the allow rule,
+  the released mail, and the SPF tip for Anchor). Not yet sent — Mike to choose send-as-him via
+  /mailbox vs. hand off text.
+- **Syncro ticket** — billable work not yet logged. Glaztech Syncro customer ID 143932. Create
+  via /syncro.
+- **Wiki update** — add the CloudFilter-layer pattern (vendor with broken SPF -> CloudFilter
+  quarantine -> domain allow rule, done via mailprotector API not manual portal) to
+  wiki/clients/glaztech.md. Run /wiki-compile separately.
+- **Sender-side recommendation to relay to Anchor Glass:** publish an SPF record for
+  anchordenver.com (ideally SPF + DKIM + DMARC) — the real root cause; fixes their deliverability
+  to all recipients, not just Glaztech.
+- Optional cleanup: remove the now-redundant per-user dhill allow rule `33590415` (low priority).
+
+## Reference Information
+
+- Original request thread subject: "FW: Email Delivery- Anchor Glass" (Robert Carnal ->
+  Mike, 2026-07-14 17:52).
+- MailProtector API base: https://emailservice.io/api/v1 ; skill: `.claude/skills/mailprotector/`.
+- Remediation tool token flow: `.claude/skills/remediation-tool/scripts/get-token.sh <tenant> exchange-op`.
+- Glaztech tenant ID: 82931e3c-de7a-4f74-87f7-fe714be1f160.
+- CloudFilter: glaztech domain 27635, account 16175; users jhitzke 1347467, dhill 1837854.
+- New CloudFilter allow rule: 33661305 (anchordenver.com, Allow, domain 27635).
+- Released message IDs: 4580147186, 4583375982.