sync: auto-sync from GURU-BEAST-ROG at 2026-07-20 10:25:51
Author: Mike Swanson Machine: GURU-BEAST-ROG Timestamp: 2026-07-20 10:25:51
This commit is contained in:
@@ -0,0 +1,139 @@
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** GURU-BEAST-ROG
|
||||
- **Role:** admin
|
||||
|
||||
## Session Summary
|
||||
|
||||
Robert Carnal (robert@glaztech.com, cc Steven Eastman) forwarded a request from Glaztech's
|
||||
Colorado (Denver) office manager Tami Weisbrod: the Colorado location was not receiving email
|
||||
from `zach@anchordenver.com` (Anchor Glass). Tami reported a specific recent failure — Zach's
|
||||
mail to Jimmie Hitzke (jhitzke@glaztech.com) was not delivering — and that a prior round of the
|
||||
same problem hitting Dave Hill (dhill@glaztech.com) had been fixed by someone she couldn't
|
||||
identify ("I thought this problem was behind us"). Steve Eastman was out of office.
|
||||
|
||||
Diagnosis proceeded outside-in. A public DNS check of anchordenver.com showed a Google Workspace
|
||||
domain (Google MX) with NO SPF record (only a google-site-verification TXT) and NO DMARC record,
|
||||
sending via an AWS relay (52.0.70.91) — i.e. effectively unauthenticated mail. An M365 message
|
||||
trace (Get-MessageTraceV2 through the Exchange Operator app via the InvokeCommand REST endpoint)
|
||||
proved the mail never reached Microsoft 365: zero messages zach->jhitzke in 10 days, while
|
||||
jhitzke received 90 other inbound messages in ~2 days (mailbox healthy) and a single
|
||||
zach->dhill message on 2026-07-14 was Delivered. That located the block upstream of Exchange, at
|
||||
the MailProtector CloudFilter inbound gateway (Glaztech's sole MX).
|
||||
|
||||
The mailprotector skill's CloudFilter API confirmed the block at the source: jhitzke's held queue
|
||||
contained two messages from zach@anchordenver.com, both `quarantine_spam`, score 400, reason
|
||||
"SPF None". The root cause of the recurrence was found in the CloudFilter allow/block rules —
|
||||
dhill had a per-user Allow rule (`33590415` = Allow zach@anchordenver.com) added during the prior
|
||||
incident, while jhitzke had no such rule. The earlier fix was scoped to one user and never
|
||||
generalized, so it recurred for the next Anchor contact.
|
||||
|
||||
Mike chose a domain-wide fix over the per-user precedent. Applied two gated CloudFilter writes:
|
||||
(1) a domain-level Allow rule for the whole `anchordenver.com` domain on the glaztech.com domain
|
||||
entity (27635), and (2) release of jhitzke's two held messages. Both verified. The domain rule
|
||||
supersedes the per-user dhill patch and prevents recurrence for all Glaztech recipients at every
|
||||
location. Sender-side, the durable fix belongs to Anchor Glass (publish SPF/DKIM/DMARC); flagged
|
||||
for relay to Robert.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Domain-wide allow for the entire anchordenver.com domain** (not per-user, not address-scoped).
|
||||
Rationale: the per-user precedent (dhill) caused the recurrence; Anchor is a known legitimate
|
||||
vendor emailing multiple Glaztech staff across locations. Mike accepted the tradeoff that a
|
||||
whole-domain allow on an SPF-less sender also permits spoofed anchordenver.com mail through.
|
||||
- **Fix at CloudFilter, not Exchange.** An EOP/Exchange transport rule (the usual Glaztech
|
||||
DMARC-reject workaround) would not help because the mail never reaches Exchange — it is
|
||||
quarantined at MailProtector first. The only effective lever was a CloudFilter allow rule.
|
||||
- **Used the mailprotector skill's CloudFilter API** rather than the manual partner-portal login
|
||||
the Glaztech wiki describes. The API key in the vault (`msp-tools/mailprotector.sops.yaml`)
|
||||
reaches Glaztech's domain, held mail, and rules — the "manual portal only" wiki note is
|
||||
superseded for these operations.
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **EOP message trace returned HTTP 401 `invalid_token` on the investigator-exo tier.** In the
|
||||
Glaztech tenant the Exchange Administrator role / `Exchange.ManageAsApp` is held only by the
|
||||
Exchange Operator SP (`b43e7342`), not the Security Investigator SP (`bfbc12a4`). Re-acquired the
|
||||
token on the `exchange-op` tier and the InvokeCommand/Get-MessageTraceV2 call succeeded. This is
|
||||
already documented in the Glaztech wiki access notes.
|
||||
- **CloudFilter `logs` endpoint returned empty** for the sender filter (likely time-boxed/index
|
||||
behavior), and the `messages --sender` filter did not reliably filter server-side. Worked around
|
||||
by pulling the full held-message queue for the user and identifying the anchordenver entries by
|
||||
eye.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
- **Created** CloudFilter allow rule `33661305`: value `anchordenver.com`, type Allow, on domain
|
||||
entity `glaztech.com` (27635). Verified present in `rules domains 27635`.
|
||||
- **Released** two held CloudFilter messages to jhitzke@glaztech.com: `4580147186`, `4583375982`
|
||||
(both were subject "Test" from zach@anchordenver.com; API returned them in `delivered_messages`).
|
||||
- No Exchange/M365 changes made. No repo files changed beyond this session log.
|
||||
- Pre-existing (not changed): per-user Allow rule `33590415` (Allow zach@anchordenver.com) on
|
||||
dhill — now redundant given the domain rule; left in place (harmless).
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- No credentials created or newly discovered this session.
|
||||
- MailProtector CloudFilter API key: vault `msp-tools/mailprotector.sops.yaml`, field
|
||||
`credentials.api_key` (used by the mailprotector skill; base URL https://emailservice.io/api/v1).
|
||||
- Exchange Operator app token acquired via
|
||||
`bash .claude/skills/remediation-tool/scripts/get-token.sh <tenant> exchange-op` (cert auth;
|
||||
SP `b43e7342-5b4b-492f-890f-bb5a4f7f40e9`, roles full_access_as_app + Exchange.ManageAsApp).
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **Glaztech M365 tenant:** glaztechindustries.onmicrosoft.com, tenant ID
|
||||
`82931e3c-de7a-4f74-87f7-fe714be1f160`, primary domain glaztech.com.
|
||||
- **MailProtector CloudFilter (emailservice.io):** glaztech.com domain entity `27635`; account
|
||||
"GlazTech" id `16175`. Inbound host `glaztech-com.inbound.emailservice.io` (sole MX, priority 5).
|
||||
- **CloudFilter users:** jhitzke@glaztech.com = user `1347467` (Jimmy Hitzke); dhill@glaztech.com
|
||||
= user `1837854` (Dave Hill).
|
||||
- **Sender — Anchor Glass (anchordenver.com):** Google Workspace (MX aspmx.l.google.com et al.);
|
||||
no SPF record (only `google-site-verification=YGN-6dSjhfxSmRfGxiXm5I_daxlM8go0u0GnR2ZpkCo`); no
|
||||
DMARC (`_dmarc.anchordenver.com` NXDOMAIN); observed sending IP 52.0.70.91 (AWS).
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
- DNS: `Resolve-DnsName -Type TXT/MX anchordenver.com` and `_dmarc.anchordenver.com` —
|
||||
confirmed no SPF, no DMARC, Google MX.
|
||||
- EOP trace (exchange-op token) via
|
||||
`POST https://outlook.office365.com/adminapi/beta/<tenant>/InvokeCommand` with
|
||||
`{"CmdletInput":{"CmdletName":"Get-MessageTraceV2","Parameters":{...}}}`:
|
||||
- sender zach@anchordenver.com, 2026-07-06..16: 1 result -> dhill, 2026-07-14 15:08Z, Delivered, FromIP 52.0.70.91.
|
||||
- recipient jhitzke + sender zach: count 0.
|
||||
- recipient jhitzke, any sender, 2 days: 90 Delivered (mailbox healthy).
|
||||
- CloudFilter (mailprotector skill, run from `.claude/skills/mailprotector/scripts`,
|
||||
`CLAUDETOOLS_ROOT=c:/Users/guru/ClaudeTools`):
|
||||
- `mp.py find-user jhitzke@glaztech.com` -> id 1347467, domain 27635.
|
||||
- `mp.py messages users 1347467` -> held queue; zach@anchordenver.com x2 (ids 4583375982,
|
||||
4580147186), decision quarantine_spam, score 400, result title "SPF None".
|
||||
- `mp.py rules users 1837854` -> included `33590415 Allow zach@anchordenver.com` (prior fix).
|
||||
- `mp.py add-rule domains 27635 --value anchordenver.com --type allow --confirm` -> rule 33661305.
|
||||
- `mp.py release-many users 1347467 --ids 4583375982,4580147186 --confirm` -> delivered both.
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **Reply to Robert Carnal** (robert@glaztech.com, cc seastman@glaztech.com) — he asked "let me
|
||||
know if you can or not." Draft prepared in-session (explains the SPF root cause, the allow rule,
|
||||
the released mail, and the SPF tip for Anchor). Not yet sent — Mike to choose send-as-him via
|
||||
/mailbox vs. hand off text.
|
||||
- **Syncro ticket** — billable work not yet logged. Glaztech Syncro customer ID 143932. Create
|
||||
via /syncro.
|
||||
- **Wiki update** — add the CloudFilter-layer pattern (vendor with broken SPF -> CloudFilter
|
||||
quarantine -> domain allow rule, done via mailprotector API not manual portal) to
|
||||
wiki/clients/glaztech.md. Run /wiki-compile separately.
|
||||
- **Sender-side recommendation to relay to Anchor Glass:** publish an SPF record for
|
||||
anchordenver.com (ideally SPF + DKIM + DMARC) — the real root cause; fixes their deliverability
|
||||
to all recipients, not just Glaztech.
|
||||
- Optional cleanup: remove the now-redundant per-user dhill allow rule `33590415` (low priority).
|
||||
|
||||
## Reference Information
|
||||
|
||||
- Original request thread subject: "FW: Email Delivery- Anchor Glass" (Robert Carnal ->
|
||||
Mike, 2026-07-14 17:52).
|
||||
- MailProtector API base: https://emailservice.io/api/v1 ; skill: `.claude/skills/mailprotector/`.
|
||||
- Remediation tool token flow: `.claude/skills/remediation-tool/scripts/get-token.sh <tenant> exchange-op`.
|
||||
- Glaztech tenant ID: 82931e3c-de7a-4f74-87f7-fe714be1f160.
|
||||
- CloudFilter: glaztech domain 27635, account 16175; users jhitzke 1347467, dhill 1837854.
|
||||
- New CloudFilter allow rule: 33661305 (anchordenver.com, Allow, domain 27635).
|
||||
- Released message IDs: 4580147186, 4583375982.
|
||||
Reference in New Issue
Block a user