sync: auto-sync from GURU-BEAST-ROG at 2026-06-04 16:05:04
Author: Mike Swanson Machine: GURU-BEAST-ROG Timestamp: 2026-06-04 16:05:04
This commit is contained in:
117
clients/cascades-tucson/session-logs/2026-06-04-session.md
Normal file
117
clients/cascades-tucson/session-logs/2026-06-04-session.md
Normal file
@@ -0,0 +1,117 @@
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** GURU-BEAST-ROG
|
||||
- **Role:** admin
|
||||
|
||||
> **Requester:** Howard Enos (@howard9645) via Discord bot
|
||||
> **Ticket:** Billed by Howard — resolved in separate channel
|
||||
|
||||
---
|
||||
|
||||
## Session Summary
|
||||
|
||||
Howard requested an investigation into why chris.knight@cascadestucson.com was not receiving verification emails from bill.com and BOK Financial (bokfinancial.com). The investigation began with a full M365 tenant analysis of the Cascades Tucson tenant (cascadestucson.com, tenant ID 207fa277-e9d8-4eb7-ada1-1064d2221498).
|
||||
|
||||
DNS records for cascadestucson.com were confirmed healthy: MX points directly to EOP (cascadestucson-com.mail.protection.outlook.com), SPF includes spf.protection.outlook.com and secureserver.net, DMARC is p=quarantine pct=100. No anomalies. The mailbox for chris.knight@cascadestucson.com was confirmed active, enabled, with no inbox rules, no forwarding, no inference overrides, no litigation hold, and an empty junk folder (0 items). A live test email from Howard (howard@azcomputerguru.com) was confirmed delivered in real time during the investigation.
|
||||
|
||||
Exchange Online was connected using the ComputerGuru Exchange Operator MSP app (client credentials / access token method) against the Cascades tenant. Message trace (Get-MessageTraceV2), quarantine, transport rules, anti-spam policies, connection filter, add-ins, inbound/outbound connectors, and Azure AD enterprise apps were all checked. No Inky deployment was found anywhere in the tenant (no connector, no transport rule, no OAuth app, no add-in). Howard had suspected Inky as a factor; this was ruled out definitively.
|
||||
|
||||
Bill.com (inform.bill.com, hq.bill.com, hello.bill.com) was confirmed delivering successfully to other Cascades users (Meredith Kuhn, Ashley Jensen, Lauren Hasselman, Zachary Nelson) throughout the investigation period. However, zero bill.com emails had ever been delivered to chris.knight@cascadestucson.com or its alias c.knight@cascadestucson.com in 90 days of message trace history. BOK Financial (bokfinancial.com) showed zero emails to any cascadestucson.com user in 90 days. Howard confirmed the bill.com account had been updated to chris.knight@cascadestucson.com; a live resend was triggered by Meredith Kuhn during the session. That resend did not arrive at the tenant within 5+ minutes of monitoring, indicating it was still routing to the old wrong email address in bill.com's backend. The issue was resolved separately in another channel and billed by Howard.
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- Used ComputerGuru Exchange Operator MSP app (b43e7342-5b4b-492f-890f-bb5a4f7f40e9) with client_credentials flow to obtain an EXO access token and connect via `Connect-ExchangeOnline -AccessToken` — admin credential auth was blocked (MFA/modern auth constraint), and the app cert was not in the Windows cert store on BEAST. Access token method bypassed both issues.
|
||||
- Searched for "bills.com" initially; corrected to "bill.com" (BILL) after fetching BILL's help article which lists inform.bill.com, hq.bill.com, hello.bill.com, mc.bill.com as their sending domains.
|
||||
- Checked c.knight@cascadestucson.com alias after Howard raised it — confirmed it is a valid secondary SMTP proxy address on the same mailbox, but also showed zero bill.com/BOK emails.
|
||||
- Treated "box.com" as a typo for "bok.com" / "bokfinancial.com" after Howard clarified. Both checked thoroughly with zero results.
|
||||
- Get-MessageTraceV2 has a 10-day max window; ran 9 consecutive 10-day windows to cover 90 days of history.
|
||||
|
||||
---
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **Get-MessageTrace deprecated:** Get-MessageTrace returned a deprecation warning and empty results. Switched to Get-MessageTraceV2. V2 has a 10-day window limit (not 30 as initially attempted); looped windows to cover 90 days.
|
||||
- **EXO credential auth blocked:** `Connect-ExchangeOnline -Credential` failed with token error (MFA/modern auth). MSP app cert was not in Windows cert store on BEAST and only the private key PEM is stored in vault (no public cert). Resolved by using client_credentials flow to get an EXO-scoped access token and passing it via `-AccessToken` parameter.
|
||||
- **Graph API inbox rule check returned 401:** Initial token was pasted as a literal string (stale). Fixed by generating a fresh token inline in the same PowerShell session block.
|
||||
- **Wildcard sender search returned false positives:** `*@bills.com` with 30-day window returned "1 message found" across all subdomains — artifact of the wildcard matching and the 30-day limit violation. Narrowed to 10-day windows and specific domains to get clean results.
|
||||
|
||||
---
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
None. Investigation only — no changes made to the tenant or any configuration.
|
||||
|
||||
---
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- **Cascades Tucson M365 admin:** `clients/cascades-tucson/m365-admin.sops.yaml` — admin@cascadestucson.com (used for reference; auth via MSP app instead)
|
||||
- **ComputerGuru Exchange Operator MSP app:** `msp-tools/computerguru-exchange-operator.sops.yaml`
|
||||
- App ID: b43e7342-5b4b-492f-890f-bb5a4f7f40e9
|
||||
- Client secret used for client_credentials token acquisition
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **Cascades Tucson M365 Tenant ID:** 207fa277-e9d8-4eb7-ada1-1064d2221498
|
||||
- **Cascades Tucson primary domain:** cascadestucson.com
|
||||
- **MX:** cascadestucson-com.mail.protection.outlook.com (EOP direct, no third-party gateway)
|
||||
- **SPF:** v=spf1 a mx ip4:72.194.62.5 include:spf.protection.outlook.com include:spf-0.secureserver.net -all
|
||||
- **DMARC cascadestucson.com:** p=quarantine; pct=100; rua=info@cascadestucson.com
|
||||
- **Bill.com sending domains:** inform.bill.com, hq.bill.com, hello.bill.com, mc.bill.com, bill.com — MX via pphosted.com (Proofpoint)
|
||||
- **BOK Financial:** bokfinancial.com — MX via pphosted.com (Proofpoint); DMARC p=reject
|
||||
- **ComputerGuru Exchange Operator MSP tenant:** ce61461e-81a0-4c84-bb4a-7b354a9a356d (ACG tenant)
|
||||
|
||||
---
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
```powershell
|
||||
# EXO connection via access token (bypasses cert/MFA requirement)
|
||||
$body = @{ grant_type='client_credentials'; client_id=$clientId; client_secret=$clientSecret; scope='https://outlook.office365.com/.default' }
|
||||
$tok = Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token" -Body $body -ContentType 'application/x-www-form-urlencoded'
|
||||
Connect-ExchangeOnline -AccessToken $tok.access_token -Organization 'cascadestucson.com' -AppId $clientId -ShowBanner:$false
|
||||
|
||||
# 90-day message trace loop (Get-MessageTraceV2 max 10-day window)
|
||||
for ($i = 0; $i -lt 9; $i++) {
|
||||
$wend = (Get-Date).AddDays(-($i * 10))
|
||||
$wstart = $wend.AddDays(-10)
|
||||
Get-MessageTraceV2 -SenderAddress "*@inform.bill.com" -RecipientAddress "chris.knight@cascadestucson.com" -StartDate $wstart -EndDate $wend
|
||||
}
|
||||
```
|
||||
|
||||
Key findings:
|
||||
- chris.knight mailbox: 30 messages in 10 days, all delivered, all non-bill.com/BOK
|
||||
- c.knight alias: 2 messages in 10 days (Howard test 5/28 "email address name change", Chris personal test)
|
||||
- bill.com to tenant: active delivery to meredith.kuhn, ashley.jensen, lauren.hasselman, zachary.nelson — confirms no tenant-wide block
|
||||
- bill.com to chris.knight: 0 messages in 90 days
|
||||
- BOK Financial to entire tenant: 0 messages in 90 days
|
||||
- Live resend by Meredith: 0 messages arrived within 5-minute monitoring window
|
||||
|
||||
Anti-spam policies:
|
||||
- Default: Spam/HighConfSpam/Bulk → MoveToJmf; HighConfPhish → Quarantine
|
||||
- Standard Preset: HighConfSpam/Phish → Quarantine (applies to all recipients per EOPProtectionPolicyRule)
|
||||
|
||||
Transport rules: "Fax Forward and Retain Copy" (P0), "Allow WordPress contact form" (P1) — neither relevant.
|
||||
|
||||
Connection filter: IPBlockList = 89.106.1.38 (single entry, unrelated to bill.com/BOK).
|
||||
|
||||
---
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- Root cause of bill.com resend not arriving was not confirmed before the session ended — most likely bill.com's backend still has the old email address despite the profile showing the new one. Resolved externally.
|
||||
- BOK Financial email address for Chris Knight was never confirmed — likely still the old wrong address.
|
||||
|
||||
---
|
||||
|
||||
## Reference Information
|
||||
|
||||
- BILL help article (sending domains): https://help.bill.com/direct/s/article/360000026246
|
||||
- BILL safe sender: account-services@inform.bill.com
|
||||
- BOK Financial: https://www.bokfinancial.com/
|
||||
- Cascades Tucson client wiki: wiki/clients/cascades-tucson.md
|
||||
- MSP Exchange Operator vault: msp-tools/computerguru-exchange-operator.sops.yaml
|
||||
Reference in New Issue
Block a user