sync: auto-sync from HOWARD-HOME at 2026-06-04 15:42:39
Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-04 15:42:39
This commit is contained in:
@@ -0,0 +1,72 @@
|
||||
# Session Log — 2026-06-04 — Howard — Chris Knight Email Delivery Investigation (bill.com / BOK)
|
||||
|
||||
## User
|
||||
- **User:** Howard Enos (howard)
|
||||
- **Machine:** Howard-Home
|
||||
- **Role:** tech
|
||||
|
||||
## Session Summary
|
||||
|
||||
Investigated why `chris.knight@cascadestucson.com` was not receiving verification / notification emails from bill.com and BOK Financial. Loaded the remediation tool, resolved the Cascades tenant (`207fa277-e9d8-4eb7-ada1-1064d2221498`), acquired a read-only Exchange Online token via the Security Investigator app (cert auth), and ran fresh message traces against Chris's mailbox and its `c.knight@` alias.
|
||||
|
||||
The trace confirmed the mailbox is healthy: 24 inbound messages in the prior two days (PrismHR, Intuit, coworkers, a test from Howard), no inbox rules, no forwarding, junk and quarantine clean, no transport rules or connectors blocking, and no Inky footprint in the tenant. Historically there were zero bill.com and zero BOK emails to either address — the senders were never delivering to the mailbox, so this was a sender-side problem, not an M365 issue.
|
||||
|
||||
The decisive evidence came mid-session: after Howard corrected the email address in BOK's portal, a BOK registration email ("Welcome to Exchange!" from `alerts@exchange.bokfinancial.com`) arrived within minutes (trace status Pending, mid-delivery). For 90 days nothing from BOK had arrived; correcting the address produced delivery immediately — proving the services simply had the wrong/unverified address on file.
|
||||
|
||||
Howard had initially proposed backing up Chris's ~40 important emails, deleting the mailbox, and recreating it to get new ID numbers. I advised against it: recreating the mailbox keeps the same email address, which is the only identifier bill.com/BOK use, so it cannot fix a sender-side suppression; and inbound delivery was already proven healthy. The recommendation was accepted and the destructive path dropped.
|
||||
|
||||
For bill.com, the email cannot be changed on the website (the user email is the locked login identity), so Cascades must call bill.com support. The bill.com root cause is the textbook signature of a SendGrid ESP suppression list — bill.com sends through SendGrid (`inform.bill.com`); once an address is on SendGrid's bounce/suppression list, every message (including verification resends) is dropped before reaching SMTP, which is exactly why repeated resends produced nothing in message trace. Closed out by creating Syncro ticket #32383, documenting the full findings, and billing 1.5h remote.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Advised against deleting/recreating Chris's mailbox.** Recreation preserves the email address (the only identifier external senders use) and inbound was proven healthy, so it could not fix a sender-side suppression — it would only risk data loss and downtime for no benefit.
|
||||
- **Used real address changes as live tests** (BOK portal email correction) rather than speculative M365 changes — produced immediate, decisive evidence.
|
||||
- **Identified the bill.com root cause as a SendGrid suppression list,** not a wrong-address-only issue, because resends to a correct address still produced nothing in trace. The fix requires bill.com support to clear the suppression, not just update the address.
|
||||
- **Billed as remote** (investigation performed remotely), regular rate, drawn from the prepaid block.
|
||||
- **Logged to the Cascades client folder** (client-specific investigation) rather than the root billing log written earlier in the day.
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **Get-MessageTrace is hard-deprecated** (as of 2025-09-01) and now returns a `BadRequest` / ValidationException instead of running. Resolved by switching to `Get-MessageTraceV2` (uses `ResultSize` instead of `PageSize`).
|
||||
- **First trace returned empty**, which looked like a true zero. Inspecting the raw response revealed the deprecation error was being swallowed by the jq filter — switching cmdlets returned full data.
|
||||
- **Working directory had drifted** into the remediation-tool skill dir from an earlier `cd`, so `whoami-block.sh` failed on a relative path. Reran with an absolute path.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
- `clients/cascades-tucson/session-logs/2026-06-04-howard-email-delivery-investigation.md` — created (this log).
|
||||
- No M365 / tenant changes made (read-only investigation).
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
None newly discovered or created. EXO read token obtained via the ComputerGuru Security Investigator app (App ID `bfbc12a4-f0dd-4e12-b06d-997e7271e10c`, cert auth), cached at `/tmp/remediation-tool/207fa277-e9d8-4eb7-ada1-1064d2221498/investigator-exo.jwt` (55-min TTL).
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- Cascades tenant ID: `207fa277-e9d8-4eb7-ada1-1064d2221498` (cascadestucson.com)
|
||||
- Affected mailbox: `chris.knight@cascadestucson.com` (+ valid alias `c.knight@cascadestucson.com`, same inbox)
|
||||
- bill.com sender: `inform.bill.com` (SendGrid ESP)
|
||||
- BOK Financial sender: `alerts@exchange.bokfinancial.com`
|
||||
- EXO REST: `https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand`
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
- `bash scripts/get-token.sh 207fa277-... investigator-exo` → cert-auth EXO token
|
||||
- `Get-MessageTrace` via InvokeCommand → `BadRequest` "Get-MessageTrace will start deprecating on September 1st, 2025... switch to Get-MessageTraceV2"
|
||||
- `Get-MessageTraceV2` (RecipientAddress / SenderAddress / StartDate / EndDate / ResultSize) → full results
|
||||
- Key result: `2026-06-04T21:59:04Z FROM alerts@exchange.bokfinancial.com SUBJ "Welcome to Exchange!" [Pending]` to chris.knight@ (and c.knight@) — arrived right after BOK address correction
|
||||
- bill.com / inform.bill.com → 0 messages to either address in 24h (and historically)
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **bill.com (open):** Cascades must CALL bill.com support to update the account email to `chris.knight@cascadestucson.com` AND clear the address from the SendGrid suppression list. Cannot be changed in the website UI. Documented on ticket #32383.
|
||||
- **BOK (near-resolved):** "Welcome to Exchange!" was Pending at last trace; Chris to complete the BOK registration. Optional follow-up: re-trace to confirm final Delivered status.
|
||||
|
||||
## Reference Information
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Ticket created | #32383 (id 112201209) — Remote - bill.com / BOK email delivery (Chris Knight) |
|
||||
| Billing | 1.5h remote (1190473) @ $150 → invoice 1650578451, $0.00 prepaid |
|
||||
| Cascades prepay block | 17.25 → 15.75 hrs |
|
||||
| Earlier same-day tickets | #32382 (Megan, id 112197080), #32381 (Tamra, id 112172438), #32298 (Desert RV, id 110582061) |
|
||||
| Ticket URL | https://computerguru.syncromsp.com/tickets/112201209 |
|
||||
Reference in New Issue
Block a user