sync: auto-sync from HOWARD-HOME at 2026-06-04 15:42:39

Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-04 15:42:39
2026-06-04 15:42:47 -07:00
parent e61966db20
commit 532be659de
3 changed files with 92 additions and 6 deletions
--- a/clients/cascades-tucson/session-logs/2026-06-04-howard-email-delivery-investigation.md
+++ b/clients/cascades-tucson/session-logs/2026-06-04-howard-email-delivery-investigation.md
@@ -0,0 +1,72 @@
+# Session Log — 2026-06-04 — Howard — Chris Knight Email Delivery Investigation (bill.com / BOK)
+
+## User
+- **User:** Howard Enos (howard)
+- **Machine:** Howard-Home
+- **Role:** tech
+
+## Session Summary
+
+Investigated why `chris.knight@cascadestucson.com` was not receiving verification / notification emails from bill.com and BOK Financial. Loaded the remediation tool, resolved the Cascades tenant (`207fa277-e9d8-4eb7-ada1-1064d2221498`), acquired a read-only Exchange Online token via the Security Investigator app (cert auth), and ran fresh message traces against Chris's mailbox and its `c.knight@` alias.
+
+The trace confirmed the mailbox is healthy: 24 inbound messages in the prior two days (PrismHR, Intuit, coworkers, a test from Howard), no inbox rules, no forwarding, junk and quarantine clean, no transport rules or connectors blocking, and no Inky footprint in the tenant. Historically there were zero bill.com and zero BOK emails to either address — the senders were never delivering to the mailbox, so this was a sender-side problem, not an M365 issue.
+
+The decisive evidence came mid-session: after Howard corrected the email address in BOK's portal, a BOK registration email ("Welcome to Exchange!" from `alerts@exchange.bokfinancial.com`) arrived within minutes (trace status Pending, mid-delivery). For 90 days nothing from BOK had arrived; correcting the address produced delivery immediately — proving the services simply had the wrong/unverified address on file.
+
+Howard had initially proposed backing up Chris's ~40 important emails, deleting the mailbox, and recreating it to get new ID numbers. I advised against it: recreating the mailbox keeps the same email address, which is the only identifier bill.com/BOK use, so it cannot fix a sender-side suppression; and inbound delivery was already proven healthy. The recommendation was accepted and the destructive path dropped.
+
+For bill.com, the email cannot be changed on the website (the user email is the locked login identity), so Cascades must call bill.com support. The bill.com root cause is the textbook signature of a SendGrid ESP suppression list — bill.com sends through SendGrid (`inform.bill.com`); once an address is on SendGrid's bounce/suppression list, every message (including verification resends) is dropped before reaching SMTP, which is exactly why repeated resends produced nothing in message trace. Closed out by creating Syncro ticket #32383, documenting the full findings, and billing 1.5h remote.
+
+## Key Decisions
+
+- **Advised against deleting/recreating Chris's mailbox.** Recreation preserves the email address (the only identifier external senders use) and inbound was proven healthy, so it could not fix a sender-side suppression — it would only risk data loss and downtime for no benefit.
+- **Used real address changes as live tests** (BOK portal email correction) rather than speculative M365 changes — produced immediate, decisive evidence.
+- **Identified the bill.com root cause as a SendGrid suppression list,** not a wrong-address-only issue, because resends to a correct address still produced nothing in trace. The fix requires bill.com support to clear the suppression, not just update the address.
+- **Billed as remote** (investigation performed remotely), regular rate, drawn from the prepaid block.
+- **Logged to the Cascades client folder** (client-specific investigation) rather than the root billing log written earlier in the day.
+
+## Problems Encountered
+
+- **Get-MessageTrace is hard-deprecated** (as of 2025-09-01) and now returns a `BadRequest` / ValidationException instead of running. Resolved by switching to `Get-MessageTraceV2` (uses `ResultSize` instead of `PageSize`).
+- **First trace returned empty**, which looked like a true zero. Inspecting the raw response revealed the deprecation error was being swallowed by the jq filter — switching cmdlets returned full data.
+- **Working directory had drifted** into the remediation-tool skill dir from an earlier `cd`, so `whoami-block.sh` failed on a relative path. Reran with an absolute path.
+
+## Configuration Changes
+
+- `clients/cascades-tucson/session-logs/2026-06-04-howard-email-delivery-investigation.md` — created (this log).
+- No M365 / tenant changes made (read-only investigation).
+
+## Credentials & Secrets
+
+None newly discovered or created. EXO read token obtained via the ComputerGuru Security Investigator app (App ID `bfbc12a4-f0dd-4e12-b06d-997e7271e10c`, cert auth), cached at `/tmp/remediation-tool/207fa277-e9d8-4eb7-ada1-1064d2221498/investigator-exo.jwt` (55-min TTL).
+
+## Infrastructure & Servers
+
+- Cascades tenant ID: `207fa277-e9d8-4eb7-ada1-1064d2221498` (cascadestucson.com)
+- Affected mailbox: `chris.knight@cascadestucson.com` (+ valid alias `c.knight@cascadestucson.com`, same inbox)
+- bill.com sender: `inform.bill.com` (SendGrid ESP)
+- BOK Financial sender: `alerts@exchange.bokfinancial.com`
+- EXO REST: `https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand`
+
+## Commands & Outputs
+
+- `bash scripts/get-token.sh 207fa277-... investigator-exo` → cert-auth EXO token
+- `Get-MessageTrace` via InvokeCommand → `BadRequest` "Get-MessageTrace will start deprecating on September 1st, 2025... switch to Get-MessageTraceV2"
+- `Get-MessageTraceV2` (RecipientAddress / SenderAddress / StartDate / EndDate / ResultSize) → full results
+- Key result: `2026-06-04T21:59:04Z FROM alerts@exchange.bokfinancial.com SUBJ "Welcome to Exchange!" [Pending]` to chris.knight@ (and c.knight@) — arrived right after BOK address correction
+- bill.com / inform.bill.com → 0 messages to either address in 24h (and historically)
+
+## Pending / Incomplete Tasks
+
+- **bill.com (open):** Cascades must CALL bill.com support to update the account email to `chris.knight@cascadestucson.com` AND clear the address from the SendGrid suppression list. Cannot be changed in the website UI. Documented on ticket #32383.
+- **BOK (near-resolved):** "Welcome to Exchange!" was Pending at last trace; Chris to complete the BOK registration. Optional follow-up: re-trace to confirm final Delivered status.
+
+## Reference Information
+
+| Item | Value |
+|---|---|
+| Ticket created | #32383 (id 112201209) — Remote - bill.com / BOK email delivery (Chris Knight) |
+| Billing | 1.5h remote (1190473) @ $150 → invoice 1650578451, $0.00 prepaid |
+| Cascades prepay block | 17.25 → 15.75 hrs |
+| Earlier same-day tickets | #32382 (Megan, id 112197080), #32381 (Tamra, id 112172438), #32298 (Desert RV, id 110582061) |
+| Ticket URL | https://computerguru.syncromsp.com/tickets/112201209 |
--- a/wiki/clients/cascades-tucson.md
+++ b/wiki/clients/cascades-tucson.md
@@ -2,8 +2,8 @@
 type: client
 name: cascades-tucson
 display_name: Cascades of Tucson
-last_compiled: 2026-06-03
-compiled_by: HOWARD-HOME/claude-main
+last_compiled: 2026-06-04
+compiled_by: Howard-Home/claude-main
 sources:
  - session-logs/2026-03-24-session.md
  - session-logs/2026-03-31-session.md
@@ -33,6 +33,7 @@ sources:
  - session-logs/2026-05-26-howard-session.md
  - clients/cascades-tucson/session-logs/2026-06-02-howard-efax-scanner-ticket.md
  - clients/cascades-tucson/session-logs/2026-06-03-session.md
+  - clients/cascades-tucson/session-logs/2026-06-04-howard-email-delivery-investigation.md
  - clients/cascades-tucson/docs/overview.md
  - clients/cascades-tucson/docs/network/topology.md
  - clients/cascades-tucson/docs/network/vlans.md
@@ -69,13 +70,16 @@ Senior living / assisted living facility in Tucson, AZ. Single 6-floor building
  - Ashley Jensen — Accountant (DESKTOP-U2DHAP0)
  - Shelby Trozzi — MemCare Director (MDIRECTOR-PC)
 - **Billing rate:** $175/hr all labor (prepaid block customer)
- **Hours remaining:** ~28.0 hrs as of 2026-05-26. Always live-check via `GET /customers/20149445` before billing — balance is unreliable across sessions.
+- **Hours remaining:** 15.75 hrs as of 2026-06-04 (after tickets #32381 0.5h onsite, #32382 1.5h onsite, #32383 1.5h remote billed 2026-06-04). Always live-check via `GET /customers/20149445` before billing — balance is unreliable across sessions.
 - **Syncro customer ID:** 20149445
 - **Active tickets:**
  - #110680053 — Dept-by-dept domain migration (primary active project; plan: `C:\Users\Howard\.claude\plans\wise-discovering-panda.md`)
  - #109412123 — Entra setup project (may be invoiced as of 2026-05-18; verify status)
  - #109035475 — John Trozzi desktop WiFi upgrade (billed)
  - #32370 — eFax setup on Karen's and Christin's machines + portable scanner setup on both (Howard onsite; no appointment scheduled yet; ticket open/pending 2026-06-02)
+  - #32381 — Tamra scanner onsite (0.5h onsite, billed 2026-06-04, prepaid block)
+  - #32382 — Megan file access onsite (1.5h onsite, billed 2026-06-04, prepaid block)
+  - #32383 — Chris Knight bill.com / BOK email delivery (1.5h remote, billed 2026-06-04, prepaid block; Syncro id 112201209)

 ---

@@ -158,6 +162,11 @@ Senior living / assisted living facility in Tucson, AZ. Single 6-floor building
 - **Billing product for prepaid block draw:** Use a real labor type (Remote, Onsite, etc.) — NOT "Prepaid project labor" (exempt, won't decrement the block).
 - **Always live-check hours before billing:** `GET /customers/20149445` in Syncro. The 2026-05-01 invoice debit may not have fired correctly — treat all cached hour counts as approximate.

+### Exchange Online / Message Tracing
+
+- **Get-MessageTrace is hard-deprecated (Sept 2025).** As of 2025-09-01, `Get-MessageTrace` returns `BadRequest` / `ValidationException` via EXO InvokeCommand. Use `Get-MessageTraceV2` instead. Key parameter change: use `ResultSize` (not `PageSize`). The deprecation error may be silently swallowed by downstream jq filters — if a trace returns unexpectedly empty, check the raw response for a deprecation error string before assuming no mail. Source: 2026-06-04 Chris Knight investigation.
+- **Sender-side suppression (SendGrid ESP):** If a user never receives mail from a specific sender despite a healthy mailbox, and message trace shows zero records (not even bounces), consider a SendGrid suppression list. Resends will also fail silently. Fix requires contacting the sender's support to clear the suppression — there is no M365 action that can resolve this. Confirmed with bill.com / inform.bill.com. Pattern also applies to other high-volume senders using SendGrid.
+
 ### Active Directory / User Management

 - **Security group assignment is always explicit.** When creating or adding any Cascades user, always ask which security group(s). OU → group auto-mirror was explicitly declined 2026-05-14. OU placement controls Entra Connect sync scope; group membership controls CA policy — two separate deliberate decisions. Source: `feedback_cascades_user_security_group.md`.
@@ -215,6 +224,7 @@ Senior living / assisted living facility in Tucson, AZ. Single 6-floor building
 - **Canva email delivery (2026-05-20):** Alma Montt not receiving Canva invites. Resolved by adding canva.com domains to AllowedSenderDomains in EOP policies.
 - **ALIS AADSTS65001 (2026-06-03):** megan.hiatt, karen.rossini, memcarereceptionist could not sign in to ALIS on non-phone devices. Root cause: missing tenant-wide admin consent on ALIS SP (`e1cae4ad`). Resolved by granting `AllPrincipals` `User.Read` via Graph API. CA was NOT the cause — all failures showed `conditionalAccessStatus: success` from trusted IPs.
 - **dunedolly21@gmail.com:** External guest invited 2026-04-14 by Lauren Hasselman from mobile. Status unknown — confirm with Lauren. [unverified]
+- **Chris Knight bill.com / BOK email delivery (2026-06-04):** `chris.knight@cascadestucson.com` (alias: `c.knight@cascadestucson.com`) not receiving bill.com or BOK Financial emails. M365 mailbox confirmed healthy: 24 inbound messages traced over prior 48h, no inbox rules, no forwarding, no junk/quarantine hits, no transport rules or connectors blocking. Root cause: SENDER-SIDE, not M365. bill.com sends via SendGrid (`inform.bill.com`); the address was on SendGrid's ESP suppression list — mail dropped before SMTP, so nothing appeared in message trace and repeated resends never arrived. BOK diagnosis confirmed: correcting the email in BOK's portal produced a "Welcome to Exchange!" delivery from `alerts@exchange.bokfinancial.com` within minutes. **bill.com fix requires calling bill.com support** — the account email cannot be changed in the web UI (it is the locked login identity); support must update it AND clear the SendGrid suppression. Ticket #32383, 1.5h remote.

 ### HIPAA Compliance

@@ -251,6 +261,7 @@ Primary active project as of 2026-05-24: dept-by-dept domain migration (Syncro #
 - Entra Connect: OU=Administrative not yet in sync scope; UPN suffix updates for that OU pending
 - NURSESTATION-PC: auto-lock GPO (HIPAA, ~10 min idle) not yet applied
 - #32370 (open): Howard onsite — eFax setup on Karen's and Christin's machines; portable scanner setup on both. No appointment scheduled as of 2026-06-02.
+- #32383 (open — pending customer action): bill.com email delivery for Chris Knight. Cascades must CALL bill.com support to update account email to `chris.knight@cascadestucson.com` AND clear it from the SendGrid suppression list (cannot be done via web UI). BOK side near-resolved (address corrected; Chris to complete registration). Ticket logged 2026-06-04; investigation billed 1.5h remote.
 - Caregiver device allow-list: 4 laptops need Entra-join + Intune-enroll + `extensionAttribute1` tagging before cutover (see Patterns section)
 - ALIS office/privileged standardization: move office/managers/nurses to ALIS SSO-only; disable ALIS-native 2FA per-user then globally (separate workstream)
 - Fix stale `SG-Caregivers-Pilot` exclude-group on `Require MFA for all users` policy (known bug, see Known Issues)
@@ -284,26 +295,29 @@ Primary active project as of 2026-05-24: dept-by-dept domain migration (Syncro #
 | 2026-05-24 | RECEPTIONIST-PC GuruRMM agent noted as 0.6.37 straggler while fleet at 0.6.38. Flaky WebSocket. |
 | 2026-05-26 | Access control vendor meeting onsite (ticket #32324). 0.5h Howard + 0.5h Mike billed against prepaid block. Block at 28.0h. Remote diagnosis of UniFi controller confirmed impossible (no Tailscale route, GuruRMM WebSocket-only, pfSense SSH blocked). |
 | 2026-06-03 | ALIS AADSTS65001 diagnosed and resolved: granted tenant-wide admin consent (`AllPrincipals` `User.Read`) on ALIS SP `e1cae4ad`. Caregiver device allow-list CA policy created in report-only (`CSC - Caregivers: allow-listed devices only (REPORT-ONLY)`, id `1b7fd025`). Allow-list = CSC- phones + 5 tagged devices (NURSESTATION-PC, Laptop2, LAPTOP-8P7HDSEI, LAPTOP-DRQ5L558, LAPTOP-E0STJJE8). Cutover pending laptop Intune enrollment + validation. Three existing enforced caregiver CA policies left untouched. |
+| 2026-06-04 | Three same-day tickets: #32381 Tamra scanner (0.5h onsite), #32382 Megan file access (1.5h onsite), #32383 Chris Knight bill.com/BOK email delivery (1.5h remote). Chris Knight mailbox investigation confirmed healthy — root cause was sender-side SendGrid suppression on bill.com side; BOK resolved by correcting email in portal (delivery within minutes). Prepay block: 17.25 → 15.75 hrs. |

 ---

 ## Compilation Notes

-**Session logs read:** 25 root session logs + client-specific logs in `clients/cascades-tucson/session-logs/` + 7 memory files + 5 structured docs. Date range: 2026-03-06 through 2026-06-03.
+**Session logs read:** 25 root session logs + client-specific logs in `clients/cascades-tucson/session-logs/` + 7 memory files + 5 structured docs. Date range: 2026-03-06 through 2026-06-04.

 **Client folder:** `clients/cascades-tucson/` (NOT `clients/cascades/` — that directory does not exist).

 **Open items flagged as unverified:**
- Hour balance — always live-check; treat cached counts as approximate
+- Hour balance — always live-check; treat cached counts as approximate (15.75 hrs derived from session log; not a live Syncro pull)
 - Break-glass accounts + YubiKeys — confirmed not created as of 2026-05-27; YubiKey arrival unconfirmed
 - Audit retention infra — approved 2026-04-29, not yet built
 - dunedolly21@gmail.com guest invite — confirm with Lauren
 - Windows MDM auto-enroll scope — confirm in portal (Entra → Devices → Mobility → Microsoft Intune → MDM user scope)
+- #32381 / #32382 ticket details (Tamra scanner, Megan file access) — referenced in 2026-06-04 session log reference table only; full ticket details not documented in session logs

 **Resolved since last compile:**
 - New tiered remediation app suite — confirmed consented 2026-04-21 (all 6 apps active)
 - DMARC — confirmed upgraded to p=quarantine;pct=100
 - ALIS AADSTS65001 sign-in failures — resolved 2026-06-03 by granting admin consent
+- BOK Financial email delivery for Chris Knight — resolved 2026-06-04 by correcting email in BOK portal (bill.com side still requires support call)

 ## Backlinks

--- a/wiki/index.md
+++ b/wiki/index.md
@@ -18,7 +18,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks.

 | Article | Summary | Last Compiled |
 |---|---|---|
-| [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, ~28.0 hrs remaining; senior living; active domain migration + HIPAA compliance project; single DC on aging R610 hardware; ALIS admin consent granted 2026-06-03 (resolved AADSTS65001); caregiver device allow-list CA policy staged (report-only); open ticket #32370 (eFax + scanner onsite) | 2026-06-03 |
+| [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, 15.75 hrs remaining; senior living; active domain migration + HIPAA compliance project; single DC on aging R610 hardware; ALIS admin consent granted 2026-06-03 (resolved AADSTS65001); caregiver device allow-list CA policy staged (report-only); open ticket #32370 (eFax + scanner onsite); #32383 bill.com/BOK email delivery (sender-side SendGrid suppression — bill.com support call pending) | 2026-06-04 |
 | [Dataforth Corporation](clients/dataforth.md) | Prepaid block ~$2,099/mo; signal conditioning manufacturer; 64 DOS test stations; 2025 crypto attack recovery; 2026-03-27 phishing incident + MFA rollout; active test datasheet pipeline project; Neptune Exchange colocated at D2; 2026-06-02 Syncro asset reconciliation (78→20 keep/21 flag/28 remove/9 verify); fleet-wide Syncro agent break ~2025-10-06; Bitdefender phase-off in progress | 2026-06-02 |
 | [Instrumental Music Center](clients/instrumental-music-center.md) | Prepaid block $175/hr, 12.5 hrs remaining; music retail/repair; AIMsi POS on SQL Server 2019; phantom DC causing slow logons; GuruRMM enrolled (IMC1) | 2026-05-24 |
 | [Valley Wide Plastering](clients/valleywide.md) | Prepaid block, 10 hrs remaining; plastering/stucco contractor; HP DL360 Gen10 + XenServer; VB6 app modernization project; RDWeb brute-force incident; 11 Yealink phones pending | 2026-05-24 |