fix(remediation): close the recurring Exchange-Admin-role gap fleet-wide

EXO email-cleanup tasks (Search-UnifiedAuditLog, Get-MessageTrace, inbox rules) kept
401/403-ing per tenant because the Exchange Operator SP was missing the Exchange Admin
directory role — admin consent grants Exchange.ManageAsApp but never the directory role.
onboard-tenant.sh assigns it, but tenants consented before that step / by hand never got
it, and nothing audited for it. Hence the recurring 'next onboarding will fix it' (false
for already-onboarded tenants).

- NEW assign-exchange-role.sh: idempotent role assignment via the authoritative
  roleManagement/directory/roleAssignments API (the legacy directoryRoles/members list
  reads back unreliably). <domain|--all> + --verify/--dry-run.
- Backfilled the whole fleet (--all): 13 stragglers ASSIGNED, 12 already OK, 20 skipped
  (tenant-admin not consented), 0 errors. Safe Site included.
- Standing audit documented (assign-exchange-role.sh --all --verify) + memory so no future
  session repeats the empty promise.
- Adds wiki/clients/safesite.md (tenant + 4-source endpoint inventory + investigation).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.claude/memory/MEMORY.md

@@ -54,6 +54,7 @@
 - [Graph CA policy reads are eventually consistent](feedback_graph_ca_policy_eventual_consistency.md) — After PATCHing a CA policy (204), wait ~5s before GET-verifying; immediate reads can be stale.
 - [Graph password reset needs a privileged role](feedback_graph_password_reset_requires_role.md) — PATCH passwordProfile on an existing user 403s without a directory role; User.ReadWrite.All alone only sets a password at CREATE.
 - [Vault writes — do the full sequence yourself](feedback_complete_vault_operations_end_to_end.md) — A vault entry = write plaintext → sops -e -i → git add/commit/push, all of it; don't stop at "encrypted on disk."
+- [Exchange role recurring gap — backfill, don't promise](feedback_exchange_role_recurring_gap.md) — EXO email-cleanup 401/403 = Exchange Operator SP missing the Exchange Admin directory role (consent never grants it). Fix: `assign-exchange-role.sh <domain|--all>` (idempotent); audit with `--all --verify`. Fleet backfilled 2026-06-08. Verify membership via roleManagement/directory/roleAssignments (not the laggy directoryRoles/members list); EXO propagation 15-60min.
 - [Syncro is the default PSA; Autotask is opt-in](feedback_psa_default_syncro.md) — Ticketing/billing/customers default to Syncro (/syncro). Only use /autotask on an explicit "in Autotask" request. /autotask kept local/undistributed.
 - [Paste-safe command formatting (Howard)](feedback_command_formatting.md) — Two clauses, one root cause: (a) multi-line scripts not semicolon one-liners (wrap breaks paste), (b) all code at column 0 inside fences (indentation breaks PowerShell paste).
 - [Autonomous infra/build setup](feedback_autonomous_infra_setup.md) — During infra/build/CI/dev setup, just install prerequisites and push through routine steps; reserve check-ins for genuine decisions (forks, destructive/outward, client/prod).

.claude/memory/feedback_exchange_role_recurring_gap.md

@@ -0,0 +1,18 @@
+---
+name: feedback_exchange_role_recurring_gap
+description: Exchange email-cleanup tasks fail with 401/403 because the EXO app SP is missing the Exchange Admin directory role — fix via the backfill script, never promise "next onboarding will fix it"
+metadata:
+  type: feedback
+---
+
+Email-cleanup / mailbox-forensic tasks (Search-UnifiedAuditLog, Get-MessageTrace, Get/Remove-InboxRule, Set-Mailbox) kept failing per-tenant with EXO 401/403, and each session hand-waved "it'll be auto-added next onboarding." Mike (2026-06-08) called this out as recurring disappointment. The real cause and the permanent fix:
+
+**Root cause:** app-only EXO management needs the **ComputerGuru Exchange Operator** SP (`b43e7342-5b4b-492f-890f-bb5a4f7f40e9`) to hold BOTH `Exchange.ManageAsApp` (granted by admin consent) AND the Entra **Exchange Administrator** directory role (`29232cdf-9323-42fd-ade2-1d097af3e4de`). Admin consent grants the API permission but NEVER the directory role. `onboard-tenant.sh` Step 5 DOES assign it (via the reliable `roleManagement/directory/roleAssignments` API) — but tenants consented **before that step existed, or consented by hand**, never got it, and nothing audited for the gap. So the recurrence was old/manual stragglers, not an onboarding bug.
+
+**The fix (do this, don't promise):**
+- `bash .claude/skills/remediation-tool/scripts/assign-exchange-role.sh <domain|--all> [--verify|--dry-run]` — assigns the role to the Exchange Operator SP. Idempotent. `--all` backfills every tenant in `references/tenants.md`; tenants where tenant-admin isn't consented are SKIPped. **Backfilled fleet-wide 2026-06-08** (~10 stragglers fixed).
+- **Standing audit:** run `assign-exchange-role.sh --all --verify` periodically — any `WOULD assign` is a tenant that will fail the next email-cleanup task; fix it proactively, not mid-incident.
+- **Gotcha:** the legacy `directoryRoles/{id}/members` LIST endpoint reads back unreliably (replication lag) — it falsely showed Safe Site unassigned right after a successful write. Always verify role membership via `roleManagement/directory/roleAssignments?$filter=principalId eq '<sp>'`, not the members list.
+- **Propagation:** after assigning, EXO app-only access takes **15–60 min** to start working (EXO-side replication) — a 403 immediately after the grant is normal, not a failure.
+
+**Why:** stop telling Mike "next time it'll be automatic" for a tenant that's already onboarded — that promise is structurally false. The durable answer is the backfill + the standing `--verify` audit. See [[reference_acg_msp_stack]] and the remediation-tool tenants reference.

.claude/skills/remediation-tool/references/tenants.md

@@ -5,6 +5,14 @@ Last updated: 2026-04-20. Source of truth: CIPP ListTenants API.
 Run `bash scripts/onboard-tenant.sh <domain>` after any tenant consents Tenant Admin.
 After full onboarding, update the Onboarded column below.
 
+**Exchange access (recurring gap — now closed):** EXO management (audit log, message trace, inbox
+rules) needs the **Exchange Operator SP** to hold the **Exchange Administrator** directory role, which
+admin consent does NOT grant. Onboarding assigns it, but tenants consented before that step / by hand
+were missing it. Fleet **backfilled 2026-06-08** (13 stragglers fixed). **Standing audit:** run
+`bash scripts/assign-exchange-role.sh --all --verify` periodically — any `WOULD assign` is a tenant
+that will fail the next email task; fix it with `assign-exchange-role.sh <domain>`. See
+[[feedback_exchange_role_recurring_gap]].
+
 ## Tenant List
 
 | Display Name | Domain | Tenant ID | Onboarded | Notes |

.claude/skills/remediation-tool/scripts/assign-exchange-role.sh

@@ -70,30 +70,25 @@ process_one() {
     sp_id="$(gget "$tok" "$GRAPH/servicePrincipals?\$filter=appId%20eq%20'$EXCHANGE_OP_APPID'&\$select=id" | jqr '.value[0].id // empty')"
     if [ -z "$sp_id" ]; then echo "SKIP  (Exchange Operator app not consented in tenant)"; return; fi
 
-    # find or (if applying) activate the Exchange Administrator directory role
-    role_id="$(gget "$tok" "$GRAPH/directoryRoles?\$filter=roleTemplateId%20eq%20'$EXCH_ADMIN_TEMPLATE'" | jqr '.value[0].id // empty')"
-    if [ -z "$role_id" ]; then
-        if [ "$MODE" = "apply" ]; then
-            role_id="$(curl -s --max-time 25 -X POST "$GRAPH/directoryRoles" \
-                -H "Authorization: Bearer $tok" -H "Content-Type: application/json" \
-                -d "{\"roleTemplateId\":\"$EXCH_ADMIN_TEMPLATE\"}" | tr -d '\000' | jqr '.id // empty')"
-            [ -z "$role_id" ] && { echo "ERROR (could not activate Exchange Admin role)"; return; }
-        else
-            echo "WOULD activate Exchange Admin role + assign SP $sp_id"; return
-        fi
-    fi
-
-    present="$(gget "$tok" "$GRAPH/directoryRoles/$role_id/members?\$select=id" | jqr --arg s "$sp_id" '[.value[]?|select(.id==$s)]|length')"
+    # Use the AUTHORITATIVE unified role-assignment API (roleManagement/directory/roleAssignments)
+    # for both the idempotency check and the write. The legacy directoryRoles/{id}/members list
+    # reads back unreliably (replication lag) and falsely reports not-assigned; roleAssignments is
+    # consistent. For built-in roles, roleDefinitionId == the roleTemplateId.
+    present="$(gget "$tok" "$GRAPH/roleManagement/directory/roleAssignments?\$filter=principalId%20eq%20'$sp_id'%20and%20roleDefinitionId%20eq%20'$EXCH_ADMIN_TEMPLATE'" | jqr '.value | length')"
     if [ "${present:-0}" -gt 0 ] 2>/dev/null; then echo "OK    (already assigned)"; return; fi
 
     if [ "$MODE" != "apply" ]; then echo "WOULD assign Exchange Admin to SP $sp_id"; return; fi
 
-    rc="$(curl -s --max-time 25 -o /tmp/aer_resp.$$ -w '%{http_code}' -X POST "$GRAPH/directoryRoles/$role_id/members/\$ref" \
+    rc="$(curl -s --max-time 25 -o /tmp/aer_resp.$$ -w '%{http_code}' -X POST "$GRAPH/roleManagement/directory/roleAssignments" \
         -H "Authorization: Bearer $tok" -H "Content-Type: application/json" \
-        -d "{\"@odata.id\":\"$GRAPH/directoryObjects/$sp_id\"}")"
-    if [ "$rc" = "204" ]; then echo "ASSIGNED (Exchange Admin -> Exchange Operator SP)";
-    else echo "ERROR  (HTTP $rc: $(tr -d '\000' </tmp/aer_resp.$$ | jqr '.error.message // .' | head -c 120))"; fi
-    rm -f /tmp/aer_resp.$$ 2>/dev/null
+        -d "{\"principalId\":\"$sp_id\",\"roleDefinitionId\":\"$EXCH_ADMIN_TEMPLATE\",\"directoryScopeId\":\"/\"}")"
+    body="$(tr -d '\000' </tmp/aer_resp.$$ 2>/dev/null)"; rm -f /tmp/aer_resp.$$ 2>/dev/null
+    case "$rc" in
+        201) echo "ASSIGNED (Exchange Admin -> Exchange Operator SP)" ;;
+        400) if echo "$body" | grep -qiE 'conflicting object|already (exist|present)'; then echo "OK    (already assigned)"
+             else echo "ERROR  (HTTP 400: $(echo "$body" | jqr '.error.message // .' | head -c 120))"; fi ;;
+        *)   echo "ERROR  (HTTP $rc: $(echo "$body" | jqr '.error.message // .' | head -c 120))" ;;
+    esac
 }
 
 echo "=== assign-exchange-role  [mode=$MODE] ==="