fix(remediation): close the recurring Exchange-Admin-role gap fleet-wide

EXO email-cleanup tasks (Search-UnifiedAuditLog, Get-MessageTrace, inbox rules) kept
401/403-ing per tenant because the Exchange Operator SP was missing the Exchange Admin
directory role — admin consent grants Exchange.ManageAsApp but never the directory role.
onboard-tenant.sh assigns it, but tenants consented before that step / by hand never got
it, and nothing audited for it. Hence the recurring 'next onboarding will fix it' (false
for already-onboarded tenants).

- NEW assign-exchange-role.sh: idempotent role assignment via the authoritative
  roleManagement/directory/roleAssignments API (the legacy directoryRoles/members list
  reads back unreliably). <domain|--all> + --verify/--dry-run.
- Backfilled the whole fleet (--all): 13 stragglers ASSIGNED, 12 already OK, 20 skipped
  (tenant-admin not consented), 0 errors. Safe Site included.
- Standing audit documented (assign-exchange-role.sh --all --verify) + memory so no future
  session repeats the empty promise.
- Adds wiki/clients/safesite.md (tenant + 4-source endpoint inventory + investigation).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.claude/skills/remediation-tool/references/tenants.md

@@ -5,6 +5,14 @@ Last updated: 2026-04-20. Source of truth: CIPP ListTenants API.
 Run `bash scripts/onboard-tenant.sh <domain>` after any tenant consents Tenant Admin.
 After full onboarding, update the Onboarded column below.
 
+**Exchange access (recurring gap — now closed):** EXO management (audit log, message trace, inbox
+rules) needs the **Exchange Operator SP** to hold the **Exchange Administrator** directory role, which
+admin consent does NOT grant. Onboarding assigns it, but tenants consented before that step / by hand
+were missing it. Fleet **backfilled 2026-06-08** (13 stragglers fixed). **Standing audit:** run
+`bash scripts/assign-exchange-role.sh --all --verify` periodically — any `WOULD assign` is a tenant
+that will fail the next email task; fix it with `assign-exchange-role.sh <domain>`. See
+[[feedback_exchange_role_recurring_gap]].
+
 ## Tenant List
 
 | Display Name | Domain | Tenant ID | Onboarded | Notes |

.claude/skills/remediation-tool/scripts/assign-exchange-role.sh

@@ -70,30 +70,25 @@ process_one() {
     sp_id="$(gget "$tok" "$GRAPH/servicePrincipals?\$filter=appId%20eq%20'$EXCHANGE_OP_APPID'&\$select=id" | jqr '.value[0].id // empty')"
     if [ -z "$sp_id" ]; then echo "SKIP  (Exchange Operator app not consented in tenant)"; return; fi
 
-    # find or (if applying) activate the Exchange Administrator directory role
-    role_id="$(gget "$tok" "$GRAPH/directoryRoles?\$filter=roleTemplateId%20eq%20'$EXCH_ADMIN_TEMPLATE'" | jqr '.value[0].id // empty')"
-    if [ -z "$role_id" ]; then
-        if [ "$MODE" = "apply" ]; then
-            role_id="$(curl -s --max-time 25 -X POST "$GRAPH/directoryRoles" \
-                -H "Authorization: Bearer $tok" -H "Content-Type: application/json" \
-                -d "{\"roleTemplateId\":\"$EXCH_ADMIN_TEMPLATE\"}" | tr -d '\000' | jqr '.id // empty')"
-            [ -z "$role_id" ] && { echo "ERROR (could not activate Exchange Admin role)"; return; }
-        else
-            echo "WOULD activate Exchange Admin role + assign SP $sp_id"; return
-        fi
-    fi
-
-    present="$(gget "$tok" "$GRAPH/directoryRoles/$role_id/members?\$select=id" | jqr --arg s "$sp_id" '[.value[]?|select(.id==$s)]|length')"
+    # Use the AUTHORITATIVE unified role-assignment API (roleManagement/directory/roleAssignments)
+    # for both the idempotency check and the write. The legacy directoryRoles/{id}/members list
+    # reads back unreliably (replication lag) and falsely reports not-assigned; roleAssignments is
+    # consistent. For built-in roles, roleDefinitionId == the roleTemplateId.
+    present="$(gget "$tok" "$GRAPH/roleManagement/directory/roleAssignments?\$filter=principalId%20eq%20'$sp_id'%20and%20roleDefinitionId%20eq%20'$EXCH_ADMIN_TEMPLATE'" | jqr '.value | length')"
     if [ "${present:-0}" -gt 0 ] 2>/dev/null; then echo "OK    (already assigned)"; return; fi
 
     if [ "$MODE" != "apply" ]; then echo "WOULD assign Exchange Admin to SP $sp_id"; return; fi
 
-    rc="$(curl -s --max-time 25 -o /tmp/aer_resp.$$ -w '%{http_code}' -X POST "$GRAPH/directoryRoles/$role_id/members/\$ref" \
+    rc="$(curl -s --max-time 25 -o /tmp/aer_resp.$$ -w '%{http_code}' -X POST "$GRAPH/roleManagement/directory/roleAssignments" \
         -H "Authorization: Bearer $tok" -H "Content-Type: application/json" \
-        -d "{\"@odata.id\":\"$GRAPH/directoryObjects/$sp_id\"}")"
-    if [ "$rc" = "204" ]; then echo "ASSIGNED (Exchange Admin -> Exchange Operator SP)";
-    else echo "ERROR  (HTTP $rc: $(tr -d '\000' </tmp/aer_resp.$$ | jqr '.error.message // .' | head -c 120))"; fi
-    rm -f /tmp/aer_resp.$$ 2>/dev/null
+        -d "{\"principalId\":\"$sp_id\",\"roleDefinitionId\":\"$EXCH_ADMIN_TEMPLATE\",\"directoryScopeId\":\"/\"}")"
+    body="$(tr -d '\000' </tmp/aer_resp.$$ 2>/dev/null)"; rm -f /tmp/aer_resp.$$ 2>/dev/null
+    case "$rc" in
+        201) echo "ASSIGNED (Exchange Admin -> Exchange Operator SP)" ;;
+        400) if echo "$body" | grep -qiE 'conflicting object|already (exist|present)'; then echo "OK    (already assigned)"
+             else echo "ERROR  (HTTP 400: $(echo "$body" | jqr '.error.message // .' | head -c 120))"; fi ;;
+        *)   echo "ERROR  (HTTP $rc: $(echo "$body" | jqr '.error.message // .' | head -c 120))" ;;
+    esac
 }
 
 echo "=== assign-exchange-role  [mode=$MODE] ==="