From 51b09dacbb59de760278186f8e63b788743bc352 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Fri, 17 Jul 2026 05:35:39 -0700 Subject: [PATCH] sync: auto-sync from GURU-5070 at 2026-07-17 05:35:10 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-17 05:35:10 --- .claude/memory/MEMORY.md | 5 +- ...ference_exchange_online_exchangeop_tier.md | 32 +++++++ .claude/skills/remediation-tool/.gitignore | 1 + .claude/skills/remediation-tool/SKILL.md | 6 +- .../remediation-tool/references/gotchas.md | 21 ++++- .../remediation-tool/scripts/.gitignore | 2 + .../scripts/breakglass-signin-alert.sh | 38 ++++++++ .../scripts/run-breakglass-alert.cmd | 5 ++ .../scripts/user-breach-check.sh | 5 +- ...-investigation-exchange-tier-breakglass.md | 88 +++++++++++++++++++ errorlog.md | 4 + 11 files changed, 200 insertions(+), 7 deletions(-) create mode 100644 .claude/memory/reference_exchange_online_exchangeop_tier.md create mode 100644 .claude/skills/remediation-tool/.gitignore create mode 100644 .claude/skills/remediation-tool/scripts/.gitignore create mode 100644 .claude/skills/remediation-tool/scripts/breakglass-signin-alert.sh create mode 100644 .claude/skills/remediation-tool/scripts/run-breakglass-alert.cmd create mode 100644 clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index 52bbc6f9..7fc1f494 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -41,6 +41,7 @@ - [GuruRMM technical reference](reference_gururmm.md) — Server (172.16.3.30) layout + downloads dir `/var/www/gururmm/downloads` + `.channel` sidecar rollout control (stable/beta) + privileged server access via the server's OWN root RMM agent (hostname `gururmm`, no SSH needed; plink fallback) + API + `context=user_session` (WTS impersonation) + build-pipeline vendoring at `deploy/build-pipeline/` + Linux agent systemd sandbox trap. - [GuruRMM command timeout_seconds](reference_gururmm_command_timeout_seconds.md) — agent command dispatch honors `timeout_seconds`, NOT `timeout`; long jobs die ~300s / go zombie (`running`, empty stdout) otherwise. Cost Birth Biologic a full day. - [SharePoint Graph large-file upload](reference_sharepoint_graph_large_file_upload.md) — <4MB simple PUT, >=4MB MUST use chunked upload session (Content-Range); `\\?\` long paths; idempotent size-check; verify counts via /root/delta; single stream ~40Mbps (SPO throttle). +- [Exchange REST uses exchange-op tier](reference_exchange_online_exchangeop_tier.md) — ALL Exchange adminapi (Get-Mailbox/InboxRule/MailboxPermission, Set-*) goes through `exchange-op`, NEVER `investigator-exo` (401s: Security Investigator app lacks `Exchange.ManageAsApp`). 401=wrong app→exchange-op; 403=missing Exchange Admin role. Recurred 6+ times. - [RMM-spawn headless Claude](reference_rmm_spawn_headless_claude.md) — run `claude -p` on any RMM-managed Windows box with Claude Code (reaches coord-isolated sites like AD2); use `context:user_session`, UNSET the stale machine `ANTHROPIC_API_KEY` (shadows OAuth → "Invalid API key"), detach + poll a DONE marker. Validated on AD2 2026-07-01. - [RMM agent update model](rmm-agent-update-model.md) — Agent updates are server-PUSH on heartbeat (no self-poll); available versions = filesystem scan needing a `.sha256`; promote flips `.channel` sidecars beta→stable globally. Two stranders: beta-first freezes stable until an explicit promote; agents older than ~0.6.50 re-enroll with a NEW device_id/agent row when updated. - [GuruRMM physical server storage](gururmm-physical-server-storage.md) — New box 172.16.1.231 (temp IP→will be .30), Ubuntu 26.04, ssh key `gururmm-physical`/alias `gururmm-new`. SSD (915G root) = HOT (PG default tablespace + WAL + builds); HDD ext4 at `/data` = COLD (`gururmm_cold` PG tablespace for aged `agent_logs` partitions + downloads + backups + archive). The #3 retention answer. @@ -230,5 +231,5 @@ - [Background tasks — no shell `&`](feedback_background_task_no_ampersand.md) — with run_in_background:true, run the command in the foreground of the shell; adding `&`/`disown` forks + exits 0 instantly, orphaning a blocking wait so it never delivers (hit twice building ask-forum) - [ask-forum — human-in-the-loop via #ct-forum](../skills/ask-forum/SKILL.md) — ask a teammate a question in the private ct-forum Discord forum, get their human answer back in-session; forum-only, any human can answer, run the --wait as a proper background task - [Claude Code Bun crash -> pin 2.1.110](reference_claude_code_bun_crash_pin.md) - v2.1.114+ segfaults on Win11 at new-session start; npm pin 2.1.110 + DISABLE_AUTOUPDATER=1; unpin on #55219 fix -- [GuruRMM build-server SSH key](reference_gururmm_build_server_ssh.md) — guru@172.16.3.30 key is ~/.ssh/gururmm-physical (not id_*); password fallback vault infrastructure/gururmm-server -- [LAB-SVR retired](feedback_labsvr_retired.md) — Len's Auto LAB-SVR replaced by LAB-SERVER; never chase LAB-SVR +- [GuruRMM build-server SSH key](reference_gururmm_build_server_ssh.md) — guru@172.16.3.30 key is ~/.ssh/gururmm-physical (not id_*); password fallback vault infrastructure/gururmm-server +- [LAB-SVR retired](feedback_labsvr_retired.md) — Len's Auto LAB-SVR replaced by LAB-SERVER; never chase LAB-SVR diff --git a/.claude/memory/reference_exchange_online_exchangeop_tier.md b/.claude/memory/reference_exchange_online_exchangeop_tier.md new file mode 100644 index 00000000..b0b78343 --- /dev/null +++ b/.claude/memory/reference_exchange_online_exchangeop_tier.md @@ -0,0 +1,32 @@ +--- +name: reference_exchange_online_exchangeop_tier +description: Exchange REST (adminapi InvokeCommand) ALWAYS uses the exchange-op tier, never investigator-exo (which 401s — lacks Exchange.ManageAsApp) +metadata: + type: reference +--- + +**Every Exchange Online REST call in the remediation-tool suite — READ and WRITE — goes through the +`exchange-op` tier. NEVER `investigator-exo`.** This friction has recurred 6+ times (Mike, 2026-07-16). + +Why: the Exchange adminapi (`https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand`, i.e. +Get-Mailbox / Get-InboxRule / Get-MailboxPermission / Get-RecipientPermission / Set-Mailbox / etc.) +requires the calling app to hold the **`Exchange.ManageAsApp`** application permission (Office 365 +Exchange Online app-role `dc50a0fb-09a3-484d-be87-e023b12c6440`) **in addition to** the Exchange +Administrator directory role. + +- **`exchange-op`** = Exchange Operator app `b43e7342-5b4b-492f-890f-bb5a4f7f40e9` — has BOTH `Exchange.ManageAsApp` + AND the Exchange Administrator role. This is the ONLY working tier for Exchange REST (read or write). +- **`investigator-exo`** = Security Investigator app `bfbc12a4-...` — has the Exchange Admin *role* but NOT + `Exchange.ManageAsApp`, so it returns **401 on every** adminapi call, including read-only `Get-*`. The + "Exchange Online read" label on this tier is aspirational/wrong. + +**Diagnosis rule:** `401` on adminapi = wrong app (missing `Exchange.ManageAsApp`) → switch to `exchange-op`. +`403` = the `exchange-op` SP is missing the Exchange Administrator *directory role* on that tenant → assign it +(Entra > Roles > Exchange Administrator > add the app). Assigning Exchange Admin to the Security Investigator +SP does NOT fix the 401 — the app-permission (baked into the app registration), not the role, is the blocker. + +Verified live on ACG (`azcomputerguru.com`, tenant `ce61461e-81a0-4c84-bb4a-7b354a9a356d`) 2026-07-16: +Investigator SP `9d242c15-6cd3-46ec-96d3-bcafaaaca333` `Exchange.ManageAsApp`=NO; Operator SP +`83c225f1-b38d-4063-9fdd-642b6b09ae8b`=YES. Docs corrected same day: `SKILL.md`, `references/gotchas.md`, +and `scripts/user-breach-check.sh` (was minting `investigator-exo` for the 03a-d mailbox checks → they +silently returned empty on every run; now uses `exchange-op`). See also [[reference_remediation_tool_365_access]]. diff --git a/.claude/skills/remediation-tool/.gitignore b/.claude/skills/remediation-tool/.gitignore new file mode 100644 index 00000000..d4c49e2b --- /dev/null +++ b/.claude/skills/remediation-tool/.gitignore @@ -0,0 +1 @@ +.bb-tenant.json diff --git a/.claude/skills/remediation-tool/SKILL.md b/.claude/skills/remediation-tool/SKILL.md index fb4e3fb8..3cb6b86a 100644 --- a/.claude/skills/remediation-tool/SKILL.md +++ b/.claude/skills/remediation-tool/SKILL.md @@ -14,7 +14,7 @@ Five multi-tenant apps cover distinct privilege tiers. Use only what the task re | Tier | App display name | App ID | Vault file | Scope | |---|---|---|---|---| | `investigator` | ComputerGuru Security Investigator | `bfbc12a4-f0dd-4e12-b06d-997e7271e10c` | `computerguru-security-investigator.sops.yaml` | Graph read-only | -| `investigator-exo` | ComputerGuru Security Investigator | `bfbc12a4-f0dd-4e12-b06d-997e7271e10c` | `computerguru-security-investigator.sops.yaml` | Exchange Online read | +| `investigator-exo` | ComputerGuru Security Investigator | `bfbc12a4-f0dd-4e12-b06d-997e7271e10c` | `computerguru-security-investigator.sops.yaml` | Exchange Online read — **BROKEN: app lacks `Exchange.ManageAsApp`, adminapi `InvokeCommand` 401s. Use `exchange-op` for ALL Exchange REST (read + write).** | | `exchange-op` | ComputerGuru Exchange Operator | `b43e7342-5b4b-492f-890f-bb5a4f7f40e9` | `computerguru-exchange-operator.sops.yaml` | Exchange Online write | | `user-manager` | ComputerGuru User Manager | `64fac46b-8b44-41ad-93ee-7da03927576c` | `computerguru-user-manager.sops.yaml` | Graph user/group write | | `tenant-admin` | ComputerGuru Tenant Admin | `709e6eed-0711-4875-9c44-2d3518c47063` | `computerguru-tenant-admin.sops.yaml` | Graph high-privilege | @@ -23,7 +23,7 @@ Five multi-tenant apps cover distinct privilege tiers. Use only what the task re **The suite has broad, working access across ALL of M365 — Graph, Exchange Online, Defender, AND SharePoint Online.** Before concluding "the tool can't do X / access denied," verify against the live permission map in `references/app-permissions-and-sharepoint.md` (decode the token `roles` claim). An `accessDenied` usually means wrong tier or wrong endpoint for a scope we DO hold — not a real gap. Two recurring traps: (1) **SharePoint app-only requires a certificate** — a `client_secret` token is rejected everywhere in SharePoint with `"Unsupported app only token"` (get-token.sh forces cert for the `sharepoint*` tiers); (2) Graph `GET /admin/sharepoint/settings` needs a scope no app holds — read/write SharePoint tenant settings via the **CSOM/REST admin API** (`sharepoint-admin` tier) instead. Full map, gotchas, and CSOM examples: `references/app-permissions-and-sharepoint.md`. -**Default for breach checks:** use `investigator` (Graph) + `investigator-exo` (Exchange read). Escalate to write tiers only when remediating. +**Default for breach checks:** use `investigator` (Graph) + **`exchange-op`** for ALL Exchange REST (read AND write). NOTE: `investigator-exo` 401s on the Exchange adminapi — the Security Investigator app holds the Exchange Admin *role* but NOT the `Exchange.ManageAsApp` *app-permission* that `InvokeCommand` requires — so even read-only `Get-*` cmdlets go through `exchange-op`. Escalate nothing extra; `exchange-op` is the read path too. Full detail: `references/gotchas.md`. ## Auto-Invocation Behavior @@ -41,7 +41,7 @@ When triggered automatically (vs. via `/remediation-tool`), follow the same work - The SOPS vault is accessible via `.claude/identity.json` `vault_path` field. The scripts auto-resolve the vault location from identity.json — no hardcoded paths. - `jq`, `curl`, `bash` are available. -- For Exchange REST checks: confirm the target tenant has **Exchange Administrator** role assigned to the **Security Investigator** SP (for reads) or **Exchange Operator** SP (for writes). If any Exchange REST call returns 403, emit the tenant-scoped Entra Roles link from `references/gotchas.md`. +- **Exchange REST (adminapi `InvokeCommand`) — read AND write — ALWAYS use the `exchange-op` tier.** The Exchange Operator app (`b43e7342-5b4b-492f-890f-bb5a4f7f40e9`) is the ONLY suite app that holds `Exchange.ManageAsApp` (required for adminapi) PLUS the Exchange Administrator directory role. `investigator-exo` returns **401** on every Exchange cmdlet — the Security Investigator app has the Exchange Admin *role* but NOT the `Exchange.ManageAsApp` *permission* (verified live on ACG 2026-07-16: Investigator SP `Exchange.ManageAsApp`=NO, Operator SP=YES). Diagnosis rule: **401 = wrong app (missing `Exchange.ManageAsApp`) → switch to `exchange-op`; 403 = the `exchange-op` SP is missing the Exchange Administrator directory role on that tenant** → emit the Entra Roles link from `references/gotchas.md`. Do NOT "assign Exchange Admin to the Security Investigator SP" — it does not fix the 401 (the app-permission, not the role, is the blocker). - For Identity Protection checks: `IdentityRiskyUser.Read.All` is in the Security Investigator manifest AND the tenant has consented to that app. If 403, emit the per-app consent URL from `references/gotchas.md`. - For Defender checks: confirm tenant has Microsoft Defender for Endpoint (MDE) license before using `defender` tier — it returns AADSTS650052 otherwise. diff --git a/.claude/skills/remediation-tool/references/gotchas.md b/.claude/skills/remediation-tool/references/gotchas.md index 9cc01ef8..aa04a6ff 100644 --- a/.claude/skills/remediation-tool/references/gotchas.md +++ b/.claude/skills/remediation-tool/references/gotchas.md @@ -38,13 +38,32 @@ Graph API permissions alone are not enough. Most privileged operations require d | Operation | App tier | Required directory role on that SP | |---|---|---| -| Exchange REST read (Get-InboxRule, Get-Mailbox) | `investigator-exo` | Exchange Administrator | +| Exchange REST read (Get-InboxRule, Get-Mailbox, Get-MailboxPermission, Get-RecipientPermission) | **`exchange-op`** (NOT `investigator-exo` — see callout) | Exchange Administrator | | Exchange REST write (Set-Mailbox, Remove-InboxRule) | `exchange-op` | Exchange Administrator | | Password reset, user property updates | `user-manager` | User Administrator | | MFA method reset | `user-manager` | Authentication Administrator | | Conditional Access reads/writes | `tenant-admin` | Conditional Access Administrator OR Security Administrator | | Teams policies | `tenant-admin` | Teams Administrator | +> **[AUTHORITATIVE — stop re-deriving this] Exchange REST goes through `exchange-op`, NEVER `investigator-exo`.** +> The Exchange adminapi (`https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand`) requires the +> calling app to hold the **`Exchange.ManageAsApp`** application permission (Office 365 Exchange Online app role +> `dc50a0fb-09a3-484d-be87-e023b12c6440`) **in addition to** the Exchange Administrator directory role. Only the +> **Exchange Operator** app (`exchange-op`, `b43e7342-5b4b-492f-890f-bb5a4f7f40e9`) has BOTH. The **Security +> Investigator** app (`investigator`/`investigator-exo`, `bfbc12a4-...`) has the Exchange Admin *role* but NOT +> `Exchange.ManageAsApp`, so **every** `investigator-exo` adminapi call returns **401** — including read-only +> `Get-*` cmdlets. Verified live on ACG (`azcomputerguru.com`) 2026-07-16: Investigator SP `Exchange.ManageAsApp`=**NO**, +> Operator SP=**YES**. **So use `exchange-op` for ALL Exchange cmdlets, read and write.** +> +> **401 vs 403:** `401` on adminapi = wrong app (missing `Exchange.ManageAsApp`) → you are on `investigator-exo`, +> switch to `exchange-op`. `403` = the `exchange-op` SP is missing the **Exchange Administrator** directory role on +> that specific tenant → assign it (steps below). Assigning Exchange Admin to the *Security Investigator* SP does +> **not** help — the missing piece there is the app-permission, which is baked into the app registration, not a +> per-tenant role. Authoritative ACG SP→role map (verified 2026-07-16): `exchange-op` SP `83c225f1-b38d-4063-9fdd-642b6b09ae8b` +> = Exchange Administrator (+ `Exchange.ManageAsApp`); `investigator` SP `9d242c15-6cd3-46ec-96d3-bcafaaaca333` = Exchange +> Administrator only (no ManageAsApp); `user-manager` SP `04020a4e-...` = Authentication Administrator + User Administrator; +> `tenant-admin` SP `046a7f70-...` = Conditional Access Administrator. + ### How to assign a role to an SP in a customer tenant 1. Sign into the customer's Entra admin center as Global Admin: diff --git a/.claude/skills/remediation-tool/scripts/.gitignore b/.claude/skills/remediation-tool/scripts/.gitignore new file mode 100644 index 00000000..2165fe63 --- /dev/null +++ b/.claude/skills/remediation-tool/scripts/.gitignore @@ -0,0 +1,2 @@ +.breakglass-lastcheck +.breakglass-alert.log diff --git a/.claude/skills/remediation-tool/scripts/breakglass-signin-alert.sh b/.claude/skills/remediation-tool/scripts/breakglass-signin-alert.sh new file mode 100644 index 00000000..cf095428 --- /dev/null +++ b/.claude/skills/remediation-tool/scripts/breakglass-signin-alert.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# breakglass-signin-alert.sh — alert #dev-alerts on ANY sign-in by the ACG M365 +# break-glass account (sysadmin@azcomputerguru.onmicrosoft.com). A break-glass login +# should be rare and always investigated. Run on a schedule (hourly/daily). +# +# State: tracks the last-checked UTC timestamp in a sidecar so it only alerts on NEW +# sign-ins. First run seeds the timestamp (no alert for history) unless --backfill . +# DRY_RUN=1 prints what it would post instead of posting. +# +# Usage: breakglass-signin-alert.sh [--since ] +set -u +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)" # skill scripts -> repo root +TEN="ce61461e-81a0-4c84-bb4a-7b354a9a356d" # azcomputerguru.com +BG="8c363d17-02ed-45e1-97d0-49e5819e6ff6" # sysadmin@azcomputerguru.onmicrosoft.com +STATE="$SCRIPT_DIR/.breakglass-lastcheck" + +NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +if [ "${1:-}" = "--since" ] && [ -n "${2:-}" ]; then SINCE="$2"; +elif [ -f "$STATE" ]; then SINCE="$(cat "$STATE")"; +else SINCE="$NOW"; echo "$NOW" > "$STATE"; echo "[seed] first run — seeded $NOW, no history alerted"; exit 0; fi + +G=$("$SCRIPT_DIR/get-token.sh" "$TEN" investigator 2>/dev/null) || { echo "[err] token"; exit 1; } +RESP=$(curl -s "https://graph.microsoft.com/v1.0/auditLogs/signIns?\$filter=userId+eq+'$BG'+and+createdDateTime+ge+$SINCE&\$top=50" -H "Authorization: Bearer $G" | tr -d '\000-\037') +N=$(echo "$RESP" | jq -r '.value | length' 2>/dev/null) +[ -z "$N" ] && { echo "[err] query"; exit 1; } + +if [ "$N" -gt 0 ]; then + while IFS= read -r line; do + [ -z "$line" ] && continue + MSG="[SECURITY] BREAK-GLASS SIGN-IN - sysadmin@azcomputerguru.onmicrosoft.com used: $line -- investigate if unplanned." + if [ "${DRY_RUN:-0}" = "1" ]; then echo "WOULD POST: $MSG"; + else bash "$ROOT/.claude/scripts/post-bot-alert.sh" "$MSG" >/dev/null 2>&1; echo "[alert] posted: $line"; fi + done < <(echo "$RESP" | jq -r '.value[] | "\(.createdDateTime) result=\(.status.errorCode) app=\(.appDisplayName) ip=\(.ipAddress) loc=\(.location.countryOrRegion)/\(.location.city // "?")"') +else + echo "[ok] no break-glass sign-ins since $SINCE" +fi +echo "$NOW" > "$STATE" diff --git a/.claude/skills/remediation-tool/scripts/run-breakglass-alert.cmd b/.claude/skills/remediation-tool/scripts/run-breakglass-alert.cmd new file mode 100644 index 00000000..f9df9e81 --- /dev/null +++ b/.claude/skills/remediation-tool/scripts/run-breakglass-alert.cmd @@ -0,0 +1,5 @@ +@echo off +REM Wrapper for the ACG M365 break-glass sign-in alert. Invoked by the +REM "ACG-BreakGlass-SignIn-Alert" scheduled task. Uses a Git login shell so the +REM SOPS/age vault env loads, then runs the check from the repo root and logs output. +"C:\Program Files\Git\bin\bash.exe" -lc "cd /d/ClaudeTools && bash .claude/skills/remediation-tool/scripts/breakglass-signin-alert.sh >> .claude/skills/remediation-tool/scripts/.breakglass-alert.log 2>&1" diff --git a/.claude/skills/remediation-tool/scripts/user-breach-check.sh b/.claude/skills/remediation-tool/scripts/user-breach-check.sh index 359dab94..8404271d 100755 --- a/.claude/skills/remediation-tool/scripts/user-breach-check.sh +++ b/.claude/skills/remediation-tool/scripts/user-breach-check.sh @@ -13,7 +13,10 @@ UPN="${2:?usage: user-breach-check.sh }" TENANT_ID=$("$SCRIPT_DIR/resolve-tenant.sh" "$TENANT_INPUT") GT=$("$SCRIPT_DIR/get-token.sh" "$TENANT_ID" investigator) -EXO=$("$SCRIPT_DIR/get-token.sh" "$TENANT_ID" investigator-exo) || EXO="" +# Exchange REST (adminapi InvokeCommand) MUST use exchange-op: it holds Exchange.ManageAsApp +# + Exchange Administrator. investigator-exo 401s (Security Investigator app lacks ManageAsApp) +# — see references/gotchas.md "Exchange REST goes through exchange-op". (fixed 2026-07-16) +EXO=$("$SCRIPT_DIR/get-token.sh" "$TENANT_ID" exchange-op) || EXO="" USER_SLUG=$(echo "$UPN" | tr '@.' '__') OUT="/tmp/remediation-tool/$TENANT_ID/user-breach/$USER_SLUG" diff --git a/clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md b/clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md new file mode 100644 index 00000000..e3d0f616 --- /dev/null +++ b/clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md @@ -0,0 +1,88 @@ +# ACG mailbox security investigation + hardening (mike@) + +**Date:** 2026-07-16 (UTC) + +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Trigger +Mike's Outlook "view" kept resetting — suspected someone else was changing his mailbox +settings. Requested a full investigation of `mike@azcomputerguru.com`. + +## Outcome (TL;DR) +- **No compromise, no delegates.** Nobody else has access to the mailbox. The changing view + is Mike's own many clients (New Outlook vs classic, OWA, Apple Mail, mobile, 3 AAD-joined PCs) + fighting over roaming view settings — not another person. +- **Real finding:** an active distributed **password-spray attack** on the account — all failing + (smart-lockout + enforced MFA + legacy-auth block). No successful foreign sign-in. +- Hardened: pruned stale credentials, made a true break-glass account, added sign-in alerting. +- **Also fixed a recurring fleet-wide friction:** Exchange REST must use the `exchange-op` tier + (docs + breach-check script corrected). + +## Investigation findings (tenant ce61461e-81a0-4c84-bb4a-7b354a9a356d) +- **Sign-ins (30d, interactive):** 200 records, **all failures** — `50053` smart-lockout (×195), + `50126` bad password (×5) — from DE (52), US-spoofed (108), CN, KR, IN, RU, BR, etc. Only 6 + successful sign-ins, all US (Phoenix/SLC), legitimate apps. **Password spray, contained.** +- **Delegates:** FullAccess = SELF only; SendAs = SELF only; SendOnBehalf = empty; forwarding = none. + **No one else can access the mailbox.** +- **OAuth grants (4):** all legit — Apple Internet Accounts (iPhone/Mac Mail, holds `EWS.AccessAsUser.All`), + Alignable (Contacts.Read), ZeroTier, Tailscale. No rogue app. +- **Inbox rules (46, all enabled):** Mike's own routing (forwarding vendor/notification mail to + howard@/rob@/wwilliams@/webmaster@/info@ + client seastman@glaztech.com). No malicious catch-all. + Two external-Gmail forwards to confirm as intentional: `energybilling@tesla.com -> ourfamily7479@gmail.com`, + `vailschools.org -> ursulaclark76@gmail.com`. +- **MFA/CA:** "Require MFA for all users" = enabled/All; "Block legacy authentication" = enabled/All. + Posture is sound — spray can't convert a lucky guess. +- **Auth methods:** Authenticator ×2 (Samsung), phone +1 520-289-1912, SSPR email superguru@gmail.com, + 21 Windows Hello keys (18 stale/unnamed). (Confirm phone + superguru@ are Mike's.) + +## Actions taken +1. **Pruned 20 stale Windows Hello keys** on mike@ (18 orphaned 2020–2025 + ACG-TECH-02L + ACG-YOGA, + both retired per Mike). **Only `GURU-5070` (2026-04-22) remains.** Other methods untouched. + (user-manager tier; 20× HTTP 204.) +2. **Break-glass exclusions** — excluded `sysadmin@azcomputerguru.onmicrosoft.com` + (`8c363d17-02ed-45e1-97d0-49e5819e6ff6`, cloud-only GA) from BOTH MFA CA policies so an MFA + outage can't lock it out, while every real user stays gated: + - "Require multifactor authentication for all users" (`1e912cc2-334a-4f12-9d81-a95dc3822b8e`) + - "Require multifactor authentication for admins" (`5f10aa47-e796-40db-89eb-fdae60dea0c7`, 14 roles preserved) + (tenant-admin tier; both HTTP 204, verified.) +3. **Break-glass sign-in alerting — LIVE.** New script + `.claude/skills/remediation-tool/scripts/breakglass-signin-alert.sh` posts `[SECURITY]` to + #dev-alerts on ANY sysadmin@ sign-in (state-tracked, seeded 2026-07-17T04:23Z, no history alerted). + Scheduled task **`ACG-BreakGlass-SignIn-Alert`** on GURU-5070, every 4h, test-run OK (result=0). + Baseline: sysadmin@ has **zero sign-ins in 2026** (break-glass untouched). + +## Blocked / pending (needs Mike) +- **Rotate the sysadmin@ break-glass password — DO IN PORTAL.** Graph reset 403'd + ("Insufficient privileges") by design: resetting a Global Admin's password needs *Privileged + Authentication Administrator*, which our apps deliberately don't hold (granting it would be worse + than the gap). Mike (a GA) resets it in Entra: Users -> sysadmin@ -> Reset password; set a long + (20+ char) password, `force change = off`. Then **vault it** (intended path + `msp-tools/acg-m365-breakglass.sops.yaml`) — hand me the password and I'll store it. +- Confirm phone `+1 520-289-1912` and SSPR email `superguru@gmail.com` are Mike's. +- Confirm the two external-Gmail forward rules are intentional. + +## Recommendations (open) +- Alert home: the task runs on GURU-5070 only while it's on — move to an always-on fleet host for + robust monitoring. +- Consider a 2nd break-glass account (Microsoft-recommended pair). +- No break-glass exclusion existed on ANY CA policy before today; now sysadmin@ is the designated one. + +## Fleet fix — Exchange REST tier (recurring friction, 6th time) +Root-caused the perennial Exchange 401: the **Security Investigator** app (`investigator-exo`) has the +Exchange Admin *role* but NOT the **`Exchange.ManageAsApp`** app-permission the adminapi requires, so it +401s on every Exchange cmdlet. Only the **Exchange Operator** app (`exchange-op`) has both → +**all Exchange REST (read + write) goes through `exchange-op`.** Corrected: +- `SKILL.md` (tier table, breach-check default, "before calling" note — which had told us to assign the + role to the wrong SP) +- `references/gotchas.md` (role table + authoritative callout + 401-vs-403 rule + SP->role map) +- `scripts/user-breach-check.sh` (was minting `investigator-exo` -> mailbox delegate/forwarding checks + silently returned empty on every run; now `exchange-op`) +- memory `reference_exchange_online_exchangeop_tier` + MEMORY.md index; errorlog `--correction` + +## Other client threads this session (separate contexts) +Khalsa Montessori RMM onboarding (17 Windows enrolled via SC, client `LOWER-TIGER-5022`); +BirthBiologic SharePoint permissions audit -> Syncro #32187 comment; Reliant Well Drilling Google +Business Profile (none exists — GBP creation outlined). Logged/tracked in their own places. diff --git a/errorlog.md b/errorlog.md index 498ab3e7..4fa85f14 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,10 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-17 | GURU-5070 | remediation-tool/exchange | [correction] Repeatedly used investigator-exo tier for Exchange REST -> 401 on every adminapi call (Security Investigator app lacks Exchange.ManageAsApp). Correct tier is exchange-op (only app with both Exchange.ManageAsApp + Exchange Admin role). Mike's 6th time hitting this. Fixed docs (SKILL.md, gotchas.md) + user-breach-check.sh (was using investigator-exo -> mailbox delegate/forwarding checks silently returned empty on every breach check) + memory reference_exchange_online_exchangeop_tier. [ctx: tenant=azcomputerguru.com ref=reference_exchange_online_exchangeop_tier] + +2026-07-17 | GURU-5070 | vault/secret-echo | [friction] grep filter to redact vault output missed the JSON field 'private_key' (filter had 'key:' with colon + 'apikey', not bare 'private_key') -> SA private key printed to transcript. Fix: when displaying vault entries, allowlist known plaintext fields (name/status/username/notes/url) instead of denylisting secret patterns; or pipe through 'sops -d' with an explicit field select. [ctx: ref=vault-skill file=msp-tools/acg-msp-access-google-workspace.sops.yaml] + 2026-07-16 | GURU-BEAST-ROG | syncro/comment | [friction] POST /tickets/{id}/comment response failed jq parse even after tr control-char strip; comment DID post (verified via GET) - check-first rule prevented duplicate [ctx: ticket=32557] 2026-07-15 | Howard-Home | msp360/backups | [correction] assumed LAB-SVR (Len's Auto) is a live machine needing attention; correct is LAB-SVR was RETIRED and replaced by LAB-SERVER — repeated correction from Howard