# Session Log — Tedards: bt@ "delete folder" root cause + DUPLICATE folder cleanup

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

---

## Session Summary

Resumed the Tedards investigation with a directive to make zero assumptions and verify live 365 access. Confirmed read access to both Tedards mailboxes (`y226@tedards.net`, `bt@tedards.net`) through the ComputerGuru app suite against tenant `4fcbb1f4-fbf9-4548-a93e-7d14a3c091e6`: Graph via the Security Investigator tier (200 on user + message reads) and Exchange Online via the Exchange Operator tier (200 on Get-Mailbox). Documented that the Security Investigator `investigator-exo` tier 401s on the EXO adminapi because its app registration carries only `full_access_as_app`, not `Exchange.ManageAsApp`; the directory role (Exchange Administrator) is already assigned, so the gap is an app-registration API permission, not a role. The `exchange-op` tier is the all-access Exchange tier and was used for all EXO work thereafter (logged as a recurring correction).

Investigated the long-standing complaint that emails from `lindsay@agencyzoomify.com` were disappearing from Bill's inbox into a "delete folder" in real time. Ruled out every server-side mechanism with live data: 18 inbox rules (all newsletter filing, none touching Lindsay/Bolton/Deleted Items), zero sweep rules, no transport rules, no forwarding, no delegates, no Inbox folder permissions, and no OAuth app holding `Mail.ReadWrite`. Located the three affected messages in Deleted Items and confirmed via timing (received at spread-out times, all deleted in one 2-minute burst) that the deletion was interactive/client-side, not an automatic delivery-time rule. A bait test (EWS-move the 3 messages back to Inbox) proved automated re-deletion: all three were re-trashed within ~1-4 seconds at an identical timestamp, by a client, keyed to Lindsay's address (a co-threaded message from brandon@agentive-one.com survived). A bisection (disabled/removed the two new Outlook-for-iOS clients, re-baited) showed the re-delete fired before the Outlook-iOS clients re-synced, exonerating them and pinning the cause to a native iOS Mail (EAS) device — iPhone16C2 or iPad15C8 — with an on-device "Block Sender -> Move to Trash" for Lindsay. On-device block lists are not server-readable, so the fix is on Bill's devices.

Enabled tenant auditing to make any recurrence attributable: the tenant was dehydrated, so ran `Enable-OrganizationCustomization` (irreversible, confirmed with the user) then `Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $true`. The config flag propagated to true. Re-baited under live ingestion and filed the 3 messages into Bill's "BOLTON, Lindsay" folder (confirmed they stay put there — the block only sweeps the Inbox). A durable cron (`ce6e3e74`, every ~19 min) rechecks `Search-UnifiedAuditLog` for the capture; through end of session it returned entries=0 (ingestion still propagating / app-only UAL not yet surfacing mailbox-item events).

Billed 2.0h remote labor to ticket #32228 ($300, new invoice #67886) after verifying the new invoice contained only the 2h line (the prior 0.5h stays on #67882). Posted a customer-visible, emailed comment to Yvonne with the device-fix steps. Note: the session-log reference to "ticket #5070" does not resolve to any Syncro ticket; #32228 ("Unable to send/receive email to/from lindsay@agencyzoomify.com") is the correct ticket for this work.

Began cleanup of Bill's "DUPLICATE need to check" folder (11,864 items, a botched-import artifact Yvonne preserved). Swept all 119,223 mailbox messages, matched by Internet Message-ID, and classified: 11,810 true duplicates (identical copy confirmed in another real folder, excluding Deleted Items) and 54 unique (only copy — kept). With explicit user approval, soft-deleting (EWS move to Deleted Items, recoverable) the 11,810. As of save, the move is still running in the background (folder at ~3,264, 8,603 already moved, converging to 54). A new no-billing notification ticket for Yvonne is drafted and user-approved but NOT yet posted — held until the folder reaches 54 so the email numbers match the mailbox.

---

## Key Decisions

- **Use the `exchange-op` tier for all Exchange writes/all-access** — it holds Exchange Administrator + `full_access_as_app` + `Exchange.ManageAsApp`. Stop claiming "no tier can write mail." (Recurring correction; saved to memory `feedback_exchange_op_all_access`.)
- **Did not add `Exchange.ManageAsApp` to the Security Investigator app** — it's a manual portal change and the read-only tier shouldn't have it; `exchange-op` covers everything.
- **Bait test over waiting** — moving the messages back to Inbox to observe behavior was the only way to prove automated re-deletion without a queryable audit log.
- **Soft-delete (move to Deleted Items), not hard purge** for the 11,810 duplicates — recoverable until Yvonne empties Deleted Items.
- **Kept the 54 uniques** — they have no copy elsewhere; deliberately did not dedupe them against each other to avoid removing a last-surviving copy.
- **Held the notification ticket until folder = 54** — avoid emailing Yvonne "54 remaining" while thousands are still visible.
- **Confirmed `Enable-OrganizationCustomization` with the user before running** — irreversible org-level change.

---

## Problems Encountered

- **`investigator-exo` 401 on EXO adminapi** — app lacks `Exchange.ManageAsApp`; used `exchange-op` instead. (Memory: `reference_investigator_exo_manageasapp_gap`.)
- **`Search-MailboxAuditLog` deprecated (Jan 2026)** — switched to `Search-UnifiedAuditLog`; found `UnifiedAuditLogIngestionEnabled=false`, enabled it (required hydrating the tenant first).
- **App-only `Search-UnifiedAuditLog` returns 0** even after ingestion enabled and after thousands of bulk MoveToDeletedItems events — ingestion propagation lag and/or app-only UAL not surfacing mailbox-item records. Recheck cron left running.
- **`Set-CASMailbox` / `Set-AdminAuditLogConfig` propagation lag** — read-backs lagged true for minutes; the `OutlookMobileEnabled=false` change never enforced during the bisection window (the timing evidence carried the conclusion instead).
- **Background dedup driver detached/died** when launched with `nohup ... &` *and* the tool's background mode (double-backgrounding); relaunched via the tool's background mechanism only. Two instances ended up running concurrently — harmless (idempotent) but wasteful.
- **Graph `$filter` on `from/emailAddress/address` returned null** intermittently — switched to client-side filtering of the message list.
- **#5070 does not exist in Syncro** — verified; used #32228 after user confirmation. Avoided posting a billable comment to a guessed ticket.
- **New invoice double-bill risk on an already-invoiced ticket** — guarded by verifying the new invoice total == $300 with auto-rollback (DELETE) if not.

---

## Configuration Changes

### Microsoft 365 — Tedards tenant (`4fcbb1f4-fbf9-4548-a93e-7d14a3c091e6`)
- `Enable-OrganizationCustomization` run (irreversible; tenant was dehydrated).
- `Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $true` (propagated to true).
- bt@ mailbox: 3 Lindsay Bolton messages moved Deleted Items -> "BOLTON, Lindsay" folder.
- bt@ mailbox: ~11,810 duplicates being moved from "DUPLICATE need to check" -> Deleted Items (in progress at save).
- bt@ `Set-CASMailbox -OutlookMobileEnabled` toggled false then reverted to true (diagnostic; revert queued, propagation lag).
- Removed two Outlook-for-iOS device partnerships (`c42bbf6e…`, `548f2cdd…`) — they auto-re-add.

### Syncro
- Ticket #32228 (internal 109697650): customer comment 420824967 (emailed); line item 43027728 (2.0h remote @ $150); invoice 1650805941 / #67886 ($300); invoice note set.

### Repo
- `.gitignore`: added session scratch patterns (`.dup*`, `.ews_*.xml`, `.x.json`).
- Memory: added `reference_tedards_tenant_facts`, `reference_investigator_exo_manageasapp_gap`, `feedback_exchange_op_all_access`; updated `MEMORY.md` index.

---

## Credentials & Secrets

No new credentials created or discovered. Tokens acquired via `get-token.sh` (cert auth) for tiers `investigator`, `investigator-exo`, `exchange-op`, `tenant-admin`. Vault unchanged.

---

## Infrastructure & Servers

- **Tedards M365 tenant:** `tedards.net` / `4fcbb1f4-fbf9-4548-a93e-7d14a3c091e6`
- **Mailboxes:** `bt@tedards.net` (Bill, owner, id `3044102c-ed9d-4777-ac8c-9d973d50e328`), `y226@tedards.net` (Yvonne, default contact, id `9c2cc39c-a523-4e73-81b7-4b818d8ea3b9`)
- **Apps:** Security Investigator `bfbc12a4-f0dd-4e12-b06d-997e7271e10c` (SP oid `2dd202e8-5f70-4826-a378-cfeeadac9cf6`); Exchange Operator `b43e7342-5b4b-492f-890f-bb5a4f7f40e9`; Tenant Admin `709e6eed-0711-4875-9c44-2d3518c47063`
- **Culprit devices (bt@):** iPhone16C2, iPad15C8 (native iOS Mail / EAS). All access from Bill's home IP `69.242.239.94`.
- **Key folder IDs (bt@):** DUPLICATE need to check `…AAABF_WUAAA=`; Deleted Items `…AAAAAAEKAAA=`; BOLTON, Lindsay `…AAEoMg_hAAA=`; Inbox `…AAAAAAEMAAA=`.

---

## Commands & Outputs

```
# verify access
get-token.sh tedards.net investigator|investigator-exo|exchange-op
investigator-exo EXO adminapi -> HTTP 401 (only full_access_as_app)
exchange-op Get-Mailbox -> 200 (bt@, y226@)

# bait test (EWS MoveItem to Inbox, then observe)
moved 03:30:58Z -> re-deleted 03:31:00Z (native device); brandon@ co-thread survived

# audit enable
Set-AdminAuditLogConfig UnifiedAuditLogIngestionEnabled -> 400 dehydrated
Enable-OrganizationCustomization -> 200
Set-AdminAuditLogConfig UnifiedAuditLogIngestionEnabled $true -> 200 (flag now true)
Search-UnifiedAuditLog (bt@, Move/Delete ops) -> entries=0 (still propagating)

# dedup
.dupscan.py: mailbox_total=119223, dup_folder=11864, true_duplicates=11810, unique_keep=54
.dupdelete.py / .dupdrive.sh: EWS MoveItem -> deleteditems, batches of 200 (in progress)
```

---

## Pending / Incomplete Tasks

- **Dedup move finishing** — folder ~3,264 at save, converging to 54; two idempotent background drivers running. Verify folder == 54 and Deleted Items ~= 11,810+ when done.
- **Post the approved notification ticket** (no billing) to Yvonne once folder == 54 — draft ready: subject "Mailbox Cleanup - Duplicate Emails Removed; Please File Remaining Items", customer 487887, status Waiting on Customer, comment emailed, then bot alert. Tells Yvonne to file the 54 remaining uniques and to empty Deleted Items only when satisfied.
- **Audit capture** — cron `ce6e3e74` still returning entries=0; let it keep retrying. If it never surfaces, attribution rests on the device-statistics timing (iPhone16C2/iPad15C8).
- **On Bill's devices (customer action):** unblock `lindsay@agencyzoomify.com` in iOS Mail (Settings - Mail - Blocked) on iPhone and iPad, or set Blocked Sender Options to None.
- **Optional:** remove scratch files (`.dup*`) after dedup completes.

---

## Reference Information

- **Syncro ticket:** #32228 (internal 109697650) — https://computerguru.syncromsp.com/tickets/109697650
- **Invoices:** #67886 (1650805941, $300, this session); #67882 (1650804914, $75, prior session)
- **Cron job:** `ce6e3e74` (durable, ~every 19 min) — UAL audit recheck
- **Prior session:** `clients/tedards/session-logs/2026-06/2026-06-25-discord-bot-agencyzoomify-dmarc-fix.md`
- **Memory:** `reference_tedards_tenant_facts`, `reference_investigator_exo_manageasapp_gap`, `feedback_exchange_op_all_access`

---

## Update: 04:14 PT (2026-06-26) — dedup completion, ticket, root-cause confirmation, second folder

### Summary
Completed the first dedup: "DUPLICATE need to check" went 11,864 -> 54 uniques; 11,813 duplicates soft-deleted to Deleted Items (recoverable). The background driver was launched twice (the first `nohup &` instance never died), so two idempotent instances ran concurrently and converged — harmless. Posted the approved no-billing notification ticket **#32467** (id 113090881) to Yvonne once the folder reached 54, with a customer-visible emailed comment listing the result and her action items (file the 54, empty Deleted Items when satisfied).

Closed the audit-capture effort as non-viable: app-only `Search-UnifiedAuditLog` returned 0 across ~8 rechecks even after ~11,800 dedup MoveToDeletedItems events and a tenant-wide `RecordType=ExchangeItem` query (also 0). Conclusion: app-only UAL cannot read mailbox-item records for this tenant/SP. Stopped cron `ce6e3e74` (it had actually persisted only as session-only despite the durable flag). Attribution rests on device-statistics timing + the bisection.

Yvonne replied confirming the root cause: she found Bolton's blocked contact on **Bill's new iPad** (matches iPad15C8), unblocked it on the phone + new iPad, and will check the other iPads. This validates the device-block diagnosis and the bisection method.

Yvonne requested a second cleanup: `/Inbox/9000 DUPLICATES - 06-26-2026` (she said it was under "8888 Client Dev" and had 3 subfolders; actually directly under Inbox with 6 loose items + 4 subfolders). Ran the generalized scan over the whole subtree (614 items): 609 true duplicates, 5 uniques. With user approval ("deal with all of them - keep all unique items"), soft-deleted the 609 to Deleted Items (0 errors), kept the 5 uniques (1 in 8376 Baines, 2 in 8390 DUPLICATES (2), 2 in 8395 Hixon). Posted a fresh emailed comment (420851846) to #32467 summarizing the second cleanup and acknowledging the device fix.

### Configuration Changes (delta)
- bt@ "DUPLICATE need to check": 11,864 -> 54 (11,813 moved to Deleted Items).
- bt@ "9000 DUPLICATES - 06-26-2026" subtree (4 subfolders + loose): 614 -> 5 (609 moved to Deleted Items).
- Syncro #32467 created (Waiting on Customer, no billing); comments 420827280 (first cleanup) + 420851846 (second cleanup), both emailed to Yvonne.
- Cron `ce6e3e74` deleted (UAL probe non-viable).
- Memory `reference_tedards_tenant_facts` updated: UAL app-only limitation + customer-confirmed root cause.
- New scan tooling (gitignored scratch, `.dup*`): `.dupscan.py`, `.dupscan2.py`, `.dupdelete.py`, `.dupdrive.sh`, `.findfolder.py`.

### Totals
- Duplicates removed across both folders: **12,419** (11,810 + 609). Unique items preserved: **59** (54 + 5). All soft-deletes recoverable in Deleted Items.

### Pending
- Yvonne: unblock Lindsay on the remaining iPads; file the 59 uniques; empty Deleted Items when satisfied.
- bt@ delete-folder fix awaiting confirmation once all devices are cleared.
- Other duplicates folders exist (e.g. `/Inbox/ZZZZZZZZZZZZZZZ CLIENTS/8376 CCD-Maine [Baines]/8376 DUPLICATES`, 422 items) — not requested; offer if relevant.

### Reference (delta)
- **Syncro:** #32467 (id 113090881) — https://computerguru.syncromsp.com/tickets/113090881
- Target folder: `/Inbox/9000 DUPLICATES - 06-26-2026` (id `…AAABGP7iAAA=`)
