azcomputerguru/claudetools
1
0
Fork 0
Code Issues 59 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dbe355a2d5dcf8263b0fbe08bb93e5c838f4e69c
claudetools/clients/tedards/session-logs/2026-06/2026-06-25-mike-bt-delete-folder-and-dedup.md
Mike Swanson 79789a8815 sync: auto-sync from GURU-5070 at 2026-06-26 04:15:16 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-26 04:15:16
2026-06-26 04:16:39 -07:00

15 KiB
Raw Blame History

Session Log — Tedards: bt@ "delete folder" root cause + DUPLICATE folder cleanup

User

  • User: Mike Swanson (mike)
  • Machine: GURU-5070
  • Role: admin

Session Summary

Resumed the Tedards investigation with a directive to make zero assumptions and verify live 365 access. Confirmed read access to both Tedards mailboxes (y226@tedards.net, bt@tedards.net) through the ComputerGuru app suite against tenant 4fcbb1f4-fbf9-4548-a93e-7d14a3c091e6: Graph via the Security Investigator tier (200 on user + message reads) and Exchange Online via the Exchange Operator tier (200 on Get-Mailbox). Documented that the Security Investigator investigator-exo tier 401s on the EXO adminapi because its app registration carries only full_access_as_app, not Exchange.ManageAsApp; the directory role (Exchange Administrator) is already assigned, so the gap is an app-registration API permission, not a role. The exchange-op tier is the all-access Exchange tier and was used for all EXO work thereafter (logged as a recurring correction).

Investigated the long-standing complaint that emails from lindsay@agencyzoomify.com were disappearing from Bill's inbox into a "delete folder" in real time. Ruled out every server-side mechanism with live data: 18 inbox rules (all newsletter filing, none touching Lindsay/Bolton/Deleted Items), zero sweep rules, no transport rules, no forwarding, no delegates, no Inbox folder permissions, and no OAuth app holding Mail.ReadWrite. Located the three affected messages in Deleted Items and confirmed via timing (received at spread-out times, all deleted in one 2-minute burst) that the deletion was interactive/client-side, not an automatic delivery-time rule. A bait test (EWS-move the 3 messages back to Inbox) proved automated re-deletion: all three were re-trashed within ~1-4 seconds at an identical timestamp, by a client, keyed to Lindsay's address (a co-threaded message from brandon@agentive-one.com survived). A bisection (disabled/removed the two new Outlook-for-iOS clients, re-baited) showed the re-delete fired before the Outlook-iOS clients re-synced, exonerating them and pinning the cause to a native iOS Mail (EAS) device — iPhone16C2 or iPad15C8 — with an on-device "Block Sender -> Move to Trash" for Lindsay. On-device block lists are not server-readable, so the fix is on Bill's devices.

Enabled tenant auditing to make any recurrence attributable: the tenant was dehydrated, so ran Enable-OrganizationCustomization (irreversible, confirmed with the user) then Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $true. The config flag propagated to true. Re-baited under live ingestion and filed the 3 messages into Bill's "BOLTON, Lindsay" folder (confirmed they stay put there — the block only sweeps the Inbox). A durable cron (ce6e3e74, every ~19 min) rechecks Search-UnifiedAuditLog for the capture; through end of session it returned entries=0 (ingestion still propagating / app-only UAL not yet surfacing mailbox-item events).

Billed 2.0h remote labor to ticket #32228 ($300, new invoice #67886) after verifying the new invoice contained only the 2h line (the prior 0.5h stays on #67882). Posted a customer-visible, emailed comment to Yvonne with the device-fix steps. Note: the session-log reference to "ticket #5070" does not resolve to any Syncro ticket; #32228 ("Unable to send/receive email to/from lindsay@agencyzoomify.com") is the correct ticket for this work.

Began cleanup of Bill's "DUPLICATE need to check" folder (11,864 items, a botched-import artifact Yvonne preserved). Swept all 119,223 mailbox messages, matched by Internet Message-ID, and classified: 11,810 true duplicates (identical copy confirmed in another real folder, excluding Deleted Items) and 54 unique (only copy — kept). With explicit user approval, soft-deleting (EWS move to Deleted Items, recoverable) the 11,810. As of save, the move is still running in the background (folder at ~3,264, 8,603 already moved, converging to 54). A new no-billing notification ticket for Yvonne is drafted and user-approved but NOT yet posted — held until the folder reaches 54 so the email numbers match the mailbox.

Key Decisions

  • Use the exchange-op tier for all Exchange writes/all-access — it holds Exchange Administrator + full_access_as_app + Exchange.ManageAsApp. Stop claiming "no tier can write mail." (Recurring correction; saved to memory feedback_exchange_op_all_access.)
  • Did not add Exchange.ManageAsApp to the Security Investigator app — it's a manual portal change and the read-only tier shouldn't have it; exchange-op covers everything.
  • Bait test over waiting — moving the messages back to Inbox to observe behavior was the only way to prove automated re-deletion without a queryable audit log.
  • Soft-delete (move to Deleted Items), not hard purge for the 11,810 duplicates — recoverable until Yvonne empties Deleted Items.
  • Kept the 54 uniques — they have no copy elsewhere; deliberately did not dedupe them against each other to avoid removing a last-surviving copy.
  • Held the notification ticket until folder = 54 — avoid emailing Yvonne "54 remaining" while thousands are still visible.
  • Confirmed Enable-OrganizationCustomization with the user before running — irreversible org-level change.

Problems Encountered

  • investigator-exo 401 on EXO adminapi — app lacks Exchange.ManageAsApp; used exchange-op instead. (Memory: reference_investigator_exo_manageasapp_gap.)
  • Search-MailboxAuditLog deprecated (Jan 2026) — switched to Search-UnifiedAuditLog; found UnifiedAuditLogIngestionEnabled=false, enabled it (required hydrating the tenant first).
  • App-only Search-UnifiedAuditLog returns 0 even after ingestion enabled and after thousands of bulk MoveToDeletedItems events — ingestion propagation lag and/or app-only UAL not surfacing mailbox-item records. Recheck cron left running.
  • Set-CASMailbox / Set-AdminAuditLogConfig propagation lag — read-backs lagged true for minutes; the OutlookMobileEnabled=false change never enforced during the bisection window (the timing evidence carried the conclusion instead).
  • Background dedup driver detached/died when launched with nohup ... & and the tool's background mode (double-backgrounding); relaunched via the tool's background mechanism only. Two instances ended up running concurrently — harmless (idempotent) but wasteful.
  • Graph $filter on from/emailAddress/address returned null intermittently — switched to client-side filtering of the message list.
  • #5070 does not exist in Syncro — verified; used #32228 after user confirmation. Avoided posting a billable comment to a guessed ticket.
  • New invoice double-bill risk on an already-invoiced ticket — guarded by verifying the new invoice total == $300 with auto-rollback (DELETE) if not.

Configuration Changes

Microsoft 365 — Tedards tenant (4fcbb1f4-fbf9-4548-a93e-7d14a3c091e6)

  • Enable-OrganizationCustomization run (irreversible; tenant was dehydrated).
  • Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $true (propagated to true).
  • bt@ mailbox: 3 Lindsay Bolton messages moved Deleted Items -> "BOLTON, Lindsay" folder.
  • bt@ mailbox: ~11,810 duplicates being moved from "DUPLICATE need to check" -> Deleted Items (in progress at save).
  • bt@ Set-CASMailbox -OutlookMobileEnabled toggled false then reverted to true (diagnostic; revert queued, propagation lag).
  • Removed two Outlook-for-iOS device partnerships (c42bbf6e…, 548f2cdd…) — they auto-re-add.

Syncro

  • Ticket #32228 (internal 109697650): customer comment 420824967 (emailed); line item 43027728 (2.0h remote @ $150); invoice 1650805941 / #67886 ($300); invoice note set.

Repo

  • .gitignore: added session scratch patterns (.dup*, .ews_*.xml, .x.json).
  • Memory: added reference_tedards_tenant_facts, reference_investigator_exo_manageasapp_gap, feedback_exchange_op_all_access; updated MEMORY.md index.

Credentials & Secrets

No new credentials created or discovered. Tokens acquired via get-token.sh (cert auth) for tiers investigator, investigator-exo, exchange-op, tenant-admin. Vault unchanged.

Infrastructure & Servers

  • Tedards M365 tenant: tedards.net / 4fcbb1f4-fbf9-4548-a93e-7d14a3c091e6
  • Mailboxes: bt@tedards.net (Bill, owner, id 3044102c-ed9d-4777-ac8c-9d973d50e328), y226@tedards.net (Yvonne, default contact, id 9c2cc39c-a523-4e73-81b7-4b818d8ea3b9)
  • Apps: Security Investigator bfbc12a4-f0dd-4e12-b06d-997e7271e10c (SP oid 2dd202e8-5f70-4826-a378-cfeeadac9cf6); Exchange Operator b43e7342-5b4b-492f-890f-bb5a4f7f40e9; Tenant Admin 709e6eed-0711-4875-9c44-2d3518c47063
  • Culprit devices (bt@): iPhone16C2, iPad15C8 (native iOS Mail / EAS). All access from Bill's home IP 69.242.239.94.
  • Key folder IDs (bt@): DUPLICATE need to check …AAABF_WUAAA=; Deleted Items …AAAAAAEKAAA=; BOLTON, Lindsay …AAEoMg_hAAA=; Inbox …AAAAAAEMAAA=.

Commands & Outputs

# verify access
get-token.sh tedards.net investigator|investigator-exo|exchange-op
investigator-exo EXO adminapi -> HTTP 401 (only full_access_as_app)
exchange-op Get-Mailbox -> 200 (bt@, y226@)

# bait test (EWS MoveItem to Inbox, then observe)
moved 03:30:58Z -> re-deleted 03:31:00Z (native device); brandon@ co-thread survived

# audit enable
Set-AdminAuditLogConfig UnifiedAuditLogIngestionEnabled -> 400 dehydrated
Enable-OrganizationCustomization -> 200
Set-AdminAuditLogConfig UnifiedAuditLogIngestionEnabled $true -> 200 (flag now true)
Search-UnifiedAuditLog (bt@, Move/Delete ops) -> entries=0 (still propagating)

# dedup
.dupscan.py: mailbox_total=119223, dup_folder=11864, true_duplicates=11810, unique_keep=54
.dupdelete.py / .dupdrive.sh: EWS MoveItem -> deleteditems, batches of 200 (in progress)

Pending / Incomplete Tasks

  • Dedup move finishing — folder ~3,264 at save, converging to 54; two idempotent background drivers running. Verify folder == 54 and Deleted Items ~= 11,810+ when done.
  • Post the approved notification ticket (no billing) to Yvonne once folder == 54 — draft ready: subject "Mailbox Cleanup - Duplicate Emails Removed; Please File Remaining Items", customer 487887, status Waiting on Customer, comment emailed, then bot alert. Tells Yvonne to file the 54 remaining uniques and to empty Deleted Items only when satisfied.
  • Audit capture — cron ce6e3e74 still returning entries=0; let it keep retrying. If it never surfaces, attribution rests on the device-statistics timing (iPhone16C2/iPad15C8).
  • On Bill's devices (customer action): unblock lindsay@agencyzoomify.com in iOS Mail (Settings - Mail - Blocked) on iPhone and iPad, or set Blocked Sender Options to None.
  • Optional: remove scratch files (.dup*) after dedup completes.

Reference Information

  • Syncro ticket: #32228 (internal 109697650) — https://computerguru.syncromsp.com/tickets/109697650
  • Invoices: #67886 (1650805941, $300, this session); #67882 (1650804914, $75, prior session)
  • Cron job: ce6e3e74 (durable, ~every 19 min) — UAL audit recheck
  • Prior session: clients/tedards/session-logs/2026-06/2026-06-25-discord-bot-agencyzoomify-dmarc-fix.md
  • Memory: reference_tedards_tenant_facts, reference_investigator_exo_manageasapp_gap, feedback_exchange_op_all_access

Update: 04:14 PT (2026-06-26) — dedup completion, ticket, root-cause confirmation, second folder

Summary

Completed the first dedup: "DUPLICATE need to check" went 11,864 -> 54 uniques; 11,813 duplicates soft-deleted to Deleted Items (recoverable). The background driver was launched twice (the first nohup & instance never died), so two idempotent instances ran concurrently and converged — harmless. Posted the approved no-billing notification ticket #32467 (id 113090881) to Yvonne once the folder reached 54, with a customer-visible emailed comment listing the result and her action items (file the 54, empty Deleted Items when satisfied).

Closed the audit-capture effort as non-viable: app-only Search-UnifiedAuditLog returned 0 across ~8 rechecks even after ~11,800 dedup MoveToDeletedItems events and a tenant-wide RecordType=ExchangeItem query (also 0). Conclusion: app-only UAL cannot read mailbox-item records for this tenant/SP. Stopped cron ce6e3e74 (it had actually persisted only as session-only despite the durable flag). Attribution rests on device-statistics timing + the bisection.

Yvonne replied confirming the root cause: she found Bolton's blocked contact on Bill's new iPad (matches iPad15C8), unblocked it on the phone + new iPad, and will check the other iPads. This validates the device-block diagnosis and the bisection method.

Yvonne requested a second cleanup: /Inbox/9000 DUPLICATES - 06-26-2026 (she said it was under "8888 Client Dev" and had 3 subfolders; actually directly under Inbox with 6 loose items + 4 subfolders). Ran the generalized scan over the whole subtree (614 items): 609 true duplicates, 5 uniques. With user approval ("deal with all of them - keep all unique items"), soft-deleted the 609 to Deleted Items (0 errors), kept the 5 uniques (1 in 8376 Baines, 2 in 8390 DUPLICATES (2), 2 in 8395 Hixon). Posted a fresh emailed comment (420851846) to #32467 summarizing the second cleanup and acknowledging the device fix.

Configuration Changes (delta)

  • bt@ "DUPLICATE need to check": 11,864 -> 54 (11,813 moved to Deleted Items).
  • bt@ "9000 DUPLICATES - 06-26-2026" subtree (4 subfolders + loose): 614 -> 5 (609 moved to Deleted Items).
  • Syncro #32467 created (Waiting on Customer, no billing); comments 420827280 (first cleanup) + 420851846 (second cleanup), both emailed to Yvonne.
  • Cron ce6e3e74 deleted (UAL probe non-viable).
  • Memory reference_tedards_tenant_facts updated: UAL app-only limitation + customer-confirmed root cause.
  • New scan tooling (gitignored scratch, .dup*): .dupscan.py, .dupscan2.py, .dupdelete.py, .dupdrive.sh, .findfolder.py.

Totals

  • Duplicates removed across both folders: 12,419 (11,810 + 609). Unique items preserved: 59 (54 + 5). All soft-deletes recoverable in Deleted Items.

Pending

  • Yvonne: unblock Lindsay on the remaining iPads; file the 59 uniques; empty Deleted Items when satisfied.
  • bt@ delete-folder fix awaiting confirmation once all devices are cleared.
  • Other duplicates folders exist (e.g. /Inbox/ZZZZZZZZZZZZZZZ CLIENTS/8376 CCD-Maine [Baines]/8376 DUPLICATES, 422 items) — not requested; offer if relevant.

Reference (delta)

Reference in New Issue View Git Blame Copy Permalink