diff --git a/clients/wolkin/session-logs/2026-07/2026-07-23-howard-duplicate-email-investigation.md b/clients/wolkin/session-logs/2026-07/2026-07-23-howard-duplicate-email-investigation.md new file mode 100644 index 00000000..fc610959 --- /dev/null +++ b/clients/wolkin/session-logs/2026-07/2026-07-23-howard-duplicate-email-investigation.md @@ -0,0 +1,83 @@ +# Wolkin - Duplicate Email Delivery Investigation & SPF Cleanup + +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Investigated a complaint from Emily Ward at Fennemore Law (EWard@fennemorelaw.com) who reported receiving the same email from Robert Wolkin (robert@rswolkin.com) 26 times. Julie Guda (julie@rswolkin.com) forwarded the complaint to howard@azcomputerguru.com. The email thread concerned "ARS v. Ability Air/Markus Kashinsky" with a PDF attachment ("plaintiff's response to defendant's motion to strike"). + +Performed a comprehensive investigation across all mail infrastructure: checked Howard's mailbox for the forwarded complaint, then pivoted to the rswolkin.com M365 tenant using the remediation tool suite. Examined Robert's and Julie's sent items, inbox rules, mailbox forwarding, transport rules, outbound/inbound connectors, and Exchange message trace (Get-MessageTraceV2). All showed clean single sends with no duplicates leaving M365. Confirmed Julie sends as Robert via Send-on-Behalf permission (GrantSendOnBehalfTo on Robert's mailbox). + +Checked the legacy mail infrastructure: ix.azcomputerguru.com (cPanel/Exim server where rswolkin.com was configured as a local domain) had zero rswolkin entries in current or rotated Exim logs and only 5 unrelated messages in queue. Could not SSH to websvr.acghosting.com (162.248.93.233) or the nfoservers VDS (162.248.93.81) - no key auth. Used RMM to check Outlook configurations on both RSW-Laptop (Julie) and DESKTOP-V1JT1SE (Robert) - both machines use Exchange/MAPI connections to M365 with OST files, no legacy POP/IMAP/SMTP accounts. + +Identified the root cause via email headers: Mimecast Targeted Threat Protection (TTP) markers (`-ttpmessagereleasesuccess`) in the References header indicated Fennemore's Mimecast gateway was re-delivering held messages. Emily's follow-up emails on 7/21 ("This was emailed again") and 7/23 ("I just received this again. The time stamp shows 7/22/2026 at 9:30 pm") confirmed ongoing re-delivery. The problem is on Fennemore's side - their Mimecast is periodically re-releasing/re-delivering a previously held message. + +Cleaned up the rswolkin.com SPF record via WHM API on ix.azcomputerguru.com, removing three legacy IPs that were no longer in use. Created Syncro ticket #32583 with public comment emailing the full investigation findings to the customer, added private work notes, billed 1 hour emergency remote at $225.00, and invoiced. + +## Key Decisions + +- Identified the investigation as remote emergency work given the active re-delivery issue and client urgency +- Determined the root cause was on Fennemore's side (Mimecast TTP) rather than any rswolkin.com infrastructure, based on: clean M365 message traces showing single sends, no legacy SMTP in use, and Mimecast TTP markers in email headers +- Cleaned up the SPF record proactively since the legacy IPs were confirmed unused and their presence was misleading (initially appeared to be the cause of duplicate sends) +- Used emergency remote rate ($225.00 = $150 x 1.5) since this was after-hours/emergency investigation work +- Sent the investigation findings as a public Syncro comment so it would email to the customer with actionable instructions for Fennemore's IT team + +## Problems Encountered + +- Get-MessageTrace cmdlet is deprecated (since Sept 2025) - switched to Get-MessageTraceV2 +- Get-MessageTraceV2 has a 10-day max window per query - had to run multiple queries to cover the full date range +- WHM API on the IX server did not recognize `remove_zone_record`/`add_zone_record` (newer API names) - used `editzonerecord` instead which worked +- Could not SSH to websvr.acghosting.com or the nfoservers VDS (162.248.93.81) - no key-based auth configured for root +- Sign-in audit logs returned 403 on rswolkin.com tenant - tenant lacks Entra ID P1/P2 license +- RMM PowerShell dispatch with inline quotes caused jq parse errors - switched to ps-encoded.sh approach + +## Configuration Changes + +- **rswolkin.com SPF record** (DNS on ix.azcomputerguru.com): Changed from `v=spf1 +a +mx +ip4:72.194.62.5 +ip4:162.248.93.233 +ip4:162.248.93.81 +include:spf.protection.outlook.com -all` to `v=spf1 +include:spf.protection.outlook.com -all` via `whmapi1 editzonerecord` (Line 16 of zone file) + +## Credentials & Secrets + +- No new credentials discovered or created + +## Infrastructure & Servers + +- **rswolkin.com M365 tenant ID:** ceb6dbe7-82c8-4d8f-9c6b-49aa26208e9b +- **ix.azcomputerguru.com:** 72.194.62.5 - cPanel/WHM server, Exim MTA, Dovecot IMAP/POP3. RMM agent ID: 4ad2e426-b03f-4c5d-817c-c8c675ba73a0. Hosts DNS for rswolkin.com (ns1/ns2/ns3.acghosting.com) +- **websvr.acghosting.com:** 162.248.93.233 - no SSH access, no RMM agent +- **nfoservers VDS:** 162.248.93.81 (v-162-248-93-81.unman-vds.premium-la.nfoservers.com) - no SSH access +- **RSW-Laptop:** RMM agent ID 043fd673-35a2-4d3d-8f91-ed73ce70cc1e, Julie's machine, Windows, Outlook 16.0.20131.20154 +- **DESKTOP-V1JT1SE:** RMM agent ID 30f6af79-ab19-4ed3-9ebc-71b2bffc2d27, Robert's machine, Windows, Outlook 16.0.20131.20154 +- **rswolkin.com users:** Robert Wolkin (robert@rswolkin.com), Julie Guda (julie@rswolkin.com), Mike Swanson (admin@rswolkin.onmicrosoft.com) +- **Julie has Send-on-Behalf for Robert's mailbox** (GrantSendOnBehalfTo) +- **Fennemore Law uses Mimecast** for email security (Targeted Threat Protection / Attachment Protection) +- **rswolkin.com MX:** rswolkin-com.mail.protection.outlook.com (M365) +- **rswolkin.com A record:** 72.194.62.5 (ix.azcomputerguru.com) + +## Commands & Outputs + +- `Get-MessageTraceV2 -SenderAddress robert@rswolkin.com -RecipientAddress *@fennemorelaw.com` - only 2 entries (7/19 reply), no original July 18 email +- `Get-MessageTraceV2 -SenderAddress julie@rswolkin.com -RecipientAddress *@fennemorelaw.com` - clean single sends on 6/25, 6/29, 7/9, 7/22 +- `whmapi1 editzonerecord domain=rswolkin.com line=16 type=TXT ttl=14400 txtdata="v=spf1 +include:spf.protection.outlook.com -all"` - result: 1 (success) +- `exim -bpc` on IX server: 5 messages in queue (none rswolkin-related) +- Mimecast TTP marker in email headers: `-ttpmessagereleasesuccess-17844237` appended to message reference ID + +## Pending / Incomplete Tasks + +- Fennemore's IT team needs to check their Mimecast TTP hold/release queue - this is on the customer to communicate +- websvr.acghosting.com and the nfoservers VDS (162.248.93.81) still in the old SPF - these servers should be audited for other domains that may reference them +- The rswolkin.com cPanel account on IX still exists (25MB, user "wolkin") with DKIM key - may want to clean up the cPanel account if no longer needed +- The rswolkin@cox.net OST file on DESKTOP-V1JT1SE is dormant since Jan 2026 - could be removed from the Outlook profile + +## Reference Information + +- **Syncro ticket:** #32583 (id 114162730) - https://computerguru.syncromsp.com/tickets/114162730 +- **Syncro invoice:** id 1651143605, $225.00 +- **Syncro customer:** Wolkin, Robert (id 796910) +- **Julie Guda Syncro contact ID:** 4504147 +- **Emily Ward email:** EWard@fennemorelaw.com +- **Amaris Aloise email:** aaloise@fennemorelaw.com +- **Robert's M365 mailbox server:** SJ2P221MB1136.NAMP221.PROD.OUTLOOK.COM +- **Julie's M365 mailbox server:** DS4PPF16391B222.NAMP221.PROD.OUTLOOK.COM