From 887a672e7d3a4ec0309939ff7ae597f7d74243ab Mon Sep 17 00:00:00 2001 From: Administrator Date: Tue, 17 Mar 2026 20:08:04 -0700 Subject: [PATCH] scc: Neptune Exchange cleanup - domain/mailbox removal, SBR routing, Mailprotector config, spam purge Co-Authored-By: Claude Opus 4.6 (1M context) --- .../2026-03-17-neptune-exchange-cleanup.md | 275 ++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 clients/internal-infrastructure/session-logs/2026-03-17-neptune-exchange-cleanup.md diff --git a/clients/internal-infrastructure/session-logs/2026-03-17-neptune-exchange-cleanup.md b/clients/internal-infrastructure/session-logs/2026-03-17-neptune-exchange-cleanup.md new file mode 100644 index 0000000..e264b75 --- /dev/null +++ b/clients/internal-infrastructure/session-logs/2026-03-17-neptune-exchange-cleanup.md @@ -0,0 +1,275 @@ +# Session Log: 2026-03-17 - Neptune Exchange Server Cleanup & Mailprotector Configuration + +## Session Summary + +Comprehensive Exchange Server maintenance on Neptune (mail.acghosting.com / 67.206.163.124). Cleaned up stale accepted domains and mailboxes, fixed outbound mail routing through Mailprotector (emailservice.io) smarthosts, created inbound restriction rules, tightened DNS security records, and purged ~20K spam messages that bypassed the filter. + +### Key Accomplishments + +1. **Accepted Domain Cleanup** - Removed 9 stale domains, disabled 23 mailboxes total (12 on removed domains, 11 orphans, 1 leftover) +2. **Send Connector Fix** - Moved all send connectors from dead MAIL server to NEPTUNE +3. **SBR Routing Restored** - Added devconllc.com and littlehearts domains to Mailprotector SBR agent config +4. **Transport Rule for Inbound Restriction** - Created rule blocking direct delivery (bypassing Mailprotector) for devcon and littlehearts domains +5. **DNS Hardening** - Added secondary MX records and tightened DMARC to p=reject for devconllc.com +6. **Spam Purge** - Soft-deleted 20,473 spam messages from littlehearts/airandspace mailboxes that bypassed filter + +### Key Decisions +- MAIL server no longer exists - all routing moved to NEPTUNE +- airandspaceacademy.com is the old domain name for littleheartslittlehands (school renamed) +- simplehost.email kept as default accepted domain (was originally slated for removal) +- littleheartslittlehands.com and acg.local kept as safe domains +- Transport rules using RouteMessageOutboundConnector are NOT supported on-prem Exchange 2016 (Multi-tenant only error) +- SBR routing uses two transport agents: messageconcept ExSBR + Microsoft Exchange SBR with config files in agents\Custom folder + +### Problems Encountered +1. **Transport rules crashed transport service** - RouteMessageOutboundConnector action throws "Multi-tenant deployments supported only" on standalone Exchange 2016. All messages got poisoned. Fixed by removing rules and using SBR agent config instead. +2. **Pickup/Replay directory messages poisoned** - Test messages injected via pickup/replay directories were marked as poison. Used real mailbox send for testing instead. +3. **Search-Mailbox can't move within same mailbox** - "source mailbox cannot be used as the target mailbox." Used -DeleteContent (soft delete to Recoverable Items) instead. + +--- + +## Infrastructure Details + +### Exchange Servers +- **NEPTUNE** (primary, this server): Exchange 2016 Standard Evaluation, Build 15.1.2507.17 +- **MAIL**: Exchange 2016 Enterprise, Build 15.1.2507.18 - **NO LONGER EXISTS** +- Both registered as Mailbox role servers + +### Server Details +- **Hostname:** neptune.acghosting.com / mail.acghosting.com +- **External IP:** 67.206.163.124 +- **Internal IP:** 172.16.3.11 +- **Domain:** acg.local +- **Let's Encrypt Cert:** CN=mail.acghosting.com, SANs: autodiscover.acghosting.com, autodiscover.amtransit.com, mail.amtransit.com, mail.devconllc.com, mail.littleheartslittlehands.org, mail.packetdial.com, mail.rieussetcorp.com, mail.tucsongoldencorral.com +- **Cert Expiry:** 2026-05-31 + +### DKIM Signer +- **Agent:** Exchange DkimSigner (C:\Program Files\Exchange DkimSigner\ExchangeDkimSigner.dll) +- **Algorithm:** RSA-SHA256, Simple/Simple canonicalization +- **Configured Domains:** + - amtransit.com (selector: s1) + - littleheartslittlehands.org (selector: default) + - tucsongoldencorral.com (selector: dkim) + - devconllc.com (selector: default) + - jparkinsonaz.com (selector: s1) + - rieussetcorp.com (selector: s1) +- **Keys:** C:\Program Files\Exchange DkimSigner\keys\ + +### SBR Agent Configuration +- **Config Path:** C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Custom\ +- **Files:** + - `Microsoft.Exchange.SBR.dll` - SBR routing agent + - `Microsoft.Exchange.SBR.InternalDomains.config` - Domain list + - `Microsoft.Exchange.SBR.OverrideSettings.config` - Domain-to-SBR mapping + - `Microsoft.Exchange.SBR.IgnoreAuthAs.config` - (empty) +- **Also installed:** messageconcept ExSBR (C:\Program Files\messageconcept\ExSBR\SenderBasedRouting.dll) + +### SBR Config (OverrideSettings.config) - Current State +``` +amtransit.com;amtransit.sbr +littleheartslittlehands.org;littleheartslittlehands.sbr +tucsonsafety.com;tucsonsafety.sbr +rieussetcorp.com;rieussetcorp.sbr +devconllc.com;devconllc.sbr +littleheartslittlehands.com;littleheartslittlehands.sbr +airandspaceacademy.com;airandspaceacademy.sbr +``` + +### SBR Config (InternalDomains.config) - Current State +``` +amtransit.com +littleheartslittlehands.org +tucsonsafety.com +rieussetcorp.com +devconllc.com +littleheartslittlehands.com +airandspaceacademy.com +``` + +--- + +## Mailprotector (emailservice.io) IPs + +### Transport Servers (US) +- 52.0.70.91 +- 52.0.74.211 +- 52.0.31.31 + +### Inbound Gateway Servers +- 52.0.43.153, 52.0.90.6, 52.0.156.43, 52.0.161.190 +- 52.1.76.196, 52.1.130.188, 52.1.217.73 +- 54.85.114.151, 54.152.152.44, 54.80.77.105 +- 52.204.186.160, 3.213.159.102, 23.20.39.50 +- 18.214.219.227, 34.233.23.45 + +### LDAP/AD Sync +- 54.152.160.142, 54.152.160.187 + +### Europe Transport +- 54.229.38.56, 54.229.197.37, 54.229.198.191 + +### Asia Pacific Transport +- 54.66.143.79, 54.66.158.252, 54.66.239.122 + +--- + +## Changes Made + +### 1. Accepted Domains Removed (9) +botapro.com, capacitance.rocks, cycloneinspiredproducts.com, gurushow.com, heieck.org, rondieyancey.com, royalweedcontrol.com, sstargroup.com, thisisnotmy.email + +### 2. Mailboxes Disabled (24 total) +**On removed domains (12):** kurt/brit/christine/mailer/orders/payments@botapro.com, info@cycloneinspiredproducts.com (acg.local primary), sheila/jjh@heieck.org, sales/admin@royalweedcontrol.com, crf@sstargroup.com + +**Orphan domains (11):** rondie@lamaddux.com, social@erinhelm.com, 8231/skeener/skeener2/y226/bt/walid@tedards.net, info@retiredpaws.org, info/katta@emoxpress.com + +**Leftover (1):** cyclone@acg.local + +### 3. Remaining Accepted Domains (19) +acg.local, acghosting.com (ExternalRelay), airandspaceacademy.com, amtransit.com, devconllc.com, farwestwell.com, goldenchoicecatering.com, jparkinsonaz.com, justsimplysmart.com, lifelonglearningacademy.com, littleheartslittlehands.com, littleheartslittlehands.org, outaboundssports.com, packetdial.com, patriotinternalmedicine.com, rieussetcorp.com, simplehost.email (Default), tucsongoldencorral.com, tucsonsafety.com + +### 4. Send Connectors (Final State) +All sourced from NEPTUNE: + +| Connector | Address Space | Smart Host | +|-----------|--------------|------------| +| Outbound.DEVCON | devconllc.sbr | devconllc-com.outbound.emailservice.io | +| Outbound.LittleHearts | littleheartslittlehands.sbr, airandspaceacademy.sbr | littleheartslittlehands-org.outbound.emailservice.io | +| Outbound.Patriot | patriotinternalmedicine.sbr | patriotinternalmedicine-com.outbound.emailservice.io | +| Outbound.Farwestwell | farwestwell.sbr | farwestwell-com.outbound.emailservice.io | +| Outbound.TGC | tucsongoldencorral.sbr | tucsongoldencorral-com.outbound.emailservice.io | +| Outbound.LLA | lifelonglearningacademy.sbr | lifelonglearningacademy-com.outbound.emailservice.io | +| Outbound.AMT | amtransit.sbr | amtransit-com.outbound.emailservice.io | +| Outbound.TucsonSafety | tucsonsafety.sbr | tucsonsafety-com.outbound.emailservice.io | +| Outbound.Sorensen | rieussetcorp.sbr | rieussetcorp-com.outbound.emailservice.io | +| Horseshoe Outbound | horseshoemgt.sbr | horseshoemgt-com.outbound.emailservice.io | +| Outbound.Avoid Filter | Q.com | webhost.acghosting.com | +| Other | * (catch-all) | DNS routing | + +**Removed:** devconllc.com_ExSBR (duplicate), AOL/YAHOO (disabled) + +### 5. Transport Rules (Final State) +| Rule | Priority | Description | +|------|----------|-------------| +| Restrict Inbound - Devcon and LittleHearts | 0 | Reject 5.7.1 if recipient is devconllc.com/littleheartslittlehands.org/.com/airandspaceacademy.com AND sender is external AND source IP not in Mailprotector list | +| Webhost Spam | 1 | Delete messages from webhost.acghosting.com or fabry | +| Bardach BCC | 2 | BCC rule for Bardach | + +### 6. DNS Changes (devconllc.com via IX WHM API) +- **Added:** MX 20 devconllc-com.inbound.emailservice.cc +- **Added:** MX 30 devconllc-com.inbound.emailservice.co +- **Updated:** DMARC from `p=none;sp=none` to `p=reject;sp=reject;fo=1` + +### 7. Spam Purge Results +20,473 messages soft-deleted (Recoverable Items, 14 days retention): +- rklem@littleheartslittlehands.org: 7,798 +- marylou@littleheartslittlehands.org: 12,594 +- sbranch@airandspaceacademy.com: 5 +- ajoseph@airandspaceacademy.com: 35 +- mrocha@airandspaceacademy.com: 33 +- tstevens@airandspaceacademy.com: 4 +- email@airandspaceacademy.com: 4 + +--- + +## Credentials Used + +### IX Server (WHM API) +- **Host:** ix.azcomputerguru.com:2087 +- **User:** root +- **Password:** Gptf*77ttb!@#!@# +- **API:** JSON API via curl with basic auth +- **Used for:** DNS zone queries and edits (dumpzone, addzonerecord, editzonerecord) + +### Neptune Exchange +- **Access:** Local PowerShell with Exchange Management Shell snapin +- **Snapin:** Microsoft.Exchange.Management.PowerShell.SnapIn +- **No credentials needed** (running as administrator.ACG) + +--- + +## Domain Status Summary + +### devconllc.com - FULLY CONFIGURED +- DNS: IX (ns1/ns2.acghosting.com) +- MX: 3x Mailprotector inbound [OK] +- SPF: Includes spf.us.emailservice.io [OK] +- DKIM: default selector, signing on Exchange [OK] +- DMARC: p=reject [OK] +- Outbound: SBR -> devconllc-com.outbound.emailservice.io [OK] +- Inbound restriction: Transport rule [OK] + +### littleheartslittlehands.org - FULLY CONFIGURED +- DNS: IX (ns1/ns2.acghosting.com) +- MX: 3x Mailprotector inbound [OK] +- SPF: Includes spf.us.emailservice.io [OK] +- DKIM: default selector, signing on Exchange [OK] +- DMARC: p=none (could tighten) +- Outbound: SBR -> littleheartslittlehands-org.outbound.emailservice.io [OK] +- Inbound restriction: Transport rule [OK] + +### airandspaceacademy.com - NEEDS DNS FIX +- DNS: GoDaddy (ns71/ns72.domaincontrol.com) +- MX: **STILL POINTS TO mail.acghosting.com (DIRECT - NO FILTER)** +- Outbound: SBR -> airandspaceacademy.sbr connector [OK] +- Inbound restriction: Transport rule now BLOCKING direct delivery +- **ACTION NEEDED:** Change MX on GoDaddy to airandspaceacademy-com.inbound.emailservice.io (if provisioned in Mailprotector) + +### littleheartslittlehands.com - PARTIAL +- DNS: Cloudflare (kristina/nile.ns.cloudflare.com) +- MX: Points to cbsolt.net (NOT Mailprotector) +- Outbound: SBR configured [OK] +- **ACTION NEEDED:** Change MX on Cloudflare to Mailprotector + +--- + +## Pending/Incomplete Tasks + +1. **airandspaceacademy.com MX** - Needs changing from mail.acghosting.com to Mailprotector inbound on GoDaddy DNS. Currently being REJECTED by the new transport rule. +2. **littleheartslittlehands.com MX** - Points to cbsolt.net on Cloudflare, needs updating to Mailprotector. +3. **littleheartslittlehands.org DMARC** - Currently p=none, should be tightened to p=reject like devcon. +4. **Missing SBR domains** - farwestwell, patriotinternalmedicine, tucsongoldencorral, goldenchoicecatering, lifelonglearningacademy not in SBR config files yet (they have send connectors but SBR agent won't route them). +5. **Transport cert expiring** - Thumbprint 5C202EE2700E34A121642FDA07190ABE907D6EAD expires 2026-05-31. +6. **Retry queues** - ~40 empty retry queues from flushed spam still visible (will auto-clean). +7. **MAIL server removal from AD/Exchange** - Dead server still registered. Should be formally decommissioned. +8. **Horseshoe Management** - Has SBR send connector but domain not in SBR config and no accepted domain. Status unknown. +9. **5 outdated WordPress sites on IX** - Security risk (from previous IX cleanup session). + +--- + +## Reference + +### Exchange PowerShell Quick Reference +```powershell +# Load snapin +Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn + +# SBR config files +C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Custom\Microsoft.Exchange.SBR.OverrideSettings.config +C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Custom\Microsoft.Exchange.SBR.InternalDomains.config + +# DKIM config +C:\Program Files\Exchange DkimSigner\settings.xml +C:\Program Files\Exchange DkimSigner\keys\ + +# Frontend protocol logs (contains real source IPs) +C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive\ + +# Restart transport after SBR config changes +Restart-Service MSExchangeTransport -Force +``` + +### WHM API (IX Server) +```bash +# Dump zone +curl -sk "https://ix.azcomputerguru.com:2087/json-api/dumpzone?domain=DOMAIN" -u "root:PASSWORD" + +# Add record +curl -sk "https://ix.azcomputerguru.com:2087/json-api/addzonerecord?domain=DOMAIN&type=TYPE&..." -u "root:PASSWORD" + +# Edit record (need Line number from dumpzone) +curl -sk "https://ix.azcomputerguru.com:2087/json-api/editzonerecord?domain=DOMAIN&Line=N&..." -u "root:PASSWORD" + +# Find cPanel user for domain +curl -sk "https://ix.azcomputerguru.com:2087/json-api/listaccts?searchtype=domain&search=DOMAIN" -u "root:PASSWORD" +```