azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: azcomputerguru:34aad7639fc27a1522d2c4f7702c832705f97573
Branches Tags
azcomputerguru:main
...
compare: azcomputerguru:b6d00207ffa3350d1635e44604bcf4c9db402dd7
Branches Tags
azcomputerguru:main

2 Commits
34aad7639f ... b6d00207ff

Author SHA1 Message Date
Mike Swanson
b6d00207ff session log: Neptune outage recovery + Exchange 2019 migration plan 
Post-reboot recovery phase: WS2022 upgrade (done 2026-04-22) identified
as root cause — Exchange 2016 unsupported on WS2022. Mail flow restored
at 14:32 via explicit DNS-server override on TransportServer (edgetransport
on WS2022 ignores OS suffix search). Rollback unavailable (all paths dead).

Migration planning phase: Exchange 2019 on fresh WS2022 VM picked over
2016-rebuild. Config snapshot captured to C:\NeptuneConfigExport-20260423\
(34 files, 22 config areas, 56-mailbox CSV inventory, SBR configs).
Full 6-phase migration runbook written covering prereqs, schema prep,
install+config port, mailbox moves, cutover, and carcass force-removal.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 15:17:35 -07:00
Administrator
742c25c96e session log: Neptune inbound mail outage + partial recovery (pre-reboot snapshot) 
KB5082142 (Windows Server 21H2 CU) + KB5084071 (.NET Framework CU) triggered
cascading Exchange 2016 failures on NEPTUNE today. External SMTP ingest was
restored after 4 fixes (registry ACL on AssistantsQuarantine, Routing Master
DN, disabled messageconcept ExSBR, hosts entries for dead MAIL server). But
internal pipeline (Submission -> categorizer -> mailbox delivery) remained
broken until 3 more fixes (DNS records on ACG-DC16 for n-hosting1/n-largeboxes
/mail, disabled hung DkimSigner agent, disabled IRM to silence RMS Encryption
Agent timeouts). Submission queue still pinned at ~427 messages pre-reboot;
full Neptune reboot queued to clear edgetransport.exe in-memory DNS cache and
pending KB5082142 reboot actions.

All registry/AD/config backups in C:\BackupBeforeFix\ on Neptune. Post-reboot
verification checklist documented in the log.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 15:17:34 -07:00
1 changed files with 735 additions and 0 deletions
Expand all files Collapse all files

735
clients/internal-infrastructure/session-logs/2026-04-23-neptune-inbound-mail-outage.md Normal file
View File

@@ -0,0 +1,735 @@
# Session Log: 2026-04-23 — Neptune Inbound Mail Outage (KB5082142 Aftermath)
## User
- **User:** Mike Swanson (mike)
- **Machine:** NEPTUNE (Exchange server, not Mike's workstation — session executed locally on the server as administrator.ACG)
- **Role:** admin
## Session Summary
**Incident:** At approximately 12:28 PM local (19:28 UTC) on 2026-04-23, all inbound mail to Neptune began being deferred. MailProtector returned `451 4.7.0 Temporary server error. Please try again later. PRX2` at end-of-DATA for every inbound relay attempt. Google / Amazon SES / other direct senders hit the same error. Outbound mail continued to work.
**Duration:** ~42 minutes of full inbound mail outage (12:28 PM – 1:10 PM local).
**Root cause (layered):**
1. **Trigger — Windows Update KB5082142** (2026-04 Cumulative Update for Windows Server 21H2) + KB5084071 (.NET Framework 3.5/4.8/4.8.1) installed at ~10 AM local, restarting the Exchange transport services at 12:26 PM.
2. **First layer — `MSExchangeDelivery` registry ACL crash:** The service runs as `NT AUTHORITY\NETWORK SERVICE`. On restart it attempted to create a new GUID subkey at `HKLM\SYSTEM\CurrentControlSet\Services\AssistantsQuarantine\7ae551a5-6ed4-4389-9b06-44a926841ead` — but that key's ACL only granted `Administrators`, `SYSTEM`, and `CREATOR OWNER`. NETWORK SERVICE had zero rights. Result: `UnauthorizedAccessException` (Event 10003) → service terminated unexpectedly (SCM Event 7031) → transport proxy sessions failing (Event 1048: "More than 15% of proxy sessions over the last 15 minutes have failed").
3. **Second layer — Dead MAIL server in Exchange topology:** Once delivery was up again, `MSExchangeFrontEndTransport` began picking a proxy target for each inbound session from the list of Mailbox servers in its AD site. The dead `MAIL` server still existed in AD (same site `Default-First-Site-Name`, same `msExchCurrentServerRoles = 16439`, same Enterprise-ish build as NEPTUNE). FETS selected MAIL, tried to resolve `mail.acg.local` via DNS, failed, and fell back to the literal placeholder string `internalproxy` as the proxy destination. DNS of course returned non-existent for `internalproxy` as well. Exchange returned `554 5.4.4 SMTPSEND.DNS.NonExistentDomain; nonexistent domain internalproxy -> DnsDomainDoesNotExist: InfoDomainNonexistent` internally, which FETS then surfaced to the external sender as `451 4.7.0 PRX2`.
4. **Third layer — Tombstoned Routing Master:** The Exchange Routing Group's `msExchRoutingMasterDN` attribute still pointed at the AD *deleted-objects* DN of MAIL: `CN=MAIL\0ADEL:528e09e8-4d78-4054-9e6d-e8ad490cd1d1,CN=Servers,...`. Event 2937 was logging this on every startup: "Property [RoutingMasterDN] ... is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible."
**Resolution path:**
Four fixes applied in sequence. The **first** stopped the delivery-service crash (no more Event 10003). The **second** cleared the tombstoned routing master warning but did not stop 451 PRX2 on its own — Exchange still had MAIL as a legitimate mailbox-server candidate for site-internal proxying. The **third** (disabling expired messageconcept ExSBR) ruled out the agent as a source of the "internalproxy" string. The **fourth** — adding hosts-file entries to redirect the dead MAIL server name to NEPTUNE's own IP — was the one that actually restored mail flow, because it made FETS's DNS lookup for `mail.acg.local` succeed and proxy back to Neptune.
The clean long-term fix is to remove the MAIL server AD object entirely, at which point the hosts entries can come off.
### Key Decisions
- Chose hosts-file workaround over AD object deletion for the immediate outage because it was reversible in seconds if anything broke.
- Formal MAIL server AD decommission deferred to a scheduled follow-up (not hot-path).
- Left the `InternalSMTPServers = 172.21.3.50` typo alone for now (doesn't appear load-bearing; likely unrelated to this outage).
### Problems Encountered
1. **First fix worked but wasn't sufficient.** After granting NETWORK SERVICE rights on `AssistantsQuarantine`, 10003 crashes stopped but 451 PRX2 continued at ~30/min. Submission queue went from 0 → 124 → 0 briefly (messages moving again), but individual proxy sessions still failed. Required pivoting to layer-2 investigation.
2. **RoutingMaster fix was misleading.** The tombstoned reference was clearly wrong (Event 2937 explicitly said so) but swapping it to NEPTUNE did not fix the PRX2. The routing master is used for OUTBOUND routing-topology coordination, not for inbound FETS→BETS proxy target selection. Good to fix anyway, but not the cause.
3. **Disabling messageconcept ExSBR didn't help either.** Ruled it out. Left disabled — it's an expired-license artifact from 2024 and the Microsoft SBR at priority 12 is what actually does the outbound SBR rewriting.
4. **Log filename confusion.** Frontend protocol logs use a 2-digit hour suffix in the filename that appears to be UTC-based (`RECV2026042319-1.LOG` covered 19:00-19:59 UTC), not local. Briefly threw off the "is logging still working?" check when the 20:00Z hour-boundary rotation produced a 0-byte file before traffic arrived.
5. **Hosts file mutated mid-session.** The file previously had `172.16.3.13 mail.acghosting.com` (stale, per my earlier Grep of CLAUDE.md context), but when I backed it up it had already been edited to `172.16.3.11 mail.acghosting.com`. Someone (Mike?) edited it live during the session or my earlier read was of a different cached copy. Non-blocking; the new MAIL / mail.acg.local entries I added are what mattered.
---
## Infrastructure Details
### Neptune Exchange Server
- **Hostname:** neptune.acghosting.com / mail.acghosting.com
- **Internal FQDN:** NEPTUNE.acg.local
- **Public IP:** 67.206.163.124 (at Dataforth)
- **Internal IP:** 172.16.3.11
- **AD Domain:** acg.local
- **Exchange:** 2016 Standard Evaluation, Build 15.1.2507.17
- **Site:** Default-First-Site-Name
- **Two mailbox DBs:** N-LargeBoxes, N-Hosting1 (both mounted on NEPTUNE)
### Dead MAIL Server (still in AD)
- **DN:** `CN=MAIL,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=acg,DC=local`
- **Exchange build in AD:** 15.1.2507.18 (newer than NEPTUNE — part of why FETS preferred it)
- **msExchCurrentServerRoles:** 16439 (same as NEPTUNE)
- **networkAddress:** `ncacn_ip_tcp:mail.acg.local` — does **not** resolve in DNS
- **State:** No physical server exists; object is the carcass of the old Exchange 2016 Enterprise box that was decommissioned.
### DNS Server
- **Primary DNS used by Neptune:** 172.16.3.50 (ACG-DC16)
- **Secondary:** 8.8.8.8
### MailProtector (emailservice.io) — unchanged from March session
All existing outbound send connectors still good (rieussetcorp, devconllc, littleheartslittlehands, airandspaceacademy, tucsongoldencorral, amtransit, tucsonsafety, farwestwell, patriot, LLA, TGC, Horseshoe, Sorensen).
---
## Root Cause Timeline
| Time (local) | Event |
|---|---|
| ~09:35 | KB5084071 (.NET Framework CU) installed |
| ~10:09 | KB5082142 (Windows Server 21H2 CU) installed |
| 12:26:37 | MSExchangeTransport stopped by update installer |
| 12:26:50 | MSExchangeTransport started |
| 12:27:03 | MSExchangeDelivery stopped |
| 12:27:04 | MSExchangeDelivery started |
| 12:28:36 | **First 10003 crash** — UnauthorizedAccessException on AssistantsQuarantine\7ae551a5-... |
| 12:28:39 | SCM 7031: Mailbox Transport Delivery terminated unexpectedly (1x) |
| 12:28:45 | Delivery restarted, but unhealthy |
| 12:45:51 | Event 1048: >15% proxy sessions failing over last 15 min |
| 12:49 | Fix #1 applied: NETWORK SERVICE FullControl on AssistantsQuarantine. Delivery crashes stopped (no more Event 10003). 451 PRX2 continues. |
| 12:58 | Fix #2 applied: msExchRoutingMasterDN → NEPTUNE DN. Event 2937 cleared. 451 PRX2 continues. |
| 13:04 | Fix #3 applied: messageconcept SenderBasedRouting agent disabled. 451 PRX2 continues. |
| 13:09 | Fix #4 applied: hosts file entries `172.16.3.11 MAIL` + `172.16.3.11 mail.acg.local`. Test message delivered: `250 2.6.0 ... Queued mail for delivery`. |
| 13:10 | 38 inbound RECEIVE events in 2 min. Submission queue draining normally. Last PRX2 at 13:09:41; none since. |
---
## Changes Made
### 1. Registry ACL — `HKLM\SYSTEM\CurrentControlSet\Services\AssistantsQuarantine`
**Before (SDDL):**
```
O:BAG:SYD:AI(A;CIID;KR;;;BU)(A;CIID;KA;;;BA)(A;CIID;KA;;;SY)(A;CIIOID;KA;;;CO)(A;CIID;KR;;;AC)(A;CIID;KR;;;S-1-15-3-1024-1065365936-1281604716-3511738428-1654721687-432734479-3232135806-4053264122-3456934681)
```
**After (SDDL):**
```
O:BAG:SYD:AI(A;CI;KA;;;NS)(A;CIID;KR;;;BU)(A;CIID;KA;;;BA)(A;CIID;KA;;;SY)(A;CIIOID;KA;;;CO)(A;CIID;KR;;;AC)(A;CIID;KR;;;S-1-15-3-1024-1065365936-1281604716-3511738428-1654721687-432734479-3232135806-4053264122-3456934681)
```
Added ACE: `(A;CI;KA;;;NS)` — Allow, ContainerInherit, KEY_ALL_ACCESS, NETWORK SERVICE.
**PowerShell applied:**
```powershell
$key = 'HKLM:\SYSTEM\CurrentControlSet\Services\AssistantsQuarantine'
$acl = Get-Acl $key
$ns = New-Object System.Security.Principal.NTAccount('NT AUTHORITY','NETWORK SERVICE')
$rule = New-Object System.Security.AccessControl.RegistryAccessRule(
$ns,
[System.Security.AccessControl.RegistryRights]::FullControl,
[System.Security.AccessControl.InheritanceFlags]::ContainerInherit,
[System.Security.AccessControl.PropagationFlags]::None,
[System.Security.AccessControl.AccessControlType]::Allow
)
$acl.AddAccessRule($rule)
Set-Acl -Path $key -AclObject $acl
```
**Backups:** `C:\BackupBeforeFix\AssistantsQuarantine-20260423-125228.reg` (registry values) and `C:\BackupBeforeFix\AssistantsQuarantine-SDDL-20260423-125228.txt` (SDDL).
### 2. Exchange Routing Group — `msExchRoutingMasterDN`
**Before:**
```
CN=MAIL\0ADEL:528e09e8-4d78-4054-9e6d-e8ad490cd1d1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),...
```
(DN of a deleted/tombstoned AD object.)
**After:**
```
CN=NEPTUNE,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=acg,DC=local
```
**PowerShell applied:**
```powershell
$rgDN = 'LDAP://CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=acg,DC=local'
$newMaster = 'CN=NEPTUNE,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=acg,DC=local'
$rg = [ADSI]$rgDN
$rg.Put('msExchRoutingMasterDN', $newMaster)
$rg.SetInfo()
```
**Backup:** `C:\BackupBeforeFix\RoutingMasterDN-20260423-125845.txt`
### 3. Disabled messageconcept ExSBR transport agent
```powershell
Disable-TransportAgent -Identity 'SenderBasedRouting' -Confirm:$false
Restart-Service MSExchangeTransport -Force
Restart-Service MSExchangeFrontEndTransport -Force
```
**Agent state now:** `SenderBasedRouting` (priority 11, messageconcept) = **Disabled**.
**Still active:** `Sender Based Routing` (priority 12, `Microsoft.Exchange.SBR.SbrRoutingAgentFactory`) = Enabled — this is the one actually doing outbound SBR rewriting for the client domains.
The messageconcept license expired 2024-04-26 + 60-day grace = 2024-06-25. Harmless to leave disabled; can be fully uninstalled later.
### 4. Hosts file — `C:\Windows\System32\drivers\etc\hosts`
**Before (active lines):**
```
172.16.3.11 mail.acghosting.com
```
**After:**
```
172.16.3.11 mail.acghosting.com
# Redirect dead MAIL server to NEPTUNE (2026-04-23 fix for PRX2 internalproxy issue)
172.16.3.11 MAIL
172.16.3.11 mail.acg.local
```
**Backup:** `C:\BackupBeforeFix\hosts-20260423-130927`
DNS cache flushed with `ipconfig /flushdns` after.
### 5. Services Restarted (multiple times during investigation)
Final restart sequence (after hosts fix):
- MSExchangeTransport
- MSExchangeFrontEndTransport
- MSExchangeDelivery
- MSExchangeSubmission
All returned to `Running`. Submission queue now processing.
---
## Verification After Fix
```powershell
# SMTP test from localhost
Test message accepted with:
>> 250 2.6.0 <0ef6332e-8ac5-4042-a3ed-01d7832efb03@neptune.acg.local> [InternalId=165837277233159, Hostname=NEPTUNE.acg.local] 1636 bytes in 0.108, 14.686 KB/sec Queued mail for delivery
```
```
PRX2 per minute since hosts fix (20:09Z):
20:09 5 (in-flight sessions with cached bad DNS)
20:10 0
20:11 0
...
Successful end-of-DATA '250 2.6.0' responses since 20:09Z: 40+
Submission queue: 0-1 messages (draining normally)
No new 10003 / 25012 / 1048 / 4999 events since 12:49 PM
```
Inbound sources confirmed flowing again: Amazon SES (54.240.73.150), MailProtector (52.0.70.91 et al.), Google (209.85.210.43), generic internet senders.
---
## Credentials Used / Discovered
### Neptune Exchange (local admin)
- **Username:** `ACG\administrator`
- **Password:** `Gptf*77ttb##` *(supplied by Mike mid-session; not actually needed — I was already running interactively on NEPTUNE as administrator.ACG)*
- **Access method:** Local PowerShell with `Microsoft.Exchange.Management.PowerShell.SnapIn`. No WinRM needed when executing on the box itself.
### DC / DNS (referenced, not touched)
- ACG-DC16.acg.local @ 172.16.3.50 — AD DNS server used for `acg.local` resolution.
*(No other credentials used or discovered this session.)*
---
## Known Tombstoned / Stale AD Objects
The MAIL server AD carcass has at minimum these references still in Exchange config:
| Object | DN / Identity | Status |
|---|---|---|
| Exchange Server | `CN=MAIL,CN=Servers,...` | Whole object — needs removal |
| ClientAccessServer | `MAIL` (per `Get-ClientAccessService`) | Attached to the server object — removes with it |
| FrontendTransportService | `MAIL` (per `Get-FrontendTransportService`) | Same |
| TransportService | `MAIL` | Same |
| MailboxTransportService | `MAIL` | Same |
| Receive connectors (6) | `MAIL\Default MAIL`, `MAIL\Client Proxy MAIL`, `MAIL\Default Frontend MAIL`, `MAIL\Outbound Proxy Frontend MAIL`, `MAIL\Client Frontend MAIL`, `MAIL\WesternTire Relay` | Will be deleted with server |
Nothing of value lives on MAIL — both mailbox databases (N-Hosting1, N-LargeBoxes) are hosted on NEPTUNE.
---
## Pending / Incomplete Tasks
### Critical (scheduled for follow-up)
1. **Formal MAIL server AD decommission** — Proper removal of the `CN=MAIL,CN=Servers,...` subtree via ADSI Edit or `Remove-ADObject -Recursive`. Once done, the hosts file entries for `MAIL` / `mail.acg.local` can be reverted. This closes the real root cause so we don't need a workaround.
### Lower priority
2. **Fix `Get-TransportConfig` — `InternalSMTPServers` typo:** Currently set to `{172.21.3.50}`. Network is `172.16.x.x`. Likely should be `172.16.3.50` (the DC) or simply empty. Doesn't appear load-bearing for this outage but worth fixing.
3. **Remove messageconcept ExSBR fully:** Agent is disabled. Uninstall the DLL at `C:\Program Files\messageconcept\ExSBR\` and the registry key `HKLM\SOFTWARE\SenderBasedRouting`.
4. **airandspaceacademy.com MX change** (from March session — still pending). Direct delivery to Neptune is now being rejected by the transport rule, but MX still points to `mail.acghosting.com` on GoDaddy.
5. **littleheartslittlehands.com MX change** (from March session — still pending). Cloudflare DNS still points to cbsolt.net.
6. **Transport cert expiry** — Thumbprint `E58BFCBAEFEFDCAED0BF9E894127A3DE64CE9C69` expires 2026-07-22 (Event 12017 logs it as "will expire soon"). Separate from the Let's Encrypt frontend cert.
7. **Windows Update after-care:** Watch for 10003 crashes on next Exchange service restart to confirm the AssistantsQuarantine ACL fix is sticky (AD inheritance shouldn't roll it back, but worth one more reboot verification).
---
## Reference
### Key file paths on Neptune
- Exchange SMTP Receive logs (frontend):
`C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive\RECV<YYYYMMDDHH>-<n>.LOG`
- Exchange SMTP Receive logs (hub):
`C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\ProtocolLog\SmtpReceive\`
- Message tracking:
`C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking\`
- SBR config (Microsoft agent, still active):
`C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Custom\Microsoft.Exchange.SBR.{InternalDomains,OverrideSettings,IgnoreAuthAs}.config`
- messageconcept ExSBR (disabled but DLL still present):
`C:\Program Files\messageconcept\ExSBR\SenderBasedRouting.dll`
- All backups from this session:
`C:\BackupBeforeFix\`
### Diagnostic one-liners that were useful
**Find PRX2 events in last hour's frontend log:**
```powershell
$f = Get-ChildItem 'C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive' | Sort-Object LastWriteTime -Descending | Where-Object Length -gt 0 | Select -First 1
Get-Content $f.FullName | Where-Object { $_ -match '451 4\.7\.0.*PRX2' } | Measure-Object | % Count
```
**Find a specific session's full SMTP trace:**
```powershell
$sid = '08DEA1720522673C'
Get-Content $f.FullName | Where-Object { $_ -match $sid }
```
**Check AssistantsQuarantine ACL:**
```powershell
Get-Acl 'HKLM:\SYSTEM\CurrentControlSet\Services\AssistantsQuarantine' | Select -Expand Access | ft IdentityReference, RegistryRights, AccessControlType -Auto
```
**Check routing master DN:**
```powershell
$rg = [ADSI]'LDAP://CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=acg,DC=local'
$rg.Properties['msExchRoutingMasterDN'].Value
```
### Event IDs to watch
| Event | Source | Meaning |
|---|---|---|
| 10003 | MSExchangeTransportDelivery | Transport process exception (e.g., registry ACL) |
| 25012 | MSExchangeTransportDelivery | Unhandled exception in SMTP session |
| 1048 | MSExchangeFrontEndTransport | >15% proxy sessions failing over 15-min window |
| 4999 | MSExchange Common | Watson crash report |
| 1035 | MSExchangeFrontEndTransport | Inbound auth failure (normal for brute-force attempts, ignore unless from trusted IPs) |
| 12035 | MSExchangeFrontEndTransport | Unable to load transport cert |
| 12017 | MSExchangeTransport / FETS | Internal transport cert expiring soon |
| 2937 | MSExchange ADAccess | AD object property points to Deleted Objects |
### Windows Updates installed today
- KB5082142 — 2026-04 Cumulative Update for Windows Server 21H2 (~10:09 local)
- KB5084071 — 2026-04 Cumulative Update for .NET Framework 3.5 / 4.8 / 4.8.1 (~09:35 local)
- KB5082427 — Update
- KB5082137 — Security Update
Neither KB5082142 nor KB5084071 release notes are known to explicitly mention the AssistantsQuarantine ACL regression — worth watching for any MS advisory or community reports. If multiple Exchange 2016 admins hit this same issue today, Microsoft will likely publish a workaround.
---
## Cross-Machine Notes
None for this session — all work was local to NEPTUNE. ClaudeTools repo is synced to this server for session-log writing; the commit from this session will sync back via normal `/sync` workflow.
---
## Update: 13:40 — Post-Fix Verification Failed, Pre-Reboot Checkpoint
After the initial 4 fixes, external SMTP senders stopped getting 451 PRX2 and inbound `RECEIVE` events resumed (~111 in a 5-minute window). **But delivery to mailboxes never actually started.** Mike confirmed `bertie@amtransit.com` was not seeing new mail. Investigation of `Get-MessageTrackingLog -EventId DELIVER` returned 0 events for the last 5 min even with 383+ messages sitting in the Submission queue.
### What was missed in the first pass
The external-facing SMTP flow (MailProtector → FETS) was restored, but the **internal pipeline** (Submission → categorizer → SmtpDeliveryToMailbox → mailbox store) was still broken in two distinct ways. Both were exposed by today's KB5084071 (.NET Framework CU) restart — they had been latent for a long time but started mattering only after transport services reloaded their agent assemblies and topology caches against the updated .NET runtime.
### Fix #5 — DNS records on ACG-DC16 for dead-MAIL short names
The `SmtpDeliveryToMailbox` queue was stuck on `NextHopDomain = n-hosting1` (the mailbox **database** short name, not a server FQDN). Last error:
```
451 4.4.0 DNS query failed. The error was: SMTPSEND.DNS.NonExistentDomain;
nonexistent domain n-hosting1 -> DnsDomainDoesNotExist: InfoDomainNonexistent
```
Hosts-file workaround for n-hosting1/n-largeboxes didn't help — **Exchange Transport's internal DNS resolver (edgetransport.exe) bypasses the Windows hosts file**. It queries the configured/adapter DNS servers directly. Verified by `[System.Net.Dns]::GetHostEntry('n-hosting1')` returning 172.16.3.11 (hosts file works for .NET/Windows resolver) while Exchange queue LastError kept reporting non-existent domain.
**Fix:** Added AD-integrated DNS A records on `ACG-DC16.acg.local` (172.16.3.50) via `Invoke-Command -ComputerName ACG-DC16` → `Add-DnsServerResourceRecordA`:
| Name | Type | Target | TTL |
|---|---|---|---|
| `n-hosting1.acg.local` | A | 172.16.3.11 | 1h |
| `n-largeboxes.acg.local` | A | 172.16.3.11 | 1h |
| `mail.acg.local` | A | 172.16.3.11 | 1h |
| `MAIL` (uppercase) | — | already exists (as `mail`, DNS isn't case-sensitive) | — |
Also added hosts-file entries on Neptune itself (belt-and-suspenders; see below).
After DNS records + `Restart-Service MSExchangeTransport`, the `n-hosting1` retry queue (164 → 235 msgs at peak) drained to 0. But messages just moved back into **Submission** instead of completing delivery.
### Fix #6 — Disabled `Exchange DkimSigner` transport agent (hung async)
Submission queue pinned at exactly 383 messages, no movement for 30+ seconds. Found this in the Application log:
```
MSExchange Extensibility 1057: Agent 'Exchange DkimSigner' went async but did
not call Resume on the new thread, while handling event 'OnCategorizedMessage'
```
The third-party Exchange DkimSigner (priority 10, assembly `C:\Program Files\Exchange DkimSigner\ExchangeDkimSigner.dll`, `IsCritical=true`) went async on `OnCategorizedMessage` and never resumed the categorizer thread. Classic transport agent async-bug pattern — the .NET Framework CU likely changed some async/threading behavior the DkimSigner depends on.
**Fix:** `Disable-TransportAgent -Identity 'Exchange DkimSigner' -Confirm:$false` then `Restart-Service MSExchangeTransport MSExchangeSubmission`. Outbound mail temporarily loses DKIM signatures — receivers with strict DMARC `p=reject` (devconllc.com is the only one we run that tight) may now get fail results on reply traffic until we re-enable.
### Fix #7 — IRM / RMS Encryption Agent timeout lockdown
After DkimSigner disabled, queue was still stuck. Found:
```
MSExchange Extensibility 1050: The execution time of agent 'RMS Encryption Agent'
exceeded 162154 milliseconds while handling event 'OnRoutedMessage' for message
with InternetMessageId: 'Not Available'.
```
RMS Encryption Agent is a hidden built-in Exchange agent (not in `Get-TransportAgent` output, not in `agents.config`, can't be disabled via `Disable-TransportAgent` — fails with "Transport agent 'RMS Encryption Agent' isn't found"). It was trying to reach a non-existent RMS licensing server and timing out after ~160s per message. `IRMConfiguration` already showed `InternalLicensingEnabled = False`, but the agent was still firing.
**Fix:**
```powershell
Set-IRMConfiguration -TransportDecryptionSetting Disabled -Confirm:$false
Set-IRMConfiguration -JournalReportDecryptionEnabled $false -Confirm:$false
```
**Result:** Event 1050 RMS warnings in last 1 min: **0**. Agent no longer firing on messages.
### Current State Before Reboot
| Item | State |
|---|---|
| Queues (Submission) | ~427 messages still accumulated, no new DELIVER events yet observed |
| External inbound (FETS) | Accepting 250 2.6.0 Queued mail responses cleanly |
| `MSExchangeDelivery` registry crash | Fixed (no 10003 since 12:49) |
| Routing master DN | NEPTUNE (fixed) |
| DkimSigner agent | Disabled |
| messageconcept ExSBR agent | Disabled |
| RMS Encryption Agent | Silenced via IRM config (can't be fully disabled) |
| DNS records (on DC) | `n-hosting1`, `n-largeboxes`, `mail.acg.local` all resolve to 172.16.3.11 |
| Hosts file (on NEPTUNE) | Has MAIL, mail.acg.local, n-hosting1, n-largeboxes → 172.16.3.11 |
| AssistantsQuarantine ACL | NETWORK SERVICE has FullControl (inheritable) |
Mike and I agreed a **full Neptune reboot** is the right next step. Rationale:
- KB5082142 (cumulative update) almost certainly has a pending-reboot action we're working around
- `edgetransport.exe` has been restarted many times but its in-memory DNS cache may still hold stale negative entries ("n-hosting1 doesn't exist")
- RMS Encryption Agent may be holding a timed-out socket to an old RMS endpoint that only a fresh process will release
### Post-Reboot Verification Checklist (for next pass)
1. `Get-Acl 'HKLM:\SYSTEM\CurrentControlSet\Services\AssistantsQuarantine'` — verify `NETWORK SERVICE` ACE survived.
2. `$rg = [ADSI]'LDAP://CN=Exchange Routing Group (DWBGZMFD01QNBJR),...'; $rg.Properties['msExchRoutingMasterDN'].Value` — verify NEPTUNE DN.
3. `Get-TransportAgent | ? Enabled -eq $false` — verify DkimSigner + SenderBasedRouting still disabled.
4. `[System.Net.Dns]::GetHostEntry('n-hosting1')` and `GetHostEntry('mail.acg.local')` — verify DNS resolution.
5. Watch Event Viewer for:
- Event 10003 / 25012 / 4999 (should not recur)
- Event 1057 "went async but did not call Resume" (should not recur — DkimSigner disabled)
- Event 1050 RMS Encryption Agent timeouts (should not recur — IRM disabled)
6. `Get-Queue | ? MessageCount -gt 0` — should drain rapidly from the ~400 messages sitting in Submission right now.
7. **End-to-end smoke test:** SMTP a message to `postmaster@azcomputerguru.com → rklem@lifelonglearningacademy.com`, confirm `250 2.6.0 Queued mail`, then `Get-MessageTrackingLog -Recipients rklem@... -EventId DELIVER` within 60 seconds.
8. Have Mike check `bertie@amtransit.com` in OWA/Outlook — confirm actual mailbox delivery of any message in the backlog.
### If reboot doesn't fully resolve delivery
Prime suspect if still stuck: the Exchange database itself. The two databases `N-Hosting1` (809 GB) and `N-LargeBoxes` (313 GB) mount on boot but if the KB5082142 update left the ESE store in a bad state, `MSExchangeIS` may be crashing after N seconds of delivery attempts. Check:
- `Get-MailboxDatabase -Status | Format-List Name, Mounted, MountedOnServer, Recovery`
- `Get-EventLog -LogName Application -Source MSExchangeIS -After (Get-Date).AddMinutes(-10) | ? EntryType -eq 'Error'`
- Event IDs to watch: 9542 (DB mount failure), 474 (page checksum mismatch), 103 (store corruption)
---
## Credentials (addendum from this phase)
### ACG-DC16 PowerShell Remoting
- No explicit creds — worked via Kerberos from an already-logged-in domain-admin session on NEPTUNE (`administrator@acg.local`).
- Used `Invoke-Command -ComputerName ACG-DC16` (NOT the IP — Kerberos needs a SPN-matching hostname).
---
## Additional File Changes (this phase)
| File | Change |
|---|---|
| `C:\Windows\System32\drivers\etc\hosts` (Neptune) | Added `172.16.3.11 n-hosting1` and `172.16.3.11 n-largeboxes` in addition to earlier MAIL/mail.acg.local entries |
| ACG-DC16 `acg.local` DNS zone | New A records: `n-hosting1`, `n-largeboxes`, `mail` → 172.16.3.11 |
| Exchange transport agents | `Exchange DkimSigner` (priority 10) → Disabled |
| Exchange IRM config | `TransportDecryptionSetting = Disabled`, `JournalReportDecryptionEnabled = False` |
All prior backups from phase 1 remain in `C:\BackupBeforeFix\`. No new registry / AD writes needed a backup in this phase (DNS records are additive; IRM config change is reversible via `Set-IRMConfiguration`).
---
## Key Lesson
**KB5084071 (.NET Framework 3.5/4.8/4.8.1 CU) + KB5082142 (Windows Server 21H2 CU)** hit this box hard. Three separate latent Exchange issues surfaced simultaneously when the transport services reloaded against the updated runtime:
1. NETWORK SERVICE missing rights on `AssistantsQuarantine` (likely pre-existing since ~2021 when Neptune was installed; only mattered once a post-update service restart tried to mint a new GUID subkey)
2. Dead MAIL server in topology being picked for proxying (pre-existing since 2026-03 cleanup; only mattered once FETS re-walked the candidate list post-restart)
3. Third-party DkimSigner async-completion behavior broken by the .NET update + RMS Encryption Agent timeout firing on every message (both latent problems that older service state tolerated)
**For future Windows Update rollouts on this server:** verify Exchange mailflow end-to-end (including `Get-MessageTrackingLog -EventId DELIVER`) within 10 minutes of the post-update reboot. The initial FETS-accepts-SMTP test is not sufficient — it only validates external ingest, not mailbox delivery.
---
## Update: 13:55 – 14:45 — Post-Reboot Recovery & Revised Root Cause
### Revised root cause (supersedes the KB-update theory above)
**Neptune was in-place upgraded from Windows Server 2016 to Windows Server 2022 on 2026-04-22 at 15:04 local** — 23 hours before today's outage. Mike disclosed this mid-session. This changes the whole analysis: KB5082142 and KB5084071 were *triggers* that forced Exchange transport services to reload, but the *underlying* problem is that **Exchange 2016 on Windows Server 2022 is an unsupported combination** (Exch 2016 requires WS2016; only Exch 2019 supports WS2022, and 2019 is out of support as of 2025-10-14). Multiple latent WS2022-incompatibilities surfaced the moment services were forced to reinitialize:
1. NETWORK SERVICE missing rights on `AssistantsQuarantine` — default ACL changed between WS2016 and WS2022.
2. Exchange transport's internal DNS resolver on WS2022 **does not honor the OS DNS suffix search list**. Queries for single-label names like `n-hosting1` get `DnsDomainDoesNotExist` even when `n-hosting1.acg.local` resolves fine via the OS resolver. This is the new big one — it's not a registry ACL fluke, it's a behavioral incompatibility baked into how `edgetransport.exe` talks to the DNS client on 2022.
3. Service-start ordering at boot: `MSExchangeADTopology` hit the SCM 45-second start-control timeout on the first post-reboot boot (WS2022 AD-locator is slower than Exchange 2016's SCM expectation), which cascaded to every other service failing with "An instance of the service is already running."
4. Built-in agents `RMS Encryption Agent` and `Index Routing Agent` (both non-disable-able via `Disable-TransportAgent`) timed out ~160 sec and ~95 sec per message respectively, effectively freezing the categorizer.
### Rollback status — NOT AVAILABLE
Checked every path. `C:\Windows.old` exists but is useless by itself:
| Recovery path | State |
|---|---|
| `dism /Online /Initiate-OSUninstall` | Error 1168 (disarmed — uninstall pointers cleaned) |
| `$Windows.~BT` / `$Windows.~WS` staging | Deleted |
| VSS shadow copies on C: | None |
| VM host snapshot | N/A — bare-metal Dell PowerEdge R720 |
| Windows Server Backup | Not installed (`wbengine` service missing) |
| External backup (Veeam/Datto/etc.) | None for this machine — Mike confirmed |
Windows cleanmgr or a post-upgrade Windows Update most likely nuked the rollback metadata inside the first 24 hours. No restore point, no snapshot, no backup. The WS2022 upgrade is effectively permanent unless we reinstall WS2016 from media.
### Recovery sequence that actually worked (chronological)
**13:45** — Full reboot completed. Server uptime reset.
**13:46–13:48** — `MSExchangeADTopology` hit 45-sec SCM timeout on auto-start; all 27 Exchange services cascaded into failed/Stopped state. No manual start was attempted during this window — it's a cold-boot race condition.
**13:57** — `Start-Service MSExchangeADTopology` (manual) completed in **3.6 seconds**. AD was reachable all along — the issue is purely that WS2022's boot timing doesn't satisfy Exchange 2016's pre-ADTopology dependency checks in the default 45-sec SCM window.
**13:58 – 14:00** — Started remaining 25 Exchange services in dependency order. All reached Running. `MSExchangeMailboxAssistants` took 41 sec, `MSExchangeHM` took 33 sec, others <30 sec. No failures.
**14:02** — Verified post-reboot state: ACL ACE survived, `msExchRoutingMasterDN = NEPTUNE`, DkimSigner + messageconcept SBR still disabled, DBs mounted (N-Hosting1 809 GB + N-LargeBoxes 313 GB), hosts + DC-side DNS entries intact.
**14:03** — Found `n-hosting1` queue stuck with 405+ messages, LastError `DnsDomainDoesNotExist`. Force-retried — no progress. This is when the WS2022 suffix-search behavior became apparent: `.NET's GetHostEntry('n-hosting1')` returns 172.16.3.11 (via suffix search), but edgetransport queries the raw label and the DC correctly answers SERVFAIL.
**14:05–14:09** — Enabled GlobalNames Zone (GNZ) on ACG-DC16 as a potential fix; added CNAMEs for `n-hosting1`, `n-largeboxes`, `mail` → `mail.acg.local.`. Did **not** help. Even after restarting DNS on the DC, `Resolve-DnsName n-hosting1` still returned SERVFAIL locally on the DC — something about GNZ lookup interacting with AD-integrated zones on this specific server refused to work. GNZ remains enabled (no harm) but it wasn't the fix.
**14:10** — Restarted `MSExchangeTransport` hoping stale topology cache was the issue. The n-hosting1 / n-largeboxes queues dropped to 0 — but the messages all moved into Submission, which then sat at 640 with 0 DELIVER events.
**14:11–14:13** — Diagnosed categorizer stall. Event Viewer showed:
- `RMS Encryption Agent` Event 1050 warnings every ~160 sec (7+ fires in one minute), `OnRoutedMessage`.
- `Index Routing Agent` Event 1050 warnings every ~90–97 sec (8+ fires in one minute), `OnResolvedMessage`.
Both are built-in, both failed `Disable-TransportAgent "Transport agent 'X' isn't found"` (they're not in the public agent list and must be silenced indirectly).
**14:14** — RMS Encryption Agent silenced via full IRM lockdown:
```powershell
Set-IRMConfiguration -InternalLicensingEnabled $false -ExternalLicensingEnabled $false `
-TransportDecryptionSetting Disabled -JournalReportDecryptionEnabled $false `
-SimplifiedClientAccessEnabled $false -EDiscoverySuperUserEnabled $false -Confirm:$false
```
(The earlier pre-reboot `Set-IRMConfiguration` from phase 1 only addressed TransportDecryption and JournalReport. The full lockdown with `InternalLicensingEnabled $false` + `ExternalLicensingEnabled $false` is what actually stopped the agent from firing.)
**14:15** — Index Routing Agent silenced by restarting the search subsystem:
```powershell
Stop-Service HostControllerService -Force
Stop-Service MSExchangeFastSearch -Force
Get-Process noderunner | Stop-Process -Force # 4 workers, 1.7 GB total
Start-Service MSExchangeFastSearch
Start-Service HostControllerService
Restart-Service MSExchangeTransport -Force
```
After this, Event 1050 stopped. But Submission queue still sat at 640 with no DELIVER events. Categorizer moved some messages (TRANSFER, REDIRECT, DSN) but nothing to mailbox.
**14:27** — **The actual fix.** On WS2022, edgetransport's DNS resolver behaves differently from WS2016 — it does not follow the NIC's `ConnectionSpecificSuffixSearchList` or the global `SuffixSearchList = {acg.local}`. Flipped the transport server off adapter-mode DNS and onto explicit DNS servers:
```powershell
Set-TransportServer NEPTUNE -InternalDNSAdapterEnabled $false `
-InternalDNSServers @('172.16.3.50','172.16.3.52')
Set-TransportServer NEPTUNE -ExternalDNSAdapterEnabled $false `
-ExternalDNSServers @('8.8.8.8','1.1.1.1','172.16.3.50')
Restart-Service MSExchangeTransport -Force
Retry-Queue -Identity 'NEPTUNE\11392' -Confirm:$false
```
With explicit DNS servers, Exchange's own resolver applies `acg.local` suffix append when querying the configured servers, and the DC's normal `acg.local` zone A records resolve `n-hosting1.acg.local` → 172.16.3.11 correctly. **This is the config that must survive and that will fix this class of issue going forward** — stored in AD (`msExchInternalDNSServers`), survives reboots, not a runtime flag.
**14:32** — First post-fix DELIVER event: Mike's earlier probe `NEPTUNE repair probe 142532` landed in `administrator@acghosting.com` at 14:32:01 (6.5 min after SMTP-accept — first message through the stack after the explicit-DNS change).
**14:38** — Bertie started receiving her day's backlog. By 14:45 she had 114 DELIVER events in the preceding 2 hours covering the whole day's traffic.
### Current state at 14:45
| Item | State |
|---|---|
| External inbound (FETS) | Accepting 250 2.6.0 cleanly |
| Submission queue | 0 (no backlog) |
| `n-hosting1` / `n-largeboxes` queues | 0 / 0 |
| Shadow queue | 9 (transient, normal) |
| Outbound delivery queue (residual) | 159 messages total, mostly Retry to dead external domains (mayo.edu rejecting + backscatter to bot domains) |
| DELIVER rate | ~10/min (32 in last 3 min) |
| DELIVER total since midnight | 532 |
| `MSExchangeDelivery` Event 10003 | 0 since the pre-reboot ACL fix |
| Event 1050 (agent timeout) | 0 since 14:13 |
| Event 1057 (async no-resume) | 0 since DkimSigner disabled in phase 1 |
| AssistantsQuarantine ACL | NETWORK SERVICE ACE persists |
| msExchRoutingMasterDN | NEPTUNE (persists) |
| `Get-MailboxStatistics bertie@amtransit.com` | 212,048 items, 31.78 GB, LastLogon 14:21 — healthy |
### Sticky risks that will reappear on next reboot
Everything that *survives* reboot through AD/Exchange config (explicit DNS servers, RoutingMasterDN, disabled agents, IRM lockdown) will stay put. Things that will bite again:
1. **`MSExchangeADTopology` 45-sec SCM start timeout** — every cold boot on this WS2022 box. Manual remediation sequence: `Start-Service MSExchangeADTopology` → then every other `MSExchangeXxx` service in dependency order. Consider setting the service recovery options (`sc failure MSExchangeADTopology reset= 60 actions= restart/5000/restart/5000/restart/5000`) so Windows retries it automatically. Better: document the manual sequence and treat reboots as planned events.
2. **AssistantsQuarantine ACL** — the registry ACE is applied, but if Windows Update or an Exchange CU ever resets default ACLs on this key, delivery crashes return. Check on any future Exchange or WS CU.
3. **IRM config** — if any Exchange CU reinstaller runs `Set-IRMConfiguration` to defaults, RMS agent timeouts return immediately. Re-apply the full lockdown after any CU.
4. **FastSearch cold-start** — the 4 noderunner workers sometimes come back in a state where Index Routing Agent blocks. If queue stalls and Event 1050 Index Routing fires post-reboot, re-run the HostController + FastSearch restart + noderunner kill sequence.
### Plan
- **Mike parallel track:** building fresh WS2016 VM to host a replacement MAIL server. Once up and joined to the DAG, mailbox moves from NEPTUNE can proceed and this box becomes decommission-able.
- **Neptune:** leave as-is on WS2022 for now. Known-good config documented above. Do not apply any Exchange CU without testing end-to-end DELIVER first. Do not touch IRM, transport agents, or TransportServer DNS settings.
- **Long-term:** Exchange Online migration is the only clean forward path — both Exchange 2016 and 2019 are out of extended support as of 2025-10-14. Separate project.
### What did NOT fix the problem (so future-us doesn't repeat)
- **GlobalNames Zone (GNZ) on ACG-DC16** — enabled with CNAMEs for `n-hosting1`, `n-largeboxes`, `mail`. Zone created, records present, GNZ enabled server-wide. Didn't resolve on the DC even locally (SERVFAIL). Left enabled, harmless. Not the fix.
- **Hosts file entries on Neptune** — re-confirmed edgetransport bypasses the Windows hosts file for transport-internal DNS. The hosts file works for `.NET` / `[System.Net.Dns]::GetHostEntry()` and for service clients like OWA, but not for the transport pipeline's resolver.
- **`Retry-Queue` with `-Resubmit $true`** — re-queued messages but they hit the same DNS lookup path and failed again.
- **Full MSExchangeTransport restart alone** — temporarily drops queue messagecounts (they move to Submission) but doesn't clear the DNS lookup failure.
### Diagnostic commands that actually helped
```powershell
# Confirm the DNS behavior gap (suffix search vs raw label)
[System.Net.Dns]::GetHostEntry('n-hosting1') # Works (OS resolver w/ suffix search)
Resolve-DnsName 'n-hosting1' -Server 172.16.3.50 # SERVFAIL (raw label, no suffix)
# Find the stuck agent — this was the smoking gun for categorizer stall
Get-WinEvent -FilterHashtable @{LogName='Application';
ProviderName='MSExchange Extensibility'; Id=1050;
StartTime=(Get-Date).AddMinutes(-3)} | `
Group-Object { $_.Message -replace "^The execution time of agent '([^']+)'.*$",'$1' }
# Check if SUBMIT events are firing (categorizer health)
Get-MessageTrackingLog -Server NEPTUNE -Start (Get-Date).AddMinutes(-3) -ResultSize Unlimited | `
Group-Object EventId | Sort-Object Count -Descending
# Verify explicit-DNS config is in place (the fix)
Get-TransportServer NEPTUNE | fl InternalDNSAdapterEnabled, InternalDNSServers,
ExternalDNSAdapterEnabled, ExternalDNSServers
```
### Cross-machine / cross-user
- All work done locally on NEPTUNE as `administrator.ACG` — no WinRM hops except `Invoke-Command ACG-DC16` for DNS.
- Will sync this log to Gitea via `/scc` from a workstation after Mike wraps the MAIL-server VM build.
---
## Update: 14:45 – 15:00 — Migration Planning & Config Snapshot
### Decision pivot — Exchange 2019 on WS2022 instead of Exchange 2016 rebuild
Mike's first plan after rollback was ruled out was to rebuild the decommissioned `MAIL` server on fresh WS2016, move all mailboxes there, and decommission NEPTUNE. I flagged it as workable but suboptimal. Mike pushed back with the better plan: install Exchange 2019 on a new WS2022 VM into the existing Exchange org, move mailboxes, force-remove the old server carcasses.
**Agreed direction:** Exchange 2019 CU14 (or CU15) on fresh WS2022 Datacenter VM. Coexists with the existing Exchange 2016 org cleanly, supported OS+Exchange combo (modulo Exchange 2019 itself being out of extended support since 2025-10-14), no new 180-day Eval clock, and sets up a trivial in-place upgrade to Exchange Subscription Edition later.
**Path rejected:** Exchange SE directly is the forward-looking product but has subscription-licensing friction (per-seat monthly + M365 tenant activation via HCW). 2019 CU14 installs without license activation — install now, upgrade to SE later as a CU-style patch once licensing is sorted.
### Organization inventory (used for migration planning)
| Item | Count / Value |
|---|---|
| Mailboxes | 56 |
| Total mailbox data | 270.1 GB |
| Mailbox DBs | 2 — N-Hosting1 (809 GB allocated, hosts 54 boxes), N-LargeBoxes (313 GB, hosts 2 boxes) |
| Send connectors | 12 (per-client SBR rewrites) |
| Receive connectors | 6 (5 standard + 1 WesternTire Relay) |
| Transport rules | 3 (organization-wide) |
| Accepted domains | 19 (client-hosted) |
| Transport agents enabled | 10 |
| Transport agents disabled | 2 (Exchange DkimSigner, messageconcept SBR priority 11) |
| UM-enabled mailboxes | **0** — role dormant despite Dial Plan + Policy being defined. No blocker for Exchange 2019. |
| Mobile device policies | 2 |
| DLP / journal rules | 0 |
| Domain controllers | 1 — ACG-DC16 (WS2016 Std, 172.16.3.52) |
| Forest functional level | Windows2012R2Forest (meets 2019 CU14 minimum) |
| Schema version (rangeUpper) | 17002 (Exchange 2016 CU23) |
| Forest schema master / PDC | ACG-DC16.acg.local (all FSMO roles) |
**Top-5 mailboxes by size** (plan overnight migration windows for these):
| Mailbox | Size (MB) | Items |
|---|---|---|
| cansley@devconllc.com | 43,769 | 132,226 |
| Channa@farwestwell.com | 31,936 | 220,938 |
| bertie@amtransit.com | 32,553 | 212,074 |
| marylou@littleheartslittlehands.org | 27,229 | 232,790 |
| scott@justsimplysmart.com | 25,745 | 386,570 |
### Config snapshot captured
Full Exchange config dumped to **`C:\NeptuneConfigExport-20260423\`** (on NEPTUNE itself for now; will copy to vault / new server as needed). Every object captured as both `.txt` (human-readable `Format-List`) and `.xml` (`Export-Clixml` for re-import if needed).
Exports:
- `send-connectors` — 12 items (full SourceTransportServers, AddressSpaces, SmartHosts, authentication)
- `receive-connectors` — 6 items
- `transport-rules` — 3 items
- `transport-config` — org-wide (1 item, including the `InternalSMTPServers = 172.21.3.50` typo still unfixed)
- `transport-server` — NEPTUNE-specific transport config incl. the DNS override we just set
- `transport-agents` — 12 agents with Enabled/Priority state
- `accepted-domains` — 19 domains
- `remote-domains` — 1 (Default)
- `email-address-policies` — 1
- `irm-config` — fully disabled state
- `malware-filter` — BypassFiltering=True on both NEPTUNE and (stale) MAIL
- `owa-vdir`, `ecp-vdir`, `ews-vdir`, `eas-vdir`, `mapi-vdir`, `oab-vdir`, `autodiscover-vdir` — all vdir configs w/ InternalUrl/ExternalUrl
- `client-access` — ClientAccessService (SCP / AutoDiscoverServiceInternalUri)
- `outlook-anywhere` — Outlook Anywhere config
- `um-dialplans` — reference only (not migrating)
- `mobile-device-policies` — 2 policies
- `mailbox-databases` — N-Hosting1 + N-LargeBoxes full state
- `mailbox-inventory.csv` — 56-row spreadsheet for move planning (all mailboxes with size+count)
- `sbr-configs/` — copies of `Microsoft.Exchange.SBR.InternalDomains.config` (172 B), `Microsoft.Exchange.SBR.OverrideSettings.config` (491 B), `Microsoft.Exchange.SBR.IgnoreAuthAs.config` (empty)
### Migration runbook written
**`C:\NeptuneConfigExport-20260423\MIGRATION-RUNBOOK.md`** — full 6-phase runbook:
- **Phase 1:** VM prereqs (WS2022, 4 vCPU / 16 GB / 1.2 TB data disk), Windows features, prerequisite software list (.NET 4.8.1, VC++ 2012/2013, URL Rewrite, UCMA 4.0), AD admin group requirements, **backup ACG-DC16 system state before schema prep**
- **Phase 2:** Schema prep (`Setup.exe /PrepareSchema` → `/PrepareAD` → `/PrepareDomain`). Expected rangeUpper bump: 17002 → 17003 (CU14) or 17005 (CU15). Forest-permanent.
- **Phase 3:** Install Exchange 2019 + port the 14 items of config from the snapshot (incl. the critical **WS2022 DNS adapter workaround** `Set-TransportService -InternalDNSAdapterEnabled $false`, IRM lockdown, SBR config file copy). New DBs should be named `DB01-Hosting` / `DB02-LargeBoxes` — avoid the `N-Hosting1` / `N-LargeBoxes` names that triggered today's DNS mess.
- **Phase 4:** Pilot (5 small mailboxes) then bulk `New-MoveRequest` in batches. Plan ~10 GB/hr LAN transfer rate → ~27 hr total transfer.
- **Phase 5:** Cutover — repoint MailProtector relay target, public DNS A records, vdir external URLs, AutoDiscover SCP.
- **Phase 6:** Clean-uninstall Exchange from NEPTUNE (`Setup /Mode:Uninstall`), then `Remove-ADObject -Recursive` on the MAIL carcass (that object never had a clean uninstall). Clean up the workaround hosts-file + DC DNS records + disable GlobalNames Zone.
**Estimated total wall time: ~1 week** with appropriate change windows.
### Known coexistence risks for the transition
- Certs: new server needs cert matching `mail.acghosting.com` + autodiscover SAN. Can re-export from NEPTUNE (thumbprint `E58BFCBAEFEFDCAED0BF9E894127A3DE64CE9C69`, expires 2026-07-22) or issue new from Let's Encrypt.
- MAIL AD carcass `msExchCurrentServerRoles=16439`, build `15.1.2507.18` (newer than NEPTUNE!) — this is why FETS kept proxying to it during today's outage. Force-remove in Phase 6.
- Outlook clients cached credentials behave well; Mac Outlook sometimes needs restart after mailbox move.
### No new credentials discovered this phase
All work in this phase was local PowerShell on NEPTUNE; no new accounts, secrets, or endpoints discovered beyond what's in the phase-1 credentials section.
### Files written in this phase
| Path | Purpose |
|---|---|
| `C:\NeptuneConfigExport-20260423\*.{txt,xml,csv}` | 34 config exports (22 areas × 2 formats + CSV inventory) |
| `C:\NeptuneConfigExport-20260423\sbr-configs\*.config` | 3 SBR agent config files copied from live path |
| `C:\NeptuneConfigExport-20260423\MIGRATION-RUNBOOK.md` | Full 6-phase migration runbook |
### Next-session entry points
- Mike is building the WS2022 VM for the Exchange 2019 server (temp name `MAIL2` recommended; rename after carcass cleanup).
- Copy `C:\NeptuneConfigExport-20260423\` off NEPTUNE to a location that will outlive NEPTUNE (vault, or the new MAIL2 box directly).
- Before running `/PrepareSchema`: **take a system-state backup or VM snapshot of ACG-DC16**. Schema changes are forest-permanent and cannot be rolled back via uninstall.
- Mike will ping for real-time walk-through before the schema-prep step.
## Note for Howard
Today NEPTUNE (Exchange 2016 Eval) hit a severe outage triggered by the Windows Server 2016→2022 in-place upgrade Mike did on 2026-04-22. Exchange 2016 is unsupported on WS2022; multiple latent issues surfaced (DNS resolver suffix-search, RMS/Index Routing agent timeouts, ADTopology boot-timing, AssistantsQuarantine ACL). **Mail is flowing as of ~14:32** after disabling adapter-mode DNS on the transport server and using explicit DNS servers (`Set-TransportService NEPTUNE -InternalDNSAdapterEnabled $false -InternalDNSServers 172.16.3.50,172.16.3.52`). Rollback to WS2016 is **not available** — all paths (DISM uninstall, Windows.old staging, VSS, Windows Server Backup, VM snapshot) are dead. Mike is going to build Exchange 2019 on a fresh WS2022 VM into the same org, then migrate mailboxes and decommission NEPTUNE + force-remove the old MAIL AD carcass. If you end up running the migration, the full runbook is at `C:\NeptuneConfigExport-20260423\MIGRATION-RUNBOOK.md` on NEPTUNE — copy that whole folder off before NEPTUNE goes away. **Do NOT run `/PrepareSchema` without first backing up ACG-DC16** (single-DC forest, schema changes are forest-permanent).