# Peaceful Spirit — PST-SERVER2 evict + re-promote runbook

**Created:** 2026-06-13 by Mike Swanson (GURU-5070)
**Why:** PST-SERVER2 is a past-tombstone-lifetime DC. AD replication dead both directions
(err 8614 "exceeded tombstone lifetime"; err 0x8009030C broken secure channel). SYSVOL +
data DFS-R in State 5 (InError), stale 200-224 days. A past-TSL DC must NOT be allowed to
resume replication (lingering-object reanimation risk). So: evict SERVER2, metadata-clean,
re-promote fresh.

**Authoritative/healthy DC:** PST-SERVER (192.168.0.2) — holds ALL 5 FSMO. Server 2016
Essentials. Domain PEACEFULSPIRIT.local (Win2016 functional level).
**DC to rebuild:** PST-SERVER2 (192.168.1.127, NW site) — Server 2019 Standard, additional DC only.

**Execution channel:** GuruRMM (SYSTEM context). PST-SERVER `87293069-33b6-45e8-a68f-6811216cdb96`,
PST-SERVER2 `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. Domain admin = `sysadmin` (vault:
clients/peaceful-spirit/server.sops.yaml). NOTE: promotion needs Domain Admin creds passed in
the RMM command — that password lands in RMM command_text/history (internal). Consider rotation
after if RMM DB exposure is a concern.

---

## Gates (confirm with Mike before each)

### Gate 0 — Pre-flight + safety backup (SAFE: read-only + backup)
- Confirm PST-SERVER is a Global Catalog. (If SERVER2 were the only GC, must GC-flag SERVER first.)
- Confirm all 5 FSMO on PST-SERVER (done: yes).
- dcdiag focused (Advertising/FSMOCheck/Services) on PST-SERVER — must be clean.
- Enable Strict Replication Consistency on PST-SERVER (protective; reg key) — *change, but safe/recommended*.
- BACK UP authoritative SYSVOL: robocopy `C:\Windows\SYSVOL\domain\Policies` -> `C:\PST-Backup\SYSVOL-Policies`
  and `Backup-GPO -All`. Insurance before any AD change.

### Gate 1 — Force-demote PST-SERVER2 (DESTRUCTIVE to SERVER2; reboots SERVER2)
- On SERVER2: `Uninstall-ADDSDomainController -ForceRemoval -DemoteOperationMasterRole -Force
  -LocalAdministratorPassword <new, vaulted>` (graceful demote impossible — replication dead).
- SERVER2 becomes a member/standalone server and reboots. Blast radius = SERVER2 only.
- Risk: AD changes made ONLY on SERVER2 during isolation are lost (already stranded; PDC authoritative).

### Gate 2 — Metadata cleanup on PST-SERVER (DESTRUCTIVE to AD metadata for SERVER2)
- Remove SERVER2 NTDS Settings / server object (ntdsutil metadata cleanup, or Remove-ADObject of the
  NTDS Settings object with -Credential domain admin).
- Remove SERVER2 from AD Sites & Services (NW site server object).
- DNS cleanup: SERVER2 host A, _msdcs CNAME/GUID, NS records, SRV records.
- DFSR cleanup: remove SERVER2 member from "Domain System Volume" (SYSVOL) and "PST-DFS" groups.
- Verify: `repadmin /viewlist *` shows only PST-SERVER; dcdiag clean.

### Gate 3 — Re-promote PST-SERVER2 (re-introduces a DC)
- Ensure SERVER2 DNS points to PST-SERVER (192.168.0.2) primary. (Currently 192.168.0.2,192.168.1.5,8.8.8.8,1.1.1.1.)
- `Install-ADDSDomainController -DomainName PEACEFULSPIRIT.local -Credential <DA> -InstallDns
  -SiteName NW -SafeModeAdministratorPassword <new, vaulted>` — fresh promotion.
- SYSVOL initializes clean via DFSR initial sync from PST-SERVER (no D2/D4 needed).
- Verify: repadmin /replsummary 0% fails; SYSVOL+NETLOGON shared on SERVER2; dcdiag clean;
  GPO count matches SERVER (11).

### Gate 4 — Rebuild data DFS-R (deferred — separate decision)
- Provision SERVER2 data volume (shrink C: / add disk / folder-on-C: — TBD after G: cleanup + sizing).
- Recreate `Shares` folder target on SERVER2 + re-establish PST-DFS replication.
- Add PST-SERVER2 as 2nd namespace ROOT target (namespace HA for VPN-outage resilience).
- Confirm backlog drains to 0.

### Gate 5 — G: cleanup on PST-SERVER (separate)
- ~160 GB candidates: G:\Windows (32), G:\Program Files (x86) (13), G:\ProgramData (10),
  G:\Users (51), G:\$Recycle.Bin (5.6), VSS in System Volume Information (~46). Confirm junk first.
- D: recovery junk (~700 GB): Recovery-EXT, Recovery2019, "Unknown folder" — confirm before delete.

---

## Rollback notes
- Gate 0 changes (strict consistency reg) are trivially reversible.
- After Gate 1 demotion, SERVER2 is a plain member server — re-promotion (Gate 3) restores it.
  No rollback needed for the eviction itself; the domain runs fine on PST-SERVER alone meanwhile.
- The SYSVOL/GPO backup from Gate 0 is the restore point if PST-SERVER's SYSVOL were ever harmed
  (it should not be touched by this procedure).