## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Resumed the Peaceful Spirit multi-site AD+DFS work from the 2026-06-11 plan. Read-only recon
(via GuruRMM) revealed the environment was far past the plan baseline AND broken: PST-SERVER2
had already been promoted as a second DC with DFS roles ~200+ days ago, then the site
disconnected for ~7 months. PST-SERVER2 was a **past-tombstone-lifetime DC** — AD replication
dead both directions (err 8614 "exceeded tombstone lifetime", err 0x8009030C broken secure
channel), SYSVOL DFSR stale 224 days (Event 4012), data DFS-R (`PST-DFS`/`Shares`) stale 200
days, and its DFS-R data target drive (a dynamic disk holding `D:\Shares`) was physically
**missing** (`diskpart` showed "Disk M0 Missing"). A past-TSL DC must never resume replication
(lingering-object reanimation risk), so the plan shifted from "fix SYSVOL" to **evict + rebuild
SERVER2**.

Executed a gated runbook (`clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`): Gate 0 pre-flight
+ SYSVOL/GPO backup; Gate 1 force-demote PST-SERVER2; Gate 2 metadata cleanup on PST-SERVER; an
authoritative **D4 SYSVOL restore** on PST-SERVER; Gate 3 re-join + re-promote PST-SERVER2 as a
fresh DC, then a **D2 non-authoritative SYSVOL sync** to fix its stale leftover DFSR database.
End state: **two healthy DCs**, AD replication 0 errors both directions, SYSVOL DFSR State 4
Normal on both (11 GPOs each), both GC, both advertising as logon servers. The 224-day-broken AD
is fully repaired and NW now has a local DC surviving a VPN outage.

Then did Gate 5 — **G: cleanup on PST-SERVER**. Investigation showed G: is the OLD server's
former C: drive (live OS runs from C:; everything OS-related on G: dated 2009-2018, profiles named
`*.NEWSERVER`). Deleted the entire dead old-OS install (~99 GB: Windows, Users, Program Files*,
ProgramData, $Recycle.Bin, 13 misc folders) using takeown + robocopy-mirror for
TrustedInstaller-protected files, and bounded G: VSS shadow storage from UNBOUNDED to 25 GB
(trimmed 41 GB -> 9.6 GB). G: free went **51 GB -> 182 GB** (~131 GB reclaimed). Live data
(`G:\Shares`, `ServerFolders`, `System Volume Information`) untouched.

Gate 4 (data DFS-R rebuild + SERVER2 data-volume decision) and several follow-ups remain.

## Key Decisions

- **Evict + rebuild SERVER2 (not repair).** Past tombstone lifetime -> resuming replication risks
  lingering-object reanimation. Force-demote -> metadata cleanup -> clean re-promote sidesteps the
  whole stale-AD/SYSVOL mess. Safe because PST-SERVER holds ALL 5 FSMO + is GC; SERVER2 was only
  an additional DC.
- **D4 authoritative on PST-SERVER, D2 non-authoritative on SERVER2.** PST-SERVER's SYSVOL was
  State 5 (InError) from the old 4012; D4 made it the authoritative source. After re-promotion,
  SERVER2's SYSVOL hit 4012 again from a **leftover DFSR database** from its prior DC life -> D2
  forced it to discard the stale DB and re-sync fresh.
- **Forced-removal method.** `Uninstall-ADDSDomainController -ForceRemoval` is blocked by a
  non-skippable test (General.21) while the DC is a DNS server + GC. Fix: clear the GC flag
  (nTDSDSA options=0) and uninstall the DNS role (reboot), THEN force-remove. `-SkipPreChecks` /
  `-Force` / `-Confirm:$false` do NOT bypass General.21.
- **AD writes need DA creds via FQDN.** Agent SYSTEM context can't do metadata cleanup. Use
  `-Credential sysadmin -Server PST-SERVER.PEACEFULSPIRIT.local` (FQDN, valid SPN). `-Server
  localhost` fails with a Kerberos SPN error; WinRM loopback with creds also failed.
- **G: cleanup confirmed as old-OS drive.** Live OS on C:; G: = old server's C: (2009-2018 dates,
  `*.NEWSERVER` profiles). Deleted whole old OS; kept Shares/ServerFolders/SVI. Bounded VSS rather
  than clearing (keep recent Previous-Versions history).

## Problems Encountered

- **`get-field` dot-notation bug:** `vault.sh get-field clients/peaceful-spirit/server credentials.password`
  returned the literal string **"null"** -> every DA attempt sent password "null" -> auth rejected
  (~6 bad attempts; harmless, domain LockoutThreshold=0). Real password `r3tr0gradE99!` was read
  from the full `vault.sh get`. **Use full `get` (or verify the field path) for nested creds.**
- **PowerShell-over-RMM quoting traps (recurring):** scripts are wrapped in bash single quotes, so
  (a) can't use single quotes inside (property names like `$obj."prop"` must use double quotes),
  (b) `\"` over-escaping breaks `-join`, (c) `$g:` parsed as a drive ref -> use `${g}`. Several
  commands failed to parse (no-ops) until fixed.
- **`Uninstall-ADDSDomainController` parameter-set / General.21** — see Key Decisions.
- **AV grabbed mingw curl** ("curl: Permission denied") intermittently; Mike disabled AV.
- **SERVER2 slow reboots** — DNS-role removal + DC promotion finalize add several minutes to boot;
  RMM `last_seen` gave false "online" before the real reboot took hold (verify actual state, not
  just the timestamp).
- **dcdiag SysVolCheck/NetLogons failed** right after SYSVOL init -> fixed by `Restart-Service
  Netlogon` (re-advertise).
- **Stubborn OS files** survived `rd /s /q` even after takeown -> robocopy-mirror-from-empty +
  `rd` cleared them.
- **Free-space readout bug:** WMI filter `DeviceID=''G:''` (doubled quotes) returned 0; used
  `(Get-PSDrive G).Free` / `Get-Volume` instead.

## Configuration Changes

AD / DC (domain PEACEFULSPIRIT.local, via GuruRMM):
- PST-SERVER: SYSVOL backup -> `C:\PST-Backup\SYSVOL-Policies-20260613-1611`; GPO backup ->
  `C:\PST-Backup\GPO-20260613-1611` (11 GPOs). Strict Replication Consistency already =1.
- PST-SERVER: **D4 authoritative SYSVOL restore** (msDFSR-Enabled FALSE+options=1, then TRUE) ->
  Event 4602, SYSVOL State 4.
- PST-SERVER: metadata cleanup of PST-SERVER2 (server object + NTDS Settings + computer account
  removed; DNS A/NS/CNAME/7xSRV removed in both zones).
- PST-SERVER2: cleared GC flag (nTDSDSA options 1->0); uninstalled DNS Server role; force-removed
  AD DS (became WORKGROUP member); re-joined PEACEFULSPIRIT.local; **re-promoted** as DC (site NW,
  InstallDns, GC); **D2 non-authoritative SYSVOL sync** (Event 4604) -> State 4, 11 GPOs, SYSVOL+
  NETLOGON shared; Netlogon restarted (dcdiag green); DNS client set to 192.168.0.2.
- PST-SERVER: **G: cleanup** — deleted old-OS dirs (Windows, Users, Program Files, Program Files
  (x86), ProgramData, $Recycle.Bin, MSOCache, v12, v13, RAPID, 8x8, Brother, Intel, inetpub,
  Support, BackupTemp, Documents and Settings, $WINDOWS.~BT). VSS shadow storage for G: resized
  to maxsize=25GB. G: free 51 -> 182 GB.

Repo / vault:
- Created `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`.
- Vault: created `clients/peaceful-spirit/server2.sops.yaml` with `local_admin_password` and
  `dsrm_password` (see below). **Pending publish** via sync.

## Credentials & Secrets

- **PST domain admin:** `PEACEFULSPIRIT\sysadmin` / `r3tr0gradE99!` (vault
  `clients/peaceful-spirit/server`, field `credentials.password`). NOTE the get-field dot bug above.
- **PST-SERVER2 new local admin + DSRM:** generated this session, stored in vault
  `clients/peaceful-spirit/server2` (`local_admin_password`, `dsrm_password`). Values are in the
  vault only (not pasted here per policy); entry is **not yet pushed** to Gitea.
- **[WARNING] DA password exposure:** the sysadmin password was passed (base64-wrapped) in RMM
  command_text for the cleanup/D4/D2/promote/join commands, so it is recoverable from the GuruRMM
  command history/DB. RMM is internal; consider rotating `sysadmin` if RMM DB exposure is a concern.

## Infrastructure & Servers

- Domain **PEACEFULSPIRIT.local** (NetBIOS PEACEFULSPIRIT? -> uses PEACEFULSPIRIT\). Win2016
  domain/forest functional level. LockoutThreshold=0 (lockout disabled).
- **PST-SERVER** — 192.168.0.2, site **CC**, holds all 5 FSMO, GC, DNS, Server 2016 Essentials.
  GuruRMM `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.66). G: 465.7 GB (182 GB free post-cleanup);
  C: 931 GB; D: 931 GB (Recovery-EXT/Recovery2019/"Unknown folder" ~700 GB of old backup junk).
- **PST-SERVER2** — 192.168.1.127 (DHCP), site **NW**, GC, DNS, Server 2019 Standard. GuruRMM
  `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65` (v0.6.66). Single 1 TB NVMe (Crucial P2), C: only (no
  D:); old dynamic data disk is MISSING.
- AD Sites & Services (pre-existing, correct): sites CC + NW; subnets 192.168.0.0/24 -> CC,
  192.168.1.0/24 -> NW. S2S VPN between sites is UP (389/445/135/88 reachable SERVER2->SERVER).
- DFS namespace `\\PEACEFULSPIRIT.local\PST-Files` -> folder `Shares` (root target only on
  PST-SERVER). DFS-R group `PST-DFS` replicated `Shares` (G:\Shares <-> D:\Shares) — currently
  broken/stale, to be rebuilt at Gate 4.
- Business data on PST-SERVER `G:\Shares`: Private ~154 GB, Scanned ~105 GB, ITServices ~5 GB
  (~265 GB). A service `SshWiaRestart` runs from `G:\Shares\ITServices\...` (Fujitsu scanner).

## Commands & Outputs

- Tombstone evidence: `repadmin /replsummary` -> 8614 / 0x8009030C, 100% fails (pre-rebuild).
- Force-demote (after clearing GC + DNS role): `Uninstall-ADDSDomainController -ForceRemoval
  -LocalAdministratorPassword <pw> -NoRebootOnCompletion -Force` -> Status=Success.
- Metadata cleanup: `Remove-ADObject -Recursive -Server PST-SERVER.PEACEFULSPIRIT.local -Credential
  $cred` on `CN=PST-SERVER2,CN=Servers,CN=NW,CN=Sites,...` + computer object; DNS via
  `Remove-DnsServerResourceRecord`.
- D4/D2: `Set-ADObject <SYSVOL Subscription DN> -Replace @{ "msDFSR-Enabled"=$false/$true;
  "msDFSR-options"=1 (D4 only) }` + `net stop/start dfsr` + `dfsrdiag pollad`. Events 4114 -> 4602
  (D4) / 4614 -> 4604 (D2).
- Re-promote: `Install-ADDSDomainController -DomainName PEACEFULSPIRIT.local -Credential $cred
  -SafeModeAdministratorPassword $dsrm -SiteName NW -InstallDns -Force -NoRebootOnCompletion`.
- Final health: `repadmin /replsummary` 0/5 fails both DCs; dcdiag PST-SERVER2 passed
  Advertising/SysVolCheck/NetLogons after `Restart-Service Netlogon`.
- G: cleanup: takeown /f <dir> /r /d y; icacls grant; robocopy <empty> <dir> /MIR; rd /s /q.
  `vssadmin resize shadowstorage /for=G: /on=G: /maxsize=25GB`.

## Pending / Incomplete Tasks

- **Gate 4 — data DFS-R rebuild:** decide PST-SERVER2 data volume (shrink C: / add disk /
  folder-on-C:), recreate `Shares` folder target on SERVER2, re-establish PST-DFS replication,
  and ADD PST-SERVER2 as a 2nd **namespace root target** (namespace HA for VPN-outage). Clean the
  orphaned PST-DFS member left from metadata cleanup.
- **PST-SERVER2 static IP** — currently DHCP 192.168.1.127; DCs should be static (dcpromo warned).
- **Stale DNS tidy:** extra `192.168.1.5` A record for PEACEFULSPIRIT.local; `.240` RRAS-pool
  registration on PST-SERVER. Identify what 192.168.1.5 is (a resolver in SERVER2's DNS list).
- **Vault publish:** push `clients/peaceful-spirit/server2.sops.yaml` (handled by this save's sync).
- **Optional:** rotate `sysadmin` (RMM command_text exposure); clean D: backup junk on PST-SERVER
  (~700 GB) if not needed; remove `C:\PST-Backup\*` once rebuild confirmed stable.
- **Wiki:** update `wiki/clients/peaceful-spirit.md` (now 2 DCs; SERVER2 rebuilt; G: cleaned).

## Reference Information

- Runbook: `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`.
- Backups on PST-SERVER: `C:\PST-Backup\SYSVOL-Policies-20260613-1611`, `C:\PST-Backup\GPO-20260613-1611`.
- Vault: `clients/peaceful-spirit/server` (DA), `clients/peaceful-spirit/server2` (new local admin+DSRM).
- GuruRMM agents: PST-SERVER `87293069-33b6-45e8-a68f-6811216cdb96`; PST-SERVER2
  `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. RMM API `http://172.16.3.30:3001`.
- KB 2218556 (authoritative/non-authoritative DFSR SYSVOL D4/D2).
