## User - **User:** Mike Swanson (mike) - **Machine:** GURU-5070 - **Role:** admin ## Session Summary Continuation of the Peaceful Spirit AD+DFS rebuild (follows `2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md`). Completed two follow-ups and the core of Gate 4 (data DFS-R rebuild). All work via GuruRMM against PST-SERVER (192.168.0.2) and PST-SERVER2. First, the SERVER2 static-IP follow-up: confirmed `192.168.1.5` was SERVER2's intended address (the lingering A record was from its prior life, not stale junk). Verified `.5` free, Mike set the static IP manually (192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2+127.0.0.1) after my scripted attempt was rejected. Cleaned up 4 stale `.127` DNS A records (`@`, DomainDnsZones, ForestDnsZones, and the old `WIN-MUM1QS0V4LN` default-hostname record); dcdiag Connectivity + Advertising then passed. Also fixed SERVER2 timezone (was Pacific -> set to US Mountain Standard, i.e. Arizona). Then Gate 4. Chose `C:\Shares` on SERVER2 as the DFS-R target (only C: exists, 634 GB free, no unallocated space; the original D: was a separate physical disk now gone). Confirmed the live data on PST-SERVER `G:\Shares` is fully intact (~265 GB: Private 154, Scanned 105, ITServices 5, qbooks 2) — the recurring "G:\Shares 0 GB" reading is a SYSTEM-cant-enumerate-the-share-root ACL artifact, not data loss (direct subpaths show the real sizes). Removed the stale `\\PST-SERVER2\Shares` namespace folder target so NW clients keep using PST-SERVER's full data. Created empty `C:\Shares` + matching `Shares` SMB share (Everyone=Full, like PST-SERVER's). Tore down the broken PST-DFS group (orphaned member + State 5) and rebuilt it clean with **PST-SERVER as PRIMARY** (G:\Shares, authoritative) and SERVER2 non-primary (C:\Shares, empty) — the data-safety-critical step, triple-checked. Replication started and SERVER2 pulled **~221 / 265 GB (~84%)** into C:\Shares before the session paused. Session ended blocked: **PST-SERVER2 began flapping** (online ~1 min, offline several min, repeating) — a NW-site stability problem (reboot loop / power / network), NOT caused by the DFS changes. PST-SERVER (the source) stayed rock-solid the whole time; no data risk. The final Gate 4 steps (re-add SERVER2 namespace folder target, add 2nd namespace root target, drain remaining ~44 GB backlog) are deferred until SERVER2 is stably online. ## Key Decisions - **SERVER2 DFS-R target = `C:\Shares`** (folder on existing volume), not a new D: partition — 634 GB free, no remote-partitioning risk; original D: physical disk is gone. - **PST-SERVER = PRIMARY member** for the rebuilt PST-DFS. Primary must be the member that HAS the data; SERVER2 (empty) is non-primary and receives. Getting this backwards would have pushed SERVER2's empty folder as authoritative and moved PST-SERVER's 265 GB to ConflictAndDeleted. - **Removed (not just disabled) the stale SERVER2 namespace folder target.** `Set-DfsnFolderTarget -State Offline` failed ("object could not be found") on this 2016 namespace; `Remove-DfsnFolderTarget` worked. Will re-add it Online after initial sync completes. - **Clean teardown+rebuild of PST-DFS** rather than repair — removed the orphaned blank member (leftover from SERVER2's metadata cleanup). `Remove-DfsReplicationGroup` needs `-RemoveReplicatedFolders`; config removal never touches the `G:\Shares` content. - **Stopped pushing commands once SERVER2 began flapping** — surfaced it as a NW-health issue for Mike rather than fighting an unstable box. ## Problems Encountered - **PST-SERVER2 flapping (UNRESOLVED):** offline 04:37-04:46 UTC (~9 min), online ~1 min, offline again 04:47+. Looks like a reboot loop / power / NW network instability. Needs investigation when it next holds a stable window — pull System log (kernel-power 41, unexpected-shutdown 6008, 1074) to find the cause. Possibly on-site (power/UPS/hardware). - **DFSR initial state confusion:** after the clean rebuild, PST-SERVER Shares RF briefly showed State 5 and SERVER2 had no Shares RF — resolved by restarting the DFSR service on both + pollad; PST-SERVER then logged event 4112 (designated primary) and reached State 4. SERVER2 already had 221 GB from the first (messy) membership config, kept as pre-existing data (hash-matched). - **Apparent time skew was a red herring** — both DCs UTC-synced; SERVER2 was just on the wrong timezone (Pacific). Fixed to Mountain. - **DA creds over RMM:** AD/DFS/DFSN/DFSR config writes need Domain Admin (agent SYSTEM can't). Working pattern: `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...}` (FQDN; loopback works now with the CORRECT password). `-Server localhost -Credential` fails with a Kerberos SPN error. - **PowerShell-over-RMM quoting (recurring):** scripts wrapped in bash single-quotes can't contain single quotes; `\"` over-escaping breaks `-join`/`-eq`; use a `$var` for literal strings instead. Several commands no-op'd on parse errors until fixed. - **IP-change-over-RMM:** a direct New-NetIPAddress command was killed mid-NIC-blip (agent dropped), leaving config unapplied (reverted to DHCP). The robust pattern is a one-shot scheduled task that applies the change a few seconds AFTER the command returns — but Mike set the IP manually instead. ## Configuration Changes - PST-SERVER2: static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2+127.0.0.1 (manual by Mike); timezone -> US Mountain Standard Time. - DNS (PEACEFULSPIRIT.local): removed stale A records `@ -> .127`, `DomainDnsZones -> .127`, `ForestDnsZones -> .127`, `WIN-MUM1QS0V4LN -> .127`. Remaining A records all point to .5. - PST-SERVER2: created `C:\Shares` + SMB share `Shares` (Everyone=Full, Unrestricted enum, Manual caching). - DFS namespace `\\PEACEFULSPIRIT.local\PST-Files\Shares`: removed the `\\PST-SERVER2\Shares` folder target (only `\\PST-SERVER\Shares` remains, Online). **TO RE-ADD after sync.** - DFS-R group **PST-DFS** rebuilt: RF `Shares`; member PST-SERVER `G:\Shares` PrimaryMember=TRUE, staging 20 GB; member PST-SERVER2 `C:\Shares` non-primary, staging 20 GB; bidirectional connection. Initial replication ~221/265 GB done. - No repo files changed (this save creates the session log). ## Credentials & Secrets - No new credentials. Used `PEACEFULSPIRIT\sysadmin` / `r3tr0gradE99!` (vault `clients/peaceful-spirit/server`, field path `credentials.password` — read via full `vault.sh get`, since `get-field credentials.password` returns the literal "null" — known bug). DA password was passed base64-wrapped in RMM command_text again (recoverable from RMM DB; rotation optional, internal). - SERVER2 local-admin + DSRM passwords remain in vault `clients/peaceful-spirit/server2` (created prior session, already pushed). ## Infrastructure & Servers - **PST-SERVER** 192.168.0.2, site CC, all FSMO, GC, DNS, Server 2016 Essentials. RMM `87293069-33b6-45e8-a68f-6811216cdb96`. Data on `G:\Shares` ~265 GB. G: now 182 GB free. - **PST-SERVER2** **192.168.1.5**/24 (static), site NW, GC, DNS, Server 2019 Standard. RMM `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. Data replica at `C:\Shares` (~221 GB so far). **Flapping as of session end.** - DFS namespace `\\PEACEFULSPIRIT.local\PST-Files` -> folder `Shares`. Root target: PST-SERVER only (SERVER2 root target still TO ADD for HA). Folder targets: PST-SERVER only currently. - S2S VPN CC<->NW confirmed up earlier (389/445/88 reachable) — but flaps when SERVER2 drops. ## Commands & Outputs - DNS cleanup: `Get-DnsServerResourceRecord -ZoneName PEACEFULSPIRIT.local -RRType A | ? {IPv4Address -eq '192.168.1.127'} | Remove-DnsServerResourceRecord`. - Namespace target remove (DA Invoke-Command): `Remove-DfsnFolderTarget -Path "\\PEACEFULSPIRIT.local\PST-Files\Shares" -TargetPath "\\PST-SERVER2.PEACEFULSPIRIT.local\Shares" -Force`. - DFS-R rebuild (DA Invoke-Command on PST-SERVER FQDN): Remove-DfsReplicationGroup -RemoveReplicatedFolders -Force; New-DfsReplicationGroup/New-DfsReplicatedFolder/Add-DfsrMember/Add-DfsrConnection; `Set-DfsrMembership -ComputerName PST-SERVER -ContentPath G:\Shares -PrimaryMember $true -StagingPathQuotaInMB 20480 -Force`; `Set-DfsrMembership -ComputerName PST-SERVER2 -ContentPath C:\Shares -StagingPathQuotaInMB 20480 -Force`. - Kickoff: `Restart-Service DFSR -Force; dfsrdiag pollad` on both. PST-SERVER event **4112** (designated primary), Shares RF State 4. SERVER2 `C:\Shares` reached 221.78 GB. - Data integrity proof: direct `Get-ChildItem G:\Shares\Private|Scanned|ITServices` => 153.77 / 104.83 / 4.73 GB. ## Pending / Incomplete Tasks 1. **PST-SERVER2 stability (BLOCKER):** diagnose why it's flapping (System log 41/6008/1074); likely NW power/UPS/hardware/network. Possibly on-site. 2. **Finish Gate 4 once SERVER2 is stable:** - Confirm DFS-R backlog drains to 0 (`Get-DfsrBacklog` via DA; ~44 GB remained). - Re-add SERVER2 folder target: `New-DfsnFolderTarget -Path "\\PEACEFULSPIRIT.local\PST-Files\Shares" -TargetPath "\\PST-SERVER2.PEACEFULSPIRIT.local\Shares"` (Online). - Add SERVER2 as 2nd namespace ROOT target: `New-DfsnRootTarget -Path "\\PEACEFULSPIRIT.local\PST-Files" -TargetPath "\\PST-SERVER2.PEACEFULSPIRIT.local\PST-Files"` (needs a PST-Files root share/DFSRoots on SERVER2) — for VPN-outage namespace HA. - Verify both RFs State 4, dcdiag clean, referral ordering site-aware. 3. **Lower priority:** D: backup-junk cleanup on PST-SERVER (~700 GB); remove `C:\PST-Backup\*` once rebuild confirmed stable; wiki update (`wiki/clients/peaceful-spirit.md` — 2 DCs, SERVER2 192.168.1.5, DFS-R rebuilt); optional sysadmin rotation. ## Reference Information - Prior log: `clients/peaceful-spirit/session-logs/2026-06/2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md`. - Runbook: `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`. - RMM API `http://172.16.3.30:3001`. Agents: PST-SERVER `87293069-...`, PST-SERVER2 `5d2d7ba0-...`. - DFS-R group `PST-DFS`, RF `Shares`. Namespace `\\PEACEFULSPIRIT.local\PST-Files`. - Vault: `clients/peaceful-spirit/server` (DA), `clients/peaceful-spirit/server2` (SERVER2 local/DSRM).