claudetools/clients/peaceful-spirit/session-logs/2026-06/2026-06-14-mike-pst-gate4-dfsr-rebuild-and-static-ip.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Continuation of the Peaceful Spirit AD+DFS rebuild (follows 2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md). Completed two follow-ups and the core of Gate 4 (data DFS-R rebuild). All work via GuruRMM against PST-SERVER (192.168.0.2) and PST-SERVER2.

First, the SERVER2 static-IP follow-up: confirmed 192.168.1.5 was SERVER2's intended address (the lingering A record was from its prior life, not stale junk). Verified .5 free, Mike set the static IP manually (192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2+127.0.0.1) after my scripted attempt was rejected. Cleaned up 4 stale .127 DNS A records (@, DomainDnsZones, ForestDnsZones, and the old WIN-MUM1QS0V4LN default-hostname record); dcdiag Connectivity + Advertising then passed. Also fixed SERVER2 timezone (was Pacific -> set to US Mountain Standard, i.e. Arizona).

Then Gate 4. Chose C:\Shares on SERVER2 as the DFS-R target (only C: exists, 634 GB free, no unallocated space; the original D: was a separate physical disk now gone). Confirmed the live data on PST-SERVER G:\Shares is fully intact (~265 GB: Private 154, Scanned 105, ITServices 5, qbooks 2) — the recurring "G:\Shares 0 GB" reading is a SYSTEM-cant-enumerate-the-share-root ACL artifact, not data loss (direct subpaths show the real sizes). Removed the stale \\PST-SERVER2\Shares namespace folder target so NW clients keep using PST-SERVER's full data. Created empty C:\Shares

• matching Shares SMB share (Everyone=Full, like PST-SERVER's). Tore down the broken PST-DFS group (orphaned member + State 5) and rebuilt it clean with PST-SERVER as PRIMARY (G:\Shares, authoritative) and SERVER2 non-primary (C:\Shares, empty) — the data-safety-critical step, triple-checked. Replication started and SERVER2 pulled ~221 / 265 GB (~84%) into C:\Shares before the session paused.

Session ended blocked: PST-SERVER2 began flapping (online ~1 min, offline several min, repeating) — a NW-site stability problem (reboot loop / power / network), NOT caused by the DFS changes. PST-SERVER (the source) stayed rock-solid the whole time; no data risk. The final Gate 4 steps (re-add SERVER2 namespace folder target, add 2nd namespace root target, drain remaining ~44 GB backlog) are deferred until SERVER2 is stably online.

Key Decisions

• SERVER2 DFS-R target = C:\Shares (folder on existing volume), not a new D: partition — 634 GB free, no remote-partitioning risk; original D: physical disk is gone.
• PST-SERVER = PRIMARY member for the rebuilt PST-DFS. Primary must be the member that HAS the data; SERVER2 (empty) is non-primary and receives. Getting this backwards would have pushed SERVER2's empty folder as authoritative and moved PST-SERVER's 265 GB to ConflictAndDeleted.
• Removed (not just disabled) the stale SERVER2 namespace folder target. Set-DfsnFolderTarget -State Offline failed ("object could not be found") on this 2016 namespace; Remove-DfsnFolderTarget worked. Will re-add it Online after initial sync completes.
• Clean teardown+rebuild of PST-DFS rather than repair — removed the orphaned blank member (leftover from SERVER2's metadata cleanup). Remove-DfsReplicationGroup needs -RemoveReplicatedFolders; config removal never touches the G:\Shares content.
• Stopped pushing commands once SERVER2 began flapping — surfaced it as a NW-health issue for Mike rather than fighting an unstable box.

Problems Encountered

• PST-SERVER2 flapping (UNRESOLVED): offline 04:37-04:46 UTC (~9 min), online ~1 min, offline again 04:47+. Looks like a reboot loop / power / NW network instability. Needs investigation when it next holds a stable window — pull System log (kernel-power 41, unexpected-shutdown 6008, 1074) to find the cause. Possibly on-site (power/UPS/hardware).
• DFSR initial state confusion: after the clean rebuild, PST-SERVER Shares RF briefly showed State 5 and SERVER2 had no Shares RF — resolved by restarting the DFSR service on both + pollad; PST-SERVER then logged event 4112 (designated primary) and reached State 4. SERVER2 already had 221 GB from the first (messy) membership config, kept as pre-existing data (hash-matched).
• Apparent time skew was a red herring — both DCs UTC-synced; SERVER2 was just on the wrong timezone (Pacific). Fixed to Mountain.
• DA creds over RMM: AD/DFS/DFSN/DFSR config writes need Domain Admin (agent SYSTEM can't). Working pattern: Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...} (FQDN; loopback works now with the CORRECT password). -Server localhost -Credential fails with a Kerberos SPN error.
• PowerShell-over-RMM quoting (recurring): scripts wrapped in bash single-quotes can't contain single quotes; \" over-escaping breaks -join/-eq; use a $var for literal strings instead. Several commands no-op'd on parse errors until fixed.
• IP-change-over-RMM: a direct New-NetIPAddress command was killed mid-NIC-blip (agent dropped), leaving config unapplied (reverted to DHCP). The robust pattern is a one-shot scheduled task that applies the change a few seconds AFTER the command returns — but Mike set the IP manually instead.

Configuration Changes

• PST-SERVER2: static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2+127.0.0.1 (manual by Mike); timezone -> US Mountain Standard Time.
• DNS (PEACEFULSPIRIT.local): removed stale A records @ -> .127, DomainDnsZones -> .127, ForestDnsZones -> .127, WIN-MUM1QS0V4LN -> .127. Remaining A records all point to .5.
• PST-SERVER2: created C:\Shares + SMB share Shares (Everyone=Full, Unrestricted enum, Manual caching).
• DFS namespace \\PEACEFULSPIRIT.local\PST-Files\Shares: removed the \\PST-SERVER2\Shares folder target (only \\PST-SERVER\Shares remains, Online). TO RE-ADD after sync.
• DFS-R group PST-DFS rebuilt: RF Shares; member PST-SERVER G:\Shares PrimaryMember=TRUE, staging 20 GB; member PST-SERVER2 C:\Shares non-primary, staging 20 GB; bidirectional connection. Initial replication ~221/265 GB done.
• No repo files changed (this save creates the session log).

Credentials & Secrets

• No new credentials. Used PEACEFULSPIRIT\sysadmin / r3tr0gradE99! (vault clients/peaceful-spirit/server, field path credentials.password — read via full vault.sh get, since get-field credentials.password returns the literal "null" — known bug). DA password was passed base64-wrapped in RMM command_text again (recoverable from RMM DB; rotation optional, internal).
• SERVER2 local-admin + DSRM passwords remain in vault clients/peaceful-spirit/server2 (created prior session, already pushed).

Infrastructure & Servers

• PST-SERVER 192.168.0.2, site CC, all FSMO, GC, DNS, Server 2016 Essentials. RMM 87293069-33b6-45e8-a68f-6811216cdb96. Data on G:\Shares ~265 GB. G: now 182 GB free.
• PST-SERVER2 192.168.1.5/24 (static), site NW, GC, DNS, Server 2019 Standard. RMM 5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65. Data replica at C:\Shares (~221 GB so far). Flapping as of session end.
• DFS namespace \\PEACEFULSPIRIT.local\PST-Files -> folder Shares. Root target: PST-SERVER only (SERVER2 root target still TO ADD for HA). Folder targets: PST-SERVER only currently.
• S2S VPN CC<->NW confirmed up earlier (389/445/88 reachable) — but flaps when SERVER2 drops.

Commands & Outputs

• DNS cleanup: Get-DnsServerResourceRecord -ZoneName PEACEFULSPIRIT.local -RRType A | ? {IPv4Address -eq '192.168.1.127'} | Remove-DnsServerResourceRecord.
• Namespace target remove (DA Invoke-Command): Remove-DfsnFolderTarget -Path "\\PEACEFULSPIRIT.local\PST-Files\Shares" -TargetPath "\\PST-SERVER2.PEACEFULSPIRIT.local\Shares" -Force.
• DFS-R rebuild (DA Invoke-Command on PST-SERVER FQDN): Remove-DfsReplicationGroup -RemoveReplicatedFolders -Force; New-DfsReplicationGroup/New-DfsReplicatedFolder/Add-DfsrMember/Add-DfsrConnection; Set-DfsrMembership -ComputerName PST-SERVER -ContentPath G:\Shares -PrimaryMember $true -StagingPathQuotaInMB 20480 -Force; Set-DfsrMembership -ComputerName PST-SERVER2 -ContentPath C:\Shares -StagingPathQuotaInMB 20480 -Force.
• Kickoff: Restart-Service DFSR -Force; dfsrdiag pollad on both. PST-SERVER event 4112 (designated primary), Shares RF State 4. SERVER2 C:\Shares reached 221.78 GB.
• Data integrity proof: direct Get-ChildItem G:\Shares\Private|Scanned|ITServices => 153.77 / 104.83 / 4.73 GB.

Pending / Incomplete Tasks

1. PST-SERVER2 stability (BLOCKER): diagnose why it's flapping (System log 41/6008/1074); likely NW power/UPS/hardware/network. Possibly on-site.
2. Finish Gate 4 once SERVER2 is stable:
   • Confirm DFS-R backlog drains to 0 (Get-DfsrBacklog via DA; ~44 GB remained).
   • Re-add SERVER2 folder target: New-DfsnFolderTarget -Path "\\PEACEFULSPIRIT.local\PST-Files\Shares" -TargetPath "\\PST-SERVER2.PEACEFULSPIRIT.local\Shares" (Online).
   • Add SERVER2 as 2nd namespace ROOT target: New-DfsnRootTarget -Path "\\PEACEFULSPIRIT.local\PST-Files" -TargetPath "\\PST-SERVER2.PEACEFULSPIRIT.local\PST-Files" (needs a PST-Files root share/DFSRoots on SERVER2) — for VPN-outage namespace HA.
   • Verify both RFs State 4, dcdiag clean, referral ordering site-aware.
3. Lower priority: D: backup-junk cleanup on PST-SERVER (~700 GB); remove C:\PST-Backup\* once rebuild confirmed stable; wiki update (wiki/clients/peaceful-spirit.md — 2 DCs, SERVER2 192.168.1.5, DFS-R rebuilt); optional sysadmin rotation.

Reference Information

• Prior log: clients/peaceful-spirit/session-logs/2026-06/2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md.
• Runbook: clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md.
• RMM API http://172.16.3.30:3001. Agents: PST-SERVER 87293069-..., PST-SERVER2 5d2d7ba0-....
• DFS-R group PST-DFS, RF Shares. Namespace \\PEACEFULSPIRIT.local\PST-Files.
• Vault: clients/peaceful-spirit/server (DA), clients/peaceful-spirit/server2 (SERVER2 local/DSRM).