azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from GURU-5070 at 2026-07-02 18:25:06

Browse Source 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-02 18:25:06
This commit is contained in:
Mike Swanson
2026-07-02 18:25:55 -07:00
parent 27b9966dfa
commit 101b95e610
5 changed files with 2624 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
wiki/clients/peaceful-spirit.md
View File

@@ -51,8 +51,8 @@ Massage therapy practice with two sites: Country Club (CC, primary — all serve
| Host | IP | Role | OS | Notes |
|---|---|---|---|---|
| PST-SERVER | 192.168.0.2 | DC (all 5 FSMO), DNS, RRAS (L2TP/IPsec VPN), NPS, Enterprise Root CA (AD CS) | Windows Server 2016 Essentials (build 14393) | Site CC. GuruRMM agent `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.75+; prior ID `6b6106a7...` retired). Win32-OpenSSH installed 2026-05-11. Machine cert: `DB71981ABE4CBA1DE96FEEEAF178F6259663B543` (CN=PST-SERVER.PEACEFULSPIRIT.local, valid 5/9/2027). Drives: C: 931 GB (OS); G: 465.7 GB data volume (ex-old-server C:, 182 GB free post-cleanup); D: 931 GB (Recovery-EXT/backup junk ~700 GB — cleanup pending). G:\Shares: Private ~154 GB, Scanned ~105 GB, ITServices ~5 GB, qbooks ~2 GB (~265 GB total). Credentials: vault `clients/peaceful-spirit/server`. |
| PST-DC-NW (shipped as PST-SERVER01) | (verify) | New DC for NW site + DFSR (replaces PST-SERVER2) | (verify) | Site NW. New physical server installed ~2026-07-02 (per Mike). To be RENAMED PST-SERVER01 -> PST-DC-NW (decided 2026-07-02) — rename BEFORE domain join/DC promotion. Mike is adding it to GuruRMM (client Peaceful Spirit; NW site may need creating). Promote as DC/GC/DNS and rebuild the DFS-R receiver role per the SERVER2 runbook once enrolled. |
| PST-SERVER2 | 192.168.1.5 | DC (additional), GC, DNS — BEING REPLACED by PST-SERVER01 | Windows Server 2019 Standard | Site NW. Static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2 + 127.0.0.1. GuruRMM agent `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. Single 1 TB NVMe, C: only (original D: physical disk gone). DFS-R replica at C:\Shares (~221 GB as of 2026-06-14; ~44 GB backlog remaining). Timezone: US Mountain Standard Time (Arizona). Rebuilt 2026-06-13 (force-demote -> metadata cleanup -> re-promote; see runbook). Credentials: vault `clients/peaceful-spirit/server2` (local admin + DSRM). [WARNING] Flapping (online ~1 min / offline several min reboot-loop pattern) at end of 2026-06-14 session — NW site power/UPS/network issue, NOT caused by DFS; PST-SERVER and data unaffected. |
| PST-DC-NW | 192.168.1.5 | DC (additional), GC, DNS, DFS-R receiver | Windows Server 2019 Standard (build 17763) | Site NW. New physical server installed 2026-07-02 (shipped as PST-SERVER01; renamed via RMM pre-join). Joined + promoted DC/GC/DNS (site NW) 2026-07-02, all via RMM. Static 192.168.1.5/24; DNS client 192.168.0.2 + 127.0.0.1; timezone US Mountain Standard (AZ). Single 1.8 TB C: (1849 GB free at setup). DFS-R replica at C:\Shares (staging 20 GB), member of PST-DFS since 2026-07-02 — initial sync of ~265 GB from PST-SERVER over S2S VPN in progress. GuruRMM agent `f60e9820-4a00-4598-83f7-c14085db5768` (v0.6.75, site "North West"). DSRM password: vault `clients/peaceful-spirit/dc-nw`. |
| PST-SERVER2 (DEAD, metadata cleaned 2026-07-02) | was 192.168.1.5 (now PST-DC-NW's) | Was DC/GC/DNS at NW — hardware DIED ~2026-06-14 (the "flapping"), never returned; replaced by PST-DC-NW. All AD/DNS/DFSR metadata removed 2026-07-02. | Windows Server 2019 Standard | Site NW. Static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2 + 127.0.0.1. GuruRMM agent `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. Single 1 TB NVMe, C: only (original D: physical disk gone). DFS-R replica at C:\Shares (~221 GB as of 2026-06-14; ~44 GB backlog remaining). Timezone: US Mountain Standard Time (Arizona). Rebuilt 2026-06-13 (force-demote -> metadata cleanup -> re-promote; see runbook). Credentials: vault `clients/peaceful-spirit/server2` (local admin + DSRM). [WARNING] Flapping (online ~1 min / offline several min reboot-loop pattern) at end of 2026-06-14 session — NW site power/UPS/network issue, NOT caused by DFS; PST-SERVER and data unaffected. |
| UCG-PST-CC | 192.168.0.10 (LAN) / 98.190.129.150 (WAN) | UniFi Cloud Gateway Ultra — perimeter router + DNAT for VPN | UniFi OS 5.1.15, kernel 5.4.213-ui-ipq5322 (aarch64) | Site CC. SSH: `root@192.168.0.10` via key `~/.ssh/pst-cc-ucg`; keyboard-interactive auth only. WAN SSH not accessible remotely. UCG VPN (strongSwan/xl2tpd) abandoned 2026-05-22; RRAS on PST-SERVER is the VPN endpoint. DNAT persistence: `/data/on_boot.d/10-vpn-portforward.sh`. Rebooted 2026-06-04 at 03:59, dropped VPN port-forward (see Known Issues). Credentials: vault `clients/peaceful-spirit/server`. |
| UCG-NW | 64.139.88.249 (old WAN; verify current) | UniFi gateway — NW site perimeter, S2S VPN | (verify) | NW site. Previously had OpenVPN at 64.139.88.249:1194 (TCP). S2S VPN CC<->NW confirmed up as of 2026-06-13 (ports 389/445/135/88 reachable SERVER2->SERVER). Details beyond this: (verify). Physical access: vault `clients/peaceful-spirit/physical-access-northwest`. |
@@ -62,10 +62,10 @@ Massage therapy practice with two sites: Country Club (CC, primary — all serve
- **Current namespace root target:** PST-SERVER only (`\\PST-SERVER\PST-Files`) — SERVER2 root target deferred pending stability
- **Current folder targets:** PST-SERVER only (`\\PST-SERVER\Shares`, Online) — SERVER2 folder target (`\\PST-SERVER2\Shares`) removed pending stability; to be re-added once SERVER2 holds stable
- **DFS-R group:** `PST-DFS`; replicated folder `Shares`
- PST-SERVER `G:\Shares` = PRIMARY / authoritative; staging 20 GB
- PST-SERVER2 `C:\Shares` = non-primary receiver; staging 20 GB
- Bidirectional connection configured; ~221/265 GB replicated as of 2026-06-14 (~44 GB backlog)
- **Gate 4 deferred items (blocked on SERVER2 stability):** drain backlog to 0; re-add SERVER2 `\\PST-SERVER2\Shares` folder target Online; add SERVER2 as 2nd namespace root target (`\\PST-SERVER2\PST-Files`) for VPN-outage HA
- PST-SERVER `G:\Shares` = authoritative content; staging 20 GB
- PST-DC-NW `C:\Shares` = receiver; staging 20 GB (member since 2026-07-02; SERVER2's dead membership removed same day)
- Bidirectional connection configured; initial sync of ~265 GB started 2026-07-02 over S2S VPN (expect days)
- **Gate 4 deferred items (blocked on initial sync completing):** RF state 4 + backlog 0 on PST-DC-NW; share C:\Shares as `Shares`; add `\\PST-DC-NW\Shares` folder target Online; add PST-DC-NW as 2nd namespace root target (`\\PST-DC-NW\PST-Files`) for VPN-outage HA
- **Runbook:** `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`
### Domain & Identity
@@ -73,7 +73,7 @@ Massage therapy practice with two sites: Country Club (CC, primary — all serve
- **Domain:** PEACEFULSPIRIT.local (NetBIOS: PEACEFULSPIRIT)
- **AD Sites & Services:** CC site (192.168.0.0/24), NW site (192.168.1.0/24); subnets correct, site link active
- **FSMO:** all 5 roles on PST-SERVER
- **Global Catalog:** both PST-SERVER and PST-SERVER2
- **Global Catalog:** both PST-SERVER and PST-DC-NW (since 2026-07-02; formerly PST-SERVER2)
- **Domain SID base:** S-1-5-21-1105246401-3156558273-4088333098
- **Domain admins:** `sysadmin` (password: vault `clients/peaceful-spirit/server`) — domain admin account. DA credentials were passed base64-wrapped in RMM command_text during June/July rebuild sessions; rotation optional (RMM is internal).
- **CA:** PEACEFULSPIRIT-PST-SERVER-CA — Enterprise Root CA on PST-SERVER. Thumbprint: 56DAF43C60F246BF2C80A671EE9812C727D8C298 (valid to 3/8/2061). `msPKI-Certificate-Name-Flag` changed 2026-05-11 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT).
@@ -124,7 +124,8 @@ Massage therapy practice with two sites: Country Club (CC, primary — all serve
| Host | Agent ID | Version | Notes |
|---|---|---|---|
| PST-SERVER | `87293069-33b6-45e8-a68f-6811216cdb96` | v0.6.75+ | Active; confirmed 2026-07-01. Prior `6b6106a7...` retired. |
| PST-SERVER2 | `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65` | — | NW site. Flapping at 2026-06-14 session end. RMM site assignment: (verify). |
| PST-SERVER2 | `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65` | — | REMOVED — agent record deleted from RMM (noticed 2026-07-02); machine being replaced by PST-DC-NW. |
| PST-DC-NW | `f60e9820-4a00-4598-83f7-c14085db5768` | v0.6.75 | NW site ("North West" in RMM). Enrolled 2026-07-02; renamed from PST-SERVER01 via RMM same day. |
| MaraHomeNew | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` | v0.6.52 | Active; confirmed 2026-06-04. |
| Maras-HP-Laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | — | — |
| PST-SURFACE | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | — | — |
@@ -219,6 +220,10 @@ A report that client files disappeared (trigger: the "Glennda" folder) prompted
- **Admin1 Delete-deny also blocks rename and delete-then-write saves.** The `(OI)(CI)(DENY)(D,DC)` ACE on G:\Shares\Scanned for Admin1 prevents deletion, rename, and any app save pattern that internally deletes-then-recreates. If CalistaA/ChristineZ/leslieW/SarahM report inability to rename or save, add an individual icacls exception. Reversal in the 2026-07-01 session log.
- **Remove-DfsrMember / Add-DfsrMember contact the member machine.** `Remove-DfsrMember` on a DEAD member fails with "network path was not found" — remove the `msDFSR-Member` (and any `msDFSR-Connection` whose `fromServer` references it) AD objects under `CN=DFSR-GlobalSettings,CN=System,...` directly. Conversely `Set-DfsrMembership` from SYSTEM on PST-SERVER fails with "Security cannot be set on the replicated folder. Access is denied" (writes to the OTHER member's AD subtree) — use the DA-cred `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local` pattern.
- **RMM inline JSON dispatch mangles backslash-containing payloads.** Inline `command` strings with `\\` (registry paths, `C:\\...`) fail the server-side parse (jq sees non-JSON error response). Use `ps-encoded.sh` (script file + -EncodedCommand) for anything with backslashes/quotes, or forward slashes (`C:/Shares`) for trivial paths.
- **vault.sh get-field returns literal "null" for nested credential fields.** `vault.sh get-field clients/peaceful-spirit/server credentials.password` returns the string `"null"`. Use `vault.sh get` (full read) and extract manually.
- **AD writes via RMM require DA creds using FQDN (not localhost).** `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...}` works; `-ComputerName localhost -Credential` fails with a Kerberos SPN error. Use the FQDN for any domain/DFS/DFSN/DFSR write over RMM.
@@ -234,7 +239,7 @@ A report that client files disappeared (trigger: the "Glennda" folder) prompted
As of 2026-07-01 session end:
- **VPN rollout: COMPLETE** across all four client machines (as of 2026-06-04).
- **[OPEN] PST-DC-NW bring-up (NEW, 2026-07-02).** New physical server (shipped as PST-SERVER01) installed at NW to replace the flapping PST-SERVER2 as the NW DC + DFSR partner. As of 2026-07-02 it is NOT in GuruRMM (and PST-SERVER2's agent record `5d2d7ba0...` has been deleted from RMM); Mike is adding the agent. Bring-up order: (1) rename PST-SERVER01 -> PST-DC-NW + reboot (BEFORE domain join — renaming a promoted DC is messy), (2) GuruRMM agent (client Peaceful Spirit, NW site), (3) static IP (192.168.1.5 free once SERVER2 is off, or new), (4) domain join, (5) promote DC/GC/DNS, (6) rebuild DFS-R receiver + finish Gate 4 targets on it. If SERVER2 is still a live DC, demote/metadata-clean it properly before or after cutover (never leave a stale DC — see tombstone-lifetime pattern).
- **[OPEN] PST-DC-NW bring-up (2026-07-02).** New physical server (shipped as PST-SERVER01) replacing PST-SERVER2 (hardware died ~2026-06-14, permanently) as the NW DC + DFSR partner. DONE: enrolled in GuruRMM (agent `f60e9820...`, client Peaceful Spirit, site "North West"); renamed PST-SERVER01 -> PST-DC-NW + rebooted via RMM 2026-07-02 (pre-domain-join); static IP set (per Mike 2026-07-02). DONE 2026-07-02, all via RMM: PST-SERVER2 metadata cleanup (see Patterns), then PST-DC-NW domain join, DC/GC/DNS promotion (site NW; verified: all DC services running, SYSVOL state 4, repadmin 0/5 fails both directions, advertising GC/KDC/DNS/timeserver), and PST-DFS re-add (member + bidirectional connection + membership C:\Shares staging 20 GB). DSRM password vaulted at `clients/peaceful-spirit/dc-nw`. REMAINING (Gate 4 finish, blocked on initial sync): (1) monitor initial DFS-R sync ~265 GB PST-SERVER -> PST-DC-NW over S2S VPN (days; watch `Get-DfsrBacklog` / RF state 2->4), (2) once state 4 + backlog 0: add `\\PST-DC-NW\Shares` folder target Online in the DFS namespace, (3) add PST-DC-NW as 2nd namespace root target (`\\PST-DC-NW\PST-Files`) for VPN-outage HA, (4) share C:\Shares on PST-DC-NW (SMB share `Shares`) before the folder target, (5) dcdiag clean both DCs. If SERVER2 is still a live DC, demote/metadata-clean it properly before or after cutover (never leave a stale DC — see tombstone-lifetime pattern).
- **[SUPERSEDED by PST-SERVER01] PST-SERVER2 NW site stability (was BLOCKER for Gate 4).** Reboot-loop flapping (System log 41/6008/1074), likely on-site power/UPS/hardware — resolved by hardware replacement rather than diagnosis.
- **[OPEN] Gate 4 finish (blocked on SERVER2 stable):** drain ~44 GB DFS-R backlog; re-add SERVER2 folder target Online; add SERVER2 as 2nd namespace root target for HA; verify both RFs State 4, dcdiag clean.
- **[OPEN] Deletion recovery — ~3,342 genuine files.** No-overwrite robocopy copy-back from `C:\PST-Recovery\PreDelete-0624` (excluding duplicate/nested-bucket trees). Awaiting Mike/Mara go — writes to live HIPAA data.
@@ -270,7 +275,7 @@ As of 2026-07-01 session end:
| 2026-06-14 | SERVER2 static IP set (192.168.1.5/24); timezone -> Mountain; stale .127 DNS records cleaned. Gate 4 DFS-R rebuilt clean with PST-SERVER G:\Shares PRIMARY and SERVER2 C:\Shares receiver; ~221/265 GB replicated. Session ended blocked: SERVER2 began flapping (NW site stability, not DFS). Gate 4 finish deferred. |
| 2026-06-29 | File-deletion investigation initiated. Stopped MSP360 backup, staged the 6/24 10:05 AM restore point. Mtime heuristic ruled out; restore-and-local-diff adopted as authoritative. |
| 2026-07-01 | Deletion-scope analysis complete: 47,749 files deleted since 6/24 10:05, ~93% duplicate cleanup, ~3,342 genuine recoverable. Incident window (10:05->12:05) had only 2 deletions. Glennda trigger = misspelled duplicate; canonical folder intact. Shelton check blocked (6/29/2025 restore point purged). Admin1/Admin2 NTFS hardening: removed incorrect Admin2-in-Admin1 nesting; Admin1 -> allow RX,W + DENY D,DC; Admin2 retained Full Control. ACL backup saved. |
| 2026-07-02 | Standing deletion audit operationalized: daily `PST Deletion Report` task (SACL 4660/4663 on G:\Shares\Scanned -> per-person HTML). Report output relocated to the legal/partner-review folder `G:\Shares\Private\Partner Review\Legal Documents - DO NOT DELETE\_Deletion Reports` (backup of the script kept). Change made via GuruRMM (site VPN was down); validated by a test run (report written, 6 items). New server installed at NW (shipped as PST-SERVER01, to be renamed PST-DC-NW) to replace flapping PST-SERVER2 as NW DC + DFSR partner; not yet in RMM (SERVER2's agent record also gone from RMM). |
| 2026-07-02 | Standing deletion audit operationalized: daily `PST Deletion Report` task (SACL 4660/4663 on G:\Shares\Scanned -> per-person HTML). Report output relocated to the legal/partner-review folder `G:\Shares\Private\Partner Review\Legal Documents - DO NOT DELETE\_Deletion Reports` (backup of the script kept). Change made via GuruRMM (site VPN was down); validated by a test run (report written, 6 items). New server installed at NW to replace flapping PST-SERVER2 as NW DC + DFSR partner: enrolled in GuruRMM (agent `f60e9820...`, site "North West") and renamed PST-SERVER01 -> PST-DC-NW + rebooted via RMM, pre-domain-join (workgroup). SERVER2's agent record found deleted from RMM. PST-SERVER2 (dead since ~6/14) metadata-cleaned from AD via RMM: DFSR member/connection objects, NTDS Settings + config server object, DC computer account, 18 stale DNS records; verified single-DC clean state. Then PST-DC-NW full bring-up, all via RMM: timezone AZ, DNS -> .0.2, domain join + reboot, AD DS role + Install-ADDSDomainController (DC/GC/DNS, site NW) + reboot, post-checks green (SYSVOL state 4, repadmin 0/5 fails both ways, advertising GC/KDC/DNS). Re-added to PST-DFS (member, bidirectional connection, membership C:\Shares staging 20 GB — Set-DfsrMembership needed DA Invoke-Command, not SYSTEM). Initial ~265 GB sync started. DSRM password vaulted (`clients/peaceful-spirit/dc-nw`). |
---