wiki: compile peaceful-spirit (full) — two-DC DFS, deletion investigation, Admin1/Admin2 ACL hardening

2026-07-01 09:09:40 -07:00
parent 02845878d6
commit e583bf43a5
2 changed files with 183 additions and 88 deletions
--- a/wiki/clients/peaceful-spirit.md
+++ b/wiki/clients/peaceful-spirit.md
@@ -2,7 +2,7 @@
 type: client
 name: peaceful-spirit
 display_name: Peaceful Spirit Therapeutic Massage
-last_compiled: 2026-06-04
+last_compiled: 2026-07-01
 compiled_by: GURU-5070/claude-main
 sources:
  - clients/peaceful-spirit/session-logs/2026-05-10-recovered-setup-radius-authentication-for-vpn-access.md
@@ -11,7 +11,14 @@ sources:
  - clients/peaceful-spirit/session-logs/2026-05-22-session.md
  - clients/peaceful-spirit/session-logs/2026-05-27-session.md
  - clients/peaceful-spirit/session-logs/2026-06-04-session.md
  - clients/peaceful-spirit/session-logs/2026-06/2026-06-11-mike-multisite-dfs-dc-plan.md
  - clients/peaceful-spirit/session-logs/2026-06/2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md
  - clients/peaceful-spirit/session-logs/2026-06/2026-06-14-mike-pst-gate4-dfsr-rebuild-and-static-ip.md
  - clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md
  - clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md
  - session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md
  - clients/peaceful-spirit/server.sops.yaml (vault)
  - clients/peaceful-spirit/server2.sops.yaml (vault)
  - clients/peaceful-spirit/vpn.sops.yaml (vault)
  - clients/peaceful-spirit/physical-access-northwest.sops.yaml (vault)
 backlinks:
@@ -20,25 +27,20 @@ backlinks:
 # Peaceful Spirit Therapeutic Massage
-Massage therapy practice with at least two sites: Country Club (primary, all work performed here) and a Northwest (NW) site. On-premises Windows Server 2016 Essentials domain environment. Domain-joined workstations for Mara (owner/operator) and other staff. L2TP/IPsec VPN fully deployed to all known machines as of 2026-05-27. Site-wide VPN outage occurred 2026-06-04 due to UDR Ultra reboot dropping VPN port-forward — resolved same day by re-adding UDP 500/4500 -> 192.168.0.2 in UniFi controller.
+Massage therapy practice with two sites: Country Club (CC, primary — all server infrastructure) and Northwest (NW). On-premises Windows Server 2016 Essentials domain (PEACEFULSPIRIT.local). As of June 2026 the environment was upgraded to a two-DC architecture: PST-SERVER (CC, Server 2016 Essentials) plus PST-SERVER2 (NW, Server 2019 Standard, rebuilt June 2026 from a past-tombstone-lifetime state). DFS namespace and DFS-R replication between sites established June 2026. L2TP/IPsec VPN fully deployed to all known client machines as of 2026-05-27.
 ---
 ## Profile
- **Contract type:** Break-fix / project [unverified — no contract details found in session logs]
+- **Business name (Syncro):** Peaceful Spirit Massage (NOT "...Therapeutic Massage" — ID-based lookup required)
- **Key contacts:**
+- **Syncro customer ID:** `278525`
-  - Mara — primary point of contact; owner/operator; personal Microsoft account `mara.concordia@gmail.com` (OneDrive). Domain user: `mara` (password reset to SpiritWalk26! on 2026-05-22, PasswordNeverExpires=true).
+- **Address:** 6650 N Oracle #100, Tucson
-  - Bridgette — staff member with home computer (BridgettePSHomeComputer); domain user `BridgetteSH`. No contact details captured.
+- **Primary contact:** Mara Concordia (owner/operator); generic contact email `info@bestmassageintucson.com`; personal Microsoft account `mara.concordia@gmail.com` (OneDrive). Domain user: `mara`.
- **Billing rate:** [unverified — not documented in session logs]
+- **Other key staff:** Bridgette (BridgetteSH); Christine Z (ChristineZ); Calista A (CalistaA); Leslie W (leslieW); Sarah M (SarahM); Katie B (katieb); Sharon S (SharonS); PSTAdmin.
- **Syncro customer ID:** `278525` (Peaceful Spirit Massage) — note the Syncro business name is "Peaceful Spirit Massage", not "...Therapeutic Massage", so a name search on "peaceful spirit" does not match; use the ID.
+- **Contract type:** Break-fix / T&M (verify — recent invoices per-ticket ~$150–300/visit, plus a recurring ~$195.19/month line item; no retainer contract confirmed)
- **Active tickets:** #32271 — "Bug - IKEv2 VPN drops and does not auto-reconnect" (the IKEv2-drops → L2TP-rebuild lineage)
+- **Managed asset count:** 31
-
+- **Open tickets:** 0 as of 2026-07-01
 ---
 ## Physical Access
 - **Northwest (NW) site:** lockbox, main-door keypad, and Mike's personal alarm-disarm code are stored in the vault at `clients/peaceful-spirit/physical-access-northwest.sops.yaml` (codes never recorded in plaintext here). Read with: `vault get-field clients/peaceful-spirit/physical-access-northwest.sops.yaml credentials.<lockbox_code|main_door_code|alarm_code>`.
 ---
@@ -48,45 +50,64 @@ Massage therapy practice with at least two sites: Country Club (primary, all wor
 | Host | IP | Role | OS | Notes |
 |---|---|---|---|---|
-| PST-SERVER | 192.168.0.2 | DC, DNS, RRAS (L2TP/IPsec VPN), NPS, Enterprise Root CA (AD CS) | Windows Server 2016 Essentials (build 14393) | GuruRMM agent ID: `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.52, confirmed 2026-06-04; prior ID `6b6106a7-8515-4b6b-857d-0dc6ede53f35` is retired/re-enrolled). Win32-OpenSSH installed 2026-05-11 (`C:\Program Files\OpenSSH\OpenSSH-Win64\`). Machine cert: `DB71981ABE4CBA1DE96FEEEAF178F6259663B543` (CN=PST-SERVER.PEACEFULSPIRIT.local, valid 5/9/2027). 30-day uptime confirmed 2026-06-04 (no reboot during VPN outage). |
+| PST-SERVER | 192.168.0.2 | DC (all 5 FSMO), DNS, RRAS (L2TP/IPsec VPN), NPS, Enterprise Root CA (AD CS) | Windows Server 2016 Essentials (build 14393) | Site CC. GuruRMM agent `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.75+; prior ID `6b6106a7...` retired). Win32-OpenSSH installed 2026-05-11. Machine cert: `DB71981ABE4CBA1DE96FEEEAF178F6259663B543` (CN=PST-SERVER.PEACEFULSPIRIT.local, valid 5/9/2027). Drives: C: 931 GB (OS); G: 465.7 GB data volume (ex-old-server C:, 182 GB free post-cleanup); D: 931 GB (Recovery-EXT/backup junk ~700 GB — cleanup pending). G:\Shares: Private ~154 GB, Scanned ~105 GB, ITServices ~5 GB, qbooks ~2 GB (~265 GB total). Credentials: vault `clients/peaceful-spirit/server`. |
-| UCG-PST-CC | 192.168.0.10 (LAN) / 98.190.129.150 (WAN) | UniFi Cloud Gateway Ultra (UDR Ultra) — perimeter router + DNAT for VPN | UniFi OS 5.1.15, kernel 5.4.213-ui-ipq5322 (aarch64) | SSH: `root@192.168.0.10` via key `~/.ssh/pst-cc-ucg` (password-auth is keyboard-interactive; password: vault). WAN SSH (98.190.129.150:22) is NOT accessible remotely — timed out from all tested sources; LAN SSH reachable via PST-SERVER jump. UCG VPN (strongSwan/xl2tpd) abandoned 2026-05-22 in favor of RRAS on PST-SERVER. DNAT persistence: `/data/on_boot.d/10-vpn-portforward.sh`. NOTE: Rebooted 2026-06-04 03:59 and dropped the VPN port-forward (see Known Issues). Port-forward re-added in UniFi controller by Mike 2026-06-04. UniFi OS 5.1.15 stores port-forward rules in a migrated schema — legacy Mongo collections read 0; use the controller UI as authoritative. |
+| PST-SERVER2 | 192.168.1.5 | DC (additional), GC, DNS | Windows Server 2019 Standard | Site NW. Static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2 + 127.0.0.1. GuruRMM agent `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. Single 1 TB NVMe, C: only (original D: physical disk gone). DFS-R replica at C:\Shares (~221 GB as of 2026-06-14; ~44 GB backlog remaining). Timezone: US Mountain Standard Time (Arizona). Rebuilt 2026-06-13 (force-demote -> metadata cleanup -> re-promote; see runbook). Credentials: vault `clients/peaceful-spirit/server2` (local admin + DSRM). [WARNING] Flapping (online ~1 min / offline several min reboot-loop pattern) at end of 2026-06-14 session — NW site power/UPS/network issue, NOT caused by DFS; PST-SERVER and data unaffected. |
 | UCG-PST-CC | 192.168.0.10 (LAN) / 98.190.129.150 (WAN) | UniFi Cloud Gateway Ultra — perimeter router + DNAT for VPN | UniFi OS 5.1.15, kernel 5.4.213-ui-ipq5322 (aarch64) | Site CC. SSH: `root@192.168.0.10` via key `~/.ssh/pst-cc-ucg`; keyboard-interactive auth only. WAN SSH not accessible remotely. UCG VPN (strongSwan/xl2tpd) abandoned 2026-05-22; RRAS on PST-SERVER is the VPN endpoint. DNAT persistence: `/data/on_boot.d/10-vpn-portforward.sh`. Rebooted 2026-06-04 at 03:59, dropped VPN port-forward (see Known Issues). Credentials: vault `clients/peaceful-spirit/server`. |
 | UCG-NW | 64.139.88.249 (old WAN; verify current) | UniFi gateway — NW site perimeter, S2S VPN | (verify) | NW site. Previously had OpenVPN at 64.139.88.249:1194 (TCP). S2S VPN CC<->NW confirmed up as of 2026-06-13 (ports 389/445/135/88 reachable SERVER2->SERVER). Details beyond this: (verify). Physical access: vault `clients/peaceful-spirit/physical-access-northwest`. |
-**Note:** An NW (Northwest) site exists with a separate UCG that previously had an OpenVPN server at 64.139.88.249:1194 (TCP). No further NW site details are documented.
+### DFS Namespace & Replication
 - **Domain-based DFS namespace:** `\\PEACEFULSPIRIT.local\PST-Files` -> folder `Shares`
 - **Current namespace root target:** PST-SERVER only (`\\PST-SERVER\PST-Files`) — SERVER2 root target deferred pending stability
 - **Current folder targets:** PST-SERVER only (`\\PST-SERVER\Shares`, Online) — SERVER2 folder target (`\\PST-SERVER2\Shares`) removed pending stability; to be re-added once SERVER2 holds stable
 - **DFS-R group:** `PST-DFS`; replicated folder `Shares`
  - PST-SERVER `G:\Shares` = PRIMARY / authoritative; staging 20 GB
  - PST-SERVER2 `C:\Shares` = non-primary receiver; staging 20 GB
  - Bidirectional connection configured; ~221/265 GB replicated as of 2026-06-14 (~44 GB backlog)
 - **Gate 4 deferred items (blocked on SERVER2 stability):** drain backlog to 0; re-add SERVER2 `\\PST-SERVER2\Shares` folder target Online; add SERVER2 as 2nd namespace root target (`\\PST-SERVER2\PST-Files`) for VPN-outage HA
 - **Runbook:** `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`
 ### Domain & Identity
- **Domain:** PEACEFULSPIRIT.local
+- **Domain:** PEACEFULSPIRIT.local (NetBIOS: PEACEFULSPIRIT)
- **Domain admins:** `sysadmin` (password: vault) — this is the domain admin account. `pst-admin` is a domain user (not domain admin) with VPN dial-in permission.
+- **AD Sites & Services:** CC site (192.168.0.0/24), NW site (192.168.1.0/24); subnets correct, site link active
- **AD domain SID base:** S-1-5-21-1105246401-3156558273-4088333098
+- **FSMO:** all 5 roles on PST-SERVER
- **CA:** PEACEFULSPIRIT-PST-SERVER-CA — Enterprise Root CA on PST-SERVER. Thumbprint: 56DAF43C60F246BF2C80A671EE9812C727D8C298 (valid to 3/8/2061).
+- **Global Catalog:** both PST-SERVER and PST-SERVER2
- **VPN-eligible users (WseRemoteAccessUsers, SID ...-1113):** Domain Admins (group), PSTAdmin, pst-admin, LMT, Mara, BridgetteSH (added 2026-05-27). NPS network policy grants VPN by group membership in WseRemoteAccessUsers — `msNPAllowDialin=TRUE` alone is not sufficient.
+- **Domain SID base:** S-1-5-21-1105246401-3156558273-4088333098
- **OneDrive:** pst-admin uses personal OneDrive (mara.concordia@gmail.com, cid: 25f0851177ceabfd). Per-machine OneDrive (v26.063.0405.0002) deployed to Maras-HP-Laptop on 2026-05-11 via `/allusers` install.
+- **Domain admins:** `sysadmin` (password: vault `clients/peaceful-spirit/server`) — domain admin account. DA credentials were passed base64-wrapped in RMM command_text during June/July rebuild sessions; rotation optional (RMM is internal).
- **Email / M365:** [unverified — no M365 tenant found; practice likely uses personal or third-party email]
+- **CA:** PEACEFULSPIRIT-PST-SERVER-CA — Enterprise Root CA on PST-SERVER. Thumbprint: 56DAF43C60F246BF2C80A671EE9812C727D8C298 (valid to 3/8/2061). `msPKI-Certificate-Name-Flag` changed 2026-05-11 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT).
 - **VPN-eligible users (WseRemoteAccessUsers, SID ...-1113):** Domain Admins (group), PSTAdmin, pst-admin, LMT, Mara, BridgetteSH. NPS grants VPN by group membership — `msNPAllowDialin=TRUE` alone is not sufficient.
 - **pst-admin:** domain user (not domain admin); in WseRemoteAccessUsers; VPN-eligible. Shared VPN credential for Mara's machines.
 - **AD security groups (custom):**
  - **Admin1** (Global Security): CalistaA, ChristineZ, leslieW, SarahM — allow `RX,W` + DENY `(D,DC)` on G:\Shares\Scanned (read/write/edit only; no delete, rename, or ownership change). Was previously Full Control; hardened 2026-07-01.
  - **Admin2** (Global Security): BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS — Full Control on G:\Shares\Scanned. Admin2 was formerly (incorrectly) nested inside Admin1; nesting removed 2026-07-01.
 - **OneDrive:** pst-admin uses personal OneDrive (mara.concordia@gmail.com, cid: 25f0851177ceabfd). Per-machine OneDrive deployed to Maras-HP-Laptop.
 - **Email / M365:** (verify — no M365 tenant found; practice likely uses personal or third-party email)
 - **GPO:** "Block New Outlook" — GUID {577028AF-0901-4BDF-A283-CD1156F313D9}, linked to domain root.
 - **SYSVOL backups (2026-06-13):** `C:\PST-Backup\SYSVOL-Policies-20260613-1611` and `C:\PST-Backup\GPO-20260613-1611` (11 GPOs) on PST-SERVER — keep until rebuild confirmed long-term stable.
 ### Network
- **WAN IP:** 98.190.129.150 (Country Club site, UCG)
+- **Country Club (CC) site:** WAN 98.190.129.150 (Cox); LAN 192.168.0.0/24; DC/DNS 192.168.0.2 (PST-SERVER); UCG 192.168.0.10
- **LAN subnet:** 192.168.0.0/24
+- **Northwest (NW) site:** LAN 192.168.1.0/24; DC/DNS 192.168.1.5 (PST-SERVER2); WAN (verify current; old OpenVPN was at 64.139.88.249); S2S VPN to CC confirmed up 2026-06-13
- **DNS / DC:** 192.168.0.2 (PST-SERVER)
+- **VPN (L2TP/IPsec, client-to-server):**
- **VPN (current — L2TP/IPsec):**
+  - Endpoint: PST-SERVER RRAS at 192.168.0.2, exposed via UCG-PST-CC DNAT (UDP 500, 4500, ESP)
  - Endpoint: PST-SERVER RRAS at 192.168.0.2, exposed via UCG DNAT (UDP 500, 4500, ESP)
  - PSK: vault (`clients/peaceful-spirit/vpn.sops.yaml`)
-  - Auth: MSCHAPv2. Mara's machines connect as shared user `pst-admin`; BridgettePSHomeComputer connects as `BridgetteSH` via SSO (no stored shared credential).
+  - Auth: MSCHAPv2. Mara's machines connect as shared user `pst-admin`; BridgettePSHomeComputer connects as `BridgetteSH` via SSO
-  - NPS RADIUS shared secret for client UCG-PST-CC (192.168.0.10): in vault (`clients/peaceful-spirit/server.sops.yaml`)
+  - NPS RADIUS shared secret for client UCG-PST-CC (192.168.0.10): vault (`clients/peaceful-spirit/server.sops.yaml`)
-  - IP pool: 192.168.0.240+ (observed: .241, .243, .248, .249 during testing)
+  - IP pool: 192.168.0.240+ (observed: .241, .242, .243, .248, .249)
-  - VPN profile name on clients: "Peaceful Spirit VPN" (AllUserConnection, split tunnel, 192.168.0.0/24 route, NRPT for .peacefulspirit.local → 192.168.0.2)
+  - VPN profile name on clients: "Peaceful Spirit VPN" (AllUserConnection, split tunnel, 192.168.0.0/24 route, NRPT for .peacefulspirit.local -> 192.168.0.2)
  - PST-SERVER registry: `AssumeUDPEncapsulationContextOnSendRule=2` (PolicyAgent), `DefaultPSK` set in L2TP parameters
  - UCG persistence: `/data/on_boot.d/10-vpn-portforward.sh`
 - **GPO:** "Block New Outlook" — GUID {577028AF-0901-4BDF-A283-CD1156F313D9}, linked to domain root. Disables new Outlook experience across all domain machines.
 ### Client Workstations
 | Machine | Role | GuruRMM Agent ID | Notes |
 |---|---|---|---|
-| MaraHomeNew | Mara's home desktop | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` (v0.6.52, confirmed 2026-06-04; prior ID `c778b6a3-c646-4454-a065-8c8bdcb1578e` retired) | Domain-joined. VPN working (confirmed via rasdial 2026-05-11; IPsec link established 2026-06-04 post-fix). Machine cert installed (D067E07B, CN=MaraHomeNew.PEACEFULSPIRIT.local, valid to 5/10/2027). Connects as pst-admin. |
+| MaraHomeNew | Mara's home desktop | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` (v0.6.52; prior `c778b6a3...` retired) | Domain-joined. VPN working. Machine cert: D067E07B (valid 5/10/2027). Connects as pst-admin. |
-| Maras-HP-Laptop | Mara's HP laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | Domain-joined. VPN deployed 2026-05-22 (PSK set on-site by Mike). OneDrive per-machine deployed 2026-05-11. pst-admin profile wiped and rebuilt 2026-05-11. Connects as pst-admin. |
+| Maras-HP-Laptop | Mara's HP laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | Domain-joined. VPN deployed 2026-05-22. OneDrive per-machine deployed. Connects as pst-admin. |
-| PST-SURFACE | Surface device | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | Domain-joined. VPN deployed 2026-05-22 (PSK set on-site by Mike). Connects as pst-admin. |
+| PST-SURFACE | Surface device | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | Domain-joined. VPN deployed 2026-05-22. Connects as pst-admin. |
-| BridgettePSHomeComputer | Bridgette's home PC | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` (v0.6.49, re-enrolled 2026-06-04; old UUID `074141d7-bd96-49ff-8f64-edf31159c00b` is dead/offline) | Domain-joined. VPN deployed remotely 2026-05-27 via GuruRMM `user_session`. Connects as BridgetteSH (SSO). Logon scheduled task `Connect Peaceful Spirit VPN` auto-connects ~20s after sign-in. NAT-T key was missing — set and rebooted 2026-05-27. Got VPN IP 192.168.0.242 after 2026-06-04 port-forward fix (event 20224 link established). |
+| BridgettePSHomeComputer | Bridgette's home PC | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` (v0.6.49; re-enrolled 2026-06-04; old `074141d7...` dead) | Domain-joined. VPN deployed remotely via GuruRMM user_session 2026-05-27. Connects as BridgetteSH (SSO). Logon scheduled task `Connect Peaceful Spirit VPN` auto-connects ~20s after sign-in. |
 ---
@@ -94,66 +115,140 @@ Massage therapy practice with at least two sites: Country Club (primary, all wor
 - **Client name in RMM:** Peaceful Spirit
 - **Client ID:** `00015eae-50e5-4102-93fa-ab0fdb135c08`
- **Site name:** Country Club
+- **Primary site name:** Country Club
- **Site ID:** `7b32983d-982a-4a5c-af07-45a23453f589`
+- **Primary site ID:** `7b32983d-982a-4a5c-af07-45a23453f589`
 **Enrolled agents:**
-| Host | Agent ID | Enrolled | Last Known Status |
+| Host | Agent ID | Version | Notes |
 |---|---|---|---|
-| PST-SERVER | `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.52) | [re-enrolled; prior `6b6106a7...` retired] | Active — confirmed 2026-06-04 |
+| PST-SERVER | `87293069-33b6-45e8-a68f-6811216cdb96` | v0.6.75+ | Active; confirmed 2026-07-01. Prior `6b6106a7...` retired. |
-| MaraHomeNew | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` (v0.6.52) | [re-enrolled; prior `c778b6a3...` retired] | Active — confirmed 2026-06-04 |
+| PST-SERVER2 | `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65` | — | NW site. Flapping at 2026-06-14 session end. RMM site assignment: (verify). |
-| Maras-HP-Laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | [unverified date] | — |
+| MaraHomeNew | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` | v0.6.52 | Active; confirmed 2026-06-04. |
-| PST-SURFACE | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | [unverified date] | — |
+| Maras-HP-Laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | — | — |
-| BridgettePSHomeComputer | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` (v0.6.49) | Re-enrolled 2026-06-04 (old `074141d7...` dead/offline) | Active — confirmed 2026-06-04 |
+| PST-SURFACE | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | — | — |
 | BridgettePSHomeComputer | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` | v0.6.49 | Re-enrolled 2026-06-04. |
 ---
 ## Data Store & Backup
 ### Data Location
 Client SOAP-note and business files reside on **PST-SERVER G:\Shares**. The @Clients tree is doubly-nested: `G:\Shares\Scanned\@Clients\@Clients`. File counts as of 2026-07-01: ~142,335 files / ~72 GB in the live @Clients tree. Total G:\Shares ~265 GB. G: is the old server's former C: drive (live PST-SERVER OS runs from C:).
 ### MSP360 / Backblaze B2 Backup
 - **Plan name:** "Files Backup 2025" (Files Backup type; ForeverForward; retention 365 days)
 - **MSP360 account:** ACG-PST `084b5069-d634-434b-84a2-971b1dcb4b43`
 - **Bunch ID:** `6a121575-84a0-4e98-9c0f-4a656d1a5132` (prefix: PST-SERVER)
 - **Destination:** Backblaze B2
 - **cbb.exe path:** `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`; logs `C:\ProgramData\Online Backup\Logs\`
 - **Known restore points:** `20260624170506` (6/24 10:05 AM, pre-incident repair source); `20260624190522` (6/24 12:05 PM). Oldest `20250629170034` (6/29/2025) **PURGED** as of 2026-07-01 (past 365-day retention) — year-ago backup unavailable.
 - **Status 2026-07-01:** running normally (the 6/29 stop-for-restores self-resumed).
 - **Caveat:** `cbb list` is unreliable on comma/space folder paths (false zeros, timeouts on large trees). Use restore-to-staging + local diff for any deletion-scope investigation.
 ### NTFS Access Control (G:\Shares\Scanned)
 ACL root is `G:\Shares\Scanned`; permissions inherit to `@Clients` and subdirectories. Hardened 2026-07-01. ACL backup on server: `C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt`.
 | Group | Members | Effective Permissions |
 |---|---|---|
 | Admin1 | CalistaA, ChristineZ, leslieW, SarahM | Allow `(OI)(CI)(RX,W)` + **DENY** `(OI)(CI)(D,DC)` — read/write/edit only; no delete, rename, permission or ownership change |
 | Admin2 | BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS | `(OI)(CI)(F)` — Full Control |
 **Caveat:** the `(D,DC)` deny on Admin1 also blocks rename and app save patterns that delete-then-write. If Admin1 users report inability to rename or save, carve an individual exception. Reversal: `Add-ADGroupMember Admin1 -Members Admin2`; `icacls "G:\Shares\Scanned" /remove:d "PEACEFULSPIRIT\Admin1"` then restore allow via `/grant`.
 ### Deletion Investigation (June–July 2026)
 A report that client files disappeared (trigger: the "Glennda" folder) prompted a staged restore-and-diff investigation. The 6/24 10:05 AM restore point was staged to `C:\PST-Recovery\PreDelete-0624` (~99 GB). Authoritative diff: **47,749 files deleted from @Clients since 6/24 10:05**; ~93% intentional duplicate cleanup (33,711 in folders labeled "duplicate DO NOT USE or delete"; ~10,696 in nested misfile-buckets A\A, D\A, P\O, H\I whose canonical client folders remain live). Genuine loss estimate: **~3,342 files**, recoverable via no-overwrite copy-back from staging (not yet executed — awaiting Mike/Mara approval; writes to live HIPAA data). The 10:05->12:05 PM window had only 2 deletions (Ballard, Kathy and Rivera, Anthony SOAP PDFs) — mass deletion occurred later. Glennda trigger: `EDWARDS, GLENDA` (single-N, 79 files, deleted) was a misspelled duplicate of the active canonical `EDWARDS, GLENNDA VA REFERRAL` (double-N, 127 files, live and growing). Shelton report: only 6 old Shelton files exist (2011–2015), loose in `S\`, CreationTime 2025-06-02 (migration), unchanged since 6/24 — not a 2026 deletion; the 6/29/2025 restore point needed for further check has been purged. Staging artifacts (~200 GB, removable after recovery decision): `C:\PST-Recovery\{PreDelete-0624, PostDelete-0624, authdiff, incidentdiff, acl-backup-scanned-20260701-072725.txt}`.
 ---
 ## Access
- **PST-SERVER SSH:** `ssh -i ~/.ssh/id_ed25519 sysadmin@192.168.0.2` — requires OpenVPN or L2TP VPN to Country Club site active. Win32-OpenSSH at `C:\Program Files\OpenSSH\OpenSSH-Win64\`. SCP paths use Unix format (`/C:/path/to/file`).
+- **PST-SERVER SSH:** `ssh -i ~/.ssh/id_ed25519 sysadmin@192.168.0.2` — requires L2TP VPN to CC site active. Win32-OpenSSH at `C:\Program Files\OpenSSH\OpenSSH-Win64\`. SCP paths use Unix format (`/C:/path/to/file`).
- **UCG SSH (LAN only):** `ssh -i ~/.ssh/pst-cc-ucg root@192.168.0.10` — UCG requires keyboard-interactive auth (paramiko with a kb_handler, or an interactive terminal; plink with `-pw` fails). WAN IP (98.190.129.150) SSH is NOT accessible remotely from any tested location. Requires VPN to LAN, on-site, or UCG cloud portal (unifi.ui.com).
+- **PST-SERVER2 SSH:** `ssh sysadmin@192.168.1.5` — requires S2S VPN or physical NW site access. Local admin creds: vault `clients/peaceful-spirit/server2`.
 - **UCG-PST-CC SSH (LAN only):** `ssh -i ~/.ssh/pst-cc-ucg root@192.168.0.10` — keyboard-interactive auth only (plink `-pw` fails; use paramiko kb_handler or interactive terminal). WAN SSH not accessible remotely. Requires VPN, on-site, or UniFi cloud portal (unifi.ui.com).
 - **GuruRMM (external):** https://rmm.azcomputerguru.com
 - **Physical access — NW site:** lockbox, main-door keypad, alarm-disarm code in vault `clients/peaceful-spirit/physical-access-northwest.sops.yaml` (codes never in plaintext here).
 - **Vault paths:**
-  - `clients/peaceful-spirit/server.sops.yaml` — PST-SERVER credentials (sysadmin) and UCG details (root, keyboard-interactive); raw secrets live in the vault entry, not here. Created during the 2026-05-10 recovered session.
+  - `clients/peaceful-spirit/server.sops.yaml` — PST-SERVER credentials (sysadmin DA) and UCG-PST-CC details.
-  - `clients/peaceful-spirit/vpn.sops.yaml` — VPN PSK (z5zkNBds2V9eIkdey09Zm6Khil3DAZs8, confirmed 2026-06-04 matches server), pst-admin credentials, network details. [WARNING] VAULT DRIFT: vault lists pst-admin password as `24Hearts$` but wiki records a reset to `SpiritWalk26!` on 2026-05-22 — needs reconciliation (verify with Mara, update whichever is stale).
+  - `clients/peaceful-spirit/server2.sops.yaml` — PST-SERVER2 local admin + DSRM passwords. Created 2026-06-13.
  - `clients/peaceful-spirit/vpn.sops.yaml` — VPN PSK, pst-admin credentials, network details. [WARNING] VAULT DRIFT: vault lists pst-admin password as one value but the wiki records a 2026-05-22 reset to another — reconcile with Mara and update the stale entry.
  - `clients/peaceful-spirit/physical-access-northwest.sops.yaml` — NW site lockbox, door, alarm codes.
 ---
 ## Patterns & Known Issues
- **Set-VpnConnection -L2tpPsk cannot run via RMM (SYSTEM context).** Windows enforces interactive mode for PSK registration. An admin must run this command manually on each machine in an interactive session. This is a one-time setup step per machine. Exception: the `user_session` command context in GuruRMM (added post-2026-05-22) does allow it — validated on BridgettePSHomeComputer 2026-05-27.
+- **Set-VpnConnection -L2tpPsk cannot run via RMM (SYSTEM context).** Windows enforces interactive mode for PSK registration. An admin must run it manually per machine in an interactive session (one-time). Exception: the `user_session` command context in GuruRMM allows it — validated on BridgettePSHomeComputer 2026-05-27.
 - **NRPT instead of VPN DNS suffix push.** `Add-VpnConnectionTriggerDnsConfiguration` fails for AllUserConnection profiles. Use `Add-DnsClientNrptRule -Namespace ".peacefulspirit.local" -NameServers "192.168.0.2"` instead.
 - **cmdkey as SYSTEM for pre-login credential persistence.** Machine credential store entries (cmdkey in SYSTEM context) are available at the Windows login screen; per-user cmdkey entries are not.
- **Stale hosts file.** During 2026-05-22 on-site, MaraHomeNew (and likely other machines) had a stale hosts entry mapping PST-SERVER to 72.194.62.5 (Mara's router's bogus DNS response). This caused name resolution failures even with VPN up. A GuruRMM cleanup script was deployed; verify no residual entries if name resolution issues recur. The hosts-file path encoding bug (`driverstc` artifact) means the cleanup script may not have fully run on all machines.
+
- **UDR Ultra reboot can silently drop the VPN port-forward (site-wide outage risk).** Confirmed 2026-06-04: the UDR Ultra (UCG-PST-CC) rebooted at 03:59 and came back without the UDP 500/4500 -> 192.168.0.2 port-forward, taking the entire site VPN offline with error 789 (IKE packets silently dropped at the edge). The `/data/on_boot.d/10-vpn-portforward.sh` persistence script was present but the UniFi OS 5.1.15 controller schema migration appears to have superseded it. **After any site-wide error 789, check the UDR port-forward in the UniFi controller FIRST** — IPsec auditing on the server (zero IKE events) is the confirmatory test. Long-term open items: (1) verify the re-added rule persists across a deliberate reboot (possible firmware bug or uncommitted rule), (2) add a DDNS hostname so the hardcoded 98.190.129.150 in client profiles is not a single point of failure for a Cox WAN-IP change.
+- **Stale hosts file.** During 2026-05-22 on-site, MaraHomeNew (and likely others) had a stale hosts entry mapping PST-SERVER to 72.194.62.5 (Mara's router's bogus DNS). A GuruRMM cleanup script was deployed; the path encoding bug (`driverstc`) means it may not have fully run on all machines — verify if resolution issues recur.
- **UCG iptables DNAT required — UniFi Traffic Rules are firewall-allow only, NOT DNAT.** Port-forward rules must be managed via the UniFi controller UI; `/data/on_boot.d/10-vpn-portforward.sh` is a legacy CLI fallback and may not persist reliably on UniFi OS 5.1.15+ (see above). Always verify iptables live after a reboot.
+
- **UCG SSH unreachable from office WAN.** All remote UCG administration must go through GuruRMM (for PST-SERVER) or the UniFi cloud portal (for UCG itself). LAN SSH (192.168.0.10) requires keyboard-interactive auth — password auth via plink fails; use paramiko with kb_handler or interactive terminal.
+- **UDR Ultra reboot can silently drop the VPN port-forward (site-wide outage risk).** Confirmed 2026-06-04: UCG-PST-CC rebooted at 03:59 and returned without the UDP 500/4500 -> 192.168.0.2 DNAT, taking the site VPN offline with error 789 (IKE packets silently dropped at the edge). The `/data/on_boot.d/10-vpn-portforward.sh` persistence script was present but the UniFi OS 5.1.15 schema migration appears to have superseded it. After any site-wide error 789, check the UDR port-forward in the UniFi controller FIRST — IPsec auditing on the server (zero IKE events) is the confirmatory test. Open: verify the re-added rule survives a deliberate reboot; add a DDNS hostname so the hardcoded Cox WAN IP is not a single point of failure.
- **GuruRMM command_type — use `powershell` or `shell`, NOT a made-up type (RESOLVED 2026-06-12).** The old advice here ("use `command_type: cmd` and call powershell.exe") was wrong on two counts: (1) the `-OutputEncoding` PowerShell failure it worked around is fixed in the agent (it sets `[Console]::OutputEncoding` inline, so `command_type: "powershell"` works on PST machines); (2) the agent's `CommandType` enum only accepts `shell`, `powershell`, `python`, `script`, `claude_task` (+ alias `cmd` → shell, added 2026-06-12). A command with an **unknown** `command_type` (e.g. the bare `cmd` before the alias) fails the agent's whole-message JSON parse and is **silently dropped** — no ack, no result — which looks exactly like a network black-hole and cost a long mis-diagnosis. Always use `powershell` (runs powershell.exe, UTF-8 fixed) or `shell`/`cmd` (runs cmd.exe). The agent now also NAKs an unparseable command so it fails fast instead of black-holing.
+
- **Machine cert template (PEACEFULSPIRIT-PST-SERVER-CA / Machine template).** `msPKI-Certificate-Name-Flag` was changed from `0x18000000` to `0x1` (ENROLLEE_SUPPLIES_SUBJECT) on 2026-05-11. This is a domain-wide template change. New machine certs will use the CSR Subject/SAN rather than the submitting machine's AD DNS identity. RRAS UserAuthProtocolAccepted now includes Certificate (added 2026-05-11).
+- **UCG iptables DNAT required — UniFi Traffic Rules are firewall-allow only, NOT DNAT.** Port-forward rules must be managed via the UniFi controller UI; the on_boot CLI script is a legacy fallback and may not persist on UniFi OS 5.1.15+. Verify iptables live after a reboot.
- **OneDrive KFM on WSE folder-redirected profiles.** Machines formerly managed by Windows Server Essentials had WSE-specific non-standard GUID variants in User Shell Folders (different from standard Known Folder GUIDs). Direct HKU writes alone do not clear the shell's internal known folder policy state — `SHSetKnownFolderPath` must be called with `flags=0` (not 0x4000) in user session context. If KFM still fails after registry cleanup, wipe the profile and redeploy with per-machine OneDrive (`/allusers`).
+
- **pst-admin vs sysadmin distinction.** `pst-admin` is a domain user (in WseRemoteAccessUsers, VPN-eligible). `sysadmin` is domain admin. Many early session failures were caused by using pst-admin credentials for domain admin operations.
+- **UCG SSH unreachable from office WAN.** Remote UCG administration goes through GuruRMM (for PST-SERVER) or the UniFi cloud portal (for UCG itself). LAN SSH (192.168.0.10) requires keyboard-interactive auth; plink password auth fails.
- **NPS grants VPN by WseRemoteAccessUsers group membership, not msNPAllowDialin alone.** The NPS network policy condition is SID-based (WseRemoteAccessUsers, `...-1113`). A user with `msNPAllowDialin=TRUE` but not in the group will get error 812 (policy denial). Both attributes are required.
+
- **cmdkey credential not used by rasdial for PPP auth.** The machine-store cmdkey entry (target = server address) is NOT consulted for PPP authentication. No-arg `rasdial` calls send the wrong principal (SYSTEM → error 691; logged-in user without explicit credential → error 812). For non-interactive auto-connect, use the logon scheduled task approach (BridgetteSH) or the AllUserConnection cmdkey path (pst-admin machines).
+- **GuruRMM command_type — use `powershell` or `shell`, NOT a made-up type (RESOLVED 2026-06-12).** The agent's `CommandType` enum accepts only `shell`, `powershell`, `python`, `script`, `claude_task` (plus alias `cmd` -> shell). An unknown `command_type` fails the agent's whole-message JSON parse and the command is silently dropped — looks like a network black-hole.
- **NAT-T registry key required on all client machines.** `AssumeUDPEncapsulationContextOnSendRule=2` under `HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent` must be set AND the machine must be rebooted (IPsec caches at boot). BridgettePSHomeComputer was missing this key; error 809 until rebooted after setting it. Verify this key is present before troubleshooting any future VPN error 809.
+
 - **Machine cert template (PEACEFULSPIRIT-PST-SERVER-CA).** `msPKI-Certificate-Name-Flag` changed from `0x18000000` to `0x1` (ENROLLEE_SUPPLIES_SUBJECT) 2026-05-11. New machine certs use the CSR Subject/SAN. RRAS UserAuthProtocolAccepted now includes Certificate.
 - **OneDrive KFM on WSE folder-redirected profiles.** WSE machines had non-standard GUID variants in User Shell Folders. `SHSetKnownFolderPath` must be called with `flags=0` (not 0x4000) in user session context. If KFM still fails after registry cleanup, wipe the profile and redeploy per-machine OneDrive (`/allusers`).
 - **pst-admin vs sysadmin distinction.** `pst-admin` is a domain user (VPN-eligible); `sysadmin` is domain admin. Many early failures came from using pst-admin creds for DA operations.
 - **NPS grants VPN by WseRemoteAccessUsers group membership, not msNPAllowDialin alone.** The NPS policy condition is SID-based (`...-1113`). Both group membership AND msNPAllowDialin are required; missing group = error 812.
 - **cmdkey credential not used by rasdial for PPP auth.** The machine-store cmdkey entry is NOT consulted for PPP auth. No-arg `rasdial` sends the wrong principal (SYSTEM -> error 691; logged-in user without explicit credential -> error 812). Use the logon scheduled task (BridgetteSH) or the AllUserConnection cmdkey path (pst-admin machines).
 - **NAT-T registry key required on all client machines.** `AssumeUDPEncapsulationContextOnSendRule=2` under `HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent` must be set AND the machine rebooted. Missing key = error 809 (was the BridgettePSHomeComputer symptom).
 - **PST-SERVER2 flapping at NW site (OPEN as of 2026-06-14).** After DFS-R initial replication reached ~221 GB, SERVER2 went into a reboot-loop (online ~1 min, offline several min). Pull System log events 41 (kernel-power), 6008 (unexpected shutdown), 1074 for cause — likely NW power/UPS/hardware/network. PST-SERVER (source) unaffected; no data risk. Gate 4 finish is blocked on this.
 - **Duplicate and misspelled client folders make deletion-scope analysis noisy.** The @Clients tree contains folders labeled "duplicate DO NOT USE or delete" and nested misfile buckets (A\A, D\A, P\O, H\I) alongside legitimate client folders. The Glennda/Glenda case (misspelled duplicate deleted, canonical intact) is recurring. When investigating any deletion report: (1) restore-to-staging + local diff is the only trustworthy method (`cbb list` is unreliable on these paths); (2) verify canonical folders before concluding data is lost; (3) count duplicate-labeled and nested-bucket files separately from genuine deletions.
 - **Admin1 Delete-deny also blocks rename and delete-then-write saves.** The `(OI)(CI)(DENY)(D,DC)` ACE on G:\Shares\Scanned for Admin1 prevents deletion, rename, and any app save pattern that internally deletes-then-recreates. If CalistaA/ChristineZ/leslieW/SarahM report inability to rename or save, add an individual icacls exception. Reversal in the 2026-07-01 session log.
 - **vault.sh get-field returns literal "null" for nested credential fields.** `vault.sh get-field clients/peaceful-spirit/server credentials.password` returns the string `"null"`. Use `vault.sh get` (full read) and extract manually.
 - **AD writes via RMM require DA creds using FQDN (not localhost).** `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...}` works; `-ComputerName localhost -Credential` fails with a Kerberos SPN error. Use the FQDN for any domain/DFS/DFSN/DFSR write over RMM.
 - **IP address change via RMM is unreliable.** A `New-NetIPAddress` over the agent was killed mid-NIC-blip and reverted to DHCP. Use a one-shot scheduled task that applies the change a few seconds after the command returns, or do it on console/on-site (as done for SERVER2 2026-06-14).
 - **Past-tombstone-lifetime DC must never resume replication.** SERVER2 was disconnected past the tombstone lifetime. Correct remediation is force-demote -> metadata cleanup on the authoritative DC -> fresh re-promote (never resume/fix replication). Runbook: `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`.
 ---
 ## Active Work
-As of 2026-06-04 session end:
+As of 2026-07-01 session end:
- **VPN rollout: COMPLETE.** All four machines (MaraHomeNew, Maras-HP-Laptop, PST-SURFACE, BridgettePSHomeComputer) have working L2TP/IPsec VPN. 2026-06-04 site-wide outage (UDR reboot) confirmed resolved: Bridgette connected (192.168.0.242), Mara IPsec established.
+- **VPN rollout: COMPLETE** across all four client machines (as of 2026-06-04).
- **[OPEN] UDR port-forward reboot-persistence test:** Confirm the re-added UDP 500/4500 -> 192.168.0.2 rule survives a deliberate UDR reboot. The prior rule vanished on the 2026-06-04 03:59 reboot — may be a firmware bug or an uncommitted rule. If it doesn't persist, re-examine `/data/on_boot.d/10-vpn-portforward.sh` or escalate to UniFi.
+- **[OPEN] PST-SERVER2 NW site stability (BLOCKER for Gate 4).** Diagnose reboot-loop flapping (System log 41/6008/1074). Likely on-site power/UPS/hardware.
- **[OPEN] DDNS for VPN endpoint:** Client profiles hardcode 98.190.129.150 (Cox WAN). A DDNS hostname would prevent a site-wide VPN breakage on a Cox IP change. Deferred — low urgency but document path.
+- **[OPEN] Gate 4 finish (blocked on SERVER2 stable):** drain ~44 GB DFS-R backlog; re-add SERVER2 folder target Online; add SERVER2 as 2nd namespace root target for HA; verify both RFs State 4, dcdiag clean.
- **[OPEN] Vault drift — pst-admin password:** `vpn.sops.yaml` lists `24Hearts$`; wiki records reset to `SpiritWalk26!` on 2026-05-22. Reconcile: verify current password with Mara, update vault to match.
+- **[OPEN] Deletion recovery — ~3,342 genuine files.** No-overwrite robocopy copy-back from `C:\PST-Recovery\PreDelete-0624` (excluding duplicate/nested-bucket trees). Awaiting Mike/Mara go — writes to live HIPAA data.
- **[OPEN] Syncro ticket #32271 update:** Resolution + 1hr warranty labor for 2026-06-04 outage (per session log, in progress at session end).
+- **[OPEN] Glennda single-N duplicate confirmation.** Verify the deleted `EDWARDS, GLENDA` (79 files) had zero unique content vs live `EDWARDS, GLENNDA` (127 files).
- **Parity decision deferred:** Mara's 3 machines connect as shared `pst-admin`; BridgetteSH connects as her own domain account via SSO. Consider aligning all to per-user auth (cleaner audit trail) or aligning Bridgette to `pst-admin`.
+- **[OPEN] Shelton SOAP notes.** Year-ago restore point purged; needs client input (were the notes ever scanned? active "Sheldon" family nearby — possible mishearing).
- **Pre-login VPN verification:** Confirmed working on MaraHomeNew via rasdial. Maras-HP-Laptop and PST-SURFACE need verification at the Windows login screen specifically.
+- **[OPEN] Admin1 ACL watch.** Monitor the 4 users for rename/save issues (Delete-deny side effect).
- **Hosts file cleanup verification:** The GuruRMM cleanup script had a path encoding bug (`driverstc` instead of `drivers\etc`) — DNS was flushed but hosts entries may not have been removed on all machines. Verify if name resolution issues recur.
+- **[OPEN] PST-Recovery staging cleanup (~200 GB)** once recovery decision finalized.
- **PST-SERVER temp file cleanup:** `C:\ProgramData\`: gen_certs.ps1, fix_acl.ps1, acl_result.txt, verify_acl.ps1, acl_verify.txt, and all *.inf, *.req, *.cer, *.pfx files. Also remove temporary firewall rules TEMP-CertEnroll-RPC (TCP 135) and TEMP-CertEnroll-DCOM (TCP 49152-65535).
+- **[OPEN] UDR port-forward reboot-persistence test.**
- **Machine cert VPN path (IKEv2) — deferred.** Machine certs were generated for MaraHomeNew (D067E07B), Maras-HP-Laptop (4CADDE8F, CA RequestId 66), and PST-SURFACE (197FF22A, CA RequestId 67) and PFXs (password: PstVpn2026!) were created. This IKEv2 machine-cert approach was superseded by the L2TP/RRAS decision on 2026-05-22. The certs and PFXs remain on PST-SERVER and DESKTOP-0O8A1RL — determine if IKEv2 path should be completed, abandoned, or the certs revoked.
+- **[OPEN] DDNS for VPN endpoint** (hardcoded Cox WAN 98.190.129.150).
- **Auto-connect task on BridgettePSHomeComputer:** Validated via `Start-ScheduledTask`; Bridgette fully connected 2026-06-04 (logon-task path confirmed end-to-end during outage resolution).
+- **[OPEN] Vault drift — pst-admin password** (vpn.sops.yaml vs 2026-05-22 reset). Verify with Mara.
 - **[OPEN] D: backup-junk cleanup on PST-SERVER** (~700 GB).
 - **[OPEN] PST-SERVER temp/staging cleanup:** `C:\PST-Backup\*` (SYSVOL/GPO backups) once rebuild confirmed stable; `C:\ProgramData\` cert-enroll scratch (*.inf/*.req/*.cer/*.pfx, gen_certs.ps1, etc.); temp firewall rules TEMP-CertEnroll-RPC / TEMP-CertEnroll-DCOM.
 - **[OPEN] Backup synthetic-full confirmation** — confirm "Files Backup 2025" completes cleanly after the stop/resume.
 - **[DEFERRED] Machine cert VPN path (IKEv2)** — certs/PFXs exist (MaraHomeNew D067E07B, Maras-HP-Laptop 4CADDE8F, PST-SURFACE 197FF22A); superseded by L2TP. Complete, abandon, or revoke.
 - **[DEFERRED] Parity decision** — Mara's machines use shared pst-admin; Bridgette uses her own account. Consider per-user auth for a cleaner audit trail.
 - **[DEFERRED] Optional sysadmin rotation** (DA passed base64 in RMM command_text; recoverable from RMM DB; RMM internal-only).
 - **[DEFERRED] Pre-login VPN verification** on Maras-HP-Laptop and PST-SURFACE.
 - **[DEFERRED] 2016 Essentials EOL** — PST-SERVER hits end of support Jan 2027; plan replacement (2022/2025 Standard, plain AD DS) vs extend-as-is.
 ---
@@ -161,19 +256,19 @@ As of 2026-06-04 session end:
 | Date | Event |
 |---|---|
-| 2026-05-10 | GuruRMM agent installed on PST-SERVER. UCG-PST-CC reconfigured for IKEv2 in prior (unlogged) session. IKEv2 error 812 diagnosed — NPS rejecting nonexistent user `apst-admin` (typo in stored credential). NPS order-0 test policy (PST-VPN-Test) added. Credential Manager corrected on DESKTOP-0O8A1RL. |
+| 2026-05-10 | GuruRMM agent installed on PST-SERVER. IKEv2 error 812 diagnosed (NPS rejecting nonexistent user `apst-admin` typo). Agents enrolled on MaraHomeNew, Maras-HP-Laptop, PST-SURFACE; IKEv2 "Peaceful Spirit VPN" profiles deployed. |
-| 2026-05-10 | GuruRMM agents enrolled on MaraHomeNew, Maras-HP-Laptop, PST-SURFACE. AllUserConnection IKEv2 "Peaceful Spirit VPN" profiles deployed to all three Mara machines. |
+| 2026-05-11 | Machine cert auth working on MaraHomeNew. Win32-OpenSSH installed on PST-SERVER. msPKI-Certificate-Name-Flag -> 0x1. OneDrive KFM WSE GUID fix. pst-admin profile on Maras-HP-Laptop wiped; per-machine OneDrive deployed. "Block New Outlook" GPO created. |
-| 2026-05-11 AM | PST-VPN-Test NPS policy removed. AutoEnroll ACL on Machine cert template fixed (Domain Computers, sysadmin scheduled task). Catch-22 identified: machine cert enrollment requires LAN access which requires a cert. OpenVPN on MaraHomeNew chosen as bootstrap path. |
+| 2026-05-22 | L2TP/IPsec VPN deployed to MaraHomeNew, Maras-HP-Laptop, PST-SURFACE on-site. UCG strongSwan/xl2tpd abandoned; RRAS on PST-SERVER became the endpoint. UCG DNAT rules created. Stale hosts entries removed. pst-admin/mara passwords reset. |
-| 2026-05-11 PM | Machine cert auth working on MaraHomeNew. Win32-OpenSSH installed on PST-SERVER. msPKI-Certificate-Name-Flag changed to 0x1 (ENROLLEE_SUPPLIES_SUBJECT). RRAS UserAuthProtocolAccepted updated to include Certificate. PFX certs generated for Maras-HP-Laptop and PST-SURFACE. |
+| 2026-05-27 | BridgettePSHomeComputer VPN deployed fully remotely via GuruRMM user_session. Logon scheduled task for auto-connect. VPN rollout complete across all four machines. |
-| 2026-05-11 PM | Maras-HP-Laptop: OneDrive KFM "Capabilities: 0x101" error troubleshooting. WSE non-standard GUID variants in User Shell Folders identified and corrected. Shell Folders cache directly updated via SYSTEM/HKU. SHSetKnownFolderPath flags=0x4000 bug identified (root cause of all prior script failures). |
+| 2026-06-04 | Site-wide VPN outage: UCG-PST-CC rebooted at 03:59, returned without the DNAT; all clients failed error 789. PST-SERVER healthy. Root cause isolated via IPsec auditing (zero IKE events). Port-forward re-added. BridgettePSHomeComputer re-enrolled (new UUID 01160fc8). |
-| 2026-05-11 Evening | pst-admin profile on Maras-HP-Laptop wiped entirely (WMI). Per-machine OneDrive deployed. "Block New Outlook" GPO created and linked to domain root. |
+| 2026-06-11 | DFS + second-DC planning session (plan-only). Flagged 2016 Essentials EOL; recommended domain-based DFS namespace + full writable DC at NW. |
-| 2026-05-22 | L2TP/IPsec VPN successfully deployed to MaraHomeNew, Maras-HP-Laptop, PST-SURFACE during on-site visit at Mara's house. UCG-hosted strongSwan/xl2tpd abandoned; RRAS on PST-SERVER became the VPN endpoint. UCG DNAT rules created for UDP 500/4500/ESP. Stale hosts file entries removed. pst-admin and mara passwords reset to SpiritWalk26!. BridgettePSHomeComputer offline — VPN pending. |
+| 2026-06-13 | PST-SERVER2 found past-tombstone-lifetime (224 days stale, AD err 8614, data disk missing). Executed evict+rebuild runbook: force-demote -> metadata cleanup -> D4 authoritative SYSVOL restore -> re-promote SERVER2 (DC/GC/DNS, site NW). Two healthy DCs, 0 replication errors both directions, SYSVOL State 4 on both. G: cleanup reclaimed ~131 GB (51 -> 182 GB free). Vault `server2.sops.yaml` created. |
-| 2026-05-27 | BridgettePSHomeComputer VPN deployed fully remotely via GuruRMM `user_session` context (no on-site visit). L2TP PSK set remotely. BridgetteSH added to WseRemoteAccessUsers and granted msNPAllowDialin. Logon scheduled task created for auto-connect. VPN rollout complete across all four machines. |
+| 2026-06-14 | SERVER2 static IP set (192.168.1.5/24); timezone -> Mountain; stale .127 DNS records cleaned. Gate 4 DFS-R rebuilt clean with PST-SERVER G:\Shares PRIMARY and SERVER2 C:\Shares receiver; ~221/265 GB replicated. Session ended blocked: SERVER2 began flapping (NW site stability, not DFS). Gate 4 finish deferred. |
-| 2026-06-01 | Crashed 2026-05-10 session transcript (9700a3c6) recovered by the auto-reconstructor. Primary-source log saved as `clients/peaceful-spirit/session-logs/2026-05-10-recovered-setup-radius-authentication-for-vpn-access.md`, cross-linked with the manual `2026-05-10-session.md`. Covers UCG SSH key generation, paramiko tunneling, RADIUS/NPS extraction, and vault `server.sops.yaml` creation. |
+| 2026-06-29 | File-deletion investigation initiated. Stopped MSP360 backup, staged the 6/24 10:05 AM restore point. Mtime heuristic ruled out; restore-and-local-diff adopted as authoritative. |
-| 2026-06-04 | Site-wide VPN outage: UDR Ultra (UCG-PST-CC) rebooted at 03:59 and returned without UDP 500/4500 -> 192.168.0.2 port-forward. All clients failed RAS error 789 (IPsec pre-auth, zero IKE packets reaching server). RRAS/PST-SERVER confirmed healthy (30-day uptime, services up, PSK correct). Root cause isolated to missing DNAT rule via IPsec auditing (zero IKE events on live dial). Mike re-added port-forward in UniFi controller. Bridgette connected (192.168.0.242, event 20224); Mara IPsec established. BridgettePSHomeComputer re-enrolled in GuruRMM (new UUID 01160fc8, old 074141d7 dead). PST-SERVER agent UUID confirmed 87293069; MaraHomeNew agent UUID confirmed e9645594. |
+| 2026-07-01 | Deletion-scope analysis complete: 47,749 files deleted since 6/24 10:05, ~93% duplicate cleanup, ~3,342 genuine recoverable. Incident window (10:05->12:05) had only 2 deletions. Glennda trigger = misspelled duplicate; canonical folder intact. Shelton check blocked (6/29/2025 restore point purged). Admin1/Admin2 NTFS hardening: removed incorrect Admin2-in-Admin1 nesting; Admin1 -> allow RX,W + DENY D,DC; Admin2 retained Full Control. ACL backup saved. |
 ---
 ## Backlinks
- [[projects/gururmm]] — PST-SERVER, MaraHomeNew, Maras-HP-Laptop, PST-SURFACE, BridgettePSHomeComputer enrolled (site: Country Club)
+- [[projects/gururmm]] — PST-SERVER, PST-SERVER2, MaraHomeNew, Maras-HP-Laptop, PST-SURFACE, BridgettePSHomeComputer enrolled (primary site: Country Club; PST-SERVER2 at NW)
--- a/wiki/index.md
+++ b/wiki/index.md
@@ -1,6 +1,6 @@
 # Wiki Index
-Last updated: 2026-06-30
+Last updated: 2026-07-01
 Compiled by: HOWARD-HOME/claude-main
 This wiki is LLM-maintained. Do not edit articles manually — run `/wiki-compile` to update.
@@ -32,7 +32,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks.
 | [Pavon](clients/pavon.md) | Former/archive client; GeoVision NVR surveillance; OwnCloud at 172.16.3.22 backed by Uranus; cron stacking fixed; Nextcloud migration deferred 3–6 months | 2026-05-24 |
 | [Rieusset Corp (Tom Sorensen)](clients/rieusset-corp.md) | Small business; email hosted on Neptune Exchange (4 mailboxes: tsorensen, tomrc, ojodeagua, csorensen @rieussetcorp.com); Mailprotector domain ID 57833; outbound via SBR Outbound.Sorensen connector; clipto.com allow rule added 2026-06-08 | 2026-06-08 |
 | [Rednour Law Offices](clients/rednour.md) | Law firm (break-fix/T&M, prepay 0); M365 rednourlaw.com (tenant 4a4ca18a) onboarded, 5 ComputerGuru SPs consented, no MDE license; 3 Win workstations GuruRMM-enrolled (all RED, prior MSP agents pending removal) — **all three now on Win 11** (LEGALASST + Carrie/REDNOURCARRIEVI upgraded 2026-06-29); REDNOURCARRIEVI hosts the firm's peer-to-peer SMB shares (Nick's Mac access done 2026-06-25); **Carrie's Win11 upgrade root cause = corrupt download (`ks.sys` 0x80070570 -> SAFE_OS 0x8007000D); fixed via fresh Media Creation Tool media — done in-shop, build 26200**; GuruRMM **works** on the Windows boxes (earlier "not working" disproved); macOS RMM agent still won't enroll (site code-vs-UUID bug, coord 6f2d22be); `endpointprotection.exe` = Datto AV (Defender RTP off by design); #32368 invoiced #67912 $669.55 (Nick = no charge); plaintext local-account creds from Syncro notes vaulted (clients/rednour/local-accounts) | 2026-06-30 |
-| [Peaceful Spirit Therapeutic Massage](clients/peaceful-spirit.md) | Massage therapy practice; PST-SERVER (192.168.0.2) + 5 GuruRMM agents; L2TP/IPsec RRAS VPN complete; 2026-06-04 site-wide outage resolved (UDR Ultra reboot dropped VPN port-forward, re-added in controller); BridgettePSHomeComputer re-enrolled (new UUID 01160fc8); vault drift open (pst-admin password); Syncro 278525 (Peaceful Spirit Massage) | 2026-06-04 |
+| [Peaceful Spirit Therapeutic Massage](clients/peaceful-spirit.md) | Massage therapy, two sites (Country Club + Northwest); break-fix, Syncro 278525, 31 assets; **two-DC domain** — PST-SERVER (192.168.0.2, 2016 Essentials, all FSMO) + PST-SERVER2 (192.168.1.5, rebuilt 6/13 from past-tombstone state, NW) with DFS-R (PST-DFS, ~221/265 GB) — **Gate 4 blocked: SERVER2 flapping (NW power/UPS/net)**; L2TP/IPsec RRAS VPN complete (6 GuruRMM agents); **June–July 2026 file-deletion investigation** — 47,749 files gone from `@Clients` since 6/24 but ~93% duplicate cleanup, **~3,342 genuine recoverable** from MSP360/B2 staging (Glennda trigger = misspelled duplicate, canonical folder intact; 6/29/2025 restore point purged by 365-day retention); **Admin1/Admin2 NTFS hardening** on G:\Shares\Scanned (fixed inverted group nesting; Admin1 = RX,W + deny-delete, Admin2 = Full); vault drift open (pst-admin password) | 2026-07-01 |
 | [Patriot Internal Medicine](clients/patriot-internal-medicine.md) | Medical practice, two locations (Tucson + Sonoita); GuruRMM client+sites provisioned 2026-06-18 (Tucson: NORTH-WOLF-6270, Sonoita: LIGHT-HARBOR-9617); no agents deployed yet; enrollment keys vaulted; infrastructure discovery pending | 2026-06-18 |
 | [Sombra Residential LLC](clients/sombra-residential.md) | Property management; Server2013 (actually WS2012 EOL, unpatched) + DESKTOP-UQRN4K3 GuruRMM enrolled; Transwiz migration artifacts cause Office credential prompts | 2026-05-24 |
 | [Stamback Septic](clients/stamback-septic.md) | Septic services; prepaid block ~3.5 hrs remaining; DESKTOP-BTR2AM3 + StambackLaptopNew GuruRMM enrolled; OneDrive identity wipe pattern documented | 2026-05-24 |