claudetools/wiki/clients/peaceful-spirit.md

---
type: client
name: peaceful-spirit
display_name: Peaceful Spirit Therapeutic Massage
last_compiled: 2026-07-03
compiled_by: HOWARD-HOME/claude-main (full: Syncro refresh + root-log provenance + VSS/address)
sources:
  - clients/peaceful-spirit/session-logs/2026-05-10-recovered-setup-radius-authentication-for-vpn-access.md
  - clients/peaceful-spirit/session-logs/2026-05-10-session.md
  - clients/peaceful-spirit/session-logs/2026-05-11-session.md
  - clients/peaceful-spirit/session-logs/2026-05-22-session.md
  - clients/peaceful-spirit/session-logs/2026-05-27-session.md
  - clients/peaceful-spirit/session-logs/2026-06-04-session.md
  - clients/peaceful-spirit/session-logs/2026-06/2026-06-11-mike-multisite-dfs-dc-plan.md
  - clients/peaceful-spirit/session-logs/2026-06/2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md
  - clients/peaceful-spirit/session-logs/2026-06/2026-06-14-mike-pst-gate4-dfsr-rebuild-and-static-ip.md
  - clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md
  - clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md
  - session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md
  - session-logs/2026-06/2026-06-29-mike-birthbio-repatriation-and-pst-soap-recovery.md
  - session-logs/2026-07/2026-07-02-mike-pst-reports-ezfag-tags-unifi-adoption-bardach.md
  - session-logs/2026-07/2026-07-02-mike-crowdstrike-rollout-365-appsuite.md
  - clients/peaceful-spirit/server.sops.yaml (vault)
  - clients/peaceful-spirit/server2.sops.yaml (vault)
  - clients/peaceful-spirit/vpn.sops.yaml (vault)
  - clients/peaceful-spirit/physical-access-northwest.sops.yaml (vault)
backlinks:
  - projects/gururmm
---

# Peaceful Spirit Therapeutic Massage

Massage therapy practice with two sites: Country Club (CC, primary — all server infrastructure) and Northwest (NW). On-premises Windows Server 2016 Essentials domain (PEACEFULSPIRIT.local). As of June 2026 the environment was upgraded to a two-DC architecture: PST-SERVER (CC, Server 2016 Essentials) plus PST-SERVER2 (NW, Server 2019 Standard, rebuilt June 2026 from a past-tombstone-lifetime state). DFS namespace and DFS-R replication between sites established June 2026. L2TP/IPsec VPN fully deployed to all known client machines as of 2026-05-27.

---

## Profile

- **Business name (Syncro):** Peaceful Spirit Massage (NOT "...Therapeutic Massage" — ID-based lookup required)
- **Syncro customer ID:** `278525`
- **Addresses (two sites):** CC / Country Club (primary — all server infrastructure): 2930 N Country Club Rd, Tucson AZ (Syncro primary address). NW / Northwest: 6650 N Oracle #100, Tucson AZ.
- **Primary contact:** Mara Concordia (owner/operator); generic contact email `info@bestmassageintucson.com`; personal Microsoft account `mara.concordia@gmail.com` (OneDrive). Domain user: `mara`.
- **Other key staff:** Bridgette (BridgetteSH); Christine Z (ChristineZ); Calista A (CalistaA); Leslie W (leslieW); Sarah M (SarahM); Katie B (katieb); Sharon S (SharonS); PSTAdmin.
- **Contract type:** Break-fix / T&M (verify — recent invoices per-ticket ~$150–300/visit, plus a recurring ~$195.19/month line item; no retainer contract confirmed)
- **Managed asset count:** 31
- **Open tickets:** 0 as of 2026-07-01

---

## Infrastructure

### Servers & Services

| Host | IP | Role | OS | Notes |
|---|---|---|---|---|
| PST-SERVER | 192.168.0.2 | DC (all 5 FSMO), DNS, RRAS (L2TP/IPsec VPN), NPS, Enterprise Root CA (AD CS) | Windows Server 2016 Essentials (build 14393) | Site CC. GuruRMM agent `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.75+; prior ID `6b6106a7...` retired). Win32-OpenSSH installed 2026-05-11. Machine cert: `DB71981ABE4CBA1DE96FEEEAF178F6259663B543` (CN=PST-SERVER.PEACEFULSPIRIT.local, valid 5/9/2027). Drives: C: 931 GB (OS); G: 465.7 GB data volume (ex-old-server C:, 182 GB free post-cleanup); D: 931 GB (Recovery-EXT/backup junk ~700 GB — cleanup pending). G:\Shares: Private ~154 GB, Scanned ~105 GB, ITServices ~5 GB, qbooks ~2 GB (~265 GB total). Credentials: vault `clients/peaceful-spirit/server`. |
| PST-DC-NW | 192.168.1.5 | DC (additional), GC, DNS, DFS-R receiver | Windows Server 2019 Standard (build 17763) | Site NW. New physical server installed 2026-07-02 (shipped as PST-SERVER01; renamed via RMM pre-join). Joined + promoted DC/GC/DNS (site NW) 2026-07-02, all via RMM. Static 192.168.1.5/24; DNS client 192.168.0.2 + 127.0.0.1; timezone US Mountain Standard (AZ). Single 1.8 TB C: (1849 GB free at setup). DFS-R replica at C:\Shares (staging 20 GB), member of PST-DFS since 2026-07-02 — initial sync of ~265 GB from PST-SERVER over S2S VPN in progress. GuruRMM agent `f60e9820-4a00-4598-83f7-c14085db5768` (v0.6.75, site "North West"). DSRM password: vault `clients/peaceful-spirit/dc-nw`. |
| PST-SERVER2 (DEAD, metadata cleaned 2026-07-02) | was 192.168.1.5 (now PST-DC-NW's) | Was DC/GC/DNS at NW — hardware DIED ~2026-06-14 (the "flapping"), never returned; replaced by PST-DC-NW. All AD/DNS/DFSR metadata removed 2026-07-02. | Windows Server 2019 Standard | Site NW. Static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2 + 127.0.0.1. GuruRMM agent `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. Single 1 TB NVMe, C: only (original D: physical disk gone). DFS-R replica at C:\Shares (~221 GB as of 2026-06-14; ~44 GB backlog remaining). Timezone: US Mountain Standard Time (Arizona). Rebuilt 2026-06-13 (force-demote -> metadata cleanup -> re-promote; see runbook). Credentials: vault `clients/peaceful-spirit/server2` (local admin + DSRM). [WARNING] Flapping (online ~1 min / offline several min reboot-loop pattern) at end of 2026-06-14 session — NW site power/UPS/network issue, NOT caused by DFS; PST-SERVER and data unaffected. |
| UCG-PST-CC | 192.168.0.10 (LAN) / 98.190.129.150 (WAN) | UniFi Cloud Gateway Ultra — perimeter router + DNAT for VPN | UniFi OS 5.1.15, kernel 5.4.213-ui-ipq5322 (aarch64) | Site CC. SSH: `root@192.168.0.10` via key `~/.ssh/pst-cc-ucg`; keyboard-interactive auth only. WAN SSH not accessible remotely. UCG VPN (strongSwan/xl2tpd) abandoned 2026-05-22; RRAS on PST-SERVER is the VPN endpoint. DNAT persistence: `/data/on_boot.d/10-vpn-portforward.sh`. Rebooted 2026-06-04 at 03:59, dropped VPN port-forward (see Known Issues). Credentials: vault `clients/peaceful-spirit/server`. |
| UCG-NW | 64.139.88.249 (old WAN; verify current) | UniFi gateway — NW site perimeter, S2S VPN | (verify) | NW site. Previously had OpenVPN at 64.139.88.249:1194 (TCP). S2S VPN CC<->NW confirmed up as of 2026-06-13 (ports 389/445/135/88 reachable SERVER2->SERVER). Details beyond this: (verify). Physical access: vault `clients/peaceful-spirit/physical-access-northwest`. |

### DFS Namespace & Replication

- **Domain-based DFS namespace:** `\\PEACEFULSPIRIT.local\PST-Files` -> folder `Shares`
- **Current namespace root target:** PST-SERVER only (`\\PST-SERVER\PST-Files`) — SERVER2 root target deferred pending stability
- **Current folder targets:** PST-SERVER only (`\\PST-SERVER\Shares`, Online) — SERVER2 folder target (`\\PST-SERVER2\Shares`) removed pending stability; to be re-added once SERVER2 holds stable
- **DFS-R group:** `PST-DFS`; replicated folder `Shares`
  - PST-SERVER `G:\Shares` = authoritative content; staging 20 GB
  - PST-DC-NW `C:\Shares` = receiver; staging 20 GB (member since 2026-07-02; SERVER2's dead membership removed same day)
  - Bidirectional connection configured; initial sync of ~265 GB started 2026-07-02 over S2S VPN (expect days)
- **Gate 4 deferred items (blocked on initial sync completing):** RF state 4 + backlog 0 on PST-DC-NW; share C:\Shares as `Shares`; add `\\PST-DC-NW\Shares` folder target Online; add PST-DC-NW as 2nd namespace root target (`\\PST-DC-NW\PST-Files`) for VPN-outage HA
- **Runbook:** `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`

### Domain & Identity

- **Domain:** PEACEFULSPIRIT.local (NetBIOS: PEACEFULSPIRIT)
- **AD Sites & Services:** CC site (192.168.0.0/24), NW site (192.168.1.0/24); subnets correct, site link active
- **FSMO:** all 5 roles on PST-SERVER
- **Global Catalog:** both PST-SERVER and PST-DC-NW (since 2026-07-02; formerly PST-SERVER2)
- **Domain SID base:** S-1-5-21-1105246401-3156558273-4088333098
- **Domain admins:** `sysadmin` (password: vault `clients/peaceful-spirit/server`) — domain admin account. DA credentials were passed base64-wrapped in RMM command_text during June/July rebuild sessions; rotation optional (RMM is internal).
- **CA:** PEACEFULSPIRIT-PST-SERVER-CA — Enterprise Root CA on PST-SERVER. Thumbprint: 56DAF43C60F246BF2C80A671EE9812C727D8C298 (valid to 3/8/2061). `msPKI-Certificate-Name-Flag` changed 2026-05-11 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT).
- **VPN-eligible users (WseRemoteAccessUsers, SID ...-1113):** Domain Admins (group), PSTAdmin, pst-admin, LMT, Mara, BridgetteSH. NPS grants VPN by group membership — `msNPAllowDialin=TRUE` alone is not sufficient.
- **pst-admin:** domain user (not domain admin); in WseRemoteAccessUsers; VPN-eligible. Shared VPN credential for Mara's machines.
- **AD security groups (custom):**
  - **Admin1** (Global Security): CalistaA, ChristineZ, leslieW, SarahM — allow `RX,W` + DENY `(D,DC)` on G:\Shares\Scanned (read/write/edit only; no delete, rename, or ownership change). Was previously Full Control; hardened 2026-07-01.
  - **Admin2** (Global Security): BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS — Full Control on G:\Shares\Scanned. Admin2 was formerly (incorrectly) nested inside Admin1; nesting removed 2026-07-01.
- **OneDrive:** pst-admin uses personal OneDrive (mara.concordia@gmail.com, cid: 25f0851177ceabfd). Per-machine OneDrive deployed to Maras-HP-Laptop.
- **Email / M365:** (verify — no M365 tenant found; practice likely uses personal or third-party email)
- **GPO:** "Block New Outlook" — GUID {577028AF-0901-4BDF-A283-CD1156F313D9}, linked to domain root.
- **SYSVOL backups (2026-06-13):** `C:\PST-Backup\SYSVOL-Policies-20260613-1611` and `C:\PST-Backup\GPO-20260613-1611` (11 GPOs) on PST-SERVER — keep until rebuild confirmed long-term stable.

### Network

- **Country Club (CC) site:** WAN 98.190.129.150 (Cox); LAN 192.168.0.0/24; DC/DNS 192.168.0.2 (PST-SERVER); UCG 192.168.0.10
- **Northwest (NW) site:** LAN 192.168.1.0/24; DC/DNS 192.168.1.5 (PST-SERVER2); WAN (verify current; old OpenVPN was at 64.139.88.249); S2S VPN to CC confirmed up 2026-06-13
- **VPN (L2TP/IPsec, client-to-server):**
  - Endpoint: PST-SERVER RRAS at 192.168.0.2, exposed via UCG-PST-CC DNAT (UDP 500, 4500, ESP)
  - PSK: vault (`clients/peaceful-spirit/vpn.sops.yaml`)
  - Auth: MSCHAPv2. Mara's machines connect as shared user `pst-admin`; BridgettePSHomeComputer connects as `BridgetteSH` via SSO
  - NPS RADIUS shared secret for client UCG-PST-CC (192.168.0.10): vault (`clients/peaceful-spirit/server.sops.yaml`)
  - IP pool: 192.168.0.240+ (observed: .241, .242, .243, .248, .249)
  - VPN profile name on clients: "Peaceful Spirit VPN" (AllUserConnection, split tunnel, 192.168.0.0/24 route, NRPT for .peacefulspirit.local -> 192.168.0.2)
  - PST-SERVER registry: `AssumeUDPEncapsulationContextOnSendRule=2` (PolicyAgent), `DefaultPSK` set in L2TP parameters
  - UCG persistence: `/data/on_boot.d/10-vpn-portforward.sh`

### Client Workstations

| Machine | Role | GuruRMM Agent ID | Notes |
|---|---|---|---|
| MaraHomeNew | Mara's home desktop | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` (v0.6.52; prior `c778b6a3...` retired) | Domain-joined. VPN working. Machine cert: D067E07B (valid 5/10/2027). Connects as pst-admin. |
| Maras-HP-Laptop | Mara's HP laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | Domain-joined. VPN deployed 2026-05-22. OneDrive per-machine deployed. Connects as pst-admin. |
| PST-SURFACE | Surface device | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | Domain-joined. VPN deployed 2026-05-22. Connects as pst-admin. |
| BridgettePSHomeComputer | Bridgette's home PC | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` (v0.6.49; re-enrolled 2026-06-04; old `074141d7...` dead) | Domain-joined. VPN deployed remotely via GuruRMM user_session 2026-05-27. Connects as BridgetteSH (SSO). Logon scheduled task `Connect Peaceful Spirit VPN` auto-connects ~20s after sign-in. |

---

## GuruRMM Enrollment

- **Client name in RMM:** Peaceful Spirit
- **Client ID:** `00015eae-50e5-4102-93fa-ab0fdb135c08`
- **Primary site name:** Country Club
- **Primary site ID:** `7b32983d-982a-4a5c-af07-45a23453f589`

**Enrolled agents:**

| Host | Agent ID | Version | Notes |
|---|---|---|---|
| PST-SERVER | `87293069-33b6-45e8-a68f-6811216cdb96` | v0.6.75+ | Active; confirmed 2026-07-01. Prior `6b6106a7...` retired. |
| PST-SERVER2 | `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65` | — | REMOVED — agent record deleted from RMM (noticed 2026-07-02); machine being replaced by PST-DC-NW. |
| PST-DC-NW | `f60e9820-4a00-4598-83f7-c14085db5768` | v0.6.75 | NW site ("North West" in RMM). Enrolled 2026-07-02; renamed from PST-SERVER01 via RMM same day. |
| MaraHomeNew | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` | v0.6.52 | Active; confirmed 2026-06-04. |
| Maras-HP-Laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | — | — |
| PST-SURFACE | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | — | — |
| BridgettePSHomeComputer | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` | v0.6.49 | Re-enrolled 2026-06-04. |

---

## Data Store & Backup

### Data Location

Client SOAP-note and business files reside on **PST-SERVER G:\Shares**. The @Clients tree is doubly-nested: `G:\Shares\Scanned\@Clients\@Clients`. File counts as of 2026-07-01: ~142,335 files / ~72 GB in the live @Clients tree. Total G:\Shares ~265 GB. G: is the old server's former C: drive (live PST-SERVER OS runs from C:).

### MSP360 / Backblaze B2 Backup

- **Plan name:** "Files Backup 2025" (Files Backup type; ForeverForward; retention 365 days)
- **MSP360 account:** ACG-PST `084b5069-d634-434b-84a2-971b1dcb4b43`
- **Bunch ID:** `6a121575-84a0-4e98-9c0f-4a656d1a5132` (prefix: PST-SERVER)
- **Destination:** Backblaze B2
- **cbb.exe path:** `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`; logs `C:\ProgramData\Online Backup\Logs\`
- **Known restore points:** `20260624170506` (6/24 10:05 AM, pre-incident repair source); `20260624190522` (6/24 12:05 PM). Oldest `20250629170034` (6/29/2025) **PURGED** as of 2026-07-01 (past 365-day retention) — year-ago backup unavailable.
- **Status 2026-07-01:** running normally (the 6/29 stop-for-restores self-resumed).
- **Caveat:** `cbb list` is unreliable on comma/space folder paths (false zeros, timeouts on large trees). Use restore-to-staging + local diff for any deletion-scope investigation.

### VSS Shadow Copies (PST-SERVER G:)

Local point-in-time recovery / self-service "Previous Versions" on the data volume, deployed as the near-line complement to the B2 backup (confirmed 2026-07-02). Volume Shadow Copies enabled on **G:** with a **69.8 GB** storage cap and roughly **4 snapshots/day** (observed schedule ~6a / 12p / 1p / 6p). Provides fast in-place rollback of individual files/folders without a B2 restore, and is the "Option 2" alternative to the (NTFS-impossible) recycle-bin design Mara requested. Note: after the 2026-06-13 trim, older snapshots were pruned — earliest snapshot ranges have been observed as recent as 6/25–6/28, so VSS is a short-window safety net, not long-term retention (that role is B2 / MSP360). The Security event log backing the deletion audit is sized to **128 MB max** on PST-SERVER.

### NTFS Access Control (G:\Shares\Scanned)

ACL root is `G:\Shares\Scanned`; permissions inherit to `@Clients` and subdirectories. Hardened 2026-07-01. ACL backup on server: `C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt`.

| Group | Members | Effective Permissions |
|---|---|---|
| Admin1 | CalistaA, ChristineZ, leslieW, SarahM | Allow `(OI)(CI)(RX,W)` + **DENY** `(OI)(CI)(D,DC)` — read/write/edit only; no delete, rename, permission or ownership change |
| Admin2 | BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS | `(OI)(CI)(F)` — Full Control |

**Caveat:** the `(D,DC)` deny on Admin1 also blocks rename and app save patterns that delete-then-write. If Admin1 users report inability to rename or save, carve an individual exception. Reversal: `Add-ADGroupMember Admin1 -Members Admin2`; `icacls "G:\Shares\Scanned" /remove:d "PEACEFULSPIRIT\Admin1"` then restore allow via `/grant`.

### 2025 Crash & File Corruption (investigated 2026-07-02)

June 4, 2025: the OLD server (hostname **NEWSERVER**, domain PSTMC) crashed; a second failure during restore corrupted file-table→data mappings. Corrupted files carry the original name/size but contain foreign data blocks (MPEG-2 video streams `00 00 01 BA`, ownCloud sync-log text, etc.). Mara's July-2026 Claude analysis of `_C_IC_Payments_2-12_to_2-26-25.xlsx` (62,993 bytes, MPEG content) is in Mike's Documents (`Data recovery covo Claude.docx`).

**Damage inventory (live G:\Shares, Mike's `[C]` filename prefix = confirmed corrupt):** 5,044 files, ~2.9 GB — 4,858 PDF, 108 xlsx, 25 docx; 5,027 under Private (~129 across Accounting Docs year folders 2016-2025). Full list: PST-SERVER `C:\PST-Recovery\corrupted-file-list-20260702.txt` (size TAB path). In the 2024/2025 payroll trees specifically only 5 of 161 Office files are corrupt, incl. Mara's IC Payments 2-12–2-26-25 (MPEG in BOTH G: `[C]` copy and D:\Shares copy — corruption predates all copies) and IC Payments 5-13–5-27-25 (content = ownCloud log text; Mara built a partial replacement "(clare and alice corrections ONLY)").

**Recovery corpus on PST-SERVER D: (931 GB, label "VM Files" — the old server's drive; FROZEN, no cleanup):**
- `D:\Shares` — crash-era share tree (2025-06-03), incl. full Accounting Docs 2013-2025; mostly intact.
- `D:\Recovery2019` — restore attempt (2025-06-26), Private+Scanned.
- `D:\Recovery-EXT` — ownCloud/Syncthing copy circa 2021 (complete IC Payments 2018 – mid-2021).
- `D:\Unknown folder` — file-carving output (2025-06-03): 101,552 files at root (`[000024].xlsx` bracket names; 20,345 .doc, 1,121 .xls, 9,274 .pdf) + hex-named subdirs; candidate source for recovering the `[C]` files.
- `D:\Users` — old-server profiles (mconcordia, hallb, lmt, pst-admin, Administrator.PSTMC/NEWSERVER).

**No cloud copy predates the crash:** MSP360 plan "Files Backup 2025" was created 2025-06-04 (crash day); B2 fully enumerated 2026-07-02 — generic bucket (MSPBackups20200311) holds other clients only (FSG-SRV-02, UC2-SERVER, LAB-BECKY, DROBO, VWP-SERVER, SALMON/TROUT), ACG-PST has only post-crash data, ACG-Internal only NEPTUNE.

**Gotcha:** `G:\Shares` ROOT denies SYSTEM directory enumeration (Access denied) — recursive scans from the root silently return nothing; enumerate the children (`Private`, `Scanned`, ...) directly.

**Carve-identification results (2026-07-02): all 4 MPEG-corrupted 2024/2025 payroll files RECOVERED** from `D:\Unknown folder` carved output. Method: fingerprint sharedStrings vocabulary from adjacent-period good files → score ~770 carved xlsx by token overlap → confirm by exact byte-size match with the corrupted original (NTFS keeps true size) + date-serial range / sheet title inside. Staged as copies (NOT yet placed into the live share — needs Excel-open validation + Mara sign-off):
`C:\PST-Recovery\carve-identified\` on PST-SERVER:
- `IC Payments 2-12 to 2-26-25` <- `[006001].xlsx` (62,993 B; serials exactly 2025-02-12..02-26) — **the file Mara has chased for a year**
- `Payroll Report 4-29-2025 - 5-12-2025` <- `[007234].xlsx` (53,480 B; sheet title contains the full period)
- `Tips Report 12-30--1-12-25` <- `[007102].xlsx` (20,302 B; serials 2024-12-30..2025-01-12; original name said "12-13" — likely Mara typo)
- `Triwest & Insurance ... 2024-11-15 check` <- `[006975].xlsx` (19,904 B; October-2024 data, consistent with 11/15 check run)
Also spotted in the carve dump with its original name: `IC Payments 2024-08-12--2024-08-27.xlsx` (253,949 B) — a period absent from the 2024 tree. Scoring artifacts: `C:\PST-Recovery\carve-match-xlsx.txt`.

**Remaining:** the 5th damaged file `IC Payments 5-13 to 5-27-25` (content = ownCloud log; no exact-size carve hit; Mara's "(clare and alice corrections ONLY)" partial rebuild exists) — nearest candidate `[005037].xlsx` (56,344 B, score 58) unverified. And the broader ~5,039 other `[C]` files (mostly PDFs) — same identify-by-size/content approach can be batch-applied to the carved .pdf pool (9,274 root PDFs) if Mike/Mara want to chase them.

### Deletion Investigation (June–July 2026)

A report that client files disappeared (trigger: the "Glennda" folder) prompted a staged restore-and-diff investigation. The 6/24 10:05 AM restore point was staged to `C:\PST-Recovery\PreDelete-0624` (~99 GB). Authoritative diff: **47,749 files deleted from @Clients since 6/24 10:05**; ~93% intentional duplicate cleanup (33,711 in folders labeled "duplicate DO NOT USE or delete"; ~10,696 in nested misfile-buckets A\A, D\A, P\O, H\I whose canonical client folders remain live). Genuine loss estimate: **~3,342 files**, recoverable via no-overwrite copy-back from staging (not yet executed — awaiting Mike/Mara approval; writes to live HIPAA data). The 10:05->12:05 PM window had only 2 deletions (Ballard, Kathy and Rivera, Anthony SOAP PDFs) — mass deletion occurred later. Glennda trigger: `EDWARDS, GLENDA` (single-N, 79 files, deleted) was a misspelled duplicate of the active canonical `EDWARDS, GLENNDA VA REFERRAL` (double-N, 127 files, live and growing). Shelton report: only 6 old Shelton files exist (2011–2015), loose in `S\`, CreationTime 2025-06-02 (migration), unchanged since 6/24 — not a 2026 deletion; the 6/29/2025 restore point needed for further check has been purged. Staging artifacts (~200 GB, removable after recovery decision): `C:\PST-Recovery\{PreDelete-0624, PostDelete-0624, authdiff, incidentdiff, acl-backup-scanned-20260701-072725.txt}`.

**Standing deletion audit (the "Mara audit log").** Object-access auditing (SACL: Everyone / Delete+DC / Success on `G:\Shares\Scanned`) feeds a daily scheduled task **`PST Deletion Report (Daily)`** → `C:\PST-Tools\PST-DeletionReport.ps1` (runs as SYSTEM, 06:30). It harvests Security events 4660/4663 into a per-day HTML report of who deleted / renamed / moved files under `G:\Shares\Scanned` (server + backup activity excluded; 90-day retention). This is the ongoing record Mara reviews for further deletions. **Report output location: `G:\Shares\Private\Partner Review\Legal Documents - DO NOT DELETE\_Deletion Reports`** — moved there 2026-07-02 (from the original `G:\Shares\Scanned\_Deletion Reports`); pre-change script backup at `C:\PST-Tools\PST-DeletionReport.ps1.bak-20260702`. Only `$OutDir` was repointed; the monitored root (`$Root = G:\Shares\Scanned`) is unchanged. PST-SERVER is reachable for this kind of change via GuruRMM (agent `87293069-...`) when the site VPN is down.

---

## Access

- **PST-SERVER SSH:** `ssh -i ~/.ssh/id_ed25519 sysadmin@192.168.0.2` — requires L2TP VPN to CC site active. Win32-OpenSSH at `C:\Program Files\OpenSSH\OpenSSH-Win64\`. SCP paths use Unix format (`/C:/path/to/file`).
- **PST-SERVER2 SSH:** `ssh sysadmin@192.168.1.5` — requires S2S VPN or physical NW site access. Local admin creds: vault `clients/peaceful-spirit/server2`.
- **UCG-PST-CC SSH (LAN only):** `ssh -i ~/.ssh/pst-cc-ucg root@192.168.0.10` — keyboard-interactive auth only (plink `-pw` fails; use paramiko kb_handler or interactive terminal). WAN SSH not accessible remotely. Requires VPN, on-site, or UniFi cloud portal (unifi.ui.com).
- **GuruRMM (external):** https://rmm.azcomputerguru.com
- **Physical access — NW site:** lockbox, main-door keypad, alarm-disarm code in vault `clients/peaceful-spirit/physical-access-northwest.sops.yaml` (codes never in plaintext here).
- **Vault paths:**
  - `clients/peaceful-spirit/server.sops.yaml` — PST-SERVER credentials (sysadmin DA) and UCG-PST-CC details.
  - `clients/peaceful-spirit/server2.sops.yaml` — PST-SERVER2 local admin + DSRM passwords. Created 2026-06-13.
  - `clients/peaceful-spirit/vpn.sops.yaml` — VPN PSK, pst-admin credentials, network details. [WARNING] VAULT DRIFT: vault lists pst-admin password as one value but the wiki records a 2026-05-22 reset to another — reconcile with Mara and update the stale entry.
  - `clients/peaceful-spirit/physical-access-northwest.sops.yaml` — NW site lockbox, door, alarm codes.

---

## Patterns & Known Issues

- **Set-VpnConnection -L2tpPsk cannot run via RMM (SYSTEM context).** Windows enforces interactive mode for PSK registration. An admin must run it manually per machine in an interactive session (one-time). Exception: the `user_session` command context in GuruRMM allows it — validated on BridgettePSHomeComputer 2026-05-27.

- **NRPT instead of VPN DNS suffix push.** `Add-VpnConnectionTriggerDnsConfiguration` fails for AllUserConnection profiles. Use `Add-DnsClientNrptRule -Namespace ".peacefulspirit.local" -NameServers "192.168.0.2"` instead.

- **cmdkey as SYSTEM for pre-login credential persistence.** Machine credential store entries (cmdkey in SYSTEM context) are available at the Windows login screen; per-user cmdkey entries are not.

- **Stale hosts file.** During 2026-05-22 on-site, MaraHomeNew (and likely others) had a stale hosts entry mapping PST-SERVER to 72.194.62.5 (Mara's router's bogus DNS). A GuruRMM cleanup script was deployed; the path encoding bug (`driverstc`) means it may not have fully run on all machines — verify if resolution issues recur.

- **UDR Ultra reboot can silently drop the VPN port-forward (site-wide outage risk).** Confirmed 2026-06-04: UCG-PST-CC rebooted at 03:59 and returned without the UDP 500/4500 -> 192.168.0.2 DNAT, taking the site VPN offline with error 789 (IKE packets silently dropped at the edge). The `/data/on_boot.d/10-vpn-portforward.sh` persistence script was present but the UniFi OS 5.1.15 schema migration appears to have superseded it. After any site-wide error 789, check the UDR port-forward in the UniFi controller FIRST — IPsec auditing on the server (zero IKE events) is the confirmatory test. Open: verify the re-added rule survives a deliberate reboot; add a DDNS hostname so the hardcoded Cox WAN IP is not a single point of failure.

- **UCG iptables DNAT required — UniFi Traffic Rules are firewall-allow only, NOT DNAT.** Port-forward rules must be managed via the UniFi controller UI; the on_boot CLI script is a legacy fallback and may not persist on UniFi OS 5.1.15+. Verify iptables live after a reboot.

- **UCG SSH unreachable from office WAN.** Remote UCG administration goes through GuruRMM (for PST-SERVER) or the UniFi cloud portal (for UCG itself). LAN SSH (192.168.0.10) requires keyboard-interactive auth; plink password auth fails.

- **GuruRMM command_type — use `powershell` or `shell`, NOT a made-up type (RESOLVED 2026-06-12).** The agent's `CommandType` enum accepts only `shell`, `powershell`, `python`, `script`, `claude_task` (plus alias `cmd` -> shell). An unknown `command_type` fails the agent's whole-message JSON parse and the command is silently dropped — looks like a network black-hole.

- **Machine cert template (PEACEFULSPIRIT-PST-SERVER-CA).** `msPKI-Certificate-Name-Flag` changed from `0x18000000` to `0x1` (ENROLLEE_SUPPLIES_SUBJECT) 2026-05-11. New machine certs use the CSR Subject/SAN. RRAS UserAuthProtocolAccepted now includes Certificate.

- **OneDrive KFM on WSE folder-redirected profiles.** WSE machines had non-standard GUID variants in User Shell Folders. `SHSetKnownFolderPath` must be called with `flags=0` (not 0x4000) in user session context. If KFM still fails after registry cleanup, wipe the profile and redeploy per-machine OneDrive (`/allusers`).

- **pst-admin vs sysadmin distinction.** `pst-admin` is a domain user (VPN-eligible); `sysadmin` is domain admin. Many early failures came from using pst-admin creds for DA operations.

- **NPS grants VPN by WseRemoteAccessUsers group membership, not msNPAllowDialin alone.** The NPS policy condition is SID-based (`...-1113`). Both group membership AND msNPAllowDialin are required; missing group = error 812.

- **cmdkey credential not used by rasdial for PPP auth.** The machine-store cmdkey entry is NOT consulted for PPP auth. No-arg `rasdial` sends the wrong principal (SYSTEM -> error 691; logged-in user without explicit credential -> error 812). Use the logon scheduled task (BridgetteSH) or the AllUserConnection cmdkey path (pst-admin machines).

- **NAT-T registry key required on all client machines.** `AssumeUDPEncapsulationContextOnSendRule=2` under `HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent` must be set AND the machine rebooted. Missing key = error 809 (was the BridgettePSHomeComputer symptom).

- **PST-SERVER2 flapping at NW site (OPEN as of 2026-06-14).** After DFS-R initial replication reached ~221 GB, SERVER2 went into a reboot-loop (online ~1 min, offline several min). Pull System log events 41 (kernel-power), 6008 (unexpected shutdown), 1074 for cause — likely NW power/UPS/hardware/network. PST-SERVER (source) unaffected; no data risk. Gate 4 finish is blocked on this.

- **Duplicate and misspelled client folders make deletion-scope analysis noisy.** The @Clients tree contains folders labeled "duplicate DO NOT USE or delete" and nested misfile buckets (A\A, D\A, P\O, H\I) alongside legitimate client folders. The Glennda/Glenda case (misspelled duplicate deleted, canonical intact) is recurring. When investigating any deletion report: (1) restore-to-staging + local diff is the only trustworthy method (`cbb list` is unreliable on these paths); (2) verify canonical folders before concluding data is lost; (3) count duplicate-labeled and nested-bucket files separately from genuine deletions.

- **Admin1 Delete-deny also blocks rename and delete-then-write saves.** The `(OI)(CI)(DENY)(D,DC)` ACE on G:\Shares\Scanned for Admin1 prevents deletion, rename, and any app save pattern that internally deletes-then-recreates. If CalistaA/ChristineZ/leslieW/SarahM report inability to rename or save, add an individual icacls exception. Reversal in the 2026-07-01 session log.

- **Remove-DfsrMember / Add-DfsrMember contact the member machine.** `Remove-DfsrMember` on a DEAD member fails with "network path was not found" — remove the `msDFSR-Member` (and any `msDFSR-Connection` whose `fromServer` references it) AD objects under `CN=DFSR-GlobalSettings,CN=System,...` directly. Conversely `Set-DfsrMembership` from SYSTEM on PST-SERVER fails with "Security cannot be set on the replicated folder. Access is denied" (writes to the OTHER member's AD subtree) — use the DA-cred `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local` pattern.

- **RMM inline JSON dispatch mangles backslash-containing payloads.** Inline `command` strings with `\\` (registry paths, `C:\\...`) fail the server-side parse (jq sees non-JSON error response). Use `ps-encoded.sh` (script file + -EncodedCommand) for anything with backslashes/quotes, or forward slashes (`C:/Shares`) for trivial paths.

- **vault.sh get-field returns literal "null" for nested credential fields.** `vault.sh get-field clients/peaceful-spirit/server credentials.password` returns the string `"null"`. Use `vault.sh get` (full read) and extract manually.

- **AD writes via RMM require DA creds using FQDN (not localhost).** `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...}` works; `-ComputerName localhost -Credential` fails with a Kerberos SPN error. Use the FQDN for any domain/DFS/DFSN/DFSR write over RMM.

- **IP address change via RMM is unreliable.** A `New-NetIPAddress` over the agent was killed mid-NIC-blip and reverted to DHCP. Use a one-shot scheduled task that applies the change a few seconds after the command returns, or do it on console/on-site (as done for SERVER2 2026-06-14).

- **Past-tombstone-lifetime DC must never resume replication.** SERVER2 was disconnected past the tombstone lifetime. Correct remediation is force-demote -> metadata cleanup on the authoritative DC -> fresh re-promote (never resume/fix replication). Runbook: `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`.

---

## Active Work

As of 2026-07-01 session end:

- **VPN rollout: COMPLETE** across all four client machines (as of 2026-06-04).
- **[OPEN] PST-DC-NW bring-up (2026-07-02).** New physical server (shipped as PST-SERVER01) replacing PST-SERVER2 (hardware died ~2026-06-14, permanently) as the NW DC + DFSR partner. DONE: enrolled in GuruRMM (agent `f60e9820...`, client Peaceful Spirit, site "North West"); renamed PST-SERVER01 -> PST-DC-NW + rebooted via RMM 2026-07-02 (pre-domain-join); static IP set (per Mike 2026-07-02). DONE 2026-07-02, all via RMM: PST-SERVER2 metadata cleanup (see Patterns), then PST-DC-NW domain join, DC/GC/DNS promotion (site NW; verified: all DC services running, SYSVOL state 4, repadmin 0/5 fails both directions, advertising GC/KDC/DNS/timeserver), and PST-DFS re-add (member + bidirectional connection + membership C:\Shares staging 20 GB). DSRM password vaulted at `clients/peaceful-spirit/dc-nw`. REMAINING (Gate 4 finish, blocked on initial sync): (1) monitor initial DFS-R sync ~265 GB PST-SERVER -> PST-DC-NW over S2S VPN (days; watch `Get-DfsrBacklog` / RF state 2->4), (2) once state 4 + backlog 0: add `\\PST-DC-NW\Shares` folder target Online in the DFS namespace, (3) add PST-DC-NW as 2nd namespace root target (`\\PST-DC-NW\PST-Files`) for VPN-outage HA, (4) share C:\Shares on PST-DC-NW (SMB share `Shares`) before the folder target, (5) dcdiag clean both DCs. If SERVER2 is still a live DC, demote/metadata-clean it properly before or after cutover (never leave a stale DC — see tombstone-lifetime pattern).
- **[SUPERSEDED by PST-SERVER01] PST-SERVER2 NW site stability (was BLOCKER for Gate 4).** Reboot-loop flapping (System log 41/6008/1074), likely on-site power/UPS/hardware — resolved by hardware replacement rather than diagnosis.
- **[OPEN] Gate 4 finish (blocked on SERVER2 stable):** drain ~44 GB DFS-R backlog; re-add SERVER2 folder target Online; add SERVER2 as 2nd namespace root target for HA; verify both RFs State 4, dcdiag clean.
- **[OPEN] Deletion recovery — ~3,342 genuine files.** No-overwrite robocopy copy-back from `C:\PST-Recovery\PreDelete-0624` (excluding duplicate/nested-bucket trees). Awaiting Mike/Mara go — writes to live HIPAA data.
- **[OPEN] Glennda single-N duplicate confirmation.** Verify the deleted `EDWARDS, GLENDA` (79 files) had zero unique content vs live `EDWARDS, GLENNDA` (127 files).
- **[OPEN] Shelton SOAP notes.** Year-ago restore point purged; needs client input (were the notes ever scanned? active "Sheldon" family nearby — possible mishearing).
- **[OPEN] Admin1 ACL watch.** Monitor the 4 users for rename/save issues (Delete-deny side effect).
- **[OPEN] PST-Recovery staging cleanup (~200 GB)** once recovery decision finalized.
- **[OPEN] UDR port-forward reboot-persistence test.**
- **[OPEN] DDNS for VPN endpoint** (hardcoded Cox WAN 98.190.129.150).
- **[OPEN] Vault drift — pst-admin password** (vpn.sops.yaml vs 2026-05-22 reset). Verify with Mara.
- **[FROZEN - DO NOT CLEAN] D: on PST-SERVER** — is NOT junk; it is the June-2025 crash recovery corpus (see "2025 Crash & Corruption" section). No deletion until the corruption-recovery effort concludes.
- **[OPEN] PST-SERVER temp/staging cleanup:** `C:\PST-Backup\*` (SYSVOL/GPO backups) once rebuild confirmed stable; `C:\ProgramData\` cert-enroll scratch (*.inf/*.req/*.cer/*.pfx, gen_certs.ps1, etc.); temp firewall rules TEMP-CertEnroll-RPC / TEMP-CertEnroll-DCOM.
- **[OPEN] Backup synthetic-full confirmation** — confirm "Files Backup 2025" completes cleanly after the stop/resume.
- **[DEFERRED] Machine cert VPN path (IKEv2)** — certs/PFXs exist (MaraHomeNew D067E07B, Maras-HP-Laptop 4CADDE8F, PST-SURFACE 197FF22A); superseded by L2TP. Complete, abandon, or revoke.
- **[DEFERRED] Parity decision** — Mara's machines use shared pst-admin; Bridgette uses her own account. Consider per-user auth for a cleaner audit trail.
- **[DEFERRED] Optional sysadmin rotation** (DA passed base64 in RMM command_text; recoverable from RMM DB; RMM internal-only).
- **[DEFERRED] Pre-login VPN verification** on Maras-HP-Laptop and PST-SURFACE.
- **[DEFERRED] 2016 Essentials EOL** — PST-SERVER hits end of support Jan 2027; plan replacement (2022/2025 Standard, plain AD DS) vs extend-as-is.

---

## History Highlights

| Date | Event |
|---|---|
| 2026-05-10 | GuruRMM agent installed on PST-SERVER. IKEv2 error 812 diagnosed (NPS rejecting nonexistent user `apst-admin` typo). Agents enrolled on MaraHomeNew, Maras-HP-Laptop, PST-SURFACE; IKEv2 "Peaceful Spirit VPN" profiles deployed. |
| 2026-05-11 | Machine cert auth working on MaraHomeNew. Win32-OpenSSH installed on PST-SERVER. msPKI-Certificate-Name-Flag -> 0x1. OneDrive KFM WSE GUID fix. pst-admin profile on Maras-HP-Laptop wiped; per-machine OneDrive deployed. "Block New Outlook" GPO created. |
| 2026-05-22 | L2TP/IPsec VPN deployed to MaraHomeNew, Maras-HP-Laptop, PST-SURFACE on-site. UCG strongSwan/xl2tpd abandoned; RRAS on PST-SERVER became the endpoint. UCG DNAT rules created. Stale hosts entries removed. pst-admin/mara passwords reset. |
| 2026-05-27 | BridgettePSHomeComputer VPN deployed fully remotely via GuruRMM user_session. Logon scheduled task for auto-connect. VPN rollout complete across all four machines. |
| 2026-06-04 | Site-wide VPN outage: UCG-PST-CC rebooted at 03:59, returned without the DNAT; all clients failed error 789. PST-SERVER healthy. Root cause isolated via IPsec auditing (zero IKE events). Port-forward re-added. BridgettePSHomeComputer re-enrolled (new UUID 01160fc8). |
| 2026-06-11 | DFS + second-DC planning session (plan-only). Flagged 2016 Essentials EOL; recommended domain-based DFS namespace + full writable DC at NW. |
| 2026-06-13 | PST-SERVER2 found past-tombstone-lifetime (224 days stale, AD err 8614, data disk missing). Executed evict+rebuild runbook: force-demote -> metadata cleanup -> D4 authoritative SYSVOL restore -> re-promote SERVER2 (DC/GC/DNS, site NW). Two healthy DCs, 0 replication errors both directions, SYSVOL State 4 on both. G: cleanup reclaimed ~131 GB (51 -> 182 GB free). Vault `server2.sops.yaml` created. |
| 2026-06-14 | SERVER2 static IP set (192.168.1.5/24); timezone -> Mountain; stale .127 DNS records cleaned. Gate 4 DFS-R rebuilt clean with PST-SERVER G:\Shares PRIMARY and SERVER2 C:\Shares receiver; ~221/265 GB replicated. Session ended blocked: SERVER2 began flapping (NW site stability, not DFS). Gate 4 finish deferred. |
| 2026-06-29 | File-deletion investigation initiated. Stopped MSP360 backup, staged the 6/24 10:05 AM restore point. Mtime heuristic ruled out; restore-and-local-diff adopted as authoritative. |
| 2026-07-01 | Deletion-scope analysis complete: 47,749 files deleted since 6/24 10:05, ~93% duplicate cleanup, ~3,342 genuine recoverable. Incident window (10:05->12:05) had only 2 deletions. Glennda trigger = misspelled duplicate; canonical folder intact. Shelton check blocked (6/29/2025 restore point purged). Admin1/Admin2 NTFS hardening: removed incorrect Admin2-in-Admin1 nesting; Admin1 -> allow RX,W + DENY D,DC; Admin2 retained Full Control. ACL backup saved. |
| 2026-07-02 | Standing deletion audit operationalized: daily `PST Deletion Report` task (SACL 4660/4663 on G:\Shares\Scanned -> per-person HTML). Report output relocated to the legal/partner-review folder `G:\Shares\Private\Partner Review\Legal Documents - DO NOT DELETE\_Deletion Reports` (backup of the script kept). Change made via GuruRMM (site VPN was down); validated by a test run (report written, 6 items). New server installed at NW to replace flapping PST-SERVER2 as NW DC + DFSR partner: enrolled in GuruRMM (agent `f60e9820...`, site "North West") and renamed PST-SERVER01 -> PST-DC-NW + rebooted via RMM, pre-domain-join (workgroup). SERVER2's agent record found deleted from RMM. PST-SERVER2 (dead since ~6/14) metadata-cleaned from AD via RMM: DFSR member/connection objects, NTDS Settings + config server object, DC computer account, 18 stale DNS records; verified single-DC clean state. Then PST-DC-NW full bring-up, all via RMM: timezone AZ, DNS -> .0.2, domain join + reboot, AD DS role + Install-ADDSDomainController (DC/GC/DNS, site NW) + reboot, post-checks green (SYSVOL state 4, repadmin 0/5 fails both ways, advertising GC/KDC/DNS). Re-added to PST-DFS (member, bidirectional connection, membership C:\Shares staging 20 GB — Set-DfsrMembership needed DA Invoke-Command, not SYSTEM). Initial ~265 GB sync started. DSRM password vaulted (`clients/peaceful-spirit/dc-nw`). |

---

## Backlinks

- [[projects/gururmm]] — PST-SERVER, PST-SERVER2, MaraHomeNew, Maras-HP-Laptop, PST-SURFACE, BridgettePSHomeComputer enrolled (primary site: Country Club; PST-SERVER2 at NW)