azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fa4867580fdd8f02ec8453f9692bb68f2c93f596
claudetools/wiki/clients/peaceful-spirit.md

40 KiB
Raw Blame History

type: client name: peaceful-spirit display_name: Peaceful Spirit Therapeutic Massage last_compiled: 2026-07-03 compiled_by: HOWARD-HOME/claude-main (full: Syncro refresh + root-log provenance + VSS/address) sources: - clients/peaceful-spirit/session-logs/2026-05-10-recovered-setup-radius-authentication-for-vpn-access.md - clients/peaceful-spirit/session-logs/2026-05-10-session.md - clients/peaceful-spirit/session-logs/2026-05-11-session.md - clients/peaceful-spirit/session-logs/2026-05-22-session.md - clients/peaceful-spirit/session-logs/2026-05-27-session.md - clients/peaceful-spirit/session-logs/2026-06-04-session.md - clients/peaceful-spirit/session-logs/2026-06/2026-06-11-mike-multisite-dfs-dc-plan.md - clients/peaceful-spirit/session-logs/2026-06/2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md - clients/peaceful-spirit/session-logs/2026-06/2026-06-14-mike-pst-gate4-dfsr-rebuild-and-static-ip.md - clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md - clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md - session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md - session-logs/2026-06/2026-06-29-mike-birthbio-repatriation-and-pst-soap-recovery.md - session-logs/2026-07/2026-07-02-mike-pst-reports-ezfag-tags-unifi-adoption-bardach.md - session-logs/2026-07/2026-07-02-mike-crowdstrike-rollout-365-appsuite.md - clients/peaceful-spirit/server.sops.yaml (vault) - clients/peaceful-spirit/server2.sops.yaml (vault) - clients/peaceful-spirit/vpn.sops.yaml (vault) - clients/peaceful-spirit/physical-access-northwest.sops.yaml (vault) backlinks: - projects/gururmm

Peaceful Spirit Therapeutic Massage

Massage therapy practice with two sites: Country Club (CC, primary — all server infrastructure) and Northwest (NW). On-premises Windows Server 2016 Essentials domain (PEACEFULSPIRIT.local). As of June 2026 the environment was upgraded to a two-DC architecture: PST-SERVER (CC, Server 2016 Essentials) plus PST-SERVER2 (NW, Server 2019 Standard, rebuilt June 2026 from a past-tombstone-lifetime state). DFS namespace and DFS-R replication between sites established June 2026. L2TP/IPsec VPN fully deployed to all known client machines as of 2026-05-27.

Profile

  • Business name (Syncro): Peaceful Spirit Massage (NOT "...Therapeutic Massage" — ID-based lookup required)
  • Syncro customer ID: 278525
  • Addresses (two sites): CC / Country Club (primary — all server infrastructure): 2930 N Country Club Rd, Tucson AZ (Syncro primary address). NW / Northwest: 6650 N Oracle #100, Tucson AZ.
  • Primary contact: Mara Concordia (owner/operator); generic contact email info@bestmassageintucson.com; personal Microsoft account mara.concordia@gmail.com (OneDrive). Domain user: mara.
  • Other key staff: Bridgette (BridgetteSH); Christine Z (ChristineZ); Calista A (CalistaA); Leslie W (leslieW); Sarah M (SarahM); Katie B (katieb); Sharon S (SharonS); PSTAdmin.
  • Contract type: Break-fix / T&M (verify — recent invoices per-ticket ~$150–300/visit, plus a recurring ~$195.19/month line item; no retainer contract confirmed)
  • Managed asset count: 31
  • Open tickets: 0 as of 2026-07-01

Infrastructure

Servers & Services

Host IP Role OS Notes
PST-SERVER 192.168.0.2 DC (all 5 FSMO), DNS, RRAS (L2TP/IPsec VPN), NPS, Enterprise Root CA (AD CS) Windows Server 2016 Essentials (build 14393) Site CC. GuruRMM agent 87293069-33b6-45e8-a68f-6811216cdb96 (v0.6.75+; prior ID 6b6106a7... retired). Win32-OpenSSH installed 2026-05-11. Machine cert: DB71981ABE4CBA1DE96FEEEAF178F6259663B543 (CN=PST-SERVER.PEACEFULSPIRIT.local, valid 5/9/2027). Drives: C: 931 GB (OS); G: 465.7 GB data volume (ex-old-server C:, 182 GB free post-cleanup); D: 931 GB (Recovery-EXT/backup junk ~700 GB — cleanup pending). G:\Shares: Private ~154 GB, Scanned ~105 GB, ITServices ~5 GB, qbooks ~2 GB (~265 GB total). Credentials: vault clients/peaceful-spirit/server.
PST-DC-NW 192.168.1.5 DC (additional), GC, DNS, DFS-R receiver Windows Server 2019 Standard (build 17763) Site NW. New physical server installed 2026-07-02 (shipped as PST-SERVER01; renamed via RMM pre-join). Joined + promoted DC/GC/DNS (site NW) 2026-07-02, all via RMM. Static 192.168.1.5/24; DNS client 192.168.0.2 + 127.0.0.1; timezone US Mountain Standard (AZ). Single 1.8 TB C: (1849 GB free at setup). DFS-R replica at C:\Shares (staging 20 GB), member of PST-DFS since 2026-07-02 — initial sync of ~265 GB from PST-SERVER over S2S VPN in progress. GuruRMM agent f60e9820-4a00-4598-83f7-c14085db5768 (v0.6.75, site "North West"). DSRM password: vault clients/peaceful-spirit/dc-nw.
PST-SERVER2 (DEAD, metadata cleaned 2026-07-02) was 192.168.1.5 (now PST-DC-NW's) Was DC/GC/DNS at NW — hardware DIED ~2026-06-14 (the "flapping"), never returned; replaced by PST-DC-NW. All AD/DNS/DFSR metadata removed 2026-07-02. Windows Server 2019 Standard Site NW. Static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2 + 127.0.0.1. GuruRMM agent 5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65. Single 1 TB NVMe, C: only (original D: physical disk gone). DFS-R replica at C:\Shares (~221 GB as of 2026-06-14; ~44 GB backlog remaining). Timezone: US Mountain Standard Time (Arizona). Rebuilt 2026-06-13 (force-demote -> metadata cleanup -> re-promote; see runbook). Credentials: vault clients/peaceful-spirit/server2 (local admin + DSRM). [WARNING] Flapping (online ~1 min / offline several min reboot-loop pattern) at end of 2026-06-14 session — NW site power/UPS/network issue, NOT caused by DFS; PST-SERVER and data unaffected.
UCG-PST-CC 192.168.0.10 (LAN) / 98.190.129.150 (WAN) UniFi Cloud Gateway Ultra — perimeter router + DNAT for VPN UniFi OS 5.1.15, kernel 5.4.213-ui-ipq5322 (aarch64) Site CC. SSH: root@192.168.0.10 via key ~/.ssh/pst-cc-ucg; keyboard-interactive auth only. WAN SSH not accessible remotely. UCG VPN (strongSwan/xl2tpd) abandoned 2026-05-22; RRAS on PST-SERVER is the VPN endpoint. DNAT persistence: /data/on_boot.d/10-vpn-portforward.sh. Rebooted 2026-06-04 at 03:59, dropped VPN port-forward (see Known Issues). Credentials: vault clients/peaceful-spirit/server.
UCG-NW 64.139.88.249 (old WAN; verify current) UniFi gateway — NW site perimeter, S2S VPN (verify) NW site. Previously had OpenVPN at 64.139.88.249:1194 (TCP). S2S VPN CC<->NW confirmed up as of 2026-06-13 (ports 389/445/135/88 reachable SERVER2->SERVER). Details beyond this: (verify). Physical access: vault clients/peaceful-spirit/physical-access-northwest.

DFS Namespace & Replication

  • Domain-based DFS namespace: \\PEACEFULSPIRIT.local\PST-Files -> folder Shares
  • Current namespace root target: PST-SERVER only (\\PST-SERVER\PST-Files) — SERVER2 root target deferred pending stability
  • Current folder targets: PST-SERVER only (\\PST-SERVER\Shares, Online) — SERVER2 folder target (\\PST-SERVER2\Shares) removed pending stability; to be re-added once SERVER2 holds stable
  • DFS-R group: PST-DFS; replicated folder Shares
    • PST-SERVER G:\Shares = authoritative content; staging 20 GB
    • PST-DC-NW C:\Shares = receiver; staging 20 GB (member since 2026-07-02; SERVER2's dead membership removed same day)
    • Bidirectional connection configured; initial sync of ~265 GB started 2026-07-02 over S2S VPN (expect days)
  • Gate 4 deferred items (blocked on initial sync completing): RF state 4 + backlog 0 on PST-DC-NW; share C:\Shares as Shares; add \\PST-DC-NW\Shares folder target Online; add PST-DC-NW as 2nd namespace root target (\\PST-DC-NW\PST-Files) for VPN-outage HA
  • Runbook: clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md

Domain & Identity

  • Domain: PEACEFULSPIRIT.local (NetBIOS: PEACEFULSPIRIT)
  • AD Sites & Services: CC site (192.168.0.0/24), NW site (192.168.1.0/24); subnets correct, site link active
  • FSMO: all 5 roles on PST-SERVER
  • Global Catalog: both PST-SERVER and PST-DC-NW (since 2026-07-02; formerly PST-SERVER2)
  • Domain SID base: S-1-5-21-1105246401-3156558273-4088333098
  • Domain admins: sysadmin (password: vault clients/peaceful-spirit/server) — domain admin account. DA credentials were passed base64-wrapped in RMM command_text during June/July rebuild sessions; rotation optional (RMM is internal).
  • CA: PEACEFULSPIRIT-PST-SERVER-CA — Enterprise Root CA on PST-SERVER. Thumbprint: 56DAF43C60F246BF2C80A671EE9812C727D8C298 (valid to 3/8/2061). msPKI-Certificate-Name-Flag changed 2026-05-11 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT).
  • VPN-eligible users (WseRemoteAccessUsers, SID ...-1113): Domain Admins (group), PSTAdmin, pst-admin, LMT, Mara, BridgetteSH. NPS grants VPN by group membership — msNPAllowDialin=TRUE alone is not sufficient.
  • pst-admin: domain user (not domain admin); in WseRemoteAccessUsers; VPN-eligible. Shared VPN credential for Mara's machines.
  • AD security groups (custom):
    • Admin1 (Global Security): CalistaA, ChristineZ, leslieW, SarahM — allow RX,W + DENY (D,DC) on G:\Shares\Scanned (read/write/edit only; no delete, rename, or ownership change). Was previously Full Control; hardened 2026-07-01.
    • Admin2 (Global Security): BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS — Full Control on G:\Shares\Scanned. Admin2 was formerly (incorrectly) nested inside Admin1; nesting removed 2026-07-01.
  • OneDrive: pst-admin uses personal OneDrive (mara.concordia@gmail.com, cid: 25f0851177ceabfd). Per-machine OneDrive deployed to Maras-HP-Laptop.
  • Email / M365: (verify — no M365 tenant found; practice likely uses personal or third-party email)
  • GPO: "Block New Outlook" — GUID {577028AF-0901-4BDF-A283-CD1156F313D9}, linked to domain root.
  • SYSVOL backups (2026-06-13): C:\PST-Backup\SYSVOL-Policies-20260613-1611 and C:\PST-Backup\GPO-20260613-1611 (11 GPOs) on PST-SERVER — keep until rebuild confirmed long-term stable.

Network

  • Country Club (CC) site: WAN 98.190.129.150 (Cox); LAN 192.168.0.0/24; DC/DNS 192.168.0.2 (PST-SERVER); UCG 192.168.0.10
  • Northwest (NW) site: LAN 192.168.1.0/24; DC/DNS 192.168.1.5 (PST-SERVER2); WAN (verify current; old OpenVPN was at 64.139.88.249); S2S VPN to CC confirmed up 2026-06-13
  • VPN (L2TP/IPsec, client-to-server):
    • Endpoint: PST-SERVER RRAS at 192.168.0.2, exposed via UCG-PST-CC DNAT (UDP 500, 4500, ESP)
    • PSK: vault (clients/peaceful-spirit/vpn.sops.yaml)
    • Auth: MSCHAPv2. Mara's machines connect as shared user pst-admin; BridgettePSHomeComputer connects as BridgetteSH via SSO
    • NPS RADIUS shared secret for client UCG-PST-CC (192.168.0.10): vault (clients/peaceful-spirit/server.sops.yaml)
    • IP pool: 192.168.0.240+ (observed: .241, .242, .243, .248, .249)
    • VPN profile name on clients: "Peaceful Spirit VPN" (AllUserConnection, split tunnel, 192.168.0.0/24 route, NRPT for .peacefulspirit.local -> 192.168.0.2)
    • PST-SERVER registry: AssumeUDPEncapsulationContextOnSendRule=2 (PolicyAgent), DefaultPSK set in L2TP parameters
    • UCG persistence: /data/on_boot.d/10-vpn-portforward.sh

Client Workstations

Machine Role GuruRMM Agent ID Notes
MaraHomeNew Mara's home desktop e9645594-6d7c-4c97-8cb4-920cb5d06c8e (v0.6.52; prior c778b6a3... retired) Domain-joined. VPN working. Machine cert: D067E07B (valid 5/10/2027). Connects as pst-admin.
Maras-HP-Laptop Mara's HP laptop 13cb3629-5043-4bd6-b977-6968eeccf804 Domain-joined. VPN deployed 2026-05-22. OneDrive per-machine deployed. Connects as pst-admin.
PST-SURFACE Surface device 4a993b61-59b3-42f4-bdb5-d4362941f7d6 Domain-joined. VPN deployed 2026-05-22. Connects as pst-admin.
BridgettePSHomeComputer Bridgette's home PC 01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7 (v0.6.49; re-enrolled 2026-06-04; old 074141d7... dead) Domain-joined. VPN deployed remotely via GuruRMM user_session 2026-05-27. Connects as BridgetteSH (SSO). Logon scheduled task Connect Peaceful Spirit VPN auto-connects ~20s after sign-in.

GuruRMM Enrollment

  • Client name in RMM: Peaceful Spirit
  • Client ID: 00015eae-50e5-4102-93fa-ab0fdb135c08
  • Primary site name: Country Club
  • Primary site ID: 7b32983d-982a-4a5c-af07-45a23453f589

Enrolled agents:

Host Agent ID Version Notes
PST-SERVER 87293069-33b6-45e8-a68f-6811216cdb96 v0.6.75+ Active; confirmed 2026-07-01. Prior 6b6106a7... retired.
PST-SERVER2 5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65 — REMOVED — agent record deleted from RMM (noticed 2026-07-02); machine being replaced by PST-DC-NW.
PST-DC-NW f60e9820-4a00-4598-83f7-c14085db5768 v0.6.75 NW site ("North West" in RMM). Enrolled 2026-07-02; renamed from PST-SERVER01 via RMM same day.
MaraHomeNew e9645594-6d7c-4c97-8cb4-920cb5d06c8e v0.6.52 Active; confirmed 2026-06-04.
Maras-HP-Laptop 13cb3629-5043-4bd6-b977-6968eeccf804 — —
PST-SURFACE 4a993b61-59b3-42f4-bdb5-d4362941f7d6 — —
BridgettePSHomeComputer 01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7 v0.6.49 Re-enrolled 2026-06-04.

Data Store & Backup

Data Location

Client SOAP-note and business files reside on PST-SERVER G:\Shares. The @Clients tree is doubly-nested: G:\Shares\Scanned\@Clients\@Clients. File counts as of 2026-07-01: ~142,335 files / ~72 GB in the live @Clients tree. Total G:\Shares ~265 GB. G: is the old server's former C: drive (live PST-SERVER OS runs from C:).

MSP360 / Backblaze B2 Backup

  • Plan name: "Files Backup 2025" (Files Backup type; ForeverForward; retention 365 days)
  • MSP360 account: ACG-PST 084b5069-d634-434b-84a2-971b1dcb4b43
  • Bunch ID: 6a121575-84a0-4e98-9c0f-4a656d1a5132 (prefix: PST-SERVER)
  • Destination: Backblaze B2
  • cbb.exe path: C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe; logs C:\ProgramData\Online Backup\Logs\
  • Known restore points: 20260624170506 (6/24 10:05 AM, pre-incident repair source); 20260624190522 (6/24 12:05 PM). Oldest 20250629170034 (6/29/2025) PURGED as of 2026-07-01 (past 365-day retention) — year-ago backup unavailable.
  • Status 2026-07-01: running normally (the 6/29 stop-for-restores self-resumed).
  • Caveat: cbb list is unreliable on comma/space folder paths (false zeros, timeouts on large trees). Use restore-to-staging + local diff for any deletion-scope investigation.

VSS Shadow Copies (PST-SERVER G:)

Local point-in-time recovery / self-service "Previous Versions" on the data volume, deployed as the near-line complement to the B2 backup (confirmed 2026-07-02). Volume Shadow Copies enabled on G: with a 69.8 GB storage cap and roughly 4 snapshots/day (observed schedule ~6a / 12p / 1p / 6p). Provides fast in-place rollback of individual files/folders without a B2 restore, and is the "Option 2" alternative to the (NTFS-impossible) recycle-bin design Mara requested. Note: after the 2026-06-13 trim, older snapshots were pruned — earliest snapshot ranges have been observed as recent as 6/25–6/28, so VSS is a short-window safety net, not long-term retention (that role is B2 / MSP360). The Security event log backing the deletion audit is sized to 128 MB max on PST-SERVER.

NTFS Access Control (G:\Shares\Scanned)

ACL root is G:\Shares\Scanned; permissions inherit to @Clients and subdirectories. Hardened 2026-07-01. ACL backup on server: C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt.

Group Members Effective Permissions
Admin1 CalistaA, ChristineZ, leslieW, SarahM Allow (OI)(CI)(RX,W) + DENY (OI)(CI)(D,DC) — read/write/edit only; no delete, rename, permission or ownership change
Admin2 BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS (OI)(CI)(F) — Full Control

Caveat: the (D,DC) deny on Admin1 also blocks rename and app save patterns that delete-then-write. If Admin1 users report inability to rename or save, carve an individual exception. Reversal: Add-ADGroupMember Admin1 -Members Admin2; icacls "G:\Shares\Scanned" /remove:d "PEACEFULSPIRIT\Admin1" then restore allow via /grant.

2025 Crash & File Corruption (investigated 2026-07-02)

June 4, 2025: the OLD server (hostname NEWSERVER, domain PSTMC) crashed; a second failure during restore corrupted file-table→data mappings. Corrupted files carry the original name/size but contain foreign data blocks (MPEG-2 video streams 00 00 01 BA, ownCloud sync-log text, etc.). Mara's July-2026 Claude analysis of _C_IC_Payments_2-12_to_2-26-25.xlsx (62,993 bytes, MPEG content) is in Mike's Documents (Data recovery covo Claude.docx).

Damage inventory (live G:\Shares, Mike's [C] filename prefix = confirmed corrupt): 5,044 files, ~2.9 GB — 4,858 PDF, 108 xlsx, 25 docx; 5,027 under Private (~129 across Accounting Docs year folders 2016-2025). Full list: PST-SERVER C:\PST-Recovery\corrupted-file-list-20260702.txt (size TAB path). In the 2024/2025 payroll trees specifically only 5 of 161 Office files are corrupt, incl. Mara's IC Payments 2-12–2-26-25 (MPEG in BOTH G: [C] copy and D:\Shares copy — corruption predates all copies) and IC Payments 5-13–5-27-25 (content = ownCloud log text; Mara built a partial replacement "(clare and alice corrections ONLY)").

Recovery corpus on PST-SERVER D: (931 GB, label "VM Files" — the old server's drive; FROZEN, no cleanup):

  • D:\Shares — crash-era share tree (2025-06-03), incl. full Accounting Docs 2013-2025; mostly intact.
  • D:\Recovery2019 — restore attempt (2025-06-26), Private+Scanned.
  • D:\Recovery-EXT — ownCloud/Syncthing copy circa 2021 (complete IC Payments 2018 – mid-2021).
  • D:\Unknown folder — file-carving output (2025-06-03): 101,552 files at root ([000024].xlsx bracket names; 20,345 .doc, 1,121 .xls, 9,274 .pdf) + hex-named subdirs; candidate source for recovering the [C] files.
  • D:\Users — old-server profiles (mconcordia, hallb, lmt, pst-admin, Administrator.PSTMC/NEWSERVER).

No cloud copy predates the crash: MSP360 plan "Files Backup 2025" was created 2025-06-04 (crash day); B2 fully enumerated 2026-07-02 — generic bucket (MSPBackups20200311) holds other clients only (FSG-SRV-02, UC2-SERVER, LAB-BECKY, DROBO, VWP-SERVER, SALMON/TROUT), ACG-PST has only post-crash data, ACG-Internal only NEPTUNE.

Gotcha: G:\Shares ROOT denies SYSTEM directory enumeration (Access denied) — recursive scans from the root silently return nothing; enumerate the children (Private, Scanned, ...) directly.

Carve-identification results (2026-07-02): all 4 MPEG-corrupted 2024/2025 payroll files RECOVERED from D:\Unknown folder carved output. Method: fingerprint sharedStrings vocabulary from adjacent-period good files → score ~770 carved xlsx by token overlap → confirm by exact byte-size match with the corrupted original (NTFS keeps true size) + date-serial range / sheet title inside. Staged as copies (NOT yet placed into the live share — needs Excel-open validation + Mara sign-off): C:\PST-Recovery\carve-identified\ on PST-SERVER:

  • IC Payments 2-12 to 2-26-25 <- [006001].xlsx (62,993 B; serials exactly 2025-02-12..02-26) — the file Mara has chased for a year
  • Payroll Report 4-29-2025 - 5-12-2025 <- [007234].xlsx (53,480 B; sheet title contains the full period)
  • Tips Report 12-30--1-12-25 <- [007102].xlsx (20,302 B; serials 2024-12-30..2025-01-12; original name said "12-13" — likely Mara typo)
  • Triwest & Insurance ... 2024-11-15 check <- [006975].xlsx (19,904 B; October-2024 data, consistent with 11/15 check run) Also spotted in the carve dump with its original name: IC Payments 2024-08-12--2024-08-27.xlsx (253,949 B) — a period absent from the 2024 tree. Scoring artifacts: C:\PST-Recovery\carve-match-xlsx.txt.

Remaining: the 5th damaged file IC Payments 5-13 to 5-27-25 (content = ownCloud log; no exact-size carve hit; Mara's "(clare and alice corrections ONLY)" partial rebuild exists) — nearest candidate [005037].xlsx (56,344 B, score 58) unverified. And the broader ~5,039 other [C] files (mostly PDFs) — same identify-by-size/content approach can be batch-applied to the carved .pdf pool (9,274 root PDFs) if Mike/Mara want to chase them.

Deletion Investigation (June–July 2026)

A report that client files disappeared (trigger: the "Glennda" folder) prompted a staged restore-and-diff investigation. The 6/24 10:05 AM restore point was staged to C:\PST-Recovery\PreDelete-0624 (~99 GB). Authoritative diff: 47,749 files deleted from @Clients since 6/24 10:05; ~93% intentional duplicate cleanup (33,711 in folders labeled "duplicate DO NOT USE or delete"; ~10,696 in nested misfile-buckets A\A, D\A, P\O, H\I whose canonical client folders remain live). Genuine loss estimate: ~3,342 files, recoverable via no-overwrite copy-back from staging (not yet executed — awaiting Mike/Mara approval; writes to live HIPAA data). The 10:05->12:05 PM window had only 2 deletions (Ballard, Kathy and Rivera, Anthony SOAP PDFs) — mass deletion occurred later. Glennda trigger: EDWARDS, GLENDA (single-N, 79 files, deleted) was a misspelled duplicate of the active canonical EDWARDS, GLENNDA VA REFERRAL (double-N, 127 files, live and growing). Shelton report: only 6 old Shelton files exist (2011–2015), loose in S\, CreationTime 2025-06-02 (migration), unchanged since 6/24 — not a 2026 deletion; the 6/29/2025 restore point needed for further check has been purged. Staging artifacts (~200 GB, removable after recovery decision): C:\PST-Recovery\{PreDelete-0624, PostDelete-0624, authdiff, incidentdiff, acl-backup-scanned-20260701-072725.txt}.

Standing deletion audit (the "Mara audit log"). Object-access auditing (SACL: Everyone / Delete+DC / Success on G:\Shares\Scanned) feeds a daily scheduled task PST Deletion Report (Daily) → C:\PST-Tools\PST-DeletionReport.ps1 (runs as SYSTEM, 06:30). It harvests Security events 4660/4663 into a per-day HTML report of who deleted / renamed / moved files under G:\Shares\Scanned (server + backup activity excluded; 90-day retention). This is the ongoing record Mara reviews for further deletions. Report output location: G:\Shares\Private\Partner Review\Legal Documents - DO NOT DELETE\_Deletion Reports — moved there 2026-07-02 (from the original G:\Shares\Scanned\_Deletion Reports); pre-change script backup at C:\PST-Tools\PST-DeletionReport.ps1.bak-20260702. Only $OutDir was repointed; the monitored root ($Root = G:\Shares\Scanned) is unchanged. PST-SERVER is reachable for this kind of change via GuruRMM (agent 87293069-...) when the site VPN is down.

Access

  • PST-SERVER SSH: ssh -i ~/.ssh/id_ed25519 sysadmin@192.168.0.2 — requires L2TP VPN to CC site active. Win32-OpenSSH at C:\Program Files\OpenSSH\OpenSSH-Win64\. SCP paths use Unix format (/C:/path/to/file).
  • PST-SERVER2 SSH: ssh sysadmin@192.168.1.5 — requires S2S VPN or physical NW site access. Local admin creds: vault clients/peaceful-spirit/server2.
  • UCG-PST-CC SSH (LAN only): ssh -i ~/.ssh/pst-cc-ucg root@192.168.0.10 — keyboard-interactive auth only (plink -pw fails; use paramiko kb_handler or interactive terminal). WAN SSH not accessible remotely. Requires VPN, on-site, or UniFi cloud portal (unifi.ui.com).
  • GuruRMM (external): https://rmm.azcomputerguru.com
  • Physical access — NW site: lockbox, main-door keypad, alarm-disarm code in vault clients/peaceful-spirit/physical-access-northwest.sops.yaml (codes never in plaintext here).
  • Vault paths:
    • clients/peaceful-spirit/server.sops.yaml — PST-SERVER credentials (sysadmin DA) and UCG-PST-CC details.
    • clients/peaceful-spirit/server2.sops.yaml — PST-SERVER2 local admin + DSRM passwords. Created 2026-06-13.
    • clients/peaceful-spirit/vpn.sops.yaml — VPN PSK, pst-admin credentials, network details. [WARNING] VAULT DRIFT: vault lists pst-admin password as one value but the wiki records a 2026-05-22 reset to another — reconcile with Mara and update the stale entry.
    • clients/peaceful-spirit/physical-access-northwest.sops.yaml — NW site lockbox, door, alarm codes.

Patterns & Known Issues

  • Set-VpnConnection -L2tpPsk cannot run via RMM (SYSTEM context). Windows enforces interactive mode for PSK registration. An admin must run it manually per machine in an interactive session (one-time). Exception: the user_session command context in GuruRMM allows it — validated on BridgettePSHomeComputer 2026-05-27.

  • NRPT instead of VPN DNS suffix push. Add-VpnConnectionTriggerDnsConfiguration fails for AllUserConnection profiles. Use Add-DnsClientNrptRule -Namespace ".peacefulspirit.local" -NameServers "192.168.0.2" instead.

  • cmdkey as SYSTEM for pre-login credential persistence. Machine credential store entries (cmdkey in SYSTEM context) are available at the Windows login screen; per-user cmdkey entries are not.

  • Stale hosts file. During 2026-05-22 on-site, MaraHomeNew (and likely others) had a stale hosts entry mapping PST-SERVER to 72.194.62.5 (Mara's router's bogus DNS). A GuruRMM cleanup script was deployed; the path encoding bug (driverstc) means it may not have fully run on all machines — verify if resolution issues recur.

  • UDR Ultra reboot can silently drop the VPN port-forward (site-wide outage risk). Confirmed 2026-06-04: UCG-PST-CC rebooted at 03:59 and returned without the UDP 500/4500 -> 192.168.0.2 DNAT, taking the site VPN offline with error 789 (IKE packets silently dropped at the edge). The /data/on_boot.d/10-vpn-portforward.sh persistence script was present but the UniFi OS 5.1.15 schema migration appears to have superseded it. After any site-wide error 789, check the UDR port-forward in the UniFi controller FIRST — IPsec auditing on the server (zero IKE events) is the confirmatory test. Open: verify the re-added rule survives a deliberate reboot; add a DDNS hostname so the hardcoded Cox WAN IP is not a single point of failure.

  • UCG iptables DNAT required — UniFi Traffic Rules are firewall-allow only, NOT DNAT. Port-forward rules must be managed via the UniFi controller UI; the on_boot CLI script is a legacy fallback and may not persist on UniFi OS 5.1.15+. Verify iptables live after a reboot.

  • UCG SSH unreachable from office WAN. Remote UCG administration goes through GuruRMM (for PST-SERVER) or the UniFi cloud portal (for UCG itself). LAN SSH (192.168.0.10) requires keyboard-interactive auth; plink password auth fails.

  • GuruRMM command_type — use powershell or shell, NOT a made-up type (RESOLVED 2026-06-12). The agent's CommandType enum accepts only shell, powershell, python, script, claude_task (plus alias cmd -> shell). An unknown command_type fails the agent's whole-message JSON parse and the command is silently dropped — looks like a network black-hole.

  • Machine cert template (PEACEFULSPIRIT-PST-SERVER-CA). msPKI-Certificate-Name-Flag changed from 0x18000000 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT) 2026-05-11. New machine certs use the CSR Subject/SAN. RRAS UserAuthProtocolAccepted now includes Certificate.

  • OneDrive KFM on WSE folder-redirected profiles. WSE machines had non-standard GUID variants in User Shell Folders. SHSetKnownFolderPath must be called with flags=0 (not 0x4000) in user session context. If KFM still fails after registry cleanup, wipe the profile and redeploy per-machine OneDrive (/allusers).

  • pst-admin vs sysadmin distinction. pst-admin is a domain user (VPN-eligible); sysadmin is domain admin. Many early failures came from using pst-admin creds for DA operations.

  • NPS grants VPN by WseRemoteAccessUsers group membership, not msNPAllowDialin alone. The NPS policy condition is SID-based (...-1113). Both group membership AND msNPAllowDialin are required; missing group = error 812.

  • cmdkey credential not used by rasdial for PPP auth. The machine-store cmdkey entry is NOT consulted for PPP auth. No-arg rasdial sends the wrong principal (SYSTEM -> error 691; logged-in user without explicit credential -> error 812). Use the logon scheduled task (BridgetteSH) or the AllUserConnection cmdkey path (pst-admin machines).

  • NAT-T registry key required on all client machines. AssumeUDPEncapsulationContextOnSendRule=2 under HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent must be set AND the machine rebooted. Missing key = error 809 (was the BridgettePSHomeComputer symptom).

  • PST-SERVER2 flapping at NW site (OPEN as of 2026-06-14). After DFS-R initial replication reached ~221 GB, SERVER2 went into a reboot-loop (online ~1 min, offline several min). Pull System log events 41 (kernel-power), 6008 (unexpected shutdown), 1074 for cause — likely NW power/UPS/hardware/network. PST-SERVER (source) unaffected; no data risk. Gate 4 finish is blocked on this.

  • Duplicate and misspelled client folders make deletion-scope analysis noisy. The @Clients tree contains folders labeled "duplicate DO NOT USE or delete" and nested misfile buckets (A\A, D\A, P\O, H\I) alongside legitimate client folders. The Glennda/Glenda case (misspelled duplicate deleted, canonical intact) is recurring. When investigating any deletion report: (1) restore-to-staging + local diff is the only trustworthy method (cbb list is unreliable on these paths); (2) verify canonical folders before concluding data is lost; (3) count duplicate-labeled and nested-bucket files separately from genuine deletions.

  • Admin1 Delete-deny also blocks rename and delete-then-write saves. The (OI)(CI)(DENY)(D,DC) ACE on G:\Shares\Scanned for Admin1 prevents deletion, rename, and any app save pattern that internally deletes-then-recreates. If CalistaA/ChristineZ/leslieW/SarahM report inability to rename or save, add an individual icacls exception. Reversal in the 2026-07-01 session log.

  • Remove-DfsrMember / Add-DfsrMember contact the member machine. Remove-DfsrMember on a DEAD member fails with "network path was not found" — remove the msDFSR-Member (and any msDFSR-Connection whose fromServer references it) AD objects under CN=DFSR-GlobalSettings,CN=System,... directly. Conversely Set-DfsrMembership from SYSTEM on PST-SERVER fails with "Security cannot be set on the replicated folder. Access is denied" (writes to the OTHER member's AD subtree) — use the DA-cred Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local pattern.

  • RMM inline JSON dispatch mangles backslash-containing payloads. Inline command strings with \\ (registry paths, C:\\...) fail the server-side parse (jq sees non-JSON error response). Use ps-encoded.sh (script file + -EncodedCommand) for anything with backslashes/quotes, or forward slashes (C:/Shares) for trivial paths.

  • vault.sh get-field returns literal "null" for nested credential fields. vault.sh get-field clients/peaceful-spirit/server credentials.password returns the string "null". Use vault.sh get (full read) and extract manually.

  • AD writes via RMM require DA creds using FQDN (not localhost). Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...} works; -ComputerName localhost -Credential fails with a Kerberos SPN error. Use the FQDN for any domain/DFS/DFSN/DFSR write over RMM.

  • IP address change via RMM is unreliable. A New-NetIPAddress over the agent was killed mid-NIC-blip and reverted to DHCP. Use a one-shot scheduled task that applies the change a few seconds after the command returns, or do it on console/on-site (as done for SERVER2 2026-06-14).

  • Past-tombstone-lifetime DC must never resume replication. SERVER2 was disconnected past the tombstone lifetime. Correct remediation is force-demote -> metadata cleanup on the authoritative DC -> fresh re-promote (never resume/fix replication). Runbook: clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md.

Active Work

As of 2026-07-01 session end:

  • VPN rollout: COMPLETE across all four client machines (as of 2026-06-04).
  • [OPEN] PST-DC-NW bring-up (2026-07-02). New physical server (shipped as PST-SERVER01) replacing PST-SERVER2 (hardware died ~2026-06-14, permanently) as the NW DC + DFSR partner. DONE: enrolled in GuruRMM (agent f60e9820..., client Peaceful Spirit, site "North West"); renamed PST-SERVER01 -> PST-DC-NW + rebooted via RMM 2026-07-02 (pre-domain-join); static IP set (per Mike 2026-07-02). DONE 2026-07-02, all via RMM: PST-SERVER2 metadata cleanup (see Patterns), then PST-DC-NW domain join, DC/GC/DNS promotion (site NW; verified: all DC services running, SYSVOL state 4, repadmin 0/5 fails both directions, advertising GC/KDC/DNS/timeserver), and PST-DFS re-add (member + bidirectional connection + membership C:\Shares staging 20 GB). DSRM password vaulted at clients/peaceful-spirit/dc-nw. REMAINING (Gate 4 finish, blocked on initial sync): (1) monitor initial DFS-R sync ~265 GB PST-SERVER -> PST-DC-NW over S2S VPN (days; watch Get-DfsrBacklog / RF state 2->4), (2) once state 4 + backlog 0: add \\PST-DC-NW\Shares folder target Online in the DFS namespace, (3) add PST-DC-NW as 2nd namespace root target (\\PST-DC-NW\PST-Files) for VPN-outage HA, (4) share C:\Shares on PST-DC-NW (SMB share Shares) before the folder target, (5) dcdiag clean both DCs. If SERVER2 is still a live DC, demote/metadata-clean it properly before or after cutover (never leave a stale DC — see tombstone-lifetime pattern).
  • [SUPERSEDED by PST-SERVER01] PST-SERVER2 NW site stability (was BLOCKER for Gate 4). Reboot-loop flapping (System log 41/6008/1074), likely on-site power/UPS/hardware — resolved by hardware replacement rather than diagnosis.
  • [OPEN] Gate 4 finish (blocked on SERVER2 stable): drain ~44 GB DFS-R backlog; re-add SERVER2 folder target Online; add SERVER2 as 2nd namespace root target for HA; verify both RFs State 4, dcdiag clean.
  • [OPEN] Deletion recovery — ~3,342 genuine files. No-overwrite robocopy copy-back from C:\PST-Recovery\PreDelete-0624 (excluding duplicate/nested-bucket trees). Awaiting Mike/Mara go — writes to live HIPAA data.
  • [OPEN] Glennda single-N duplicate confirmation. Verify the deleted EDWARDS, GLENDA (79 files) had zero unique content vs live EDWARDS, GLENNDA (127 files).
  • [OPEN] Shelton SOAP notes. Year-ago restore point purged; needs client input (were the notes ever scanned? active "Sheldon" family nearby — possible mishearing).
  • [OPEN] Admin1 ACL watch. Monitor the 4 users for rename/save issues (Delete-deny side effect).
  • [OPEN] PST-Recovery staging cleanup (~200 GB) once recovery decision finalized.
  • [OPEN] UDR port-forward reboot-persistence test.
  • [OPEN] DDNS for VPN endpoint (hardcoded Cox WAN 98.190.129.150).
  • [OPEN] Vault drift — pst-admin password (vpn.sops.yaml vs 2026-05-22 reset). Verify with Mara.
  • [FROZEN - DO NOT CLEAN] D: on PST-SERVER — is NOT junk; it is the June-2025 crash recovery corpus (see "2025 Crash & Corruption" section). No deletion until the corruption-recovery effort concludes.
  • [OPEN] PST-SERVER temp/staging cleanup: C:\PST-Backup\* (SYSVOL/GPO backups) once rebuild confirmed stable; C:\ProgramData\ cert-enroll scratch (.inf/.req/.cer/.pfx, gen_certs.ps1, etc.); temp firewall rules TEMP-CertEnroll-RPC / TEMP-CertEnroll-DCOM.
  • [OPEN] Backup synthetic-full confirmation — confirm "Files Backup 2025" completes cleanly after the stop/resume.
  • [DEFERRED] Machine cert VPN path (IKEv2) — certs/PFXs exist (MaraHomeNew D067E07B, Maras-HP-Laptop 4CADDE8F, PST-SURFACE 197FF22A); superseded by L2TP. Complete, abandon, or revoke.
  • [DEFERRED] Parity decision — Mara's machines use shared pst-admin; Bridgette uses her own account. Consider per-user auth for a cleaner audit trail.
  • [DEFERRED] Optional sysadmin rotation (DA passed base64 in RMM command_text; recoverable from RMM DB; RMM internal-only).
  • [DEFERRED] Pre-login VPN verification on Maras-HP-Laptop and PST-SURFACE.
  • [DEFERRED] 2016 Essentials EOL — PST-SERVER hits end of support Jan 2027; plan replacement (2022/2025 Standard, plain AD DS) vs extend-as-is.

History Highlights

Date Event
2026-05-10 GuruRMM agent installed on PST-SERVER. IKEv2 error 812 diagnosed (NPS rejecting nonexistent user apst-admin typo). Agents enrolled on MaraHomeNew, Maras-HP-Laptop, PST-SURFACE; IKEv2 "Peaceful Spirit VPN" profiles deployed.
2026-05-11 Machine cert auth working on MaraHomeNew. Win32-OpenSSH installed on PST-SERVER. msPKI-Certificate-Name-Flag -> 0x1. OneDrive KFM WSE GUID fix. pst-admin profile on Maras-HP-Laptop wiped; per-machine OneDrive deployed. "Block New Outlook" GPO created.
2026-05-22 L2TP/IPsec VPN deployed to MaraHomeNew, Maras-HP-Laptop, PST-SURFACE on-site. UCG strongSwan/xl2tpd abandoned; RRAS on PST-SERVER became the endpoint. UCG DNAT rules created. Stale hosts entries removed. pst-admin/mara passwords reset.
2026-05-27 BridgettePSHomeComputer VPN deployed fully remotely via GuruRMM user_session. Logon scheduled task for auto-connect. VPN rollout complete across all four machines.
2026-06-04 Site-wide VPN outage: UCG-PST-CC rebooted at 03:59, returned without the DNAT; all clients failed error 789. PST-SERVER healthy. Root cause isolated via IPsec auditing (zero IKE events). Port-forward re-added. BridgettePSHomeComputer re-enrolled (new UUID 01160fc8).
2026-06-11 DFS + second-DC planning session (plan-only). Flagged 2016 Essentials EOL; recommended domain-based DFS namespace + full writable DC at NW.
2026-06-13 PST-SERVER2 found past-tombstone-lifetime (224 days stale, AD err 8614, data disk missing). Executed evict+rebuild runbook: force-demote -> metadata cleanup -> D4 authoritative SYSVOL restore -> re-promote SERVER2 (DC/GC/DNS, site NW). Two healthy DCs, 0 replication errors both directions, SYSVOL State 4 on both. G: cleanup reclaimed ~131 GB (51 -> 182 GB free). Vault server2.sops.yaml created.
2026-06-14 SERVER2 static IP set (192.168.1.5/24); timezone -> Mountain; stale .127 DNS records cleaned. Gate 4 DFS-R rebuilt clean with PST-SERVER G:\Shares PRIMARY and SERVER2 C:\Shares receiver; ~221/265 GB replicated. Session ended blocked: SERVER2 began flapping (NW site stability, not DFS). Gate 4 finish deferred.
2026-06-29 File-deletion investigation initiated. Stopped MSP360 backup, staged the 6/24 10:05 AM restore point. Mtime heuristic ruled out; restore-and-local-diff adopted as authoritative.
2026-07-01 Deletion-scope analysis complete: 47,749 files deleted since 6/24 10:05, ~93% duplicate cleanup, ~3,342 genuine recoverable. Incident window (10:05->12:05) had only 2 deletions. Glennda trigger = misspelled duplicate; canonical folder intact. Shelton check blocked (6/29/2025 restore point purged). Admin1/Admin2 NTFS hardening: removed incorrect Admin2-in-Admin1 nesting; Admin1 -> allow RX,W + DENY D,DC; Admin2 retained Full Control. ACL backup saved.
2026-07-02 Standing deletion audit operationalized: daily PST Deletion Report task (SACL 4660/4663 on G:\Shares\Scanned -> per-person HTML). Report output relocated to the legal/partner-review folder G:\Shares\Private\Partner Review\Legal Documents - DO NOT DELETE\_Deletion Reports (backup of the script kept). Change made via GuruRMM (site VPN was down); validated by a test run (report written, 6 items). New server installed at NW to replace flapping PST-SERVER2 as NW DC + DFSR partner: enrolled in GuruRMM (agent f60e9820..., site "North West") and renamed PST-SERVER01 -> PST-DC-NW + rebooted via RMM, pre-domain-join (workgroup). SERVER2's agent record found deleted from RMM. PST-SERVER2 (dead since ~6/14) metadata-cleaned from AD via RMM: DFSR member/connection objects, NTDS Settings + config server object, DC computer account, 18 stale DNS records; verified single-DC clean state. Then PST-DC-NW full bring-up, all via RMM: timezone AZ, DNS -> .0.2, domain join + reboot, AD DS role + Install-ADDSDomainController (DC/GC/DNS, site NW) + reboot, post-checks green (SYSVOL state 4, repadmin 0/5 fails both ways, advertising GC/KDC/DNS). Re-added to PST-DFS (member, bidirectional connection, membership C:\Shares staging 20 GB — Set-DfsrMembership needed DA Invoke-Command, not SYSTEM). Initial ~265 GB sync started. DSRM password vaulted (clients/peaceful-spirit/dc-nw).

Backlinks

  • projects/gururmm — PST-SERVER, PST-SERVER2, MaraHomeNew, Maras-HP-Laptop, PST-SURFACE, BridgettePSHomeComputer enrolled (primary site: Country Club; PST-SERVER2 at NW)
Reference in New Issue View Git Blame Copy Permalink