azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c3aeef60fbf10491f392bc5de2f23ccf407564e8
claudetools/wiki/clients/peaceful-spirit.md

29 KiB
Raw Blame History

type, name, display_name, last_compiled, compiled_by, sources, backlinks
type name display_name last_compiled compiled_by sources backlinks
client peaceful-spirit Peaceful Spirit Therapeutic Massage 2026-07-01 GURU-5070/claude-main
clients/peaceful-spirit/session-logs/2026-05-10-recovered-setup-radius-authentication-for-vpn-access.md
clients/peaceful-spirit/session-logs/2026-05-10-session.md
clients/peaceful-spirit/session-logs/2026-05-11-session.md
clients/peaceful-spirit/session-logs/2026-05-22-session.md
clients/peaceful-spirit/session-logs/2026-05-27-session.md
clients/peaceful-spirit/session-logs/2026-06-04-session.md
clients/peaceful-spirit/session-logs/2026-06/2026-06-11-mike-multisite-dfs-dc-plan.md
clients/peaceful-spirit/session-logs/2026-06/2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md
clients/peaceful-spirit/session-logs/2026-06/2026-06-14-mike-pst-gate4-dfsr-rebuild-and-static-ip.md
clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md
clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md
session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md
clients/peaceful-spirit/server.sops.yaml (vault)
clients/peaceful-spirit/server2.sops.yaml (vault)
clients/peaceful-spirit/vpn.sops.yaml (vault)
clients/peaceful-spirit/physical-access-northwest.sops.yaml (vault)
projects/gururmm

Peaceful Spirit Therapeutic Massage

Massage therapy practice with two sites: Country Club (CC, primary — all server infrastructure) and Northwest (NW). On-premises Windows Server 2016 Essentials domain (PEACEFULSPIRIT.local). As of June 2026 the environment was upgraded to a two-DC architecture: PST-SERVER (CC, Server 2016 Essentials) plus PST-SERVER2 (NW, Server 2019 Standard, rebuilt June 2026 from a past-tombstone-lifetime state). DFS namespace and DFS-R replication between sites established June 2026. L2TP/IPsec VPN fully deployed to all known client machines as of 2026-05-27.

Profile

  • Business name (Syncro): Peaceful Spirit Massage (NOT "...Therapeutic Massage" — ID-based lookup required)
  • Syncro customer ID: 278525
  • Address: 6650 N Oracle #100, Tucson
  • Primary contact: Mara Concordia (owner/operator); generic contact email info@bestmassageintucson.com; personal Microsoft account mara.concordia@gmail.com (OneDrive). Domain user: mara.
  • Other key staff: Bridgette (BridgetteSH); Christine Z (ChristineZ); Calista A (CalistaA); Leslie W (leslieW); Sarah M (SarahM); Katie B (katieb); Sharon S (SharonS); PSTAdmin.
  • Contract type: Break-fix / T&M (verify — recent invoices per-ticket ~$150–300/visit, plus a recurring ~$195.19/month line item; no retainer contract confirmed)
  • Managed asset count: 31
  • Open tickets: 0 as of 2026-07-01

Infrastructure

Servers & Services

Host IP Role OS Notes
PST-SERVER 192.168.0.2 DC (all 5 FSMO), DNS, RRAS (L2TP/IPsec VPN), NPS, Enterprise Root CA (AD CS) Windows Server 2016 Essentials (build 14393) Site CC. GuruRMM agent 87293069-33b6-45e8-a68f-6811216cdb96 (v0.6.75+; prior ID 6b6106a7... retired). Win32-OpenSSH installed 2026-05-11. Machine cert: DB71981ABE4CBA1DE96FEEEAF178F6259663B543 (CN=PST-SERVER.PEACEFULSPIRIT.local, valid 5/9/2027). Drives: C: 931 GB (OS); G: 465.7 GB data volume (ex-old-server C:, 182 GB free post-cleanup); D: 931 GB (Recovery-EXT/backup junk ~700 GB — cleanup pending). G:\Shares: Private ~154 GB, Scanned ~105 GB, ITServices ~5 GB, qbooks ~2 GB (~265 GB total). Credentials: vault clients/peaceful-spirit/server.
PST-SERVER2 192.168.1.5 DC (additional), GC, DNS Windows Server 2019 Standard Site NW. Static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2 + 127.0.0.1. GuruRMM agent 5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65. Single 1 TB NVMe, C: only (original D: physical disk gone). DFS-R replica at C:\Shares (~221 GB as of 2026-06-14; ~44 GB backlog remaining). Timezone: US Mountain Standard Time (Arizona). Rebuilt 2026-06-13 (force-demote -> metadata cleanup -> re-promote; see runbook). Credentials: vault clients/peaceful-spirit/server2 (local admin + DSRM). [WARNING] Flapping (online ~1 min / offline several min reboot-loop pattern) at end of 2026-06-14 session — NW site power/UPS/network issue, NOT caused by DFS; PST-SERVER and data unaffected.
UCG-PST-CC 192.168.0.10 (LAN) / 98.190.129.150 (WAN) UniFi Cloud Gateway Ultra — perimeter router + DNAT for VPN UniFi OS 5.1.15, kernel 5.4.213-ui-ipq5322 (aarch64) Site CC. SSH: root@192.168.0.10 via key ~/.ssh/pst-cc-ucg; keyboard-interactive auth only. WAN SSH not accessible remotely. UCG VPN (strongSwan/xl2tpd) abandoned 2026-05-22; RRAS on PST-SERVER is the VPN endpoint. DNAT persistence: /data/on_boot.d/10-vpn-portforward.sh. Rebooted 2026-06-04 at 03:59, dropped VPN port-forward (see Known Issues). Credentials: vault clients/peaceful-spirit/server.
UCG-NW 64.139.88.249 (old WAN; verify current) UniFi gateway — NW site perimeter, S2S VPN (verify) NW site. Previously had OpenVPN at 64.139.88.249:1194 (TCP). S2S VPN CC<->NW confirmed up as of 2026-06-13 (ports 389/445/135/88 reachable SERVER2->SERVER). Details beyond this: (verify). Physical access: vault clients/peaceful-spirit/physical-access-northwest.

DFS Namespace & Replication

  • Domain-based DFS namespace: \\PEACEFULSPIRIT.local\PST-Files -> folder Shares
  • Current namespace root target: PST-SERVER only (\\PST-SERVER\PST-Files) — SERVER2 root target deferred pending stability
  • Current folder targets: PST-SERVER only (\\PST-SERVER\Shares, Online) — SERVER2 folder target (\\PST-SERVER2\Shares) removed pending stability; to be re-added once SERVER2 holds stable
  • DFS-R group: PST-DFS; replicated folder Shares
    • PST-SERVER G:\Shares = PRIMARY / authoritative; staging 20 GB
    • PST-SERVER2 C:\Shares = non-primary receiver; staging 20 GB
    • Bidirectional connection configured; ~221/265 GB replicated as of 2026-06-14 (~44 GB backlog)
  • Gate 4 deferred items (blocked on SERVER2 stability): drain backlog to 0; re-add SERVER2 \\PST-SERVER2\Shares folder target Online; add SERVER2 as 2nd namespace root target (\\PST-SERVER2\PST-Files) for VPN-outage HA
  • Runbook: clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md

Domain & Identity

  • Domain: PEACEFULSPIRIT.local (NetBIOS: PEACEFULSPIRIT)
  • AD Sites & Services: CC site (192.168.0.0/24), NW site (192.168.1.0/24); subnets correct, site link active
  • FSMO: all 5 roles on PST-SERVER
  • Global Catalog: both PST-SERVER and PST-SERVER2
  • Domain SID base: S-1-5-21-1105246401-3156558273-4088333098
  • Domain admins: sysadmin (password: vault clients/peaceful-spirit/server) — domain admin account. DA credentials were passed base64-wrapped in RMM command_text during June/July rebuild sessions; rotation optional (RMM is internal).
  • CA: PEACEFULSPIRIT-PST-SERVER-CA — Enterprise Root CA on PST-SERVER. Thumbprint: 56DAF43C60F246BF2C80A671EE9812C727D8C298 (valid to 3/8/2061). msPKI-Certificate-Name-Flag changed 2026-05-11 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT).
  • VPN-eligible users (WseRemoteAccessUsers, SID ...-1113): Domain Admins (group), PSTAdmin, pst-admin, LMT, Mara, BridgetteSH. NPS grants VPN by group membership — msNPAllowDialin=TRUE alone is not sufficient.
  • pst-admin: domain user (not domain admin); in WseRemoteAccessUsers; VPN-eligible. Shared VPN credential for Mara's machines.
  • AD security groups (custom):
    • Admin1 (Global Security): CalistaA, ChristineZ, leslieW, SarahM — allow RX,W + DENY (D,DC) on G:\Shares\Scanned (read/write/edit only; no delete, rename, or ownership change). Was previously Full Control; hardened 2026-07-01.
    • Admin2 (Global Security): BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS — Full Control on G:\Shares\Scanned. Admin2 was formerly (incorrectly) nested inside Admin1; nesting removed 2026-07-01.
  • OneDrive: pst-admin uses personal OneDrive (mara.concordia@gmail.com, cid: 25f0851177ceabfd). Per-machine OneDrive deployed to Maras-HP-Laptop.
  • Email / M365: (verify — no M365 tenant found; practice likely uses personal or third-party email)
  • GPO: "Block New Outlook" — GUID {577028AF-0901-4BDF-A283-CD1156F313D9}, linked to domain root.
  • SYSVOL backups (2026-06-13): C:\PST-Backup\SYSVOL-Policies-20260613-1611 and C:\PST-Backup\GPO-20260613-1611 (11 GPOs) on PST-SERVER — keep until rebuild confirmed long-term stable.

Network

  • Country Club (CC) site: WAN 98.190.129.150 (Cox); LAN 192.168.0.0/24; DC/DNS 192.168.0.2 (PST-SERVER); UCG 192.168.0.10
  • Northwest (NW) site: LAN 192.168.1.0/24; DC/DNS 192.168.1.5 (PST-SERVER2); WAN (verify current; old OpenVPN was at 64.139.88.249); S2S VPN to CC confirmed up 2026-06-13
  • VPN (L2TP/IPsec, client-to-server):
    • Endpoint: PST-SERVER RRAS at 192.168.0.2, exposed via UCG-PST-CC DNAT (UDP 500, 4500, ESP)
    • PSK: vault (clients/peaceful-spirit/vpn.sops.yaml)
    • Auth: MSCHAPv2. Mara's machines connect as shared user pst-admin; BridgettePSHomeComputer connects as BridgetteSH via SSO
    • NPS RADIUS shared secret for client UCG-PST-CC (192.168.0.10): vault (clients/peaceful-spirit/server.sops.yaml)
    • IP pool: 192.168.0.240+ (observed: .241, .242, .243, .248, .249)
    • VPN profile name on clients: "Peaceful Spirit VPN" (AllUserConnection, split tunnel, 192.168.0.0/24 route, NRPT for .peacefulspirit.local -> 192.168.0.2)
    • PST-SERVER registry: AssumeUDPEncapsulationContextOnSendRule=2 (PolicyAgent), DefaultPSK set in L2TP parameters
    • UCG persistence: /data/on_boot.d/10-vpn-portforward.sh

Client Workstations

Machine Role GuruRMM Agent ID Notes
MaraHomeNew Mara's home desktop e9645594-6d7c-4c97-8cb4-920cb5d06c8e (v0.6.52; prior c778b6a3... retired) Domain-joined. VPN working. Machine cert: D067E07B (valid 5/10/2027). Connects as pst-admin.
Maras-HP-Laptop Mara's HP laptop 13cb3629-5043-4bd6-b977-6968eeccf804 Domain-joined. VPN deployed 2026-05-22. OneDrive per-machine deployed. Connects as pst-admin.
PST-SURFACE Surface device 4a993b61-59b3-42f4-bdb5-d4362941f7d6 Domain-joined. VPN deployed 2026-05-22. Connects as pst-admin.
BridgettePSHomeComputer Bridgette's home PC 01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7 (v0.6.49; re-enrolled 2026-06-04; old 074141d7... dead) Domain-joined. VPN deployed remotely via GuruRMM user_session 2026-05-27. Connects as BridgetteSH (SSO). Logon scheduled task Connect Peaceful Spirit VPN auto-connects ~20s after sign-in.

GuruRMM Enrollment

  • Client name in RMM: Peaceful Spirit
  • Client ID: 00015eae-50e5-4102-93fa-ab0fdb135c08
  • Primary site name: Country Club
  • Primary site ID: 7b32983d-982a-4a5c-af07-45a23453f589

Enrolled agents:

Host Agent ID Version Notes
PST-SERVER 87293069-33b6-45e8-a68f-6811216cdb96 v0.6.75+ Active; confirmed 2026-07-01. Prior 6b6106a7... retired.
PST-SERVER2 5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65 — NW site. Flapping at 2026-06-14 session end. RMM site assignment: (verify).
MaraHomeNew e9645594-6d7c-4c97-8cb4-920cb5d06c8e v0.6.52 Active; confirmed 2026-06-04.
Maras-HP-Laptop 13cb3629-5043-4bd6-b977-6968eeccf804 — —
PST-SURFACE 4a993b61-59b3-42f4-bdb5-d4362941f7d6 — —
BridgettePSHomeComputer 01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7 v0.6.49 Re-enrolled 2026-06-04.

Data Store & Backup

Data Location

Client SOAP-note and business files reside on PST-SERVER G:\Shares. The @Clients tree is doubly-nested: G:\Shares\Scanned\@Clients\@Clients. File counts as of 2026-07-01: ~142,335 files / ~72 GB in the live @Clients tree. Total G:\Shares ~265 GB. G: is the old server's former C: drive (live PST-SERVER OS runs from C:).

MSP360 / Backblaze B2 Backup

  • Plan name: "Files Backup 2025" (Files Backup type; ForeverForward; retention 365 days)
  • MSP360 account: ACG-PST 084b5069-d634-434b-84a2-971b1dcb4b43
  • Bunch ID: 6a121575-84a0-4e98-9c0f-4a656d1a5132 (prefix: PST-SERVER)
  • Destination: Backblaze B2
  • cbb.exe path: C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe; logs C:\ProgramData\Online Backup\Logs\
  • Known restore points: 20260624170506 (6/24 10:05 AM, pre-incident repair source); 20260624190522 (6/24 12:05 PM). Oldest 20250629170034 (6/29/2025) PURGED as of 2026-07-01 (past 365-day retention) — year-ago backup unavailable.
  • Status 2026-07-01: running normally (the 6/29 stop-for-restores self-resumed).
  • Caveat: cbb list is unreliable on comma/space folder paths (false zeros, timeouts on large trees). Use restore-to-staging + local diff for any deletion-scope investigation.

NTFS Access Control (G:\Shares\Scanned)

ACL root is G:\Shares\Scanned; permissions inherit to @Clients and subdirectories. Hardened 2026-07-01. ACL backup on server: C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt.

Group Members Effective Permissions
Admin1 CalistaA, ChristineZ, leslieW, SarahM Allow (OI)(CI)(RX,W) + DENY (OI)(CI)(D,DC) — read/write/edit only; no delete, rename, permission or ownership change
Admin2 BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS (OI)(CI)(F) — Full Control

Caveat: the (D,DC) deny on Admin1 also blocks rename and app save patterns that delete-then-write. If Admin1 users report inability to rename or save, carve an individual exception. Reversal: Add-ADGroupMember Admin1 -Members Admin2; icacls "G:\Shares\Scanned" /remove:d "PEACEFULSPIRIT\Admin1" then restore allow via /grant.

Deletion Investigation (June–July 2026)

A report that client files disappeared (trigger: the "Glennda" folder) prompted a staged restore-and-diff investigation. The 6/24 10:05 AM restore point was staged to C:\PST-Recovery\PreDelete-0624 (~99 GB). Authoritative diff: 47,749 files deleted from @Clients since 6/24 10:05; ~93% intentional duplicate cleanup (33,711 in folders labeled "duplicate DO NOT USE or delete"; ~10,696 in nested misfile-buckets A\A, D\A, P\O, H\I whose canonical client folders remain live). Genuine loss estimate: ~3,342 files, recoverable via no-overwrite copy-back from staging (not yet executed — awaiting Mike/Mara approval; writes to live HIPAA data). The 10:05->12:05 PM window had only 2 deletions (Ballard, Kathy and Rivera, Anthony SOAP PDFs) — mass deletion occurred later. Glennda trigger: EDWARDS, GLENDA (single-N, 79 files, deleted) was a misspelled duplicate of the active canonical EDWARDS, GLENNDA VA REFERRAL (double-N, 127 files, live and growing). Shelton report: only 6 old Shelton files exist (2011–2015), loose in S\, CreationTime 2025-06-02 (migration), unchanged since 6/24 — not a 2026 deletion; the 6/29/2025 restore point needed for further check has been purged. Staging artifacts (~200 GB, removable after recovery decision): C:\PST-Recovery\{PreDelete-0624, PostDelete-0624, authdiff, incidentdiff, acl-backup-scanned-20260701-072725.txt}.

Access

  • PST-SERVER SSH: ssh -i ~/.ssh/id_ed25519 sysadmin@192.168.0.2 — requires L2TP VPN to CC site active. Win32-OpenSSH at C:\Program Files\OpenSSH\OpenSSH-Win64\. SCP paths use Unix format (/C:/path/to/file).
  • PST-SERVER2 SSH: ssh sysadmin@192.168.1.5 — requires S2S VPN or physical NW site access. Local admin creds: vault clients/peaceful-spirit/server2.
  • UCG-PST-CC SSH (LAN only): ssh -i ~/.ssh/pst-cc-ucg root@192.168.0.10 — keyboard-interactive auth only (plink -pw fails; use paramiko kb_handler or interactive terminal). WAN SSH not accessible remotely. Requires VPN, on-site, or UniFi cloud portal (unifi.ui.com).
  • GuruRMM (external): https://rmm.azcomputerguru.com
  • Physical access — NW site: lockbox, main-door keypad, alarm-disarm code in vault clients/peaceful-spirit/physical-access-northwest.sops.yaml (codes never in plaintext here).
  • Vault paths:
    • clients/peaceful-spirit/server.sops.yaml — PST-SERVER credentials (sysadmin DA) and UCG-PST-CC details.
    • clients/peaceful-spirit/server2.sops.yaml — PST-SERVER2 local admin + DSRM passwords. Created 2026-06-13.
    • clients/peaceful-spirit/vpn.sops.yaml — VPN PSK, pst-admin credentials, network details. [WARNING] VAULT DRIFT: vault lists pst-admin password as one value but the wiki records a 2026-05-22 reset to another — reconcile with Mara and update the stale entry.
    • clients/peaceful-spirit/physical-access-northwest.sops.yaml — NW site lockbox, door, alarm codes.

Patterns & Known Issues

  • Set-VpnConnection -L2tpPsk cannot run via RMM (SYSTEM context). Windows enforces interactive mode for PSK registration. An admin must run it manually per machine in an interactive session (one-time). Exception: the user_session command context in GuruRMM allows it — validated on BridgettePSHomeComputer 2026-05-27.

  • NRPT instead of VPN DNS suffix push. Add-VpnConnectionTriggerDnsConfiguration fails for AllUserConnection profiles. Use Add-DnsClientNrptRule -Namespace ".peacefulspirit.local" -NameServers "192.168.0.2" instead.

  • cmdkey as SYSTEM for pre-login credential persistence. Machine credential store entries (cmdkey in SYSTEM context) are available at the Windows login screen; per-user cmdkey entries are not.

  • Stale hosts file. During 2026-05-22 on-site, MaraHomeNew (and likely others) had a stale hosts entry mapping PST-SERVER to 72.194.62.5 (Mara's router's bogus DNS). A GuruRMM cleanup script was deployed; the path encoding bug (driverstc) means it may not have fully run on all machines — verify if resolution issues recur.

  • UDR Ultra reboot can silently drop the VPN port-forward (site-wide outage risk). Confirmed 2026-06-04: UCG-PST-CC rebooted at 03:59 and returned without the UDP 500/4500 -> 192.168.0.2 DNAT, taking the site VPN offline with error 789 (IKE packets silently dropped at the edge). The /data/on_boot.d/10-vpn-portforward.sh persistence script was present but the UniFi OS 5.1.15 schema migration appears to have superseded it. After any site-wide error 789, check the UDR port-forward in the UniFi controller FIRST — IPsec auditing on the server (zero IKE events) is the confirmatory test. Open: verify the re-added rule survives a deliberate reboot; add a DDNS hostname so the hardcoded Cox WAN IP is not a single point of failure.

  • UCG iptables DNAT required — UniFi Traffic Rules are firewall-allow only, NOT DNAT. Port-forward rules must be managed via the UniFi controller UI; the on_boot CLI script is a legacy fallback and may not persist on UniFi OS 5.1.15+. Verify iptables live after a reboot.

  • UCG SSH unreachable from office WAN. Remote UCG administration goes through GuruRMM (for PST-SERVER) or the UniFi cloud portal (for UCG itself). LAN SSH (192.168.0.10) requires keyboard-interactive auth; plink password auth fails.

  • GuruRMM command_type — use powershell or shell, NOT a made-up type (RESOLVED 2026-06-12). The agent's CommandType enum accepts only shell, powershell, python, script, claude_task (plus alias cmd -> shell). An unknown command_type fails the agent's whole-message JSON parse and the command is silently dropped — looks like a network black-hole.

  • Machine cert template (PEACEFULSPIRIT-PST-SERVER-CA). msPKI-Certificate-Name-Flag changed from 0x18000000 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT) 2026-05-11. New machine certs use the CSR Subject/SAN. RRAS UserAuthProtocolAccepted now includes Certificate.

  • OneDrive KFM on WSE folder-redirected profiles. WSE machines had non-standard GUID variants in User Shell Folders. SHSetKnownFolderPath must be called with flags=0 (not 0x4000) in user session context. If KFM still fails after registry cleanup, wipe the profile and redeploy per-machine OneDrive (/allusers).

  • pst-admin vs sysadmin distinction. pst-admin is a domain user (VPN-eligible); sysadmin is domain admin. Many early failures came from using pst-admin creds for DA operations.

  • NPS grants VPN by WseRemoteAccessUsers group membership, not msNPAllowDialin alone. The NPS policy condition is SID-based (...-1113). Both group membership AND msNPAllowDialin are required; missing group = error 812.

  • cmdkey credential not used by rasdial for PPP auth. The machine-store cmdkey entry is NOT consulted for PPP auth. No-arg rasdial sends the wrong principal (SYSTEM -> error 691; logged-in user without explicit credential -> error 812). Use the logon scheduled task (BridgetteSH) or the AllUserConnection cmdkey path (pst-admin machines).

  • NAT-T registry key required on all client machines. AssumeUDPEncapsulationContextOnSendRule=2 under HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent must be set AND the machine rebooted. Missing key = error 809 (was the BridgettePSHomeComputer symptom).

  • PST-SERVER2 flapping at NW site (OPEN as of 2026-06-14). After DFS-R initial replication reached ~221 GB, SERVER2 went into a reboot-loop (online ~1 min, offline several min). Pull System log events 41 (kernel-power), 6008 (unexpected shutdown), 1074 for cause — likely NW power/UPS/hardware/network. PST-SERVER (source) unaffected; no data risk. Gate 4 finish is blocked on this.

  • Duplicate and misspelled client folders make deletion-scope analysis noisy. The @Clients tree contains folders labeled "duplicate DO NOT USE or delete" and nested misfile buckets (A\A, D\A, P\O, H\I) alongside legitimate client folders. The Glennda/Glenda case (misspelled duplicate deleted, canonical intact) is recurring. When investigating any deletion report: (1) restore-to-staging + local diff is the only trustworthy method (cbb list is unreliable on these paths); (2) verify canonical folders before concluding data is lost; (3) count duplicate-labeled and nested-bucket files separately from genuine deletions.

  • Admin1 Delete-deny also blocks rename and delete-then-write saves. The (OI)(CI)(DENY)(D,DC) ACE on G:\Shares\Scanned for Admin1 prevents deletion, rename, and any app save pattern that internally deletes-then-recreates. If CalistaA/ChristineZ/leslieW/SarahM report inability to rename or save, add an individual icacls exception. Reversal in the 2026-07-01 session log.

  • vault.sh get-field returns literal "null" for nested credential fields. vault.sh get-field clients/peaceful-spirit/server credentials.password returns the string "null". Use vault.sh get (full read) and extract manually.

  • AD writes via RMM require DA creds using FQDN (not localhost). Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...} works; -ComputerName localhost -Credential fails with a Kerberos SPN error. Use the FQDN for any domain/DFS/DFSN/DFSR write over RMM.

  • IP address change via RMM is unreliable. A New-NetIPAddress over the agent was killed mid-NIC-blip and reverted to DHCP. Use a one-shot scheduled task that applies the change a few seconds after the command returns, or do it on console/on-site (as done for SERVER2 2026-06-14).

  • Past-tombstone-lifetime DC must never resume replication. SERVER2 was disconnected past the tombstone lifetime. Correct remediation is force-demote -> metadata cleanup on the authoritative DC -> fresh re-promote (never resume/fix replication). Runbook: clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md.

Active Work

As of 2026-07-01 session end:

  • VPN rollout: COMPLETE across all four client machines (as of 2026-06-04).
  • [OPEN] PST-SERVER2 NW site stability (BLOCKER for Gate 4). Diagnose reboot-loop flapping (System log 41/6008/1074). Likely on-site power/UPS/hardware.
  • [OPEN] Gate 4 finish (blocked on SERVER2 stable): drain ~44 GB DFS-R backlog; re-add SERVER2 folder target Online; add SERVER2 as 2nd namespace root target for HA; verify both RFs State 4, dcdiag clean.
  • [OPEN] Deletion recovery — ~3,342 genuine files. No-overwrite robocopy copy-back from C:\PST-Recovery\PreDelete-0624 (excluding duplicate/nested-bucket trees). Awaiting Mike/Mara go — writes to live HIPAA data.
  • [OPEN] Glennda single-N duplicate confirmation. Verify the deleted EDWARDS, GLENDA (79 files) had zero unique content vs live EDWARDS, GLENNDA (127 files).
  • [OPEN] Shelton SOAP notes. Year-ago restore point purged; needs client input (were the notes ever scanned? active "Sheldon" family nearby — possible mishearing).
  • [OPEN] Admin1 ACL watch. Monitor the 4 users for rename/save issues (Delete-deny side effect).
  • [OPEN] PST-Recovery staging cleanup (~200 GB) once recovery decision finalized.
  • [OPEN] UDR port-forward reboot-persistence test.
  • [OPEN] DDNS for VPN endpoint (hardcoded Cox WAN 98.190.129.150).
  • [OPEN] Vault drift — pst-admin password (vpn.sops.yaml vs 2026-05-22 reset). Verify with Mara.
  • [OPEN] D: backup-junk cleanup on PST-SERVER (~700 GB).
  • [OPEN] PST-SERVER temp/staging cleanup: C:\PST-Backup\* (SYSVOL/GPO backups) once rebuild confirmed stable; C:\ProgramData\ cert-enroll scratch (.inf/.req/.cer/.pfx, gen_certs.ps1, etc.); temp firewall rules TEMP-CertEnroll-RPC / TEMP-CertEnroll-DCOM.
  • [OPEN] Backup synthetic-full confirmation — confirm "Files Backup 2025" completes cleanly after the stop/resume.
  • [DEFERRED] Machine cert VPN path (IKEv2) — certs/PFXs exist (MaraHomeNew D067E07B, Maras-HP-Laptop 4CADDE8F, PST-SURFACE 197FF22A); superseded by L2TP. Complete, abandon, or revoke.
  • [DEFERRED] Parity decision — Mara's machines use shared pst-admin; Bridgette uses her own account. Consider per-user auth for a cleaner audit trail.
  • [DEFERRED] Optional sysadmin rotation (DA passed base64 in RMM command_text; recoverable from RMM DB; RMM internal-only).
  • [DEFERRED] Pre-login VPN verification on Maras-HP-Laptop and PST-SURFACE.
  • [DEFERRED] 2016 Essentials EOL — PST-SERVER hits end of support Jan 2027; plan replacement (2022/2025 Standard, plain AD DS) vs extend-as-is.

History Highlights

Date Event
2026-05-10 GuruRMM agent installed on PST-SERVER. IKEv2 error 812 diagnosed (NPS rejecting nonexistent user apst-admin typo). Agents enrolled on MaraHomeNew, Maras-HP-Laptop, PST-SURFACE; IKEv2 "Peaceful Spirit VPN" profiles deployed.
2026-05-11 Machine cert auth working on MaraHomeNew. Win32-OpenSSH installed on PST-SERVER. msPKI-Certificate-Name-Flag -> 0x1. OneDrive KFM WSE GUID fix. pst-admin profile on Maras-HP-Laptop wiped; per-machine OneDrive deployed. "Block New Outlook" GPO created.
2026-05-22 L2TP/IPsec VPN deployed to MaraHomeNew, Maras-HP-Laptop, PST-SURFACE on-site. UCG strongSwan/xl2tpd abandoned; RRAS on PST-SERVER became the endpoint. UCG DNAT rules created. Stale hosts entries removed. pst-admin/mara passwords reset.
2026-05-27 BridgettePSHomeComputer VPN deployed fully remotely via GuruRMM user_session. Logon scheduled task for auto-connect. VPN rollout complete across all four machines.
2026-06-04 Site-wide VPN outage: UCG-PST-CC rebooted at 03:59, returned without the DNAT; all clients failed error 789. PST-SERVER healthy. Root cause isolated via IPsec auditing (zero IKE events). Port-forward re-added. BridgettePSHomeComputer re-enrolled (new UUID 01160fc8).
2026-06-11 DFS + second-DC planning session (plan-only). Flagged 2016 Essentials EOL; recommended domain-based DFS namespace + full writable DC at NW.
2026-06-13 PST-SERVER2 found past-tombstone-lifetime (224 days stale, AD err 8614, data disk missing). Executed evict+rebuild runbook: force-demote -> metadata cleanup -> D4 authoritative SYSVOL restore -> re-promote SERVER2 (DC/GC/DNS, site NW). Two healthy DCs, 0 replication errors both directions, SYSVOL State 4 on both. G: cleanup reclaimed ~131 GB (51 -> 182 GB free). Vault server2.sops.yaml created.
2026-06-14 SERVER2 static IP set (192.168.1.5/24); timezone -> Mountain; stale .127 DNS records cleaned. Gate 4 DFS-R rebuilt clean with PST-SERVER G:\Shares PRIMARY and SERVER2 C:\Shares receiver; ~221/265 GB replicated. Session ended blocked: SERVER2 began flapping (NW site stability, not DFS). Gate 4 finish deferred.
2026-06-29 File-deletion investigation initiated. Stopped MSP360 backup, staged the 6/24 10:05 AM restore point. Mtime heuristic ruled out; restore-and-local-diff adopted as authoritative.
2026-07-01 Deletion-scope analysis complete: 47,749 files deleted since 6/24 10:05, ~93% duplicate cleanup, ~3,342 genuine recoverable. Incident window (10:05->12:05) had only 2 deletions. Glennda trigger = misspelled duplicate; canonical folder intact. Shelton check blocked (6/29/2025 restore point purged). Admin1/Admin2 NTFS hardening: removed incorrect Admin2-in-Admin1 nesting; Admin1 -> allow RX,W + DENY D,DC; Admin2 retained Full Control. ACL backup saved.

Backlinks

  • projects/gururmm — PST-SERVER, PST-SERVER2, MaraHomeNew, Maras-HP-Laptop, PST-SURFACE, BridgettePSHomeComputer enrolled (primary site: Country Club; PST-SERVER2 at NW)
Reference in New Issue View Git Blame Copy Permalink