275 lines
29 KiB
Markdown
275 lines
29 KiB
Markdown
---
|
||
type: client
|
||
name: peaceful-spirit
|
||
display_name: Peaceful Spirit Therapeutic Massage
|
||
last_compiled: 2026-07-01
|
||
compiled_by: GURU-5070/claude-main
|
||
sources:
|
||
- clients/peaceful-spirit/session-logs/2026-05-10-recovered-setup-radius-authentication-for-vpn-access.md
|
||
- clients/peaceful-spirit/session-logs/2026-05-10-session.md
|
||
- clients/peaceful-spirit/session-logs/2026-05-11-session.md
|
||
- clients/peaceful-spirit/session-logs/2026-05-22-session.md
|
||
- clients/peaceful-spirit/session-logs/2026-05-27-session.md
|
||
- clients/peaceful-spirit/session-logs/2026-06-04-session.md
|
||
- clients/peaceful-spirit/session-logs/2026-06/2026-06-11-mike-multisite-dfs-dc-plan.md
|
||
- clients/peaceful-spirit/session-logs/2026-06/2026-06-13-mike-pst-server2-dc-rebuild-and-g-cleanup.md
|
||
- clients/peaceful-spirit/session-logs/2026-06/2026-06-14-mike-pst-gate4-dfsr-rebuild-and-static-ip.md
|
||
- clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md
|
||
- clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md
|
||
- session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md
|
||
- clients/peaceful-spirit/server.sops.yaml (vault)
|
||
- clients/peaceful-spirit/server2.sops.yaml (vault)
|
||
- clients/peaceful-spirit/vpn.sops.yaml (vault)
|
||
- clients/peaceful-spirit/physical-access-northwest.sops.yaml (vault)
|
||
backlinks:
|
||
- projects/gururmm
|
||
---
|
||
|
||
# Peaceful Spirit Therapeutic Massage
|
||
|
||
Massage therapy practice with two sites: Country Club (CC, primary — all server infrastructure) and Northwest (NW). On-premises Windows Server 2016 Essentials domain (PEACEFULSPIRIT.local). As of June 2026 the environment was upgraded to a two-DC architecture: PST-SERVER (CC, Server 2016 Essentials) plus PST-SERVER2 (NW, Server 2019 Standard, rebuilt June 2026 from a past-tombstone-lifetime state). DFS namespace and DFS-R replication between sites established June 2026. L2TP/IPsec VPN fully deployed to all known client machines as of 2026-05-27.
|
||
|
||
---
|
||
|
||
## Profile
|
||
|
||
- **Business name (Syncro):** Peaceful Spirit Massage (NOT "...Therapeutic Massage" — ID-based lookup required)
|
||
- **Syncro customer ID:** `278525`
|
||
- **Address:** 6650 N Oracle #100, Tucson
|
||
- **Primary contact:** Mara Concordia (owner/operator); generic contact email `info@bestmassageintucson.com`; personal Microsoft account `mara.concordia@gmail.com` (OneDrive). Domain user: `mara`.
|
||
- **Other key staff:** Bridgette (BridgetteSH); Christine Z (ChristineZ); Calista A (CalistaA); Leslie W (leslieW); Sarah M (SarahM); Katie B (katieb); Sharon S (SharonS); PSTAdmin.
|
||
- **Contract type:** Break-fix / T&M (verify — recent invoices per-ticket ~$150–300/visit, plus a recurring ~$195.19/month line item; no retainer contract confirmed)
|
||
- **Managed asset count:** 31
|
||
- **Open tickets:** 0 as of 2026-07-01
|
||
|
||
---
|
||
|
||
## Infrastructure
|
||
|
||
### Servers & Services
|
||
|
||
| Host | IP | Role | OS | Notes |
|
||
|---|---|---|---|---|
|
||
| PST-SERVER | 192.168.0.2 | DC (all 5 FSMO), DNS, RRAS (L2TP/IPsec VPN), NPS, Enterprise Root CA (AD CS) | Windows Server 2016 Essentials (build 14393) | Site CC. GuruRMM agent `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.75+; prior ID `6b6106a7...` retired). Win32-OpenSSH installed 2026-05-11. Machine cert: `DB71981ABE4CBA1DE96FEEEAF178F6259663B543` (CN=PST-SERVER.PEACEFULSPIRIT.local, valid 5/9/2027). Drives: C: 931 GB (OS); G: 465.7 GB data volume (ex-old-server C:, 182 GB free post-cleanup); D: 931 GB (Recovery-EXT/backup junk ~700 GB — cleanup pending). G:\Shares: Private ~154 GB, Scanned ~105 GB, ITServices ~5 GB, qbooks ~2 GB (~265 GB total). Credentials: vault `clients/peaceful-spirit/server`. |
|
||
| PST-SERVER2 | 192.168.1.5 | DC (additional), GC, DNS | Windows Server 2019 Standard | Site NW. Static IP 192.168.1.5/24, GW 192.168.1.1, DNS 192.168.0.2 + 127.0.0.1. GuruRMM agent `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65`. Single 1 TB NVMe, C: only (original D: physical disk gone). DFS-R replica at C:\Shares (~221 GB as of 2026-06-14; ~44 GB backlog remaining). Timezone: US Mountain Standard Time (Arizona). Rebuilt 2026-06-13 (force-demote -> metadata cleanup -> re-promote; see runbook). Credentials: vault `clients/peaceful-spirit/server2` (local admin + DSRM). [WARNING] Flapping (online ~1 min / offline several min reboot-loop pattern) at end of 2026-06-14 session — NW site power/UPS/network issue, NOT caused by DFS; PST-SERVER and data unaffected. |
|
||
| UCG-PST-CC | 192.168.0.10 (LAN) / 98.190.129.150 (WAN) | UniFi Cloud Gateway Ultra — perimeter router + DNAT for VPN | UniFi OS 5.1.15, kernel 5.4.213-ui-ipq5322 (aarch64) | Site CC. SSH: `root@192.168.0.10` via key `~/.ssh/pst-cc-ucg`; keyboard-interactive auth only. WAN SSH not accessible remotely. UCG VPN (strongSwan/xl2tpd) abandoned 2026-05-22; RRAS on PST-SERVER is the VPN endpoint. DNAT persistence: `/data/on_boot.d/10-vpn-portforward.sh`. Rebooted 2026-06-04 at 03:59, dropped VPN port-forward (see Known Issues). Credentials: vault `clients/peaceful-spirit/server`. |
|
||
| UCG-NW | 64.139.88.249 (old WAN; verify current) | UniFi gateway — NW site perimeter, S2S VPN | (verify) | NW site. Previously had OpenVPN at 64.139.88.249:1194 (TCP). S2S VPN CC<->NW confirmed up as of 2026-06-13 (ports 389/445/135/88 reachable SERVER2->SERVER). Details beyond this: (verify). Physical access: vault `clients/peaceful-spirit/physical-access-northwest`. |
|
||
|
||
### DFS Namespace & Replication
|
||
|
||
- **Domain-based DFS namespace:** `\\PEACEFULSPIRIT.local\PST-Files` -> folder `Shares`
|
||
- **Current namespace root target:** PST-SERVER only (`\\PST-SERVER\PST-Files`) — SERVER2 root target deferred pending stability
|
||
- **Current folder targets:** PST-SERVER only (`\\PST-SERVER\Shares`, Online) — SERVER2 folder target (`\\PST-SERVER2\Shares`) removed pending stability; to be re-added once SERVER2 holds stable
|
||
- **DFS-R group:** `PST-DFS`; replicated folder `Shares`
|
||
- PST-SERVER `G:\Shares` = PRIMARY / authoritative; staging 20 GB
|
||
- PST-SERVER2 `C:\Shares` = non-primary receiver; staging 20 GB
|
||
- Bidirectional connection configured; ~221/265 GB replicated as of 2026-06-14 (~44 GB backlog)
|
||
- **Gate 4 deferred items (blocked on SERVER2 stability):** drain backlog to 0; re-add SERVER2 `\\PST-SERVER2\Shares` folder target Online; add SERVER2 as 2nd namespace root target (`\\PST-SERVER2\PST-Files`) for VPN-outage HA
|
||
- **Runbook:** `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`
|
||
|
||
### Domain & Identity
|
||
|
||
- **Domain:** PEACEFULSPIRIT.local (NetBIOS: PEACEFULSPIRIT)
|
||
- **AD Sites & Services:** CC site (192.168.0.0/24), NW site (192.168.1.0/24); subnets correct, site link active
|
||
- **FSMO:** all 5 roles on PST-SERVER
|
||
- **Global Catalog:** both PST-SERVER and PST-SERVER2
|
||
- **Domain SID base:** S-1-5-21-1105246401-3156558273-4088333098
|
||
- **Domain admins:** `sysadmin` (password: vault `clients/peaceful-spirit/server`) — domain admin account. DA credentials were passed base64-wrapped in RMM command_text during June/July rebuild sessions; rotation optional (RMM is internal).
|
||
- **CA:** PEACEFULSPIRIT-PST-SERVER-CA — Enterprise Root CA on PST-SERVER. Thumbprint: 56DAF43C60F246BF2C80A671EE9812C727D8C298 (valid to 3/8/2061). `msPKI-Certificate-Name-Flag` changed 2026-05-11 to 0x1 (ENROLLEE_SUPPLIES_SUBJECT).
|
||
- **VPN-eligible users (WseRemoteAccessUsers, SID ...-1113):** Domain Admins (group), PSTAdmin, pst-admin, LMT, Mara, BridgetteSH. NPS grants VPN by group membership — `msNPAllowDialin=TRUE` alone is not sufficient.
|
||
- **pst-admin:** domain user (not domain admin); in WseRemoteAccessUsers; VPN-eligible. Shared VPN credential for Mara's machines.
|
||
- **AD security groups (custom):**
|
||
- **Admin1** (Global Security): CalistaA, ChristineZ, leslieW, SarahM — allow `RX,W` + DENY `(D,DC)` on G:\Shares\Scanned (read/write/edit only; no delete, rename, or ownership change). Was previously Full Control; hardened 2026-07-01.
|
||
- **Admin2** (Global Security): BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS — Full Control on G:\Shares\Scanned. Admin2 was formerly (incorrectly) nested inside Admin1; nesting removed 2026-07-01.
|
||
- **OneDrive:** pst-admin uses personal OneDrive (mara.concordia@gmail.com, cid: 25f0851177ceabfd). Per-machine OneDrive deployed to Maras-HP-Laptop.
|
||
- **Email / M365:** (verify — no M365 tenant found; practice likely uses personal or third-party email)
|
||
- **GPO:** "Block New Outlook" — GUID {577028AF-0901-4BDF-A283-CD1156F313D9}, linked to domain root.
|
||
- **SYSVOL backups (2026-06-13):** `C:\PST-Backup\SYSVOL-Policies-20260613-1611` and `C:\PST-Backup\GPO-20260613-1611` (11 GPOs) on PST-SERVER — keep until rebuild confirmed long-term stable.
|
||
|
||
### Network
|
||
|
||
- **Country Club (CC) site:** WAN 98.190.129.150 (Cox); LAN 192.168.0.0/24; DC/DNS 192.168.0.2 (PST-SERVER); UCG 192.168.0.10
|
||
- **Northwest (NW) site:** LAN 192.168.1.0/24; DC/DNS 192.168.1.5 (PST-SERVER2); WAN (verify current; old OpenVPN was at 64.139.88.249); S2S VPN to CC confirmed up 2026-06-13
|
||
- **VPN (L2TP/IPsec, client-to-server):**
|
||
- Endpoint: PST-SERVER RRAS at 192.168.0.2, exposed via UCG-PST-CC DNAT (UDP 500, 4500, ESP)
|
||
- PSK: vault (`clients/peaceful-spirit/vpn.sops.yaml`)
|
||
- Auth: MSCHAPv2. Mara's machines connect as shared user `pst-admin`; BridgettePSHomeComputer connects as `BridgetteSH` via SSO
|
||
- NPS RADIUS shared secret for client UCG-PST-CC (192.168.0.10): vault (`clients/peaceful-spirit/server.sops.yaml`)
|
||
- IP pool: 192.168.0.240+ (observed: .241, .242, .243, .248, .249)
|
||
- VPN profile name on clients: "Peaceful Spirit VPN" (AllUserConnection, split tunnel, 192.168.0.0/24 route, NRPT for .peacefulspirit.local -> 192.168.0.2)
|
||
- PST-SERVER registry: `AssumeUDPEncapsulationContextOnSendRule=2` (PolicyAgent), `DefaultPSK` set in L2TP parameters
|
||
- UCG persistence: `/data/on_boot.d/10-vpn-portforward.sh`
|
||
|
||
### Client Workstations
|
||
|
||
| Machine | Role | GuruRMM Agent ID | Notes |
|
||
|---|---|---|---|
|
||
| MaraHomeNew | Mara's home desktop | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` (v0.6.52; prior `c778b6a3...` retired) | Domain-joined. VPN working. Machine cert: D067E07B (valid 5/10/2027). Connects as pst-admin. |
|
||
| Maras-HP-Laptop | Mara's HP laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | Domain-joined. VPN deployed 2026-05-22. OneDrive per-machine deployed. Connects as pst-admin. |
|
||
| PST-SURFACE | Surface device | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | Domain-joined. VPN deployed 2026-05-22. Connects as pst-admin. |
|
||
| BridgettePSHomeComputer | Bridgette's home PC | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` (v0.6.49; re-enrolled 2026-06-04; old `074141d7...` dead) | Domain-joined. VPN deployed remotely via GuruRMM user_session 2026-05-27. Connects as BridgetteSH (SSO). Logon scheduled task `Connect Peaceful Spirit VPN` auto-connects ~20s after sign-in. |
|
||
|
||
---
|
||
|
||
## GuruRMM Enrollment
|
||
|
||
- **Client name in RMM:** Peaceful Spirit
|
||
- **Client ID:** `00015eae-50e5-4102-93fa-ab0fdb135c08`
|
||
- **Primary site name:** Country Club
|
||
- **Primary site ID:** `7b32983d-982a-4a5c-af07-45a23453f589`
|
||
|
||
**Enrolled agents:**
|
||
|
||
| Host | Agent ID | Version | Notes |
|
||
|---|---|---|---|
|
||
| PST-SERVER | `87293069-33b6-45e8-a68f-6811216cdb96` | v0.6.75+ | Active; confirmed 2026-07-01. Prior `6b6106a7...` retired. |
|
||
| PST-SERVER2 | `5d2d7ba0-3903-4aa3-9e97-6ca4424ffe65` | — | NW site. Flapping at 2026-06-14 session end. RMM site assignment: (verify). |
|
||
| MaraHomeNew | `e9645594-6d7c-4c97-8cb4-920cb5d06c8e` | v0.6.52 | Active; confirmed 2026-06-04. |
|
||
| Maras-HP-Laptop | `13cb3629-5043-4bd6-b977-6968eeccf804` | — | — |
|
||
| PST-SURFACE | `4a993b61-59b3-42f4-bdb5-d4362941f7d6` | — | — |
|
||
| BridgettePSHomeComputer | `01160fc8-4c2e-4e47-a591-e4e0f9ba5ea7` | v0.6.49 | Re-enrolled 2026-06-04. |
|
||
|
||
---
|
||
|
||
## Data Store & Backup
|
||
|
||
### Data Location
|
||
|
||
Client SOAP-note and business files reside on **PST-SERVER G:\Shares**. The @Clients tree is doubly-nested: `G:\Shares\Scanned\@Clients\@Clients`. File counts as of 2026-07-01: ~142,335 files / ~72 GB in the live @Clients tree. Total G:\Shares ~265 GB. G: is the old server's former C: drive (live PST-SERVER OS runs from C:).
|
||
|
||
### MSP360 / Backblaze B2 Backup
|
||
|
||
- **Plan name:** "Files Backup 2025" (Files Backup type; ForeverForward; retention 365 days)
|
||
- **MSP360 account:** ACG-PST `084b5069-d634-434b-84a2-971b1dcb4b43`
|
||
- **Bunch ID:** `6a121575-84a0-4e98-9c0f-4a656d1a5132` (prefix: PST-SERVER)
|
||
- **Destination:** Backblaze B2
|
||
- **cbb.exe path:** `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`; logs `C:\ProgramData\Online Backup\Logs\`
|
||
- **Known restore points:** `20260624170506` (6/24 10:05 AM, pre-incident repair source); `20260624190522` (6/24 12:05 PM). Oldest `20250629170034` (6/29/2025) **PURGED** as of 2026-07-01 (past 365-day retention) — year-ago backup unavailable.
|
||
- **Status 2026-07-01:** running normally (the 6/29 stop-for-restores self-resumed).
|
||
- **Caveat:** `cbb list` is unreliable on comma/space folder paths (false zeros, timeouts on large trees). Use restore-to-staging + local diff for any deletion-scope investigation.
|
||
|
||
### NTFS Access Control (G:\Shares\Scanned)
|
||
|
||
ACL root is `G:\Shares\Scanned`; permissions inherit to `@Clients` and subdirectories. Hardened 2026-07-01. ACL backup on server: `C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt`.
|
||
|
||
| Group | Members | Effective Permissions |
|
||
|---|---|---|
|
||
| Admin1 | CalistaA, ChristineZ, leslieW, SarahM | Allow `(OI)(CI)(RX,W)` + **DENY** `(OI)(CI)(D,DC)` — read/write/edit only; no delete, rename, permission or ownership change |
|
||
| Admin2 | BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS | `(OI)(CI)(F)` — Full Control |
|
||
|
||
**Caveat:** the `(D,DC)` deny on Admin1 also blocks rename and app save patterns that delete-then-write. If Admin1 users report inability to rename or save, carve an individual exception. Reversal: `Add-ADGroupMember Admin1 -Members Admin2`; `icacls "G:\Shares\Scanned" /remove:d "PEACEFULSPIRIT\Admin1"` then restore allow via `/grant`.
|
||
|
||
### Deletion Investigation (June–July 2026)
|
||
|
||
A report that client files disappeared (trigger: the "Glennda" folder) prompted a staged restore-and-diff investigation. The 6/24 10:05 AM restore point was staged to `C:\PST-Recovery\PreDelete-0624` (~99 GB). Authoritative diff: **47,749 files deleted from @Clients since 6/24 10:05**; ~93% intentional duplicate cleanup (33,711 in folders labeled "duplicate DO NOT USE or delete"; ~10,696 in nested misfile-buckets A\A, D\A, P\O, H\I whose canonical client folders remain live). Genuine loss estimate: **~3,342 files**, recoverable via no-overwrite copy-back from staging (not yet executed — awaiting Mike/Mara approval; writes to live HIPAA data). The 10:05->12:05 PM window had only 2 deletions (Ballard, Kathy and Rivera, Anthony SOAP PDFs) — mass deletion occurred later. Glennda trigger: `EDWARDS, GLENDA` (single-N, 79 files, deleted) was a misspelled duplicate of the active canonical `EDWARDS, GLENNDA VA REFERRAL` (double-N, 127 files, live and growing). Shelton report: only 6 old Shelton files exist (2011–2015), loose in `S\`, CreationTime 2025-06-02 (migration), unchanged since 6/24 — not a 2026 deletion; the 6/29/2025 restore point needed for further check has been purged. Staging artifacts (~200 GB, removable after recovery decision): `C:\PST-Recovery\{PreDelete-0624, PostDelete-0624, authdiff, incidentdiff, acl-backup-scanned-20260701-072725.txt}`.
|
||
|
||
---
|
||
|
||
## Access
|
||
|
||
- **PST-SERVER SSH:** `ssh -i ~/.ssh/id_ed25519 sysadmin@192.168.0.2` — requires L2TP VPN to CC site active. Win32-OpenSSH at `C:\Program Files\OpenSSH\OpenSSH-Win64\`. SCP paths use Unix format (`/C:/path/to/file`).
|
||
- **PST-SERVER2 SSH:** `ssh sysadmin@192.168.1.5` — requires S2S VPN or physical NW site access. Local admin creds: vault `clients/peaceful-spirit/server2`.
|
||
- **UCG-PST-CC SSH (LAN only):** `ssh -i ~/.ssh/pst-cc-ucg root@192.168.0.10` — keyboard-interactive auth only (plink `-pw` fails; use paramiko kb_handler or interactive terminal). WAN SSH not accessible remotely. Requires VPN, on-site, or UniFi cloud portal (unifi.ui.com).
|
||
- **GuruRMM (external):** https://rmm.azcomputerguru.com
|
||
- **Physical access — NW site:** lockbox, main-door keypad, alarm-disarm code in vault `clients/peaceful-spirit/physical-access-northwest.sops.yaml` (codes never in plaintext here).
|
||
- **Vault paths:**
|
||
- `clients/peaceful-spirit/server.sops.yaml` — PST-SERVER credentials (sysadmin DA) and UCG-PST-CC details.
|
||
- `clients/peaceful-spirit/server2.sops.yaml` — PST-SERVER2 local admin + DSRM passwords. Created 2026-06-13.
|
||
- `clients/peaceful-spirit/vpn.sops.yaml` — VPN PSK, pst-admin credentials, network details. [WARNING] VAULT DRIFT: vault lists pst-admin password as one value but the wiki records a 2026-05-22 reset to another — reconcile with Mara and update the stale entry.
|
||
- `clients/peaceful-spirit/physical-access-northwest.sops.yaml` — NW site lockbox, door, alarm codes.
|
||
|
||
---
|
||
|
||
## Patterns & Known Issues
|
||
|
||
- **Set-VpnConnection -L2tpPsk cannot run via RMM (SYSTEM context).** Windows enforces interactive mode for PSK registration. An admin must run it manually per machine in an interactive session (one-time). Exception: the `user_session` command context in GuruRMM allows it — validated on BridgettePSHomeComputer 2026-05-27.
|
||
|
||
- **NRPT instead of VPN DNS suffix push.** `Add-VpnConnectionTriggerDnsConfiguration` fails for AllUserConnection profiles. Use `Add-DnsClientNrptRule -Namespace ".peacefulspirit.local" -NameServers "192.168.0.2"` instead.
|
||
|
||
- **cmdkey as SYSTEM for pre-login credential persistence.** Machine credential store entries (cmdkey in SYSTEM context) are available at the Windows login screen; per-user cmdkey entries are not.
|
||
|
||
- **Stale hosts file.** During 2026-05-22 on-site, MaraHomeNew (and likely others) had a stale hosts entry mapping PST-SERVER to 72.194.62.5 (Mara's router's bogus DNS). A GuruRMM cleanup script was deployed; the path encoding bug (`driverstc`) means it may not have fully run on all machines — verify if resolution issues recur.
|
||
|
||
- **UDR Ultra reboot can silently drop the VPN port-forward (site-wide outage risk).** Confirmed 2026-06-04: UCG-PST-CC rebooted at 03:59 and returned without the UDP 500/4500 -> 192.168.0.2 DNAT, taking the site VPN offline with error 789 (IKE packets silently dropped at the edge). The `/data/on_boot.d/10-vpn-portforward.sh` persistence script was present but the UniFi OS 5.1.15 schema migration appears to have superseded it. After any site-wide error 789, check the UDR port-forward in the UniFi controller FIRST — IPsec auditing on the server (zero IKE events) is the confirmatory test. Open: verify the re-added rule survives a deliberate reboot; add a DDNS hostname so the hardcoded Cox WAN IP is not a single point of failure.
|
||
|
||
- **UCG iptables DNAT required — UniFi Traffic Rules are firewall-allow only, NOT DNAT.** Port-forward rules must be managed via the UniFi controller UI; the on_boot CLI script is a legacy fallback and may not persist on UniFi OS 5.1.15+. Verify iptables live after a reboot.
|
||
|
||
- **UCG SSH unreachable from office WAN.** Remote UCG administration goes through GuruRMM (for PST-SERVER) or the UniFi cloud portal (for UCG itself). LAN SSH (192.168.0.10) requires keyboard-interactive auth; plink password auth fails.
|
||
|
||
- **GuruRMM command_type — use `powershell` or `shell`, NOT a made-up type (RESOLVED 2026-06-12).** The agent's `CommandType` enum accepts only `shell`, `powershell`, `python`, `script`, `claude_task` (plus alias `cmd` -> shell). An unknown `command_type` fails the agent's whole-message JSON parse and the command is silently dropped — looks like a network black-hole.
|
||
|
||
- **Machine cert template (PEACEFULSPIRIT-PST-SERVER-CA).** `msPKI-Certificate-Name-Flag` changed from `0x18000000` to `0x1` (ENROLLEE_SUPPLIES_SUBJECT) 2026-05-11. New machine certs use the CSR Subject/SAN. RRAS UserAuthProtocolAccepted now includes Certificate.
|
||
|
||
- **OneDrive KFM on WSE folder-redirected profiles.** WSE machines had non-standard GUID variants in User Shell Folders. `SHSetKnownFolderPath` must be called with `flags=0` (not 0x4000) in user session context. If KFM still fails after registry cleanup, wipe the profile and redeploy per-machine OneDrive (`/allusers`).
|
||
|
||
- **pst-admin vs sysadmin distinction.** `pst-admin` is a domain user (VPN-eligible); `sysadmin` is domain admin. Many early failures came from using pst-admin creds for DA operations.
|
||
|
||
- **NPS grants VPN by WseRemoteAccessUsers group membership, not msNPAllowDialin alone.** The NPS policy condition is SID-based (`...-1113`). Both group membership AND msNPAllowDialin are required; missing group = error 812.
|
||
|
||
- **cmdkey credential not used by rasdial for PPP auth.** The machine-store cmdkey entry is NOT consulted for PPP auth. No-arg `rasdial` sends the wrong principal (SYSTEM -> error 691; logged-in user without explicit credential -> error 812). Use the logon scheduled task (BridgetteSH) or the AllUserConnection cmdkey path (pst-admin machines).
|
||
|
||
- **NAT-T registry key required on all client machines.** `AssumeUDPEncapsulationContextOnSendRule=2` under `HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent` must be set AND the machine rebooted. Missing key = error 809 (was the BridgettePSHomeComputer symptom).
|
||
|
||
- **PST-SERVER2 flapping at NW site (OPEN as of 2026-06-14).** After DFS-R initial replication reached ~221 GB, SERVER2 went into a reboot-loop (online ~1 min, offline several min). Pull System log events 41 (kernel-power), 6008 (unexpected shutdown), 1074 for cause — likely NW power/UPS/hardware/network. PST-SERVER (source) unaffected; no data risk. Gate 4 finish is blocked on this.
|
||
|
||
- **Duplicate and misspelled client folders make deletion-scope analysis noisy.** The @Clients tree contains folders labeled "duplicate DO NOT USE or delete" and nested misfile buckets (A\A, D\A, P\O, H\I) alongside legitimate client folders. The Glennda/Glenda case (misspelled duplicate deleted, canonical intact) is recurring. When investigating any deletion report: (1) restore-to-staging + local diff is the only trustworthy method (`cbb list` is unreliable on these paths); (2) verify canonical folders before concluding data is lost; (3) count duplicate-labeled and nested-bucket files separately from genuine deletions.
|
||
|
||
- **Admin1 Delete-deny also blocks rename and delete-then-write saves.** The `(OI)(CI)(DENY)(D,DC)` ACE on G:\Shares\Scanned for Admin1 prevents deletion, rename, and any app save pattern that internally deletes-then-recreates. If CalistaA/ChristineZ/leslieW/SarahM report inability to rename or save, add an individual icacls exception. Reversal in the 2026-07-01 session log.
|
||
|
||
- **vault.sh get-field returns literal "null" for nested credential fields.** `vault.sh get-field clients/peaceful-spirit/server credentials.password` returns the string `"null"`. Use `vault.sh get` (full read) and extract manually.
|
||
|
||
- **AD writes via RMM require DA creds using FQDN (not localhost).** `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock {...}` works; `-ComputerName localhost -Credential` fails with a Kerberos SPN error. Use the FQDN for any domain/DFS/DFSN/DFSR write over RMM.
|
||
|
||
- **IP address change via RMM is unreliable.** A `New-NetIPAddress` over the agent was killed mid-NIC-blip and reverted to DHCP. Use a one-shot scheduled task that applies the change a few seconds after the command returns, or do it on console/on-site (as done for SERVER2 2026-06-14).
|
||
|
||
- **Past-tombstone-lifetime DC must never resume replication.** SERVER2 was disconnected past the tombstone lifetime. Correct remediation is force-demote -> metadata cleanup on the authoritative DC -> fresh re-promote (never resume/fix replication). Runbook: `clients/peaceful-spirit/AD-DC2-REBUILD-RUNBOOK.md`.
|
||
|
||
---
|
||
|
||
## Active Work
|
||
|
||
As of 2026-07-01 session end:
|
||
|
||
- **VPN rollout: COMPLETE** across all four client machines (as of 2026-06-04).
|
||
- **[OPEN] PST-SERVER2 NW site stability (BLOCKER for Gate 4).** Diagnose reboot-loop flapping (System log 41/6008/1074). Likely on-site power/UPS/hardware.
|
||
- **[OPEN] Gate 4 finish (blocked on SERVER2 stable):** drain ~44 GB DFS-R backlog; re-add SERVER2 folder target Online; add SERVER2 as 2nd namespace root target for HA; verify both RFs State 4, dcdiag clean.
|
||
- **[OPEN] Deletion recovery — ~3,342 genuine files.** No-overwrite robocopy copy-back from `C:\PST-Recovery\PreDelete-0624` (excluding duplicate/nested-bucket trees). Awaiting Mike/Mara go — writes to live HIPAA data.
|
||
- **[OPEN] Glennda single-N duplicate confirmation.** Verify the deleted `EDWARDS, GLENDA` (79 files) had zero unique content vs live `EDWARDS, GLENNDA` (127 files).
|
||
- **[OPEN] Shelton SOAP notes.** Year-ago restore point purged; needs client input (were the notes ever scanned? active "Sheldon" family nearby — possible mishearing).
|
||
- **[OPEN] Admin1 ACL watch.** Monitor the 4 users for rename/save issues (Delete-deny side effect).
|
||
- **[OPEN] PST-Recovery staging cleanup (~200 GB)** once recovery decision finalized.
|
||
- **[OPEN] UDR port-forward reboot-persistence test.**
|
||
- **[OPEN] DDNS for VPN endpoint** (hardcoded Cox WAN 98.190.129.150).
|
||
- **[OPEN] Vault drift — pst-admin password** (vpn.sops.yaml vs 2026-05-22 reset). Verify with Mara.
|
||
- **[OPEN] D: backup-junk cleanup on PST-SERVER** (~700 GB).
|
||
- **[OPEN] PST-SERVER temp/staging cleanup:** `C:\PST-Backup\*` (SYSVOL/GPO backups) once rebuild confirmed stable; `C:\ProgramData\` cert-enroll scratch (*.inf/*.req/*.cer/*.pfx, gen_certs.ps1, etc.); temp firewall rules TEMP-CertEnroll-RPC / TEMP-CertEnroll-DCOM.
|
||
- **[OPEN] Backup synthetic-full confirmation** — confirm "Files Backup 2025" completes cleanly after the stop/resume.
|
||
- **[DEFERRED] Machine cert VPN path (IKEv2)** — certs/PFXs exist (MaraHomeNew D067E07B, Maras-HP-Laptop 4CADDE8F, PST-SURFACE 197FF22A); superseded by L2TP. Complete, abandon, or revoke.
|
||
- **[DEFERRED] Parity decision** — Mara's machines use shared pst-admin; Bridgette uses her own account. Consider per-user auth for a cleaner audit trail.
|
||
- **[DEFERRED] Optional sysadmin rotation** (DA passed base64 in RMM command_text; recoverable from RMM DB; RMM internal-only).
|
||
- **[DEFERRED] Pre-login VPN verification** on Maras-HP-Laptop and PST-SURFACE.
|
||
- **[DEFERRED] 2016 Essentials EOL** — PST-SERVER hits end of support Jan 2027; plan replacement (2022/2025 Standard, plain AD DS) vs extend-as-is.
|
||
|
||
---
|
||
|
||
## History Highlights
|
||
|
||
| Date | Event |
|
||
|---|---|
|
||
| 2026-05-10 | GuruRMM agent installed on PST-SERVER. IKEv2 error 812 diagnosed (NPS rejecting nonexistent user `apst-admin` typo). Agents enrolled on MaraHomeNew, Maras-HP-Laptop, PST-SURFACE; IKEv2 "Peaceful Spirit VPN" profiles deployed. |
|
||
| 2026-05-11 | Machine cert auth working on MaraHomeNew. Win32-OpenSSH installed on PST-SERVER. msPKI-Certificate-Name-Flag -> 0x1. OneDrive KFM WSE GUID fix. pst-admin profile on Maras-HP-Laptop wiped; per-machine OneDrive deployed. "Block New Outlook" GPO created. |
|
||
| 2026-05-22 | L2TP/IPsec VPN deployed to MaraHomeNew, Maras-HP-Laptop, PST-SURFACE on-site. UCG strongSwan/xl2tpd abandoned; RRAS on PST-SERVER became the endpoint. UCG DNAT rules created. Stale hosts entries removed. pst-admin/mara passwords reset. |
|
||
| 2026-05-27 | BridgettePSHomeComputer VPN deployed fully remotely via GuruRMM user_session. Logon scheduled task for auto-connect. VPN rollout complete across all four machines. |
|
||
| 2026-06-04 | Site-wide VPN outage: UCG-PST-CC rebooted at 03:59, returned without the DNAT; all clients failed error 789. PST-SERVER healthy. Root cause isolated via IPsec auditing (zero IKE events). Port-forward re-added. BridgettePSHomeComputer re-enrolled (new UUID 01160fc8). |
|
||
| 2026-06-11 | DFS + second-DC planning session (plan-only). Flagged 2016 Essentials EOL; recommended domain-based DFS namespace + full writable DC at NW. |
|
||
| 2026-06-13 | PST-SERVER2 found past-tombstone-lifetime (224 days stale, AD err 8614, data disk missing). Executed evict+rebuild runbook: force-demote -> metadata cleanup -> D4 authoritative SYSVOL restore -> re-promote SERVER2 (DC/GC/DNS, site NW). Two healthy DCs, 0 replication errors both directions, SYSVOL State 4 on both. G: cleanup reclaimed ~131 GB (51 -> 182 GB free). Vault `server2.sops.yaml` created. |
|
||
| 2026-06-14 | SERVER2 static IP set (192.168.1.5/24); timezone -> Mountain; stale .127 DNS records cleaned. Gate 4 DFS-R rebuilt clean with PST-SERVER G:\Shares PRIMARY and SERVER2 C:\Shares receiver; ~221/265 GB replicated. Session ended blocked: SERVER2 began flapping (NW site stability, not DFS). Gate 4 finish deferred. |
|
||
| 2026-06-29 | File-deletion investigation initiated. Stopped MSP360 backup, staged the 6/24 10:05 AM restore point. Mtime heuristic ruled out; restore-and-local-diff adopted as authoritative. |
|
||
| 2026-07-01 | Deletion-scope analysis complete: 47,749 files deleted since 6/24 10:05, ~93% duplicate cleanup, ~3,342 genuine recoverable. Incident window (10:05->12:05) had only 2 deletions. Glennda trigger = misspelled duplicate; canonical folder intact. Shelton check blocked (6/29/2025 restore point purged). Admin1/Admin2 NTFS hardening: removed incorrect Admin2-in-Admin1 nesting; Admin1 -> allow RX,W + DENY D,DC; Admin2 retained Full Control. ACL backup saved. |
|
||
|
||
---
|
||
|
||
## Backlinks
|
||
|
||
- [[projects/gururmm]] — PST-SERVER, PST-SERVER2, MaraHomeNew, Maras-HP-Laptop, PST-SURFACE, BridgettePSHomeComputer enrolled (primary site: Country Club; PST-SERVER2 at NW)
|