## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Resumed the Peaceful Spirit **PST-SERVER file-deletion investigation** (initially misread as the
DFS rebuild thread; corrected to the deleted-files thread). All work via GuruRMM against PST-SERVER
(192.168.0.2, agent `87293069-33b6-45e8-a68f-6811216cdb96`). Confirmed the prior session's 6/24
10:05 AM pre-incident restore (`C:\PST-Recovery\PreDelete-0624`, 188,399 files / 99 GB) was complete,
then ran the authoritative staged-vs-live diff: **47,749 files deleted** from the `@Clients` tree
since 10:05 AM 6/24, 1,685 added. Spot-checks confirmed the deletions are real (present in staging,
absent from live). Classification showed the loss is overwhelmingly **duplicate cleanup**: 33,711 in
folders literally labeled "duplicate DO NOT USE or delete", plus 10,696 in nested-misfile buckets
(`A\A`, `D\A`, `P\O`, `H\I`) whose canonical client folders were verified still present in live —
leaving only **~3,342 genuinely-deleted client/training files** as the real loss.

Disproved the prior session's premise that the deletion happened in the 6/24 10:05->12:05 window.
Restored the 6/24 12:05 PM post-deletion point (`C:\PST-Recovery\PostDelete-0624`, 188,621 files)
and diffed it against the 10:05 point: **only 2 files were deleted in that window** (Ballard, Kathy
and Rivera, Anthony SOAP PDFs), 32 added — the mass deletion occurred later. Resolved the trigger:
there were two Glennda folders — `EDWARDS, GLENDA` (single-N, 79 files, deleted) and
`EDWARDS, GLENNDA` (double-N, 121->127 files, alive and actively growing). Filename analysis (176
"Glennda" vs 27 "Glenda" occurrences) plus the live/active canonical folder confirm proper spelling
= **Glennda**; the deleted single-N folder was a misspelled duplicate. So the alarm folder was a
duplicate; her real records are intact.

Investigated the **Shelton missing-SOAP-notes** report (the actual reason the year-ago backup was
pulled). Found only 6 Shelton files (Linda 2015, Nancy x3 2011, Tina 2015, Roger 2015), all loose in
the `S\` root (no Shelton client folder), identical across live / 6/24-pre / 6/24-post — not a 2026
deletion. All 6 share CreationTime 2025-06-02 (a data-recovery/migration event). Attempted a scoped
restore of the `S\` subtree at the 6/29/2025 oldest point to check the year-ago state; it **failed —
the restore point has been purged** by the 365-day retention (today is 367 days past 6/29/2025). The
year-ago backup no longer exists, so any Shelton notes lost before ~2025-06-29 are unrecoverable via
backup.

Pivoted to access hardening. Listed the 62 AD security groups; the only custom ones are **Admin1**
and **Admin2**, both granted Full Control on `G:\Shares\Scanned`. Per Mike's direction, restricted
Admin1 from deleting client files, then fixed the group nesting (Admin2 was nested inside Admin1 —
inverted, since Admin1 is the less-entitled group and Admin2 the data-owner/superuser group), and
finally reduced Admin1 to true least privilege. End state on `G:\Shares\Scanned` (inherited across
the whole store): **Admin1 = allow `RX,W` + Deny `D,DC`** (read/write/edit only; no delete, rename,
permission-change, or ownership); **Admin2 = Full Control** (unchanged).

## Key Decisions

- Used **restore-and-local-diff** as the only trustworthy method; `cbb list` proven unreliable on
  the comma/space folder paths (false zeros and server-side timeouts), so per-restore-point folder
  counting was abandoned.
- Classified the 47,749 deletions as ~93% duplicate/intentional cleanup by (a) folder labels and
  (b) verifying nested-bucket clients still exist at their canonical letter in live — so the real
  loss is ~3,342 files, not a catastrophe.
- Restored the 12:05 PM point to precisely bound the incident window rather than trust the prior
  session's assumption; the 2-file result invalidated that assumption.
- Determined proper spelling from document filenames + which folder is live/active, not from the
  folder name (which was itself the misspelling).
- Denied delete to the **4 direct Admin1 users individually** first (CalistaA, ChristineZ, leslieW,
  SarahM) to avoid the Admin2 nesting cascade, then after decoupling the groups, **consolidated to a
  single Admin1 group Deny** (future-proof) and reduced the Admin1 allow to `RX,W`.
- Fixed nesting by **decoupling** (Remove-ADGroupMember Admin1 -Members Admin2), not by re-nesting
  the other direction — re-nesting Admin1 into Admin2 would have made the base admins transitive
  superusers.
- Kept the explicit Deny `D,DC` as defense-in-depth even though the reduced allow (`RX,W`) already
  excludes delete.

## Problems Encountered

- **Misread the resume target** — assumed "PST-SERVER investigation" = the Peaceful Spirit DFS
  rebuild; Mike corrected to the deleted-files scope. Logged as a correction to errorlog.
- **6/29/2025 oldest restore point purged** — the scoped `S\` restore failed with "Specified restore
  point not found"; 365-day retention aged it out. Year-ago backup unavailable for the Shelton check.
- **Backup-load command timeouts** — the MSP360 "Files Backup 2025" synthetic full was running
  (~294 GB), so several RMM commands lagged past their server timeout and were marked failed even
  though the icacls/AD operations actually applied. Worked around by verifying state after each and
  re-applying idempotent operations. One stale mid-propagation ACL read (root showed
  `RX,W,WDAC,WO` while the child already showed `RX,W`) was resolved by re-applying `/grant:r`.
- **Prior CRITICAL "backup stopped" item RESOLVED** — the 6/29 `cbb plan -s` only stopped that one
  run; the schedule resumed on its own. "Files Backup 2025" is running normally (retention 365 days,
  Deleted:0 on recent runs).

## Configuration Changes

- **AD (PEACEFULSPIRIT.local):** removed group **Admin2** from group **Admin1** (decoupled the
  nesting). Admin1 now = {CalistaA, ChristineZ, leslieW, SarahM}. Admin2 unchanged.
- **NTFS ACL `G:\Shares\Scanned`** (inheritance root; propagates to `@Clients` and all children):
  - Removed the 4 interim per-user Deny ACEs (CalistaA/ChristineZ/leslieW/SarahM).
  - Admin1 final: `(OI)(CI)(DENY)(D,DC)` + allow `(OI)(CI)(RX,W)` (was `(OI)(CI)(F)`).
  - Admin2 unchanged: `(OI)(CI)(F)`.
  - ACL backup saved on server: `C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt`.
- **PST-SERVER restore plans (cbb):** created `ZPostDelete0624` (RP 20260624190522 ->
  `C:\PST-Recovery\PostDelete-0624`, completed, auto-deleted on success); attempted `ZOldestS`
  (RP 20250629170034 -> failed, point purged).
- **Server staging artifacts** under `C:\PST-Recovery\`: `PreDelete-0624\` (99 GB),
  `PostDelete-0624\` (99 GB), `authdiff\` (deleted-files.txt, clean-client-deletions.txt, rollup.txt,
  summaries), `incidentdiff\` (deleted-in-window.txt, incident-summary.txt), `acl-backup-scanned-*.txt`.
- **Repo:** this session log only. Logged one `--correction` to `errorlog.md`.

## Credentials & Secrets

- No new credentials. Domain Admin used for the AD group change: `PEACEFULSPIRIT\sysadmin` /
  `r3tr0gradE99!` (vault `clients/peaceful-spirit/server`, field `credentials.password` — read via
  full `vault.sh get`; `get-field credentials.password` returns literal "null", known bug). Passed
  base64-wrapped in the RMM command_text (recoverable from RMM DB; rotation optional, internal).

## Infrastructure & Servers

- **PST-SERVER** 192.168.0.2, DC/DNS/RRAS/CA, Server 2016 Essentials. RMM agent
  `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.75). Data on `G:\Shares\Scanned\@Clients\@Clients`
  (doubly-nested). Live @Clients ~142,335 files / ~72 GB. C: 705 GB free.
- **MSP360/cbb:** account ACG-PST `084b5069-d634-434b-84a2-971b1dcb4b43`, bunch
  `6a121575-84a0-4e98-9c0f-4a656d1a5132`, prefix PST-SERVER, exe
  `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`, logs
  `C:\ProgramData\Online Backup\Logs\`. Retention **365 days**.
- **Restore points:** pre-incident `20260624170506` (6/24 10:05 AM), post `20260624190522`
  (6/24 12:05 PM). Oldest `20250629170034` (6/29/2025) **now purged**.
- **AD security groups (62 total).** Custom: Admin1 (Global) = CalistaA, ChristineZ, leslieW, SarahM;
  Admin2 (Global) = BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS. Both formerly Full
  Control on Scanned. All staff passwords reset ~2026-05-04/05.

## Commands & Outputs

- Authoritative diff (10:05 staging vs live): `stage=188,399 live=142,335 DELETED=47,749 ADDED=1,685`.
  Breakdown: 33,711 "duplicate DO NOT USE"; A\A=5,614 / D\A=2,532 / P\O=1,901 / H\I=649 (all verified
  duplicates, canonicals live); ~3,342 genuine.
- Incident-window diff (10:05 vs 12:05): `DELETED=2 (Ballard/Rivera), ADDED=32`; Glennda folder 121
  files at both points.
- Glennda spelling tally in filenames: `Glennda(double-N)=176, Glenda(single-N)=27`; live canonical
  `EDWARDS, GLENNDA VA REFERRAL` = 127 (growing), single-N deleted.
- Shelton: 6 files, all loose in `S\`, CreationTime 2025-06-02, content dates 2011-2015; identical
  across all three snapshots. Nearby active "Sheldon" family (Bill 2024, Krista 2023).
- AD decouple: `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred
  -ScriptBlock { Remove-ADGroupMember -Identity Admin1 -Members Admin2 -Confirm:$false }`.
- ACL: `icacls "G:\Shares\Scanned" /deny "PEACEFULSPIRIT\Admin1:(OI)(CI)(DE,DC)"` then
  `icacls "G:\Shares\Scanned" /grant:r "PEACEFULSPIRIT\Admin1:(OI)(CI)(RX,W)"`. Final verified:
  `Admin1:(OI)(CI)(DENY)(D,DC)` + `Admin1:(OI)(CI)(RX,W)`; `Admin2:(OI)(CI)(F)`.
- Reversal: `Add-ADGroupMember Admin1 -Members Admin2`; `icacls "G:\Shares\Scanned" /remove:d
  "PEACEFULSPIRIT\Admin1"`; restore allow via `/grant` or the saved ACL backup.

## Pending / Incomplete Tasks

1. **Deletion recovery (NOT started):** ~3,342 genuinely-deleted client/training files are
   recoverable from `C:\PST-Recovery\PreDelete-0624` staging via no-overwrite copy-back
   (robocopy `/XC /XN /XO`), excluding the duplicate/nested buckets. Awaiting Mike/Mara go — writes
   to live production HIPAA data.
2. **Glennda single-N duplicate:** confirm the deleted single-N folder had zero unique files vs the
   live double-N folder before writing it off entirely (offered, not run).
3. **Shelton:** year-ago backup purged; if recent Shelton notes ever existed and were lost before
   ~2025-06-29, they are unrecoverable via backup. Open question: were they ever scanned, or is
   "Shelton" a mishearing of the active "Sheldon" family? Needs client input.
4. **Admin1 ACL watch:** RX,W + Deny(D,DC) also blocks rename and delete-then-write "save" patterns
   for those 4 users. If any report inability to rename/save, carve an exception.
5. **Cleanup:** ~200 GB of staging on PST-SERVER `C:\PST-Recovery\` (PreDelete + PostDelete) can be
   removed once recovery is decided.
6. **Backup:** confirm the running synthetic-full "Files Backup 2025" completes cleanly.
7. **Wiki:** rebuild `wiki/clients/peaceful-spirit.md` (requested this session).

## Reference Information

- RMM API `http://172.16.3.30:3001`. Agent PST-SERVER `87293069-33b6-45e8-a68f-6811216cdb96`.
- Prior thread log: `session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md`.
- Server artifacts: `C:\PST-Recovery\{PreDelete-0624,PostDelete-0624,authdiff,incidentdiff,acl-backup-scanned-20260701-072725.txt}`.
- Vault: `clients/peaceful-spirit/server` (DA sysadmin).
- Data root: `G:\Shares\Scanned\@Clients\@Clients` (doubly-nested @Clients).