azcomputerguru/claudetools
1
0
Fork 0
Code Issues 58 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ecc0c8cd82f8bafd2f6537f72b7d9e8ff3e7b31b
claudetools/clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md
Mike Swanson 6f676672a8 sync: auto-sync from GURU-5070 at 2026-07-01 08:54:46 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-01 08:54:46
2026-07-01 08:55:39 -07:00

11 KiB
Raw Blame History

User

  • User: Mike Swanson (mike)
  • Machine: GURU-5070
  • Role: admin

Session Summary

Resumed the Peaceful Spirit PST-SERVER file-deletion investigation (initially misread as the DFS rebuild thread; corrected to the deleted-files thread). All work via GuruRMM against PST-SERVER (192.168.0.2, agent 87293069-33b6-45e8-a68f-6811216cdb96). Confirmed the prior session's 6/24 10:05 AM pre-incident restore (C:\PST-Recovery\PreDelete-0624, 188,399 files / 99 GB) was complete, then ran the authoritative staged-vs-live diff: 47,749 files deleted from the @Clients tree since 10:05 AM 6/24, 1,685 added. Spot-checks confirmed the deletions are real (present in staging, absent from live). Classification showed the loss is overwhelmingly duplicate cleanup: 33,711 in folders literally labeled "duplicate DO NOT USE or delete", plus 10,696 in nested-misfile buckets (A\A, D\A, P\O, H\I) whose canonical client folders were verified still present in live — leaving only ~3,342 genuinely-deleted client/training files as the real loss.

Disproved the prior session's premise that the deletion happened in the 6/24 10:05->12:05 window. Restored the 6/24 12:05 PM post-deletion point (C:\PST-Recovery\PostDelete-0624, 188,621 files) and diffed it against the 10:05 point: only 2 files were deleted in that window (Ballard, Kathy and Rivera, Anthony SOAP PDFs), 32 added — the mass deletion occurred later. Resolved the trigger: there were two Glennda folders — EDWARDS, GLENDA (single-N, 79 files, deleted) and EDWARDS, GLENNDA (double-N, 121->127 files, alive and actively growing). Filename analysis (176 "Glennda" vs 27 "Glenda" occurrences) plus the live/active canonical folder confirm proper spelling = Glennda; the deleted single-N folder was a misspelled duplicate. So the alarm folder was a duplicate; her real records are intact.

Investigated the Shelton missing-SOAP-notes report (the actual reason the year-ago backup was pulled). Found only 6 Shelton files (Linda 2015, Nancy x3 2011, Tina 2015, Roger 2015), all loose in the S\ root (no Shelton client folder), identical across live / 6/24-pre / 6/24-post — not a 2026 deletion. All 6 share CreationTime 2025-06-02 (a data-recovery/migration event). Attempted a scoped restore of the S\ subtree at the 6/29/2025 oldest point to check the year-ago state; it failed — the restore point has been purged by the 365-day retention (today is 367 days past 6/29/2025). The year-ago backup no longer exists, so any Shelton notes lost before ~2025-06-29 are unrecoverable via backup.

Pivoted to access hardening. Listed the 62 AD security groups; the only custom ones are Admin1 and Admin2, both granted Full Control on G:\Shares\Scanned. Per Mike's direction, restricted Admin1 from deleting client files, then fixed the group nesting (Admin2 was nested inside Admin1 — inverted, since Admin1 is the less-entitled group and Admin2 the data-owner/superuser group), and finally reduced Admin1 to true least privilege. End state on G:\Shares\Scanned (inherited across the whole store): Admin1 = allow RX,W + Deny D,DC (read/write/edit only; no delete, rename, permission-change, or ownership); Admin2 = Full Control (unchanged).

Key Decisions

  • Used restore-and-local-diff as the only trustworthy method; cbb list proven unreliable on the comma/space folder paths (false zeros and server-side timeouts), so per-restore-point folder counting was abandoned.
  • Classified the 47,749 deletions as ~93% duplicate/intentional cleanup by (a) folder labels and (b) verifying nested-bucket clients still exist at their canonical letter in live — so the real loss is ~3,342 files, not a catastrophe.
  • Restored the 12:05 PM point to precisely bound the incident window rather than trust the prior session's assumption; the 2-file result invalidated that assumption.
  • Determined proper spelling from document filenames + which folder is live/active, not from the folder name (which was itself the misspelling).
  • Denied delete to the 4 direct Admin1 users individually first (CalistaA, ChristineZ, leslieW, SarahM) to avoid the Admin2 nesting cascade, then after decoupling the groups, consolidated to a single Admin1 group Deny (future-proof) and reduced the Admin1 allow to RX,W.
  • Fixed nesting by decoupling (Remove-ADGroupMember Admin1 -Members Admin2), not by re-nesting the other direction — re-nesting Admin1 into Admin2 would have made the base admins transitive superusers.
  • Kept the explicit Deny D,DC as defense-in-depth even though the reduced allow (RX,W) already excludes delete.

Problems Encountered

  • Misread the resume target — assumed "PST-SERVER investigation" = the Peaceful Spirit DFS rebuild; Mike corrected to the deleted-files scope. Logged as a correction to errorlog.
  • 6/29/2025 oldest restore point purged — the scoped S\ restore failed with "Specified restore point not found"; 365-day retention aged it out. Year-ago backup unavailable for the Shelton check.
  • Backup-load command timeouts — the MSP360 "Files Backup 2025" synthetic full was running (~294 GB), so several RMM commands lagged past their server timeout and were marked failed even though the icacls/AD operations actually applied. Worked around by verifying state after each and re-applying idempotent operations. One stale mid-propagation ACL read (root showed RX,W,WDAC,WO while the child already showed RX,W) was resolved by re-applying /grant:r.
  • Prior CRITICAL "backup stopped" item RESOLVED — the 6/29 cbb plan -s only stopped that one run; the schedule resumed on its own. "Files Backup 2025" is running normally (retention 365 days, Deleted:0 on recent runs).

Configuration Changes

  • AD (PEACEFULSPIRIT.local): removed group Admin2 from group Admin1 (decoupled the nesting). Admin1 now = {CalistaA, ChristineZ, leslieW, SarahM}. Admin2 unchanged.
  • NTFS ACL G:\Shares\Scanned (inheritance root; propagates to @Clients and all children):
    • Removed the 4 interim per-user Deny ACEs (CalistaA/ChristineZ/leslieW/SarahM).
    • Admin1 final: (OI)(CI)(DENY)(D,DC) + allow (OI)(CI)(RX,W) (was (OI)(CI)(F)).
    • Admin2 unchanged: (OI)(CI)(F).
    • ACL backup saved on server: C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt.
  • PST-SERVER restore plans (cbb): created ZPostDelete0624 (RP 20260624190522 -> C:\PST-Recovery\PostDelete-0624, completed, auto-deleted on success); attempted ZOldestS (RP 20250629170034 -> failed, point purged).
  • Server staging artifacts under C:\PST-Recovery\: PreDelete-0624\ (99 GB), PostDelete-0624\ (99 GB), authdiff\ (deleted-files.txt, clean-client-deletions.txt, rollup.txt, summaries), incidentdiff\ (deleted-in-window.txt, incident-summary.txt), acl-backup-scanned-*.txt.
  • Repo: this session log only. Logged one --correction to errorlog.md.

Credentials & Secrets

  • No new credentials. Domain Admin used for the AD group change: PEACEFULSPIRIT\sysadmin / r3tr0gradE99! (vault clients/peaceful-spirit/server, field credentials.password — read via full vault.sh get; get-field credentials.password returns literal "null", known bug). Passed base64-wrapped in the RMM command_text (recoverable from RMM DB; rotation optional, internal).

Infrastructure & Servers

  • PST-SERVER 192.168.0.2, DC/DNS/RRAS/CA, Server 2016 Essentials. RMM agent 87293069-33b6-45e8-a68f-6811216cdb96 (v0.6.75). Data on G:\Shares\Scanned\@Clients\@Clients (doubly-nested). Live @Clients ~142,335 files / ~72 GB. C: 705 GB free.
  • MSP360/cbb: account ACG-PST 084b5069-d634-434b-84a2-971b1dcb4b43, bunch 6a121575-84a0-4e98-9c0f-4a656d1a5132, prefix PST-SERVER, exe C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe, logs C:\ProgramData\Online Backup\Logs\. Retention 365 days.
  • Restore points: pre-incident 20260624170506 (6/24 10:05 AM), post 20260624190522 (6/24 12:05 PM). Oldest 20250629170034 (6/29/2025) now purged.
  • AD security groups (62 total). Custom: Admin1 (Global) = CalistaA, ChristineZ, leslieW, SarahM; Admin2 (Global) = BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS. Both formerly Full Control on Scanned. All staff passwords reset ~2026-05-04/05.

Commands & Outputs

  • Authoritative diff (10:05 staging vs live): stage=188,399 live=142,335 DELETED=47,749 ADDED=1,685. Breakdown: 33,711 "duplicate DO NOT USE"; A\A=5,614 / D\A=2,532 / P\O=1,901 / H\I=649 (all verified duplicates, canonicals live); ~3,342 genuine.
  • Incident-window diff (10:05 vs 12:05): DELETED=2 (Ballard/Rivera), ADDED=32; Glennda folder 121 files at both points.
  • Glennda spelling tally in filenames: Glennda(double-N)=176, Glenda(single-N)=27; live canonical EDWARDS, GLENNDA VA REFERRAL = 127 (growing), single-N deleted.
  • Shelton: 6 files, all loose in S\, CreationTime 2025-06-02, content dates 2011-2015; identical across all three snapshots. Nearby active "Sheldon" family (Bill 2024, Krista 2023).
  • AD decouple: Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred -ScriptBlock { Remove-ADGroupMember -Identity Admin1 -Members Admin2 -Confirm:$false }.
  • ACL: icacls "G:\Shares\Scanned" /deny "PEACEFULSPIRIT\Admin1:(OI)(CI)(DE,DC)" then icacls "G:\Shares\Scanned" /grant:r "PEACEFULSPIRIT\Admin1:(OI)(CI)(RX,W)". Final verified: Admin1:(OI)(CI)(DENY)(D,DC) + Admin1:(OI)(CI)(RX,W); Admin2:(OI)(CI)(F).
  • Reversal: Add-ADGroupMember Admin1 -Members Admin2; icacls "G:\Shares\Scanned" /remove:d "PEACEFULSPIRIT\Admin1"; restore allow via /grant or the saved ACL backup.

Pending / Incomplete Tasks

  1. Deletion recovery (NOT started): ~3,342 genuinely-deleted client/training files are recoverable from C:\PST-Recovery\PreDelete-0624 staging via no-overwrite copy-back (robocopy /XC /XN /XO), excluding the duplicate/nested buckets. Awaiting Mike/Mara go — writes to live production HIPAA data.
  2. Glennda single-N duplicate: confirm the deleted single-N folder had zero unique files vs the live double-N folder before writing it off entirely (offered, not run).
  3. Shelton: year-ago backup purged; if recent Shelton notes ever existed and were lost before ~2025-06-29, they are unrecoverable via backup. Open question: were they ever scanned, or is "Shelton" a mishearing of the active "Sheldon" family? Needs client input.
  4. Admin1 ACL watch: RX,W + Deny(D,DC) also blocks rename and delete-then-write "save" patterns for those 4 users. If any report inability to rename/save, carve an exception.
  5. Cleanup: ~200 GB of staging on PST-SERVER C:\PST-Recovery\ (PreDelete + PostDelete) can be removed once recovery is decided.
  6. Backup: confirm the running synthetic-full "Files Backup 2025" completes cleanly.
  7. Wiki: rebuild wiki/clients/peaceful-spirit.md (requested this session).

Reference Information

  • RMM API http://172.16.3.30:3001. Agent PST-SERVER 87293069-33b6-45e8-a68f-6811216cdb96.
  • Prior thread log: session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md.
  • Server artifacts: C:\PST-Recovery\{PreDelete-0624,PostDelete-0624,authdiff,incidentdiff,acl-backup-scanned-20260701-072725.txt}.
  • Vault: clients/peaceful-spirit/server (DA sysadmin).
  • Data root: G:\Shares\Scanned\@Clients\@Clients (doubly-nested @Clients).
Reference in New Issue View Git Blame Copy Permalink