From 6f676672a8b395717d4042b51a65abff1a06fac0 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Wed, 1 Jul 2026 08:55:39 -0700 Subject: [PATCH] sync: auto-sync from GURU-5070 at 2026-07-01 08:54:46 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-01 08:54:46 --- ...-30-mike-birthbio-quality-sync-complete.md | 30 ++++ ...ke-pst-deletion-scope-shelton-admin-acl.md | 164 ++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md diff --git a/clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md b/clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md index ed4ff2eb..a992432e 100644 --- a/clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md +++ b/clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md @@ -139,3 +139,33 @@ Modified: `4c978424-03cf-401c-805a-45162ff52be2`; big-file test `fb3a6c4b-7158-4b77-9988-4326503753d8`. - Memories: `gururmm-command-timeout-seconds`, `sharepoint-graph-large-file-upload`. + +## Update: 09:00 PT (2026-07-01) — Ticket #32187 documented + rename scheduled + +Followed the migration completion with client-facing ticket work and scheduling. + +**Ticket #32187** (SharePoint Migration - Datto Workplace to SharePoint Online; id `109277420`; +customer Birth Biologic `17983014`; owner Mike 1735): +- Posted customer-visible + emailed completion note (comment `421583525`): Quality Systems + Department final sync to Datto complete — all 3,768 Datto files verified present (0 missing), + including the ~30 GB of large training/video files earlier passes missed; 4 live-work files + (new Temperature Excursion Log + 3 open docs) preserved, nothing overwritten. +- Posted customer-visible + emailed reply to Annise (comment `421593667`) re: her 2026-06-29 + request to rename "Quality Systems Department" back to "Quality Department" — confirmed we'll + rename the Team + SharePoint site + Staff Portal link off-hours. +- Set ticket status -> **Scheduled**. +- Created Remote appointment `5628749055` for **tonight Wed 2026-07-01, 7:00-8:00 PM MST** for + the off-hours rename. + +**Coord fleet todo** `c051e97d` (project birth-biologic) captures the rename with the gotcha: +renaming the Team changes the M365 Group/site **display name only**; the site URL +(`/sites/QualitySystemsDepartment`) does NOT auto-change. Changing the site address to +`/QualityDepartment` requires updating the Staff Portal link that points to it, or it breaks. +Do off-hours (Quality staff use these docs during the day). + +**Caveat:** Syncro "Do Not Invite" (suppress the customer calendar-invite email) is not +API-controllable — toggle in the GUI on appointment `5628749055` if a customer invite is +unwanted. + +All Syncro writes posted to #bot-alerts. Next step: perform the rename tonight per the todo, +then confirm on the ticket. diff --git a/clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md b/clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md new file mode 100644 index 00000000..459646a2 --- /dev/null +++ b/clients/peaceful-spirit/session-logs/2026-07/2026-07-01-mike-pst-deletion-scope-shelton-admin-acl.md @@ -0,0 +1,164 @@ +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Session Summary + +Resumed the Peaceful Spirit **PST-SERVER file-deletion investigation** (initially misread as the +DFS rebuild thread; corrected to the deleted-files thread). All work via GuruRMM against PST-SERVER +(192.168.0.2, agent `87293069-33b6-45e8-a68f-6811216cdb96`). Confirmed the prior session's 6/24 +10:05 AM pre-incident restore (`C:\PST-Recovery\PreDelete-0624`, 188,399 files / 99 GB) was complete, +then ran the authoritative staged-vs-live diff: **47,749 files deleted** from the `@Clients` tree +since 10:05 AM 6/24, 1,685 added. Spot-checks confirmed the deletions are real (present in staging, +absent from live). Classification showed the loss is overwhelmingly **duplicate cleanup**: 33,711 in +folders literally labeled "duplicate DO NOT USE or delete", plus 10,696 in nested-misfile buckets +(`A\A`, `D\A`, `P\O`, `H\I`) whose canonical client folders were verified still present in live — +leaving only **~3,342 genuinely-deleted client/training files** as the real loss. + +Disproved the prior session's premise that the deletion happened in the 6/24 10:05->12:05 window. +Restored the 6/24 12:05 PM post-deletion point (`C:\PST-Recovery\PostDelete-0624`, 188,621 files) +and diffed it against the 10:05 point: **only 2 files were deleted in that window** (Ballard, Kathy +and Rivera, Anthony SOAP PDFs), 32 added — the mass deletion occurred later. Resolved the trigger: +there were two Glennda folders — `EDWARDS, GLENDA` (single-N, 79 files, deleted) and +`EDWARDS, GLENNDA` (double-N, 121->127 files, alive and actively growing). Filename analysis (176 +"Glennda" vs 27 "Glenda" occurrences) plus the live/active canonical folder confirm proper spelling += **Glennda**; the deleted single-N folder was a misspelled duplicate. So the alarm folder was a +duplicate; her real records are intact. + +Investigated the **Shelton missing-SOAP-notes** report (the actual reason the year-ago backup was +pulled). Found only 6 Shelton files (Linda 2015, Nancy x3 2011, Tina 2015, Roger 2015), all loose in +the `S\` root (no Shelton client folder), identical across live / 6/24-pre / 6/24-post — not a 2026 +deletion. All 6 share CreationTime 2025-06-02 (a data-recovery/migration event). Attempted a scoped +restore of the `S\` subtree at the 6/29/2025 oldest point to check the year-ago state; it **failed — +the restore point has been purged** by the 365-day retention (today is 367 days past 6/29/2025). The +year-ago backup no longer exists, so any Shelton notes lost before ~2025-06-29 are unrecoverable via +backup. + +Pivoted to access hardening. Listed the 62 AD security groups; the only custom ones are **Admin1** +and **Admin2**, both granted Full Control on `G:\Shares\Scanned`. Per Mike's direction, restricted +Admin1 from deleting client files, then fixed the group nesting (Admin2 was nested inside Admin1 — +inverted, since Admin1 is the less-entitled group and Admin2 the data-owner/superuser group), and +finally reduced Admin1 to true least privilege. End state on `G:\Shares\Scanned` (inherited across +the whole store): **Admin1 = allow `RX,W` + Deny `D,DC`** (read/write/edit only; no delete, rename, +permission-change, or ownership); **Admin2 = Full Control** (unchanged). + +## Key Decisions + +- Used **restore-and-local-diff** as the only trustworthy method; `cbb list` proven unreliable on + the comma/space folder paths (false zeros and server-side timeouts), so per-restore-point folder + counting was abandoned. +- Classified the 47,749 deletions as ~93% duplicate/intentional cleanup by (a) folder labels and + (b) verifying nested-bucket clients still exist at their canonical letter in live — so the real + loss is ~3,342 files, not a catastrophe. +- Restored the 12:05 PM point to precisely bound the incident window rather than trust the prior + session's assumption; the 2-file result invalidated that assumption. +- Determined proper spelling from document filenames + which folder is live/active, not from the + folder name (which was itself the misspelling). +- Denied delete to the **4 direct Admin1 users individually** first (CalistaA, ChristineZ, leslieW, + SarahM) to avoid the Admin2 nesting cascade, then after decoupling the groups, **consolidated to a + single Admin1 group Deny** (future-proof) and reduced the Admin1 allow to `RX,W`. +- Fixed nesting by **decoupling** (Remove-ADGroupMember Admin1 -Members Admin2), not by re-nesting + the other direction — re-nesting Admin1 into Admin2 would have made the base admins transitive + superusers. +- Kept the explicit Deny `D,DC` as defense-in-depth even though the reduced allow (`RX,W`) already + excludes delete. + +## Problems Encountered + +- **Misread the resume target** — assumed "PST-SERVER investigation" = the Peaceful Spirit DFS + rebuild; Mike corrected to the deleted-files scope. Logged as a correction to errorlog. +- **6/29/2025 oldest restore point purged** — the scoped `S\` restore failed with "Specified restore + point not found"; 365-day retention aged it out. Year-ago backup unavailable for the Shelton check. +- **Backup-load command timeouts** — the MSP360 "Files Backup 2025" synthetic full was running + (~294 GB), so several RMM commands lagged past their server timeout and were marked failed even + though the icacls/AD operations actually applied. Worked around by verifying state after each and + re-applying idempotent operations. One stale mid-propagation ACL read (root showed + `RX,W,WDAC,WO` while the child already showed `RX,W`) was resolved by re-applying `/grant:r`. +- **Prior CRITICAL "backup stopped" item RESOLVED** — the 6/29 `cbb plan -s` only stopped that one + run; the schedule resumed on its own. "Files Backup 2025" is running normally (retention 365 days, + Deleted:0 on recent runs). + +## Configuration Changes + +- **AD (PEACEFULSPIRIT.local):** removed group **Admin2** from group **Admin1** (decoupled the + nesting). Admin1 now = {CalistaA, ChristineZ, leslieW, SarahM}. Admin2 unchanged. +- **NTFS ACL `G:\Shares\Scanned`** (inheritance root; propagates to `@Clients` and all children): + - Removed the 4 interim per-user Deny ACEs (CalistaA/ChristineZ/leslieW/SarahM). + - Admin1 final: `(OI)(CI)(DENY)(D,DC)` + allow `(OI)(CI)(RX,W)` (was `(OI)(CI)(F)`). + - Admin2 unchanged: `(OI)(CI)(F)`. + - ACL backup saved on server: `C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt`. +- **PST-SERVER restore plans (cbb):** created `ZPostDelete0624` (RP 20260624190522 -> + `C:\PST-Recovery\PostDelete-0624`, completed, auto-deleted on success); attempted `ZOldestS` + (RP 20250629170034 -> failed, point purged). +- **Server staging artifacts** under `C:\PST-Recovery\`: `PreDelete-0624\` (99 GB), + `PostDelete-0624\` (99 GB), `authdiff\` (deleted-files.txt, clean-client-deletions.txt, rollup.txt, + summaries), `incidentdiff\` (deleted-in-window.txt, incident-summary.txt), `acl-backup-scanned-*.txt`. +- **Repo:** this session log only. Logged one `--correction` to `errorlog.md`. + +## Credentials & Secrets + +- No new credentials. Domain Admin used for the AD group change: `PEACEFULSPIRIT\sysadmin` / + `r3tr0gradE99!` (vault `clients/peaceful-spirit/server`, field `credentials.password` — read via + full `vault.sh get`; `get-field credentials.password` returns literal "null", known bug). Passed + base64-wrapped in the RMM command_text (recoverable from RMM DB; rotation optional, internal). + +## Infrastructure & Servers + +- **PST-SERVER** 192.168.0.2, DC/DNS/RRAS/CA, Server 2016 Essentials. RMM agent + `87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.75). Data on `G:\Shares\Scanned\@Clients\@Clients` + (doubly-nested). Live @Clients ~142,335 files / ~72 GB. C: 705 GB free. +- **MSP360/cbb:** account ACG-PST `084b5069-d634-434b-84a2-971b1dcb4b43`, bunch + `6a121575-84a0-4e98-9c0f-4a656d1a5132`, prefix PST-SERVER, exe + `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`, logs + `C:\ProgramData\Online Backup\Logs\`. Retention **365 days**. +- **Restore points:** pre-incident `20260624170506` (6/24 10:05 AM), post `20260624190522` + (6/24 12:05 PM). Oldest `20250629170034` (6/29/2025) **now purged**. +- **AD security groups (62 total).** Custom: Admin1 (Global) = CalistaA, ChristineZ, leslieW, SarahM; + Admin2 (Global) = BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS. Both formerly Full + Control on Scanned. All staff passwords reset ~2026-05-04/05. + +## Commands & Outputs + +- Authoritative diff (10:05 staging vs live): `stage=188,399 live=142,335 DELETED=47,749 ADDED=1,685`. + Breakdown: 33,711 "duplicate DO NOT USE"; A\A=5,614 / D\A=2,532 / P\O=1,901 / H\I=649 (all verified + duplicates, canonicals live); ~3,342 genuine. +- Incident-window diff (10:05 vs 12:05): `DELETED=2 (Ballard/Rivera), ADDED=32`; Glennda folder 121 + files at both points. +- Glennda spelling tally in filenames: `Glennda(double-N)=176, Glenda(single-N)=27`; live canonical + `EDWARDS, GLENNDA VA REFERRAL` = 127 (growing), single-N deleted. +- Shelton: 6 files, all loose in `S\`, CreationTime 2025-06-02, content dates 2011-2015; identical + across all three snapshots. Nearby active "Sheldon" family (Bill 2024, Krista 2023). +- AD decouple: `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred + -ScriptBlock { Remove-ADGroupMember -Identity Admin1 -Members Admin2 -Confirm:$false }`. +- ACL: `icacls "G:\Shares\Scanned" /deny "PEACEFULSPIRIT\Admin1:(OI)(CI)(DE,DC)"` then + `icacls "G:\Shares\Scanned" /grant:r "PEACEFULSPIRIT\Admin1:(OI)(CI)(RX,W)"`. Final verified: + `Admin1:(OI)(CI)(DENY)(D,DC)` + `Admin1:(OI)(CI)(RX,W)`; `Admin2:(OI)(CI)(F)`. +- Reversal: `Add-ADGroupMember Admin1 -Members Admin2`; `icacls "G:\Shares\Scanned" /remove:d + "PEACEFULSPIRIT\Admin1"`; restore allow via `/grant` or the saved ACL backup. + +## Pending / Incomplete Tasks + +1. **Deletion recovery (NOT started):** ~3,342 genuinely-deleted client/training files are + recoverable from `C:\PST-Recovery\PreDelete-0624` staging via no-overwrite copy-back + (robocopy `/XC /XN /XO`), excluding the duplicate/nested buckets. Awaiting Mike/Mara go — writes + to live production HIPAA data. +2. **Glennda single-N duplicate:** confirm the deleted single-N folder had zero unique files vs the + live double-N folder before writing it off entirely (offered, not run). +3. **Shelton:** year-ago backup purged; if recent Shelton notes ever existed and were lost before + ~2025-06-29, they are unrecoverable via backup. Open question: were they ever scanned, or is + "Shelton" a mishearing of the active "Sheldon" family? Needs client input. +4. **Admin1 ACL watch:** RX,W + Deny(D,DC) also blocks rename and delete-then-write "save" patterns + for those 4 users. If any report inability to rename/save, carve an exception. +5. **Cleanup:** ~200 GB of staging on PST-SERVER `C:\PST-Recovery\` (PreDelete + PostDelete) can be + removed once recovery is decided. +6. **Backup:** confirm the running synthetic-full "Files Backup 2025" completes cleanly. +7. **Wiki:** rebuild `wiki/clients/peaceful-spirit.md` (requested this session). + +## Reference Information + +- RMM API `http://172.16.3.30:3001`. Agent PST-SERVER `87293069-33b6-45e8-a68f-6811216cdb96`. +- Prior thread log: `session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md`. +- Server artifacts: `C:\PST-Recovery\{PreDelete-0624,PostDelete-0624,authdiff,incidentdiff,acl-backup-scanned-20260701-072725.txt}`. +- Vault: `clients/peaceful-spirit/server` (DA sysadmin). +- Data root: `G:\Shares\Scanned\@Clients\@Clients` (doubly-nested @Clients).