sync: auto-sync from GURU-5070 at 2026-07-01 08:54:46
Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-01 08:54:46
This commit is contained in:
@@ -139,3 +139,33 @@ Modified:
|
||||
`4c978424-03cf-401c-805a-45162ff52be2`; big-file test
|
||||
`fb3a6c4b-7158-4b77-9988-4326503753d8`.
|
||||
- Memories: `gururmm-command-timeout-seconds`, `sharepoint-graph-large-file-upload`.
|
||||
|
||||
## Update: 09:00 PT (2026-07-01) — Ticket #32187 documented + rename scheduled
|
||||
|
||||
Followed the migration completion with client-facing ticket work and scheduling.
|
||||
|
||||
**Ticket #32187** (SharePoint Migration - Datto Workplace to SharePoint Online; id `109277420`;
|
||||
customer Birth Biologic `17983014`; owner Mike 1735):
|
||||
- Posted customer-visible + emailed completion note (comment `421583525`): Quality Systems
|
||||
Department final sync to Datto complete — all 3,768 Datto files verified present (0 missing),
|
||||
including the ~30 GB of large training/video files earlier passes missed; 4 live-work files
|
||||
(new Temperature Excursion Log + 3 open docs) preserved, nothing overwritten.
|
||||
- Posted customer-visible + emailed reply to Annise (comment `421593667`) re: her 2026-06-29
|
||||
request to rename "Quality Systems Department" back to "Quality Department" — confirmed we'll
|
||||
rename the Team + SharePoint site + Staff Portal link off-hours.
|
||||
- Set ticket status -> **Scheduled**.
|
||||
- Created Remote appointment `5628749055` for **tonight Wed 2026-07-01, 7:00-8:00 PM MST** for
|
||||
the off-hours rename.
|
||||
|
||||
**Coord fleet todo** `c051e97d` (project birth-biologic) captures the rename with the gotcha:
|
||||
renaming the Team changes the M365 Group/site **display name only**; the site URL
|
||||
(`/sites/QualitySystemsDepartment`) does NOT auto-change. Changing the site address to
|
||||
`/QualityDepartment` requires updating the Staff Portal link that points to it, or it breaks.
|
||||
Do off-hours (Quality staff use these docs during the day).
|
||||
|
||||
**Caveat:** Syncro "Do Not Invite" (suppress the customer calendar-invite email) is not
|
||||
API-controllable — toggle in the GUI on appointment `5628749055` if a customer invite is
|
||||
unwanted.
|
||||
|
||||
All Syncro writes posted to #bot-alerts. Next step: perform the rename tonight per the todo,
|
||||
then confirm on the ticket.
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** GURU-5070
|
||||
- **Role:** admin
|
||||
|
||||
## Session Summary
|
||||
|
||||
Resumed the Peaceful Spirit **PST-SERVER file-deletion investigation** (initially misread as the
|
||||
DFS rebuild thread; corrected to the deleted-files thread). All work via GuruRMM against PST-SERVER
|
||||
(192.168.0.2, agent `87293069-33b6-45e8-a68f-6811216cdb96`). Confirmed the prior session's 6/24
|
||||
10:05 AM pre-incident restore (`C:\PST-Recovery\PreDelete-0624`, 188,399 files / 99 GB) was complete,
|
||||
then ran the authoritative staged-vs-live diff: **47,749 files deleted** from the `@Clients` tree
|
||||
since 10:05 AM 6/24, 1,685 added. Spot-checks confirmed the deletions are real (present in staging,
|
||||
absent from live). Classification showed the loss is overwhelmingly **duplicate cleanup**: 33,711 in
|
||||
folders literally labeled "duplicate DO NOT USE or delete", plus 10,696 in nested-misfile buckets
|
||||
(`A\A`, `D\A`, `P\O`, `H\I`) whose canonical client folders were verified still present in live —
|
||||
leaving only **~3,342 genuinely-deleted client/training files** as the real loss.
|
||||
|
||||
Disproved the prior session's premise that the deletion happened in the 6/24 10:05->12:05 window.
|
||||
Restored the 6/24 12:05 PM post-deletion point (`C:\PST-Recovery\PostDelete-0624`, 188,621 files)
|
||||
and diffed it against the 10:05 point: **only 2 files were deleted in that window** (Ballard, Kathy
|
||||
and Rivera, Anthony SOAP PDFs), 32 added — the mass deletion occurred later. Resolved the trigger:
|
||||
there were two Glennda folders — `EDWARDS, GLENDA` (single-N, 79 files, deleted) and
|
||||
`EDWARDS, GLENNDA` (double-N, 121->127 files, alive and actively growing). Filename analysis (176
|
||||
"Glennda" vs 27 "Glenda" occurrences) plus the live/active canonical folder confirm proper spelling
|
||||
= **Glennda**; the deleted single-N folder was a misspelled duplicate. So the alarm folder was a
|
||||
duplicate; her real records are intact.
|
||||
|
||||
Investigated the **Shelton missing-SOAP-notes** report (the actual reason the year-ago backup was
|
||||
pulled). Found only 6 Shelton files (Linda 2015, Nancy x3 2011, Tina 2015, Roger 2015), all loose in
|
||||
the `S\` root (no Shelton client folder), identical across live / 6/24-pre / 6/24-post — not a 2026
|
||||
deletion. All 6 share CreationTime 2025-06-02 (a data-recovery/migration event). Attempted a scoped
|
||||
restore of the `S\` subtree at the 6/29/2025 oldest point to check the year-ago state; it **failed —
|
||||
the restore point has been purged** by the 365-day retention (today is 367 days past 6/29/2025). The
|
||||
year-ago backup no longer exists, so any Shelton notes lost before ~2025-06-29 are unrecoverable via
|
||||
backup.
|
||||
|
||||
Pivoted to access hardening. Listed the 62 AD security groups; the only custom ones are **Admin1**
|
||||
and **Admin2**, both granted Full Control on `G:\Shares\Scanned`. Per Mike's direction, restricted
|
||||
Admin1 from deleting client files, then fixed the group nesting (Admin2 was nested inside Admin1 —
|
||||
inverted, since Admin1 is the less-entitled group and Admin2 the data-owner/superuser group), and
|
||||
finally reduced Admin1 to true least privilege. End state on `G:\Shares\Scanned` (inherited across
|
||||
the whole store): **Admin1 = allow `RX,W` + Deny `D,DC`** (read/write/edit only; no delete, rename,
|
||||
permission-change, or ownership); **Admin2 = Full Control** (unchanged).
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- Used **restore-and-local-diff** as the only trustworthy method; `cbb list` proven unreliable on
|
||||
the comma/space folder paths (false zeros and server-side timeouts), so per-restore-point folder
|
||||
counting was abandoned.
|
||||
- Classified the 47,749 deletions as ~93% duplicate/intentional cleanup by (a) folder labels and
|
||||
(b) verifying nested-bucket clients still exist at their canonical letter in live — so the real
|
||||
loss is ~3,342 files, not a catastrophe.
|
||||
- Restored the 12:05 PM point to precisely bound the incident window rather than trust the prior
|
||||
session's assumption; the 2-file result invalidated that assumption.
|
||||
- Determined proper spelling from document filenames + which folder is live/active, not from the
|
||||
folder name (which was itself the misspelling).
|
||||
- Denied delete to the **4 direct Admin1 users individually** first (CalistaA, ChristineZ, leslieW,
|
||||
SarahM) to avoid the Admin2 nesting cascade, then after decoupling the groups, **consolidated to a
|
||||
single Admin1 group Deny** (future-proof) and reduced the Admin1 allow to `RX,W`.
|
||||
- Fixed nesting by **decoupling** (Remove-ADGroupMember Admin1 -Members Admin2), not by re-nesting
|
||||
the other direction — re-nesting Admin1 into Admin2 would have made the base admins transitive
|
||||
superusers.
|
||||
- Kept the explicit Deny `D,DC` as defense-in-depth even though the reduced allow (`RX,W`) already
|
||||
excludes delete.
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **Misread the resume target** — assumed "PST-SERVER investigation" = the Peaceful Spirit DFS
|
||||
rebuild; Mike corrected to the deleted-files scope. Logged as a correction to errorlog.
|
||||
- **6/29/2025 oldest restore point purged** — the scoped `S\` restore failed with "Specified restore
|
||||
point not found"; 365-day retention aged it out. Year-ago backup unavailable for the Shelton check.
|
||||
- **Backup-load command timeouts** — the MSP360 "Files Backup 2025" synthetic full was running
|
||||
(~294 GB), so several RMM commands lagged past their server timeout and were marked failed even
|
||||
though the icacls/AD operations actually applied. Worked around by verifying state after each and
|
||||
re-applying idempotent operations. One stale mid-propagation ACL read (root showed
|
||||
`RX,W,WDAC,WO` while the child already showed `RX,W`) was resolved by re-applying `/grant:r`.
|
||||
- **Prior CRITICAL "backup stopped" item RESOLVED** — the 6/29 `cbb plan -s` only stopped that one
|
||||
run; the schedule resumed on its own. "Files Backup 2025" is running normally (retention 365 days,
|
||||
Deleted:0 on recent runs).
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
- **AD (PEACEFULSPIRIT.local):** removed group **Admin2** from group **Admin1** (decoupled the
|
||||
nesting). Admin1 now = {CalistaA, ChristineZ, leslieW, SarahM}. Admin2 unchanged.
|
||||
- **NTFS ACL `G:\Shares\Scanned`** (inheritance root; propagates to `@Clients` and all children):
|
||||
- Removed the 4 interim per-user Deny ACEs (CalistaA/ChristineZ/leslieW/SarahM).
|
||||
- Admin1 final: `(OI)(CI)(DENY)(D,DC)` + allow `(OI)(CI)(RX,W)` (was `(OI)(CI)(F)`).
|
||||
- Admin2 unchanged: `(OI)(CI)(F)`.
|
||||
- ACL backup saved on server: `C:\PST-Recovery\acl-backup-scanned-20260701-072725.txt`.
|
||||
- **PST-SERVER restore plans (cbb):** created `ZPostDelete0624` (RP 20260624190522 ->
|
||||
`C:\PST-Recovery\PostDelete-0624`, completed, auto-deleted on success); attempted `ZOldestS`
|
||||
(RP 20250629170034 -> failed, point purged).
|
||||
- **Server staging artifacts** under `C:\PST-Recovery\`: `PreDelete-0624\` (99 GB),
|
||||
`PostDelete-0624\` (99 GB), `authdiff\` (deleted-files.txt, clean-client-deletions.txt, rollup.txt,
|
||||
summaries), `incidentdiff\` (deleted-in-window.txt, incident-summary.txt), `acl-backup-scanned-*.txt`.
|
||||
- **Repo:** this session log only. Logged one `--correction` to `errorlog.md`.
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- No new credentials. Domain Admin used for the AD group change: `PEACEFULSPIRIT\sysadmin` /
|
||||
`r3tr0gradE99!` (vault `clients/peaceful-spirit/server`, field `credentials.password` — read via
|
||||
full `vault.sh get`; `get-field credentials.password` returns literal "null", known bug). Passed
|
||||
base64-wrapped in the RMM command_text (recoverable from RMM DB; rotation optional, internal).
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **PST-SERVER** 192.168.0.2, DC/DNS/RRAS/CA, Server 2016 Essentials. RMM agent
|
||||
`87293069-33b6-45e8-a68f-6811216cdb96` (v0.6.75). Data on `G:\Shares\Scanned\@Clients\@Clients`
|
||||
(doubly-nested). Live @Clients ~142,335 files / ~72 GB. C: 705 GB free.
|
||||
- **MSP360/cbb:** account ACG-PST `084b5069-d634-434b-84a2-971b1dcb4b43`, bunch
|
||||
`6a121575-84a0-4e98-9c0f-4a656d1a5132`, prefix PST-SERVER, exe
|
||||
`C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`, logs
|
||||
`C:\ProgramData\Online Backup\Logs\`. Retention **365 days**.
|
||||
- **Restore points:** pre-incident `20260624170506` (6/24 10:05 AM), post `20260624190522`
|
||||
(6/24 12:05 PM). Oldest `20250629170034` (6/29/2025) **now purged**.
|
||||
- **AD security groups (62 total).** Custom: Admin1 (Global) = CalistaA, ChristineZ, leslieW, SarahM;
|
||||
Admin2 (Global) = BridgetteSH, katieb, Mara, PSTAdmin, pst-admin, SharonS. Both formerly Full
|
||||
Control on Scanned. All staff passwords reset ~2026-05-04/05.
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
- Authoritative diff (10:05 staging vs live): `stage=188,399 live=142,335 DELETED=47,749 ADDED=1,685`.
|
||||
Breakdown: 33,711 "duplicate DO NOT USE"; A\A=5,614 / D\A=2,532 / P\O=1,901 / H\I=649 (all verified
|
||||
duplicates, canonicals live); ~3,342 genuine.
|
||||
- Incident-window diff (10:05 vs 12:05): `DELETED=2 (Ballard/Rivera), ADDED=32`; Glennda folder 121
|
||||
files at both points.
|
||||
- Glennda spelling tally in filenames: `Glennda(double-N)=176, Glenda(single-N)=27`; live canonical
|
||||
`EDWARDS, GLENNDA VA REFERRAL` = 127 (growing), single-N deleted.
|
||||
- Shelton: 6 files, all loose in `S\`, CreationTime 2025-06-02, content dates 2011-2015; identical
|
||||
across all three snapshots. Nearby active "Sheldon" family (Bill 2024, Krista 2023).
|
||||
- AD decouple: `Invoke-Command -ComputerName PST-SERVER.PEACEFULSPIRIT.local -Credential $cred
|
||||
-ScriptBlock { Remove-ADGroupMember -Identity Admin1 -Members Admin2 -Confirm:$false }`.
|
||||
- ACL: `icacls "G:\Shares\Scanned" /deny "PEACEFULSPIRIT\Admin1:(OI)(CI)(DE,DC)"` then
|
||||
`icacls "G:\Shares\Scanned" /grant:r "PEACEFULSPIRIT\Admin1:(OI)(CI)(RX,W)"`. Final verified:
|
||||
`Admin1:(OI)(CI)(DENY)(D,DC)` + `Admin1:(OI)(CI)(RX,W)`; `Admin2:(OI)(CI)(F)`.
|
||||
- Reversal: `Add-ADGroupMember Admin1 -Members Admin2`; `icacls "G:\Shares\Scanned" /remove:d
|
||||
"PEACEFULSPIRIT\Admin1"`; restore allow via `/grant` or the saved ACL backup.
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
1. **Deletion recovery (NOT started):** ~3,342 genuinely-deleted client/training files are
|
||||
recoverable from `C:\PST-Recovery\PreDelete-0624` staging via no-overwrite copy-back
|
||||
(robocopy `/XC /XN /XO`), excluding the duplicate/nested buckets. Awaiting Mike/Mara go — writes
|
||||
to live production HIPAA data.
|
||||
2. **Glennda single-N duplicate:** confirm the deleted single-N folder had zero unique files vs the
|
||||
live double-N folder before writing it off entirely (offered, not run).
|
||||
3. **Shelton:** year-ago backup purged; if recent Shelton notes ever existed and were lost before
|
||||
~2025-06-29, they are unrecoverable via backup. Open question: were they ever scanned, or is
|
||||
"Shelton" a mishearing of the active "Sheldon" family? Needs client input.
|
||||
4. **Admin1 ACL watch:** RX,W + Deny(D,DC) also blocks rename and delete-then-write "save" patterns
|
||||
for those 4 users. If any report inability to rename/save, carve an exception.
|
||||
5. **Cleanup:** ~200 GB of staging on PST-SERVER `C:\PST-Recovery\` (PreDelete + PostDelete) can be
|
||||
removed once recovery is decided.
|
||||
6. **Backup:** confirm the running synthetic-full "Files Backup 2025" completes cleanly.
|
||||
7. **Wiki:** rebuild `wiki/clients/peaceful-spirit.md` (requested this session).
|
||||
|
||||
## Reference Information
|
||||
|
||||
- RMM API `http://172.16.3.30:3001`. Agent PST-SERVER `87293069-33b6-45e8-a68f-6811216cdb96`.
|
||||
- Prior thread log: `session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md`.
|
||||
- Server artifacts: `C:\PST-Recovery\{PreDelete-0624,PostDelete-0624,authdiff,incidentdiff,acl-backup-scanned-20260701-072725.txt}`.
|
||||
- Vault: `clients/peaceful-spirit/server` (DA sysadmin).
|
||||
- Data root: `G:\Shares\Scanned\@Clients\@Clients` (doubly-nested @Clients).
|
||||
Reference in New Issue
Block a user