sync: auto-sync from GURU-5070 at 2026-06-29 11:45:50

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-29 11:45:50

session-logs/2026-06/2026-06-29-mike-birthbio-repatriation-and-pst-soap-recovery.md

@@ -0,0 +1,166 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Three distinct workstreams. First, a submodule pointer sync: committed five drifted submodule
+pointers (discord-bot, ampipit, guru-rmm, guru-scan, security-assessment) via
+`sync.sh --with-submodules` after an initial plain sync only captured the session log.
+
+Second, Birth Biologic data repatriation. Files in the Google Shared Drive
+`G:\Shared drives\Calm Ops  BirthBiologic` (Calm Ops's Google tenant; BirthBio owns the data,
+Calm Ops currently custodies it) needed to land in BirthBio's SharePoint. The drive held 46 items:
+17 real binaries (10.14 GB of .mov/.mp4/.m4a meeting recordings + 7 .docx) and 28 Google-native
+pointer files (.gsheet/.gdoc, 178-byte stubs with no local content). Because Calm Ops owns the
+source tenant, SPMT's Google connector was unavailable (no super-admin there); the chosen path was
+export via Mike's member access. Mike ran a Google Drive web "Download" of the native docs, which
+auto-converted gsheet->xlsx and gdoc->docx into a 33-file zip (the 9 gdoc + 19 gsheet minus 2
+Time Off Calendars that didn't export, plus the 7 already-real docx). A flat PowerShell uploader
+(Graph API, vaulted Tenant Admin app) was built. Initial run created a separate "Calm Ops - From
+Google" document library; Mike clarified the root site redirects to Admin Documents, so the library
+was deleted and the upload re-pointed to a `Calm Ops - From Google` folder inside the Documents
+library of the root Admin site. The 33 docs uploaded; the 10 GB of media was uploading in the
+background at session end (task bgmkd97rt; status unconfirmed).
+
+Third, and the bulk of the session: Peaceful Spirit forensic file-recovery. Mara Concordia
+reported client Glennda Edwards's SOAP notes (dated 4/29/2024-6/5/2026) missing from the practice's
+scanned-records share, and asked whether an audit trail could show when/who deleted them. Loaded
+PST context (wiki + the 6/13 DC-rebuild and 6/14 DFSR-rebuild logs). Located the records on
+PST-SERVER at `G:\Shares\Scanned\@Clients\@Clients\E\EDWARDS, GLENNDA VA REFERRAL`. Confirmed via
+GuruRMM recon that PST-SERVER runs MSP360/CloudBerry ("Online Backup 8.6", ACG white-label) backing
+up G:\Shares to Backblaze B2 (account ACG-PST), forever-forward-incremental archive format with
+2-hourly restore points back a full year. Also confirmed twice-daily VSS shadow copies (only back
+to 6/25 after the 6/13 trim) and that file-system auditing was OFF (no SACL, no "who").
+
+Using the MSP360 CLI `list` against restore points (reads the B2 index, no download), established
+that Glennda's folder held 105-107 files on 6/5 vs 8 live. Restored the deleted files: the
+new-format restore required `addRestorePlan` with -bunch + -restorePoint (legacy `-r` failed
+"not found"), and collided with the running 2-hourly backup ("backup and restore cannot run
+simultaneously"). After stopping the backup, restored to her original folder with no-overwrite,
+taking it from 8 -> 126 files (deleted SOAPs back, recent scans preserved). Enabled file-system
+delete auditing (auditpol + a Delete/DeleteSubdirectoriesAndFiles SACL on G:\Shares\Scanned) so
+future deletions are logged. Finally, binary-searched the restore points to date the deletion:
+the folder was stable at ~106 files from 6/5 through 6/24 10:05 AM, then collapsed to 1 file by
+6/24 12:05 PM. Conclusion: the bulk deletion (~106 files) occurred 6/24/2026 between 10:05 AM and
+12:05 PM, by an admin-rights account (the LMT shared login lacks delete rights). An initial wrong
+assumption that the 6/29 folder timestamp meant a same-day deletion was corrected by Mike and
+disproven by the restore-point data (6/29 timestamp was a later re-scan).
+
+## Key Decisions
+
+- BirthBio: chose export-via-member-access over SPMT because Calm Ops owns the source Google tenant
+  and we lack super-admin there; SPMT's Google connector needs the owning org's admin.
+- BirthBio: landed content in a `Calm Ops - From Google` folder inside the root Admin site's
+  Documents library (not a separate library) because the root site home redirects to Admin Documents.
+- PST: used MSP360 `list` against restore points as the diff/dating engine (reads the B2-stored
+  index, no egress) rather than restoring 70 GB to compare, and rather than per-folder VSS diffs
+  (shadow copies only reach 6/25, before the deletion).
+- PST: restored to original with no `-o` (no-overwrite) + `-deleted yes` so the deleted SOAPs came
+  back while the 8 surviving/newer files were untouched.
+- PST: stopped the running backup to free the engine for restores (forever-forward incrementals
+  resume safely on the next 2-hour cycle). Backup intentionally left paused at session end pending
+  further recovery — MUST be re-enabled.
+- PST: enabled delete auditing immediately (before further restores) given the deletion was by an
+  admin account and could recur.
+
+## Problems Encountered
+
+- MSP360 legacy `cbb -r` restore failed "not found in cloud storage" for the new backup format.
+  Fix: new format requires `addRestorePlan -bunch <planID> -restorePoint <id>` navigation.
+- `addRestorePlan -rt "point in time"` rejected ("Unexpected value of Restore type"). Fix: `-rt`
+  takes the date itself (e.g. "6/5/2026 10:04:49 PM"); the date signals point-in-time mode.
+- Restore plans exited with Warning / 0 files due to `GenerationSimultaneousUsageException` -
+  backup and restore can't run on the same data simultaneously. Fix: `cbb plan -s "Files Backup 2025"`
+  to stop the backup, then `cbb plan -r <restoreplan>`.
+- Restore plans are async (created then run by the service); `cbb plan -r` triggers; files appear
+  after a delay (polling the target folder count is the reliable signal, not the CLI return).
+- Wrong timeline assumption: anchored on Glennda's folder LastWriteTime of 6/29 9:12 AM as the
+  deletion time. Corrected by Mike ("Mara reported it days ago"); restore-point counts proved the
+  deletion was 6/24, and the 6/29 timestamp was a single later re-scan (7->8 files).
+- RMM commands doing many sequential `cbb` calls (each ~10-20s CLI startup) repeatedly hit the
+  server-side reaper timeout. Fix: cached the full restore-point list to a file on the box
+  (`C:\PST-Recovery\rps.txt`) and ran small targeted probe sets per command.
+- Bash tool's 120s ceiling kept cutting off local poll loops while RMM commands ran longer; polled
+  the command_id across follow-up calls.
+- Vault field name drift: wiki says the Tenant Admin secret is at `credentials.credential`; actual
+  field is `credentials.client_secret`.
+- BirthBio media glob `*.mov` (lowercase) missed the 5 uppercase `IMG_*.MOV`; fixed with a
+  case-insensitive extension filter.
+
+## Configuration Changes
+
+- Parent repo: committed 5 submodule pointer advances (discord-bot, ampipit, guru-rmm, guru-scan,
+  security-assessment) via `sync.sh --with-submodules`.
+- BirthBio SharePoint (root Admin site, `birthbiologic.sharepoint.com`): created then DELETED a
+  stray `Calm Ops - From Google` document library; created `Calm Ops - From Google` FOLDER in the
+  Documents library; uploaded 33 converted docs (+ 10 GB media in progress).
+- PST-SERVER `G:\Shares\Scanned\@Clients\@Clients\E\EDWARDS, GLENNDA VA REFERRAL`: restored from
+  MSP360 6/5 restore point, 8 -> 126 files.
+- PST-SERVER: file-system auditing ENABLED — `auditpol /set /subcategory:"File System"
+  /success:enable /failure:enable`; SACL added on `G:\Shares\Scanned` (Everyone,
+  Delete + DeleteSubdirectoriesAndFiles, Success, inheritable).
+- PST-SERVER: MSP360 backup plan "Files Backup 2025" STOPPED (paused) — needs re-enable.
+- Scratch on PST-SERVER: `C:\PST-Recovery\` (Gtest staging ~119 files, rps.txt, missing/touched
+  lists); leftover run-once restore plan ZRecGlenndaTest. Cleanup pending.
+
+## Credentials & Secrets
+
+- MSP360 Managed Backup console login for PST: `info@bestmassageintucson.com` (password NOT in
+  vault; managed-agent, console-side). Backup is UNENCRYPTED (no encryption key).
+- MSP360 API creds exist at vault `msp-tools/msp360-api.sops.yaml`; B2 storage key at
+  `projects/claudetools/backblaze-b2.sops.yaml`.
+- Tenant Admin app secret (BirthBio Graph): vault `msp-tools/computerguru-tenant-admin.sops.yaml`,
+  field `credentials.client_secret` (NOT `credentials.credential` as the wiki states).
+- PST domain admin: `PEACEFULSPIRIT\sysadmin` / `r3tr0gradE99!` (vault
+  `clients/peaceful-spirit/server`, field `credentials.password`).
+
+## Infrastructure & Servers
+
+- PST-SERVER 192.168.0.2, GuruRMM agent `87293069-33b6-45e8-a68f-6811216cdb96`. PST-SERVER2 is
+  DEAD per Mike ("won't come back with that name") — DFSR partner gone; recovery rests on MSP360
+  (B2) + VSS only.
+- MSP360 plan "Files Backup 2025" id `6a121575-84a0-4e98-9c0f-4a656d1a5132`; engine CLI
+  `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`; destination account ACG-PST id
+  `084b5069-d634-434b-84a2-971b1dcb4b43`; backup prefix `PST-SERVER`; source set includes
+  C:\DFSRoots, C:\Recovery, C:\Users\Public, C:\Users\sysadmin, D:\Unknown folder, G:\Shares;
+  2-hourly schedule; forever-forward archive.
+- 6/5 EOD restore point id `20260606050449` (6/5/2026 10:04:49 PM). 6/24 pre-delete RP near
+  10:05 AM = 107 files; 6/24 12:05 PM = 1 file.
+- PST delete-capable identities on G:\Shares\Scanned: SYSTEM, BUILTIN\Administrators, Domain Admins,
+  sysadmin, Admin1, Admin2 (NOT LMT).
+- BirthBio: tenant `19a568e8-9e88-413b-9341-cbc224b39145`; Tenant Admin app client_id
+  `709e6eed-0711-4875-9c44-2d3518c47063`; root Admin site id
+  `birthbiologic.sharepoint.com,1baf65c1-c4b3-4602-9111-1f99ae800023,4a1a8706-42a5-4d72-bb7d-75a846036623`;
+  Documents drive id `b!wWWvG7PEAkaRER-ZroAAIwaHGkqlQnJNu311qEYDZiN0Lx5ebwuIQZsGh07QO2dt`.
+
+## Commands & Outputs
+
+- MSP360 list at restore point (no download): `cbb.exe list -a ACG-PST -path "Files Backup 2025\<RP>\G:\Shares\Scanned\@Clients\@Clients\E\EDWARDS, GLENNDA VA REFERRAL"`.
+- New-format restore to original, no-overwrite: `cbb.exe addRestorePlan -n <name> -aid 084b5069-... -bp PST-SERVER -bunch 6a121575-... -restorePoint 20260606050449 -rt "6/5/2026 10:04:49 PM" -d "<dir>" -rl original -deleted yes -ro yes` then `cbb.exe plan -r "<name>"`.
+- Free engine: `cbb.exe plan -s "Files Backup 2025"`.
+- Glennda count by RP: 6/5-6/21 ~105-106; 6/24 10:05AM=107; 6/24 12:05PM=1; 6/27=7; live now=126 post-restore.
+
+## Pending / Incomplete Tasks
+
+- **CRITICAL: re-enable the PST MSP360 backup** ("Files Backup 2025" is stopped). Verify it resumes.
+- PST scope: diff restore points 6/24 10:05 AM vs 12:05 PM to find ALL files deleted in that event
+  (other clients beyond Glennda — Mara's concern). More targeted than the 6/5 full-tree diff.
+- PST "who": pull Security log logon events (4624) for 6/24 10:00-12:00 to identify which admin
+  account/workstation was active (no file-delete audit existed pre-today).
+- PST cleanup: remove `C:\PST-Recovery\Gtest` staging, rps.txt, and the run-once restore plan
+  ZRecGlenndaTest.
+- BirthBio: confirm the 10 GB media upload (task bgmkd97rt) finished; backfill the 2 Time Off
+  Calendar sheets that didn't export (likely Calm Ops-owned shortcuts).
+- Verify the restored Glennda files open correctly / notify Mara.
+
+## Reference Information
+
+- RMM API `http://172.16.3.30:3001`; agent `87293069-33b6-45e8-a68f-6811216cdb96`.
+- PST records root: `G:\Shares\Scanned\@Clients\@Clients\<Letter>\<CLIENT>` (provider + client
+  folders; ~2,536 client dirs; 141,054 live files).
+- Wiki: `wiki/clients/peaceful-spirit.md`, `wiki/clients/birth-biologic.md`.
+- BirthBio source: `G:\Shared drives\Calm Ops  BirthBiologic` (Calm Ops Google tenant, Mike is a member).
+- Errorlog-worthy: vault wiki field drift (tenant-admin secret), MSP360 new-format restore syntax,
+  simultaneous backup/restore lock.