claudetools/session-logs/2026-06/2026-06-29-mike-birthbio-repatriation-and-pst-soap-recovery.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Three distinct workstreams. First, a submodule pointer sync: committed five drifted submodule pointers (discord-bot, ampipit, guru-rmm, guru-scan, security-assessment) via sync.sh --with-submodules after an initial plain sync only captured the session log.

Second, Birth Biologic data repatriation. Files in the Google Shared Drive G:\Shared drives\Calm Ops BirthBiologic (Calm Ops's Google tenant; BirthBio owns the data, Calm Ops currently custodies it) needed to land in BirthBio's SharePoint. The drive held 46 items: 17 real binaries (10.14 GB of .mov/.mp4/.m4a meeting recordings + 7 .docx) and 28 Google-native pointer files (.gsheet/.gdoc, 178-byte stubs with no local content). Because Calm Ops owns the source tenant, SPMT's Google connector was unavailable (no super-admin there); the chosen path was export via Mike's member access. Mike ran a Google Drive web "Download" of the native docs, which auto-converted gsheet->xlsx and gdoc->docx into a 33-file zip (the 9 gdoc + 19 gsheet minus 2 Time Off Calendars that didn't export, plus the 7 already-real docx). A flat PowerShell uploader (Graph API, vaulted Tenant Admin app) was built. Initial run created a separate "Calm Ops - From Google" document library; Mike clarified the root site redirects to Admin Documents, so the library was deleted and the upload re-pointed to a Calm Ops - From Google folder inside the Documents library of the root Admin site. The 33 docs uploaded; the 10 GB of media was uploading in the background at session end (task bgmkd97rt; status unconfirmed).

Third, and the bulk of the session: Peaceful Spirit forensic file-recovery. Mara Concordia reported client Glennda Edwards's SOAP notes (dated 4/29/2024-6/5/2026) missing from the practice's scanned-records share, and asked whether an audit trail could show when/who deleted them. Loaded PST context (wiki + the 6/13 DC-rebuild and 6/14 DFSR-rebuild logs). Located the records on PST-SERVER at G:\Shares\Scanned\@Clients\@Clients\E\EDWARDS, GLENNDA VA REFERRAL. Confirmed via GuruRMM recon that PST-SERVER runs MSP360/CloudBerry ("Online Backup 8.6", ACG white-label) backing up G:\Shares to Backblaze B2 (account ACG-PST), forever-forward-incremental archive format with 2-hourly restore points back a full year. Also confirmed twice-daily VSS shadow copies (only back to 6/25 after the 6/13 trim) and that file-system auditing was OFF (no SACL, no "who").

Using the MSP360 CLI list against restore points (reads the B2 index, no download), established that Glennda's folder held 105-107 files on 6/5 vs 8 live. Restored the deleted files: the new-format restore required addRestorePlan with -bunch + -restorePoint (legacy -r failed "not found"), and collided with the running 2-hourly backup ("backup and restore cannot run simultaneously"). After stopping the backup, restored to her original folder with no-overwrite, taking it from 8 -> 126 files (deleted SOAPs back, recent scans preserved). Enabled file-system delete auditing (auditpol + a Delete/DeleteSubdirectoriesAndFiles SACL on G:\Shares\Scanned) so future deletions are logged. Finally, binary-searched the restore points to date the deletion: the folder was stable at ~106 files from 6/5 through 6/24 10:05 AM, then collapsed to 1 file by 6/24 12:05 PM. Conclusion: the bulk deletion (~106 files) occurred 6/24/2026 between 10:05 AM and 12:05 PM, by an admin-rights account (the LMT shared login lacks delete rights). An initial wrong assumption that the 6/29 folder timestamp meant a same-day deletion was corrected by Mike and disproven by the restore-point data (6/29 timestamp was a later re-scan).

Key Decisions

• BirthBio: chose export-via-member-access over SPMT because Calm Ops owns the source Google tenant and we lack super-admin there; SPMT's Google connector needs the owning org's admin.
• BirthBio: landed content in a Calm Ops - From Google folder inside the root Admin site's Documents library (not a separate library) because the root site home redirects to Admin Documents.
• PST: used MSP360 list against restore points as the diff/dating engine (reads the B2-stored index, no egress) rather than restoring 70 GB to compare, and rather than per-folder VSS diffs (shadow copies only reach 6/25, before the deletion).
• PST: restored to original with no -o (no-overwrite) + -deleted yes so the deleted SOAPs came back while the 8 surviving/newer files were untouched.
• PST: stopped the running backup to free the engine for restores (forever-forward incrementals resume safely on the next 2-hour cycle). Backup intentionally left paused at session end pending further recovery — MUST be re-enabled.
• PST: enabled delete auditing immediately (before further restores) given the deletion was by an admin account and could recur.

Problems Encountered

• MSP360 legacy cbb -r restore failed "not found in cloud storage" for the new backup format. Fix: new format requires addRestorePlan -bunch <planID> -restorePoint <id> navigation.
• addRestorePlan -rt "point in time" rejected ("Unexpected value of Restore type"). Fix: -rt takes the date itself (e.g. "6/5/2026 10:04:49 PM"); the date signals point-in-time mode.
• Restore plans exited with Warning / 0 files due to GenerationSimultaneousUsageException - backup and restore can't run on the same data simultaneously. Fix: cbb plan -s "Files Backup 2025" to stop the backup, then cbb plan -r <restoreplan>.
• Restore plans are async (created then run by the service); cbb plan -r triggers; files appear after a delay (polling the target folder count is the reliable signal, not the CLI return).
• Wrong timeline assumption: anchored on Glennda's folder LastWriteTime of 6/29 9:12 AM as the deletion time. Corrected by Mike ("Mara reported it days ago"); restore-point counts proved the deletion was 6/24, and the 6/29 timestamp was a single later re-scan (7->8 files).
• RMM commands doing many sequential cbb calls (each ~10-20s CLI startup) repeatedly hit the server-side reaper timeout. Fix: cached the full restore-point list to a file on the box (C:\PST-Recovery\rps.txt) and ran small targeted probe sets per command.
• Bash tool's 120s ceiling kept cutting off local poll loops while RMM commands ran longer; polled the command_id across follow-up calls.
• Vault field name drift: wiki says the Tenant Admin secret is at credentials.credential; actual field is credentials.client_secret.
• BirthBio media glob *.mov (lowercase) missed the 5 uppercase IMG_*.MOV; fixed with a case-insensitive extension filter.

Configuration Changes

• Parent repo: committed 5 submodule pointer advances (discord-bot, ampipit, guru-rmm, guru-scan, security-assessment) via sync.sh --with-submodules.
• BirthBio SharePoint (root Admin site, birthbiologic.sharepoint.com): created then DELETED a stray Calm Ops - From Google document library; created Calm Ops - From Google FOLDER in the Documents library; uploaded 33 converted docs (+ 10 GB media in progress).
• PST-SERVER G:\Shares\Scanned\@Clients\@Clients\E\EDWARDS, GLENNDA VA REFERRAL: restored from MSP360 6/5 restore point, 8 -> 126 files.
• PST-SERVER: file-system auditing ENABLED — auditpol /set /subcategory:"File System" /success:enable /failure:enable; SACL added on G:\Shares\Scanned (Everyone, Delete + DeleteSubdirectoriesAndFiles, Success, inheritable).
• PST-SERVER: MSP360 backup plan "Files Backup 2025" STOPPED (paused) — needs re-enable.
• Scratch on PST-SERVER: C:\PST-Recovery\ (Gtest staging ~119 files, rps.txt, missing/touched lists); leftover run-once restore plan ZRecGlenndaTest. Cleanup pending.

Credentials & Secrets

• MSP360 Managed Backup console login for PST: info@bestmassageintucson.com (password NOT in vault; managed-agent, console-side). Backup is UNENCRYPTED (no encryption key).
• MSP360 API creds exist at vault msp-tools/msp360-api.sops.yaml; B2 storage key at projects/claudetools/backblaze-b2.sops.yaml.
• Tenant Admin app secret (BirthBio Graph): vault msp-tools/computerguru-tenant-admin.sops.yaml, field credentials.client_secret (NOT credentials.credential as the wiki states).
• PST domain admin: PEACEFULSPIRIT\sysadmin / r3tr0gradE99! (vault clients/peaceful-spirit/server, field credentials.password).

Infrastructure & Servers

• PST-SERVER 192.168.0.2, GuruRMM agent 87293069-33b6-45e8-a68f-6811216cdb96. PST-SERVER2 is DEAD per Mike ("won't come back with that name") — DFSR partner gone; recovery rests on MSP360 (B2) + VSS only.
• MSP360 plan "Files Backup 2025" id 6a121575-84a0-4e98-9c0f-4a656d1a5132; engine CLI C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe; destination account ACG-PST id 084b5069-d634-434b-84a2-971b1dcb4b43; backup prefix PST-SERVER; source set includes C:\DFSRoots, C:\Recovery, C:\Users\Public, C:\Users\sysadmin, D:\Unknown folder, G:\Shares; 2-hourly schedule; forever-forward archive.
• 6/5 EOD restore point id 20260606050449 (6/5/2026 10:04:49 PM). 6/24 pre-delete RP near 10:05 AM = 107 files; 6/24 12:05 PM = 1 file.
• PST delete-capable identities on G:\Shares\Scanned: SYSTEM, BUILTIN\Administrators, Domain Admins, sysadmin, Admin1, Admin2 (NOT LMT).
• BirthBio: tenant 19a568e8-9e88-413b-9341-cbc224b39145; Tenant Admin app client_id 709e6eed-0711-4875-9c44-2d3518c47063; root Admin site id birthbiologic.sharepoint.com,1baf65c1-c4b3-4602-9111-1f99ae800023,4a1a8706-42a5-4d72-bb7d-75a846036623; Documents drive id b!wWWvG7PEAkaRER-ZroAAIwaHGkqlQnJNu311qEYDZiN0Lx5ebwuIQZsGh07QO2dt.

Commands & Outputs

• MSP360 list at restore point (no download): cbb.exe list -a ACG-PST -path "Files Backup 2025\<RP>\G:\Shares\Scanned\@Clients\@Clients\E\EDWARDS, GLENNDA VA REFERRAL".
• New-format restore to original, no-overwrite: cbb.exe addRestorePlan -n <name> -aid 084b5069-... -bp PST-SERVER -bunch 6a121575-... -restorePoint 20260606050449 -rt "6/5/2026 10:04:49 PM" -d "<dir>" -rl original -deleted yes -ro yes then cbb.exe plan -r "<name>".
• Free engine: cbb.exe plan -s "Files Backup 2025".
• Glennda count by RP: 6/5-6/21 ~105-106; 6/24 10:05AM=107; 6/24 12:05PM=1; 6/27=7; live now=126 post-restore.

Pending / Incomplete Tasks

• CRITICAL: re-enable the PST MSP360 backup ("Files Backup 2025" is stopped). Verify it resumes.
• PST scope: diff restore points 6/24 10:05 AM vs 12:05 PM to find ALL files deleted in that event (other clients beyond Glennda — Mara's concern). More targeted than the 6/5 full-tree diff.
• PST "who": pull Security log logon events (4624) for 6/24 10:00-12:00 to identify which admin account/workstation was active (no file-delete audit existed pre-today).
• PST cleanup: remove C:\PST-Recovery\Gtest staging, rps.txt, and the run-once restore plan ZRecGlenndaTest.
• BirthBio: confirm the 10 GB media upload (task bgmkd97rt) finished; backfill the 2 Time Off Calendar sheets that didn't export (likely Calm Ops-owned shortcuts).
• Verify the restored Glennda files open correctly / notify Mara.

Reference Information

• RMM API http://172.16.3.30:3001; agent 87293069-33b6-45e8-a68f-6811216cdb96.
• PST records root: G:\Shares\Scanned\@Clients\@Clients\<Letter>\<CLIENT> (provider + client folders; ~2,536 client dirs; 141,054 live files).
• Wiki: wiki/clients/peaceful-spirit.md, wiki/clients/birth-biologic.md.
• BirthBio source: G:\Shared drives\Calm Ops BirthBiologic (Calm Ops Google tenant, Mike is a member).
• Errorlog-worthy: vault wiki field drift (tenant-admin secret), MSP360 new-format restore syntax, simultaneous backup/restore lock.