azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from GURU-5070 at 2026-06-29 11:05:52

Browse Source 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-29 11:05:52
This commit is contained in:
Mike Swanson
2026-06-29 11:06:44 -07:00
parent eee062be3f
commit 1f15a6bc79
10 changed files with 3778 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

145
clients/birth-biologic/session-logs/2026-06/2026-06-29-mike-birthbio-google-audit-corruption-restore-mailgroups.md Normal file
View File

@@ -0,0 +1,145 @@
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
## Session Summary
Began with a request to enumerate BirthBiologic's Google Workspace email addresses. Confirmed the
client is mid-migration (Google -> M365), located the `acg-msp-access` service account with
domain-wide delegation, and built a Python puller (`bb_google_roster.py`) using the SA key from the
vault. The initial pull failed `unauthorized_client` because the DWD app had been removed from
Google Admin; Mike re-added it. Per-scope diagnosis showed every migration + `admin.directory.user`
scope worked but `admin.directory.group.readonly` would not issue a token (propagation lag that
never cleared within the session). Delivered the full user roster (20 mailboxes: 15 active, 5
suspended, one alias `mmaher@`) and stood up a 15-minute self-rescheduling timer to retry the group
pull until the group scope propagates (still pending at session save).
Mid-session, Mike reported an Excel "file format or extension is not valid" error on the Master
Donor Spreadsheet and an Inventory file. Diagnosed via RMM on ACG-DWP-X-BB: the 2026-06-26
Datto->SharePoint migration pass had written files' bytes as space-separated DECIMAL TEXT instead of
binary (a `"$bytes"` byte-array stringification bug in the base64/stdout-capture upload). The bug is
format-agnostic (xlsx/docx/pdf). Ran a full Phase-1 inventory of the 3,371 files modified 06-26:
84 confirmed corrupt (83 with intact Datto source, 1 without), ~3,197 clean, 89 deferred (cloud-only
files at >=260-char MAX_PATH paths, no Datto source). Wrote a recovery plan, full manifest CSV, and
remediation list to the client docs. On Mike's go, restored all 84 (83 copied from the verified
Datto source, 1 recovered by decoding the corrupt file's own decimal bytes back to a valid docx).
Verified the email migration: corrected a stale assumption (MX was still believed to be on Google;
Mike noted it was cut to M365 on Sat 06-27 — confirmed live MX `birthbiologic-com.mail.protection.outlook.com`).
Migration batch BB-Batch1 is fully Synced (14/14 mailboxes, 0 failures, 7 skipped items total).
Created the requested mail groups: `medicalrecords@` distribution group (14 members, external
senders allowed for processors), and granted Full Access + Send As on the existing `info@` and
`quality@` shared mailboxes.
Closed out by posting two customer-visible ticket updates (#32187 SharePoint migration / file
restore; #32451 email migration / mail groups), billing 2.0h remote split across both, building a
Datto->SharePoint "where files landed" map for Calm Ops (posted to #32187), and pinging Winter to
close the tickets and handle the low prepaid block.
## Key Decisions
- Used the SA + DWD path for the Google roster rather than interactive Admin console; made the puller
resilient (per-scope tokens) so users pull even while the group scope is unauthorized.
- Restored corrupted files via local source->OneDrive copy (native sync) rather than the buggy
custom-script upload path; the byte-stringify bug must never be reused.
- For the 1 corrupt file with no Datto source, decoded its decimal-text content back to binary
(lossless) rather than treating it as unrecoverable.
- Kept `info@`/`quality@` as shared mailboxes (granted access) instead of converting to distribution
groups — they already existed as shared mailboxes with potential migrated mail; conversion would be
destructive.
- `medicalrecords@` built as a distribution group with `RequireSenderAuthenticationEnabled=$false`
so external processors can reach all staff for time-sensitive donation needs.
- Left the unexpected 5.0h block deduction in place (Mike confirmed that pre-existing migration
charge was going to be billed anyway).
- 89 long-path/no-source files deferred (not silently dropped) for a robocopy/SPMT pass.
## Problems Encountered
- **Google DWD `unauthorized_client`:** app was missing from Google Admin (re-added by Mike); then
`admin.directory.group.readonly` would not propagate within the session. Per-scope diagnostic
isolated it; user scope works, group pending. Timer polling continues.
- **Tenant-admin vault field wrong in wiki:** documented `credentials.credential` returns 4-char
`null`; actual field is `credentials.client_secret`. Used the correct field; flagged for wiki fix.
- **Graph token tenant:** the tenant-admin vault `tenant_id` is ACG's home tenant; had to request the
token against BirthBio's tenant (`19a568e8-...`) for the multi-tenant app to access their sites.
- **Graph driveItem-by-path 404 + local MAX_PATH:** the 89 deep cloud-only files could not be
header-checked by Graph or local IO (even with `\\?\` and .NET long-path switches). Deferred.
- **Huge-arg friction:** passed ~20KB base64 inline via `jq --arg` -> "Argument list too long";
switched to reading the manifest CSV already staged on the endpoint. Logged (--friction).
- **Billing over-deduction:** creating the invoice on #32187 swept a pre-existing unbilled 5.0h
migration line item alongside the new 1.0h, deducting 6.0h (block 10->3 total, intended 2). Caught
immediately by re-reading the block, surfaced to Mike, left as-is on his OK. Logged (--friction):
always GET ticket line_items before invoicing a prepaid customer.
- **Stale MX assumption:** corrected via live `nslookup`; logged (--correction). MX cut to M365 06-27.
## Configuration Changes
- Created `clients/birth-biologic/docs/migration/2026-06-26-corruption-recovery-plan.md` (plan + Phase-1 results).
- Created `clients/birth-biologic/docs/migration/2026-06-26-corruption-manifest.csv` (full 3,371-row inventory).
- Created `clients/birth-biologic/docs/migration/2026-06-26-remediation-list.csv` (the 84 corrupt files + route).
- Created `clients/birth-biologic/docs/migration/datto-to-sharepoint-map.md` (Calm Ops deliverable).
- Updated `wiki/clients/birth-biologic.md` (MX -> M365 06-27; new mail groups; shared-mailbox access).
- Appended errorlog entries: migration byte-stringify defect; MX correction; huge-arg friction;
long-path friction; billing over-deduction friction.
- M365 (birthbiologic.com): created distribution group `medicalrecords@`; granted FullAccess+SendAs on
`info@` and `quality@` shared mailboxes.
- ACG-DWP-X-BB: restored 84 files in `OneDrive - Birth Biologic, LLC\` libraries from Datto source.
## Credentials & Secrets
- No new credentials created. Used existing vaulted secrets:
- `msp-tools/acg-msp-access-google-workspace.sops.yaml` -> `credentials.credential` (Google SA JSON key).
- `msp-tools/computerguru-tenant-admin.sops.yaml` -> `credentials.client_secret` (Graph app secret;
NOTE: correct field is `client_secret`, not the wiki's `credentials.credential`). `client_id`
`709e6eed-0711-4875-9c44-2d3518c47063`.
- `infrastructure/gururmm-server.sops.yaml` (RMM API admin).
- Syncro per-user key (mike) from the syncro skill.
## Infrastructure & Servers
- **Google Workspace:** domain birthbiologic.com; SA `acg-msp-access@acg-msp-access.iam.gserviceaccount.com`,
OAuth2 client ID `102231607889615995452` (the DWD key). Super-admin impersonated: `sysadmin@birthbiologic.com`.
DWD scopes present at save: mail, calendar, contacts, gmail.settings.sharing, admin.directory.user.readonly,
admin.directory.group.readonly. MISSING: `https://www.google.com/m8/feeds/` (re-add before final migration delta).
- **M365 tenant:** birthbiologic.com / `19a568e8-9e88-413b-9341-cbc224b39145`. MX (live):
`birthbiologic-com.mail.protection.outlook.com` (cut over 2026-06-27).
- **SharePoint sites:** Admin `https://birthbiologic.sharepoint.com/sites/admin`; Donor Services `/sites/donorservices`;
Supply Management `/sites/SupplyManagement`; Quality Department `/sites/QualityDepartment`.
- **RMM agent (Datto/SPMT VM):** ACG-DWP-X-BB, id `a4524e85-8a07-45d0-91b1-51ce7e2ca74a`, 172.16.3.45.
Datto source tree: `C:\Users\Public\Desktop\Datto Workplace Server Projects`. OneDrive libraries:
`C:\Users\Administrator\OneDrive - Birth Biologic, LLC\<Site> - Documents`.
## Commands & Outputs
- Google roster pull: `GOOGLE_SA_JSON="$(vault.sh get-field msp-tools/acg-msp-access-google-workspace.sops.yaml credentials.credential)" python bb_google_roster.py` -> 20 users.
- Per-scope DWD test isolated `admin.directory.group.readonly` as the only failing scope.
- Corruption signature: corrupt xlsx header `80 75 3 4...` = decimal of `0x50 0x4B 0x03 0x04` (PK);
corrupt pdf `37 80 68 70...` = decimal of `%PDF`. Format-agnostic stringified byte array.
- Restore result: `RESTORED OK: 83 / 83 FAILURES: 0`. Decode recovery: `RECOVERED via DECODE: 29938 bytes, header PK OK`.
- Migration batch: `BB-Batch1 Status=Synced Total=14 Synced=14 Failed=0`; 7 skipped items (jbeck 1, ksteen 6).
- EXO via Exchange Operator token: `get-token.sh <tenant> exchange-op` (needs `VAULT_ROOT_ENV=D:/vault`
since the skill reads the home identity.json which lacks vault_path). InvokeCommand for
New-DistributionGroup / Add-MailboxPermission / Add-RecipientPermission.
- Syncro: comments on #32187 (ids 421237477, 421244515) and #32451 (421237513); line items 43059746/43059748;
invoices 1650837688 ($0, swept 5.0h+1.0h) and 1650837689 ($0, 1.0h); prepaid block 10.0 -> 3.0.
## Pending / Incomplete Tasks
- **Google group pull** — waiting on `admin.directory.group.readonly` propagation; 15-min timer running.
When live, list groups incl. `quality@birthbiologic.com` (Mike confirmed groups exist).
- **m8/feeds scope** — re-add to DWD before the final Gmail migration delta (currently missing).
- **89 deferred files** — long-path/no-source cloud-only files modified 06-26; verify + remediate via
robocopy/SPMT (long-path native). Lower risk (no Datto source).
- **Migration batch finalize** — BB-Batch1 is Synced; finalize/complete when ready (review skipped items;
DataConsistencyScore=Investigate is from the 7 skipped items).
- **Tickets #32187 / #32451** — handed to Winter to close; block renewal outreach.
- **Wiki recompile** — run `/wiki-compile client:birth-biologic --full` to fold this session in.
## Reference Information
- Tickets: #32187 (id 109277420) SharePoint migration; #32451 (id 112986219) point Google->Outlook.
Customer ID 17983014. Prepaid block 3.0h after this session.
- Scratchpad artifacts (this session): bb_google_roster.py, bb_graph_check.py, bb_mig_parse.py,
bb_make_remlist.py (under the session scratchpad).
- Corruption root cause logged in errorlog.md (migration/datto-to-sharepoint, 2026-06-29).