diff --git a/wiki/clients/birth-biologic.md b/wiki/clients/birth-biologic.md index 747d3d95..f59b3a8f 100644 --- a/wiki/clients/birth-biologic.md +++ b/wiki/clients/birth-biologic.md @@ -2,13 +2,20 @@ type: client name: birth-biologic display_name: BirthBiologic -last_compiled: 2026-06-26 +last_compiled: 2026-07-01 compiled_by: GURU-5070/claude-main sources: - clients/birth-biologic/session-logs/2026-04-21-session.md - clients/birth-biologic/session-logs/2026-06-02-session.md - clients/birth-biologic/session-logs/2026-06/2026-06-26-mike-birthbio-mail-migration-and-datto-vm.md + - clients/birth-biologic/session-logs/2026-06/2026-06-29-mike-birthbio-google-audit-corruption-restore-mailgroups.md + - clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-consolidation-corruption.md + - clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md + - clients/birth-biologic/docs/migration/2026-06-30-quality-sync-COMPLETE.md + - clients/birth-biologic/docs/migration/datto-to-sharepoint-map.md - clients/birth-biologic/docs/migration/google-to-m365-scope.md + - clients/birth-biologic/docs/migration/2026-06-26-corruption-recovery-plan.md + - clients/birth-biologic/docs/migration/2026-06-29-quality-dept-archival-plan.md backlinks: - projects/gururmm aliases: [birthbiologic] @@ -18,18 +25,17 @@ aliases: [birthbiologic] ## Profile -- **Company type:** Biological/healthcare services (cord blood / donor services implied by site structure: Donor Services, Quality Department, Birth Biologic Activity Reports); Stilwell, KS -- **Contract type:** Prepaid hour block +- **Company type:** Biological/healthcare services (cord blood / donor services); 19546 Metcalf Avenue, Stilwell KS +- **Contract type:** Prepaid hour block (~$132.03/month recurring + separate project/labor invoices) - **Key contacts:** - Annise — primary client contact for migration work; no last name or email documented - - Kristin Steen — ksteen@birthbiologic.com (known Syncro contact; workstation KSTEENBB2025) + - Kristin Steen — ksteen@birthbiologic.com, 316-833-9803 (known Syncro contact; workstation KSTEENBB2025) - sysadmin@birthbiologic.com — M365/Google shared admin account (ACG-managed); M365 Business Premium license assigned 2026-04-21; SharePoint admin role confirmed -- **Billing rate:** (verify — check Syncro invoices) -- **Hours remaining (prepaid):** 10.0 hrs as of 2026-06-26 +- **Billing rate:** (verify — recent labor invoices ~$150/hr remote; confirm in Syncro) +- **Hours remaining (prepaid):** 3.0 hrs as of 2026-07-01 (was 10.0 on 2026-06-26; dropped due to 5.0h migration billing + 2.0h sessions on 2026-06-29) - **Syncro customer ID:** 17983014 - **Managed assets (Syncro):** 13 -- **Open tickets:** 0 as of 2026-06-26 -- **Historical ticket:** #109277420 — Datto Workplace to SharePoint Migration; assigned Mike Swanson; contact Annise; closed/historical +- **Active ticket:** #32187 (Scheduled) — SharePoint Migration rename, 2026-07-01 7-8 PM MST ## Infrastructure @@ -38,19 +44,20 @@ aliases: [birthbiologic] | Host | IP | Role | OS | Notes | |---|---|---|---|---| | BB-SERVER | (verify) | On-premise Windows server | Windows Server 2016 | GuruRMM agent `6c02baa7-0f1c-4990-b466-c9ab9eaefd3b` installed 2026-04-21; Datto Workplace Server installed; custom Datto→SP migration script artifacts at `C:\GuruMigration`; state file shows 160 Supply Mgmt + 49 ITSvcs uploaded April 2026 | -| ACG-DWP-X-BB | 172.16.3.45 | ACG-owned Datto/SPMT migration VM (Jupiter libvirt) | Windows Server 2019 build 17763 (libvirt domain label "Windows Server 2016") | Static IP /22, GW 172.16.0.1, DNS 172.16.0.1+1.1.1.1; virtio NIC 52:54:00:d4:8e:59 on br0 (vnet14); Datto Workplace Server (svc `datto_workplace_server.default`) + SPMT (under Administrator profile); source tree `C:\Users\Public\Desktop\Datto Workplace Server Projects`; GuruRMM agent `a4524e85-8a07-45d0-91b1-51ce7e2ca74a` enrolled 2026-06-26 | +| ACG-DWP-X-BB | 172.16.3.45 | ACG-owned Datto/SPMT migration VM (Jupiter libvirt) | Windows Server 2019 build 17763 (libvirt domain label "Windows Server 2016") | Static IP /22, GW 172.16.0.1, DNS 172.16.0.1+1.1.1.1; virtio NIC 52:54:00:d4:8e:59 on br0 (vnet14); Datto Workplace Server (svc `datto_workplace_server.default`) **stopped + disabled 2026-06-27** (source frozen post-migration); SPMT under Administrator profile; source tree `C:\Users\Public\Desktop\Datto Workplace Server Projects`; GuruRMM agent `a4524e85-8a07-45d0-91b1-51ce7e2ca74a` enrolled 2026-06-26 | ### Email & Identity - **M365 tenant:** birthbiologic.com / tenant ID `19a568e8-9e88-413b-9341-cbc224b39145` - **Target delivery domain (migration):** birthbiologic.onmicrosoft.com - **Accepted domains:** birthbiologic.com (default), birthbiologic.onmicrosoft.com -- **MX (as of 2026-06-29):** **M365** (`birthbiologic-com.mail.protection.outlook.com`) — **cutover done 2026-06-27 (Sat)**; live mail now on M365 (was Google Workspace through 06-26). Always verify MX live; do not trust the 06-26 migration-scope docs. -- **Mail groups / shared mailboxes (created/configured 2026-06-29):** - - `medicalrecords@` — **distribution group**, 14 members (12 core staff + `medicaldirector@` + `mmerritt@`), `RequireSenderAuthenticationEnabled=$false` (external processors can email it). Functions as all-staff but is a distinct named group for time-sensitive processor outreach. +- **MX (as of 2026-06-29, confirmed live):** **M365** (`birthbiologic-com.mail.protection.outlook.com`) — **cutover done 2026-06-27 (Sat)**; live mail now on M365. Do not trust pre-2026-06-27 assumptions. +- **SPF / DKIM / autodiscover / DMARC:** (verify — should have been updated at MX cutover 2026-06-27; no session log confirms) +- **Mail groups / shared mailboxes (configured 2026-06-29):** + - `medicalrecords@` — **distribution group**, 14 members (12 core staff + `medicaldirector@` + `mmerritt@`), `RequireSenderAuthenticationEnabled=$false` (external processors can email it). - `info@` — **shared mailbox**; Full Access + Send As: Brandy Burgess, Julie Beck. - `quality@` — **shared mailbox**; Full Access + Send As: Brandy Burgess, Julie Beck, Mary Ster, Alicia Meneely, Kristin Steen, Vicki Fountain. - - Other existing shared mailboxes: `accounting@`, `operations@` (user mailbox). + - Other shared mailboxes: `accounting@`, `operations@` (user mailbox). - **DNS host:** SiteGround (`ns1/ns2.us92.siteground.us`); Registrar: Name.com; `www` → GCP 35.215.115.203 (not in scope) - **M365 licensing (all consumed as of 2026-06-26):** - Business Premium (skuId `cbdc14ab-d96c-4c30-b9f4-6ada7cdc1d46`): 14/14 @@ -62,11 +69,11 @@ aliases: [birthbiologic] - Security Investigator: consented (SP `bf684a4b-…`) - Tenant Admin: consented (app client_id `709e6eed-0711-4875-9c44-2d3518c47063`; SP object `7a199b11-97fb-4e65-917d-f8d29a53ba49`; consent redirect URI must be `https://azcomputerguru.com`, NOT `https://rmm.azcomputerguru.com`) - Exchange Operator: consented 2026-06-26 (SP `bab4699b-32a3-4434-9cad-7a4a08cc4d9e`; Exchange Administrator role) - - User Manager: consented 2026-06-26 (SP `3347ebcc-…`) + - User Manager: consented 2026-06-26 (SP `3347ebcc-…`; has Group.ReadWrite.All — use this app for M365 group deletes, not Tenant Admin) - Defender Add-on: consented 2026-06-26 (SP `161b8f61-…`) - **Note:** sysadmin@birthbiologic.com did not have a SharePoint/M365 license prior to 2026-04-21. For SharePoint app-only access, use Tenant Admin app with `Sites.ReadWrite.All` (no user license required for app-only). -### Google Workspace (source tenant — migration in progress) +### Google Workspace (source tenant — mail migration completed for live users) - **Super-admin:** sysadmin@birthbiologic.com; password vaulted at `clients/birth-biologic/google-workspace.sops.yaml` (`credentials.password`) - **Domain-wide delegation:** acg-msp-access SA (`acg-msp-access@acg-msp-access.iam.gserviceaccount.com`); OAuth2 client ID `102231607889615995452`; GCP project `acg-msp-access` (number 806899474449) @@ -74,37 +81,38 @@ aliases: [birthbiologic] `https://mail.google.com/,https://www.googleapis.com/auth/calendar,https://www.google.com/m8/feeds/,https://www.googleapis.com/auth/gmail.settings.sharing,https://www.googleapis.com/auth/contacts` - **GCP APIs enabled on acg-msp-access:** Gmail, Calendar (calendar-json), People - **Google roster (DWD pull, 2026-06-26):** 20 accounts — 15 active, 5 suspended +- **DWD status (as of 2026-06-29):** `m8/feeds` scope was missing at that point (was present on 06-26 when migration ran, then dropped); must be re-added before running any final Gmail migration delta or Batch 2. -### Gmail Migration Status (as of 2026-06-26) +### Gmail Migration Status - **Method:** Native MS "Migration from Google Workspace" via Exchange Operator REST InvokeCommand - **Endpoint:** `BB-Gmail` (type: Gmail; impersonation admin: sysadmin@birthbiologic.com) -- **Batch 1 (BB-Batch1):** 14 live mailboxes, mail + calendar + contacts, TargetDeliveryDomain `birthbiologic.onmicrosoft.com`, AutoStart, NotificationEmails sysadmin@; **Status: Syncing** (created 2026-06-26) -- **Batch 2:** Not started — 5 former employees; pending un-suspend in Google + free Workspace seats +- **Batch 1 (BB-Batch1):** 14 live mailboxes, mail + calendar + contacts, TargetDeliveryDomain `birthbiologic.onmicrosoft.com`; **Status: Synced** (created + auto-started 2026-06-26; confirmed Synced 14/14, 0 failures, 7 skipped items as of 2026-06-29); DataConsistencyScore=Investigate (from 7 skipped items); **batch not yet finalized/completed** +- **Batch 2:** Not started — 5 former employees (`aboutte`, `araso`, `khoffman`, `pnelson`, `sabron`); pending un-suspend in Google + free Workspace seats ### File Storage -- **Pre-migration source:** Datto Workplace (server on ACG-DWP-X-BB; original custom-script artifacts on BB-SERVER at `C:\GuruMigration`) +- **Pre-migration source:** Datto Workplace (server on ACG-DWP-X-BB; original custom-script artifacts on BB-SERVER at `C:\GuruMigration`); Datto service stopped + disabled 2026-06-27 - **Post-migration target:** Microsoft SharePoint (M365) -- **Migration tools:** Custom PowerShell script (`clients/birth-biologic/scripts/migrate-datto-to-sharepoint.ps1`) + SPMT (on ACG-DWP-X-BB under Administrator profile) +- **Migration tools:** Custom PowerShell scripts (see `clients/birth-biologic/scripts/`) + SPMT (on ACG-DWP-X-BB under Administrator profile) ### SharePoint Site Map | Datto Folder | SharePoint Site | Size / Files | Status | |---|---|---|---| -| Admin | birthbiologic.sharepoint.com/sites/Admin | 5.8 GB / 6,279 files | SPMT last ran 2026-04-29; completion UNCONFIRMED | -| Birth Biologic Activity Reports | birthbiologic.sharepoint.com/sites/Admin (subfolder) | 1 file | SPMT; SPMT preserves source folder name as subfolder; UNCONFIRMED | -| Donor Services | birthbiologic.sharepoint.com/sites/DonorServices | 109 GB / 56,826 files | SPMT last ran 2026-04-29; completion UNCONFIRMED | -| Quality Department | **canonical: birthbiologic.sharepoint.com/sites/QualitySystemsDepartment** (interim/orphan site: /sites/QualityDepartment) | 28 GB / 3,714 files | Quality content split across TWO sites (corrected 2026-06-29). Canonical working site = Quality Systems Department; the OneDrive-sync target on ACG-DWP-X-BB is the interim QualityDepartment site. ~1,006 files (~2.1 GB) orphaned in QualityDepartment, missing from Quality Systems Department (LOGS 485/1.6GB, Bone Bank Onsite Audit 2025 427/440MB) — migration/reconciliation in progress. 4 Quality Systems Department-* spoke sites exist but empty. Orphan list: clients/birth-biologic/docs/migration/quality-orphaned-files.txt | +| Admin | birthbiologic.sharepoint.com/sites/Admin | 5.8 GB / ~6,300 files | Reconciled to 0 missing 2026-06-27 (delta-recon-v2 + delta-upload-v3) — COMPLETE | +| Birth Biologic Activity Reports | birthbiologic.sharepoint.com/sites/Admin (subfolder) | small / 1 file | SPMT; preserves source folder name as subfolder; reconciled 0 missing 2026-06-27 — COMPLETE | +| Donor Services | birthbiologic.sharepoint.com/sites/DonorServices | 109 GB / ~56,800 files | Reconciled to 0 missing 2026-06-27 — COMPLETE | +| Quality Department (Datto) | **canonical: birthbiologic.sharepoint.com/sites/QualitySystemsDepartment** | ~29.7 GB / 3,768 Datto files | COMPLETE 2026-06-30: all 3,768 Datto files present (0 missing); 1 staff-created file also in SP (3,769 total); 4 live-work files preserved. Old /sites/QualityDepartment duplicate site soft-deleted 2026-06-29 (group restorable ~30 days, site ~93 days from that date). | | Supply Management | birthbiologic.sharepoint.com/sites/SupplyManagement | 33 MB / 160 files | 160/160 migrated via custom PS script 2026-04-21 — COMPLETE | | ITSvcs | EXCLUDED | 52 files | ACG-owned folder; never client data | -Site IDs hardcoded in `$SITE_MAP` hashtable in the migration script. +Site IDs hardcoded in `$SITE_MAP` hashtable in the migration script. QSD site ID: `3173c017-58bd-406a-8858-2c969667336f` (drive `b!F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9`). Graph app for all SharePoint work: vault `msp-tools/computerguru-tenant-admin` (tenant `19a568e8-9e88-413b-9341-cbc224b39145`). ### Network - **ACG Jupiter (Datto VM host):** LAN 172.16.0.0/22, GW pfSense 172.16.0.1; Jupiter at 172.16.3.20 (Unraid, virsh); guest-exec helper `/root/gx.sh` -- **ACG-DWP-X-BB:** 172.16.3.45/22 static (was APIPA after ~2 months parked; pfSense DHCP not leasing that MAC; fixed 2026-06-26) +- **ACG-DWP-X-BB:** 172.16.3.45/22 static (was APIPA after ~2 months parked; pfSense DHCP not leasing that MAC; fixed 2026-06-26); pfSense DHCP reservation for MAC `52:54:00:d4:8e:59` not yet confirmed - **ISP / WAN (BirthBio site):** (verify) - **Firewall (BirthBio site):** (verify) - **VPN:** (verify) @@ -129,7 +137,7 @@ Site IDs hardcoded in `$SITE_MAP` hashtable in the migration script. | KSTEENBB2025 | KSTEENBB2025 | Windows 11 | `ee3c6aea-e9cc-4d2f-9e79-a38dd0eb129e` | — | Kristin Steen's workstation | | EVO-X1 | EVO-X1 | Windows 11 | `9595f002-5cfe-4db6-b7aa-1df4a20e9f9b` | — | Vicki Fountain's workstation; SmartBadge fleet reference machine | | BB-Office2 | BB-Office2 | Windows 11 | `48763401-4859-49f9-b64a-7a50d0148b23` | — | Shared/office workstation | -| ACG-DWP-X-BB | ACG-DWP-X-BB | Windows Server 2019 | `a4524e85-8a07-45d0-91b1-51ce7e2ca74a` | 172.16.3.45 | ACG-owned; Jupiter libvirt VM; Datto Workplace Server + SPMT migration host; enrolled 2026-06-26 under BirthBiologic/Main Office | +| ACG-DWP-X-BB | ACG-DWP-X-BB | Windows Server 2019 | `a4524e85-8a07-45d0-91b1-51ce7e2ca74a` | 172.16.3.45 | ACG-owned; Jupiter libvirt VM; Datto source + SPMT migration host; enrolled 2026-06-26; Datto service stopped 2026-06-27 | ## Access @@ -138,15 +146,16 @@ Site IDs hardcoded in `$SITE_MAP` hashtable in the migration script. - **Google Workspace admin:** sysadmin@birthbiologic.com (same account; password vaulted) - **Vault paths:** - `clients/birthbiologic/gururmm-site-main.sops.yaml` — GuruRMM site enrollment key - - `msp-tools/computerguru-tenant-admin.sops.yaml` → `credentials.credential` — Tenant Admin app secret + - `msp-tools/computerguru-tenant-admin.sops.yaml` → `credentials.client_secret` — Tenant Admin app secret (NOTE: field is `client_secret`, NOT `credential`; `credential` returns 4-char null) - `msp-tools/computerguru-exchange-operator.sops.yaml` → `credentials.client_secret` — Exchange Operator app secret + - `msp-tools/computerguru-user-manager.sops.yaml` → `credentials.client_secret` — User Manager app secret (use for M365 group deletes) - `msp-tools/acg-msp-access-google-workspace.sops.yaml` → `credentials.credential` — Google SA JSON key (full) - `clients/birth-biologic/google-workspace.sops.yaml` → `credentials.password` — Google Workspace super-admin password - `clients/birth-biologic/m365-medicaldirector.sops.yaml` — Dr. Chris Gillis M365 initial password (forceChangePasswordNextSignIn=true) - `clients/birth-biologic/m365-mmerritt.sops.yaml` — Michael Merritt M365 initial password (forceChangePasswordNextSignIn=true) - **Tenant Admin app:** client_id `709e6eed-0711-4875-9c44-2d3518c47063`; consent redirect URI must be `https://azcomputerguru.com` (NOT `https://rmm.azcomputerguru.com`) - **Exchange Operator SP:** `bab4699b-32a3-4434-9cad-7a4a08cc4d9e`; Exchange Administrator role; drive via REST InvokeCommand (see Patterns) -- **Migration script:** `clients/birth-biologic/scripts/migrate-datto-to-sharepoint.ps1` +- **Migration scripts:** `clients/birth-biologic/scripts/` (migrate-datto-to-sharepoint.ps1, enumerate-datto.ps1, upload-quality-final.ps1, bb-recover.py) - **Migration runbook:** `projects/msp-tools/runbooks/google-workspace-to-m365-migration.md` (updated 2026-06-26 — exact 5-scope string, all-or-nothing gotcha, Contacts API retired/People API, GCP-owner requirement) ## Patterns & Known Issues @@ -154,11 +163,16 @@ Site IDs hardcoded in `$SITE_MAP` hashtable in the migration script. - **Datto Workplace fleet standard = "Datto Workplace" v10.53.4 (installs to `C:\Program Files\Datto\Workplace2\`).** EVO-X1 and BB-Office2 run this version only. **Never** run the older "Datto Workplace **Desktop**" v8.50.13 (folder `…\Workplace Desktop\`) alongside it — having both installed breaks the Excel SmartBadge add-in (see below). Note the confusing naming: despite "Desktop" sounding newer, v8 Desktop is the *older* product; plain "Datto Workplace" v10 is current. - **SmartBadge Excel add-in failure from dual Datto Workplace installs:** When both Workplace2 (v10) and Workplace Desktop (v8) are present, the `_CC` COM class `{3C639243-95A2-400D-B4B4-4384DA7F61D3}` gets a 64-bit InprocServer32 pointing at the wrong DLL (or only a 32-bit WOW64 entry), so 64-bit Excel can't load the shim and silently drops the SmartBadge ribbon tab. Excel then auto-disables the add-in (per-user `LoadBehavior=2`). **Fix = align to fleet:** remove Workplace Desktop v8 (Revo for a full leftover sweep), install Workplace v10.53.4, ensure only the `_CC` add-in (HKLM+WOW64, `LoadBehavior=3`) with the `_CC` CLSID → `…\Workplace2\SmartBadge\DattoSmartBadgeShim_x64/x86.dll`, and reset the user's `LoadBehavior` to 3 + clear Excel Resiliency. Reference machine: EVO-X1. Scripts: `.claude/scripts/ksteen-smartbadge-verify.ps1`, `.claude/scripts/ksteen-smartbadge-fix.ps1`. - **Windows Server 2016 TLS:** BB-SERVER defaults to TLS 1.0. PowerShell scripts must include `[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12` at the top or Graph API calls will fail. -- **GuruRMM command timeout on long-running processes:** The RMM command channel times out on operations running longer than ~300 seconds. An 8 MB PDF upload at ~77 KB/s exceeded this limit during the migration. Workaround: base64-encode file on server, capture stdout, decode and upload locally. +- **GuruRMM command dispatch: use `timeout_seconds`, not `timeout`:** The RMM agent **ignores** the `timeout` field and caps commands at ~300 seconds. For any long-running upload/migration command, use `timeout_seconds` (e.g. 10800 for 3h) — sending both fields is safe. Commands dispatched with only `timeout` go zombie ("running", no output, never complete). Root cause confirmed 2026-06-30 during Quality sync. Memory: `gururmm-command-timeout-seconds`. +- **SharePoint Graph uploads: chunked upload sessions required for files >=4MB:** Simple PUT to `/content` works only for files <4MB. Files >=4MB **must** use Graph upload sessions (`POST .../createUploadSession`, then PUT chunks with `Content-Range`; 10 MB chunks work reliably). Failing to use upload sessions silently skips large files — the Quality sync Mac session failed all day because 301 large files (~29.7 GB) were skipped this way. Memory: `sharepoint-graph-large-file-upload`. - **SharePoint 409 Conflict on retry:** If a chunked upload session is interrupted, a partial item remains in SharePoint. Subsequent upload sessions against the same path return 409 Conflict. Fix: DELETE the item before creating a new upload session. +- **Long Windows paths (>260 chars) require `\\?\` prefix:** The Datto source tree contains paths exceeding MAX_PATH. Use `\\?\` prefix for `[IO.File]` reads in PowerShell. Note: `Rename-Item` and `File.Move` in PS5.1 do NOT support `\\?\` — use `robocopy` or SPMT for long-path rename/move operations. +- **SharePoint single-session upload throttles ~40 Mbps:** For large migrations, parallel-stream uploaders (multiple concurrent file uploads, larger chunk sizes) would significantly improve throughput. +- **Tenant Admin app cannot delete M365 groups (403):** The Tenant Admin app has GroupMember write only, not Group.ReadWrite.All. `DELETE /groups/{id}` returns 403 via Tenant Admin app. Use the **User Manager app** for group deletes (returns HTTP 204). Also: the Tenant Admin app cannot manage SP site lock/spoke-site grants (`Unsupported app only token` on SP REST) — use PnP PowerShell as SharePoint Admin. +- **Byte-array stringification bug — RETIRED path:** The 2026-06-26 custom-script upload path passed file bytes as `"$bytes"`, which stringified the .NET byte array to space-separated decimal text instead of raw binary. Corrupt files are inflated ~3-4x; headers are decimal (e.g. `80 75 3 4...` for PK, `37 80 68 70...` for %PDF). 84 files were corrupted and restored from Datto source. This code path is permanently retired. **Never stringify a byte array in PowerShell** — use `[IO.File]::WriteAllBytes` for binary output. - **SPMT requires sysadmin to be SharePoint admin:** SPMT destination access requires the running account to have SharePoint admin rights. Confirm before scheduling future SPMT runs. - **Syncro comment rendering:** Use `
` for line breaks in Syncro comments. `