Files
claudetools/clients/birth-biologic/session-logs/2026-06/2026-06-26-mike-birthbio-mail-migration-and-datto-vm.md
Mike Swanson 4321dbbbc0 sync: auto-sync from GURU-5070 at 2026-06-27 04:42:51
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-27 04:42:51
2026-06-27 04:44:53 -07:00

287 lines
21 KiB
Markdown

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
## Session Summary
Worked two BirthBiologic migrations in parallel: the Datto Workplace → SharePoint migration and a
new Google Workspace → M365 mail migration. The Datto/SharePoint thread started as a "verify current
state" task but Mike clarified the migration host is a Jupiter VM, not BB-SERVER. Located it as the
libvirt domain "Windows Server 2016" (actual Windows hostname **ACG-DWP-X-BB**, actually Server 2019
build 17763) — an ACG-owned migration box running Datto Workplace Server + SPMT, **not enrolled in
RMM** and sitting on an APIPA address (no LAN). Diagnosed: host bridging was fine (vnet14 enslaved to
br0, carrier up); the guest simply wasn't getting a DHCP lease from pfSense after ~2 months parked.
Fixed with a static IP (172.16.3.45/22), installed the GuruRMM agent (enrolled under BirthBiologic /
Main Office), and confirmed Datto Workplace Server reconnected and is re-syncing. Established (via the
qemu guest agent and SPMT job storage) that the April 2026 migration only completed Supply Management
(160 files, custom script) + ITSvcs (excluded); the four large folders (Admin 5.8 GB, Donor Services
109 GB, Quality 28 GB, Activity Reports) were SPMT's job and last ran 2026-04-29 — completion still
unconfirmed. Per Mike, full reconciliation waits until Datto finishes re-syncing.
The larger thread was standing up the **Google Workspace → M365 mail migration** end-to-end. Confirmed
the Google super-admin (`sysadmin@birthbiologic.com`) lives in 1Password (Clients vault item "Google");
read it via the SOPS-vaulted 1Password service-account token and mirrored it into SOPS. Onboarded
BirthBio's tenant for **Exchange Operator** (already had Tenant Admin consented, so the suite was
provisioned programmatically — Exchange Operator SP created + Exchange Administrator role). Pulled the
authoritative Google roster via domain-wide delegation (20 accounts: 15 active, 5 suspended),
reconciled against M365, and surfaced two active accounts not on Mike's list (Dr. Chris Gillis
`medicaldirector@`, Michael Merritt `mmerritt@`) plus an address mismatch (Mindi is `mindim@` in
Google, `mmaher@` in M365).
Provisioned the M365 target side to Mike's licensing rules: active-12 → Business Premium (assigned BP
to Mei Mei + Valerie, freed Savanna's BP by moving her to Exchange-only); created Gillis + Merritt with
Exchange-only and vaulted their passwords; licensed the 4 disabled former employees with Exchange-only
(kept sign-in disabled) as future shared-mailbox targets. License math closed exactly: 14 Business
Premium + 7 Exchange Online Plan 1, all consumed.
Hit a real blocker creating the Gmail migration endpoint: Google returned `unauthorized_client … not
authorized for any of the scopes requested`. Root cause = the DWD grant had only 3 of Microsoft's
required **5** scopes (missing `m8/feeds` and `gmail.settings.sharing`); Google rejects the migration
token all-or-nothing. Verified the exact 5-scope string against live MS Learn + a Grok live-search
cross-check (Gemini CLI was down on this box), updated our runbook, and Mike re-authorized all 5 in the
BB Google console. After that the endpoint (`BB-Gmail`) created cleanly and **Batch 1 (14 live
mailboxes, mail + calendar + contacts) was created and auto-started — Status: Syncing**.
## Key Decisions
- **Datto VM gets a static IP (172.16.3.45), not a pfSense DHCP fix.** The fault was pfSense not
leasing this MAC after a long park; a static on the ACG server range (172.16.3.x) is the reliable,
convention-consistent fix. Follow-up: add a pfSense reservation or confirm it's outside the DHCP pool.
- **Enrolled the Datto VM under BirthBiologic / Main Office** (not AZ Computer Guru) since the box exists
solely for BirthBio's migration and we had that site key; reversible (agents can be moved).
- **Former employees migrate to shared mailboxes via a temp Exchange-only license** (migrate into a
licensed mailbox → convert to shared ≤50 GB = free → reclaim license). Source Google accounts must be
**un-suspended** during migration (Gmail API can't read suspended accounts).
- **Licensing tiers (Mike's rules):** active-users-list → Business Premium; live Google accounts not yet
in M365 (Gillis, Merritt) → Exchange-only ("E1" = Exchange Online Plan 1); formers → Exchange-only
(reclaimable). `operations@` stays BP through migration. Existing BP users left on BP (not downgraded).
- **Batch sequencing:** live users first (Batch 1); formers as Batch 2 after un-suspending them in Google
and freeing Workspace seats by suspending already-migrated live users.
- **Mindi mapped via the CSV `Username` column** (`EmailAddress=mmaher@`, `Username=mindim@`) — the
proper MS mechanism — plus a belt-and-suspenders `mindim@` proxy on her mailbox.
- **Target delivery domain = `birthbiologic.onmicrosoft.com`** for Batch 1 (no routing subdomain exists;
acceptable for a near-term cutover; MS prefers a subdomain for long coexistence).
- **Drove Exchange via REST `InvokeCommand`** (Exchange Operator app token) — the EXO PowerShell module
isn't installed and the app has no vaulted cert, so app-only Connect-ExchangeOnline wasn't available.
## Problems Encountered
- **Datto VM on APIPA (no LAN).** Host bridging fine; pfSense wasn't leasing the MAC. Fixed with static
172.16.3.45/22, GW 172.16.0.1, DNS 172.16.0.1+1.1.1.1. Verified gateway/internet/DNS + RMM check-in.
- **`vault.sh get-field` returned `null` (len 4)** for nested secrets until the field arg used dotted
path: `credentials.client_secret`, `credentials.credential`. Plain leaf names don't resolve.
- **SPB skuId mismatch.** The scope doc's BP GUID (`cbdc14ab-d96c-4132-b7f4-1f3a3a819bb4`) was stale; the
tenant's real SPB skuId is `cbdc14ab-d96c-4c30-b9f4-6ada7cdc1d46`. License assign 400'd until corrected.
- **License seat propagation lag** — Valerie's BP assign 400'd ("no available licenses") immediately after
freeing Savanna's seat; succeeded on retry seconds later.
- **`proxyAddresses` read-only via Graph** — adding Mindi's alias required Exchange `Set-Mailbox` (EXO),
not a Graph PATCH.
- **Gmail migration endpoint failed: `unauthorized_client … not authorized for any of the scopes
requested`.** DWD had 3 of 5 required scopes. Got the verbatim 5-scope string from MS Learn + Grok;
Mike re-authorized; endpoint then created.
- **`onboard365.sh` vault path** — looked at `/c/Users/guru/.claude/identity.json`; fixed by exporting
`VAULT_ROOT_ENV=/d/vault` (logged as friction).
- **GCP API enable initially run as the wrong identity** — Mike first ran `gcloud services enable` as
`sysadmin@birthbiologic.com` (no rights to ACG's project); succeeded once run as the ACG owner of
`acg-msp-access`.
- **Gemini CLI down** (`throwIneligibleOrProjectIdError`, needs interactive re-login) — used Grok for the
live-doc cross-check instead. Logged to errorlog.
## Configuration Changes
- **ACG-DWP-X-BB (Jupiter "Windows Server 2016" VM):** static IP 172.16.3.45/22, GW 172.16.0.1, DNS
172.16.0.1 + 1.1.1.1 (persistent). GuruRMM agent installed (universal installer), enrolled BirthBiologic
/ Main Office, agent `a4524e85-8a07-45d0-91b1-51ce7e2ca74a`.
- **BirthBio M365 tenant (19a568e8-…):** onboarded Exchange Operator (+ Defender Add-on) SPs via
`onboard365.sh provision`; roles assigned (Exchange Admin on Exchange Operator + Security Investigator,
CA Admin on Tenant Admin, User Admin + Auth Admin on User Manager).
- License changes: Mei Mei (`msenthavy`) +BP; Valerie (`vvaneaton`) +BP; Savanna (`sabron`) BP→EXO;
created `medicaldirector@` (Gillis) +EXO and `mmerritt@` (Merritt) +EXO; licensed `aboutte`, `araso`,
`khoffman`, `pnelson` with EXO (kept sign-in disabled).
- `Set-Mailbox mmaher@` added secondary `smtp:mindim@birthbiologic.com`.
- Created Gmail migration endpoint `BB-Gmail`; created + auto-started migration batch `BB-Batch1` (14
users, TargetDeliveryDomain `birthbiologic.onmicrosoft.com`, NotificationEmails sysadmin@).
- **Vault (pushed):** `clients/birth-biologic/google-workspace.sops.yaml`,
`clients/birth-biologic/m365-medicaldirector.sops.yaml`, `clients/birth-biologic/m365-mmerritt.sops.yaml`.
- **Repo:** updated `projects/msp-tools/runbooks/google-workspace-to-m365-migration.md` (exact 5-scope
string + all-or-nothing gotcha + Contacts-API-retired/People-API + GCP-owner notes).
- **errorlog.md:** gemini CLI failure entry (+ onboard365 vault-path friction).
## Credentials & Secrets
- **Google Workspace super-admin** `sysadmin@birthbiologic.com` (source tenant) — sourced from 1Password
Clients vault item "Google"; mirrored to SOPS `clients/birth-biologic/google-workspace.sops.yaml`
(`credentials.password`, 19 chars). Used for admin.google.com console (DWD/API) + as the migration
impersonation admin.
- **M365 mailbox — Dr. Chris Gillis** `medicaldirector@birthbiologic.com` — created this session; password
vaulted at `clients/birth-biologic/m365-medicaldirector.sops.yaml` (forceChangePasswordNextSignIn=true).
- **M365 mailbox — Michael Merritt** `mmerritt@birthbiologic.com` — created this session; password vaulted
at `clients/birth-biologic/m365-mmerritt.sops.yaml` (forceChangePasswordNextSignIn=true).
- App secrets used (already vaulted): Tenant Admin `msp-tools/computerguru-tenant-admin`
(`credentials.client_secret`); Exchange Operator `msp-tools/computerguru-exchange-operator`
(`credentials.client_secret`); Google SA `msp-tools/acg-msp-access-google-workspace`
(`credentials.credential`, full JSON); 1Password service token
`infrastructure/1password-service-account.sops.yaml`.
## Infrastructure & Servers
- **ACG-DWP-X-BB** — Jupiter libvirt domain "Windows Server 2016" (actually WS2019, build 17763). Windows
hostname ACG-DWP-X-BB. NIC virtio 52:54:00:d4:8e:59 on br0 (vnet14). Static 172.16.3.45/22. Runs Datto
Workplace Server (svc `datto_workplace_server.default`, proc WorkplaceServer) + SPMT (under
Administrator profile). RMM agent `a4524e85-8a07-45d0-91b1-51ce7e2ca74a`. Datto source tree
`C:\Users\Public\Desktop\Datto Workplace Server Projects`.
- **Jupiter** 172.16.3.20 (Unraid, virsh host). LAN 172.16.0.0/22, GW pfSense 172.16.0.1. guest-exec
helper at `/root/gx.sh` on Jupiter.
- **BB-SERVER** — RMM agent `6c02baa7-0f1c-4990-b466-c9ab9eaefd3b`. Also has Datto Workplace Server + the
original custom-script artifacts at `C:\GuruMigration` (bb-migration-state.json shows 160 Supply Mgmt +
49 ITSvcs uploaded in April).
- **BirthBio M365 tenant** `birthbiologic.com` / `19a568e8-9e88-413b-9341-cbc224b39145`.
- SPs: Tenant Admin `7a199b11-97fb-4e65-917d-f8d29a53ba49`; Exchange Operator
`bab4699b-32a3-4434-9cad-7a4a08cc4d9e`; Security Investigator `bf684a4b-…`; User Manager `3347ebcc-…`;
Defender Add-on `161b8f61-…`. New user objects: Gillis `1bd491e1-3ba6-4214-8c6d-46426f8681da`, Merritt
`117a3367-cd5f-4565-af11-af5ff089224f`.
- SKUs: Business Premium (SPB) `cbdc14ab-d96c-4c30-b9f4-6ada7cdc1d46` (14/14); Exchange Online Plan 1
(EXCHANGESTANDARD) `4b9405b0-7788-4568-add1-99614e613b69` (7/7). Accepted domains: birthbiologic.com
(default), birthbiologic.onmicrosoft.com.
- **Google project** `acg-msp-access` (number 806899474449). SA `acg-msp-access@acg-msp-access.iam.gserviceaccount.com`,
OAuth2 client ID `102231607889615995452`. APIs enabled: Gmail, Calendar (calendar-json), People.
- **Google roster (DWD pull):** 15 active, 5 suspended. Active staff emails per `clients/birth-biologic/
docs/migration/google-to-m365-scope.md`; Mindi = `mindim@` (Google) ↔ `mmaher@` (M365).
## Commands & Outputs
- **Required Google DWD scopes (exact, 5, comma-separated, no spaces):**
`https://mail.google.com/,https://www.googleapis.com/auth/calendar,https://www.google.com/m8/feeds/,https://www.googleapis.com/auth/gmail.settings.sharing,https://www.googleapis.com/auth/contacts`
(`m8/feeds` is a still-valid alias for the contacts scope, served by People API; legacy Contacts API
retired 2022, not enableable, not needed.)
- EXO via REST: `POST https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand` with Exchange
Operator app token (`scope=https://outlook.office365.com/.default`), body
`{"CmdletInput":{"CmdletName":"…","Parameters":{…}}}`. byte[] params (ServiceAccountKeyFileData, CSVData)
passed as **base64 strings**.
- `New-MigrationEndpoint -Gmail -Name BB-Gmail -ServiceAccountKeyFileData <b64> -EmailAddress sysadmin@birthbiologic.com` → created.
- `New-MigrationBatch -Name BB-Batch1 -SourceEndpoint BB-Gmail -CSVData <b64> -TargetDeliveryDomain birthbiologic.onmicrosoft.com -AutoStart -NotificationEmails sysadmin@` → Status=Syncing, Total=14.
- Get-MigrationUser BB-Batch1 → 14 Provisioning, 0 skipped (normal initial state).
- Datto source counts (ACG-DWP-X-BB): Admin 6,279/5.8GB · Donor Services 56,826/109GB · Quality 3,714/28GB
· Supply Mgmt 160/33MB · Activity Reports 1 · ITSvcs 52 (excluded).
## Pending / Incomplete Tasks
- **Batch 1 monitor → MX cutover.** Watch `BB-Batch1` Provisioning→Syncing→Synced. When Synced: flip MX in
SiteGround → M365, update SPF (`include:spf.protection.outlook.com`), enable/publish DKIM (2 CNAMEs),
autodiscover CNAME → autodiscover.outlook.com, run final delta, then **complete** the batch.
- **Batch 2 — 5 former employees → shared.** Un-suspend each in Google (free Workspace seats by suspending
migrated live users), run a Gmail batch (targets already EXO-licensed: aboutte, araso, khoffman, pnelson,
sabron), then convert to shared mailboxes and reclaim the 5 EXO licenses.
- **Datto → SharePoint reconciliation.** After ACG-DWP-X-BB finishes re-syncing with Datto cloud, compare
source vs each SharePoint site to confirm what the April SPMT run left unfinished (Admin / Donor Services
/ Quality / Activity Reports).
- **pfSense:** add a DHCP reservation for 172.16.3.45 (MAC 52:54:00:d4:8e:59) or confirm it's outside the pool.
- **Valerie VanEaton** — active (receiving daily; last *sent* 2026-05-13). Julie to confirm whether the
mid-May send drop-off = leave/departure; if departed, move her to the former→shared track.
- **Decisions still open:** confirm Merritt's long-term tier; whether `operations@` becomes shared post-migration.
- **Wiki:** BirthBio article is stale (says migration incomplete / 13 mailboxes) — recompile.
## Reference Information
- Migration scope doc: `clients/birth-biologic/docs/migration/google-to-m365-scope.md`.
- Runbook (updated): `projects/msp-tools/runbooks/google-workspace-to-m365-migration.md`.
- MS Learn: `manually-configuring-gsuite-for-migration` (scope string), `automated-migration-neweac`,
`google-workspace-migration-prerequisites`, `perform-g-suite-migration`.
- RMM install one-liner (BirthBio site): `irm https://rmm.azcomputerguru.com/install/BRIGHT-PEAK-5980/windows | iex`.
- Discord DMs to Mike: message_id 1520034139900739627 (initial DWD), 1520055625302675537 (corrected 5-scope).
- Vault enrollment key: `clients/birth-biologic/gururmm-site-main` (site BRIGHT-PEAK-5980, id 3b20ef97-…).
---
## Update: 04:42 PT (2026-06-27) — Datto->SharePoint delta completion, Quality recovery, April-vs-now reconcile, ticket #32187 billed
Continuation of the same session. Covers the SharePoint side: completing the additive delta, recovering an
accidentally-deleted Quality site, reconciling SharePoint to match Datto (source of truth), freezing the
Datto source, and updating/billing the Datto migration ticket.
### Session Summary (update)
After ACG-DWP-X-BB finished re-syncing with Datto cloud, ran the **Datto -> SharePoint delta**. The April
SPMT run was additive and never re-synchronized, so the delta only needed to add files that had never
transferred. Built `delta-recon-v2.ps1` (sanitize-aware reconcile: matches on both raw and sanitized
paths to find GENUINELY_MISSING files) and `delta-upload-v3.ps1` (simple-PUT for <=244MB auto-creating
parent folders, EnsureFolder + chunked for larger, FileShare.ReadWrite shared reads for Datto-locked
files, long-path `\\?\` for [IO.File] reads, SanRemote trim of leading/trailing spaces + trailing dots).
Reconciled to **0 missing** across Supply Management, Admin, Birth Biologic Activity Reports, Donor
Services (107 GB / ~57K files), and Quality. Renamed 19 Datto source files to match SharePoint (stripped
leading/trailing spaces + trailing dots).
**Quality site recovery.** The Quality Department SharePoint site was deleted 6/26. Unified audit log
showed `operations@` deleted the connected M365 Group, which cascaded (AAD -> SharePoint sync) to remove
the site. Restored from the SharePoint deleted-site recycle bin (cert-based app token; SP REST rejects
app-only tokens). Since Quality is being reorganized into the **Quality Systems Department (QSD)** site,
relocated the migrated Quality content there via server-side copy, then filled 44 missing + 3 file-lock
stragglers. Old `/sites/QualityDepartment` auto-purges ~7/26.
**April-vs-now divergence + mirror.** Because the April push was additive (not a sync), anything deleted,
moved, or changed in Datto after April was stale in SharePoint. Treating Datto as source of truth, built
a consolidated change-list (`consolidated_changelist.csv`): **1,583 deleted/moved + 161 modified (~1,744
differences)**. Cross-checked the SP unified audit log to find files users had created/edited directly in
SharePoint (operations@, ksteen, jbeck, etc. on live sites) and **flagged 11 to protect**. Ran
`mirror-execute.ps1` (re-validates each row against a frozen Datto set, DELETEs stale by path-addressed
Graph DELETE -> recycle bin, refreshes modified via PUT, skips protected): **deleted=1,564, refreshed=160,
protected-skip=11, fail=0**. For the 1 modified protected file, pushed the Datto version beside the user's
edit as "...Datto Copy.docx"; the 10 deleted/moved protected are SP-only (no Datto copy) -> left as-is.
**Froze the source.** Stopped + disabled the Datto Workplace Server service on ACG-DWP-X-BB so the source
no longer changes (also resolves the "reappearing files" complaint by removing the stale SP copies).
**Ticket #32187 (Datto, Syncro 109277420).** Posted a highly-detailed public+email completion/remediation
note and billed **5.0h Labor - Remote Business ($150/hr = $750)**. Posted #bot-alerts notifications.
### Key Decisions (update)
- **Datto = source of truth** for the reconcile; SharePoint mirrored to it. Deletes go to the SP recycle
bin (recoverable 93 days), never hard-deleted.
- **Protect user-touched SP files** — never overwrite/delete the 11 flagged via audit log; for the one
edited file, keep both (Datto pushed as "Datto Copy") rather than overwrite.
- **Relocate Quality content to QSD** rather than rebuild under the restored old site, matching the planned
reorg; let the old site auto-purge.
- **Root-cause correction (Mike):** the "reappearing" files were NOT a Datto resurrection. They were stale
SP copies sitting in SharePoint since the April additive push (files deleted from Datto after April were
never removed from SP). Rewrote the ticket note's root-cause section accordingly.
### Problems Encountered (update)
- **SP REST "Unsupported app only token"** -> SharePoint requires a cert-based token; granted
Sites.FullControl.All and used a client_assertion JWT (x5t = cert thumbprint b64url). Fixed.
- **Chunked-upload 400 into brand-new folders** -> switched to simple-PUT (auto-creates parents). Fixed.
- **Long-path SKIP-nofile** -> `\\?\` prefix for [IO.File] reads (not for Rename-Item/File.Move in PS5.1).
- **Filename 400s** = leading/trailing spaces / trailing dots -> SanRemote trim; renamed 19 source files.
- **Datto file-locks** -> FileShare.ReadWrite shared read. Fixed.
- **Background poller broke** (curl --data-binary @file errored each iteration due to $0-relative temp paths
under run_in_background) -> read the mirror log directly instead. Logged as friction.
- **bot-alert missing link** -> first #bot-alerts post for #32187 omitted the mandated `-> <link>` tail; the
helper posts text verbatim and does not auto-append. Reposted correctly + logged friction.
### Configuration Changes (update)
- Created on ACG-DWP-X-BB / scratchpad: `delta-recon-v2.ps1`, `delta-upload-v3.ps1`, `mirror-execute.ps1`,
`consolidated_changelist.csv`, divergence CSVs, `mirror-execute.log`.
- Stopped + disabled Datto Workplace Server service on ACG-DWP-X-BB.
- Renamed 19 Datto source files (whitespace/trailing-dot cleanup) under
`C:\Users\Public\Desktop\Datto Workplace Server Projects`.
### Pending / Incomplete Tasks (update)
- **Mail:** MX cutover still pending (Batch 1 complete). Then authorize Workspace write scopes
(apps.licensing + admin.directory.user + Licensing API), unlicense migrated Google users, run Batch 2.
- **SP-only user files** (Shift Coms / DEMO and similar) — decide whether to fold into Datto.
- Old `/sites/QualityDepartment` auto-purges ~7/26 (no action needed).
### Reference Information (update)
- Datto migration ticket: #32187 (Syncro id 109277420). Comment id 420992239 (public+email);
line item id 43043687 (5.0h Labor - Remote Business, product 1190473, $150).
- #bot-alerts: message_id 1520266361996316802 (corrected, with link).
- SP site IDs — Donor: `birthbiologic.sharepoint.com,bcbfa272-dc85-424c-af66-3f14c75ffeb4,8b0975dd-...`;
Admin: `...,1baf65c1-c4b3-4602-9111-1f99ae800023,...`; Supply: `...,4700ecf3-25ba-41b6-918c-9fe620038172,...`;
QSD: `...,3173c017-58bd-406a-8858-2c969667336f,...`.
- Tenant 19a568e8-9e88-413b-9341-cbc224b39145; Graph app client 709e6eed-0711-4875-9c44-2d3518c47063.