sync: auto-sync from GURU-5070 at 2026-06-30 17:21:06

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-30 17:21:06

clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md

@@ -0,0 +1,141 @@
+# Birth Biologic — Quality Systems Department -> Datto SharePoint sync (COMPLETED)
+
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Picked up the Birth Biologic Quality Systems Department -> Datto SharePoint migration after a
+Mac session (Mikes-MacBook-Air) worked it all day and failed to complete. The goal: make the
+SharePoint "Quality Systems Department" document library contain every file from the Datto
+Workplace source (3,768 files) on agent ACG-DWP-X-BB. Started by loading the pulled-in
+continuation docs and reconciling two contradictory records — an earlier one claiming a
+surgical 5-file delete "COMPLETE," and a later `CONTINUE-QUALITY-SYNC.md` describing a
+wipe-and-reupload stuck at 3,249/3,768.
+
+Rather than trust the stale docs, pulled live ground truth via the Graph delta endpoint:
+SharePoint held 3,337 files (only 17 >=4MB), gap of 431. Re-enumerated the Datto source via
+RMM, which revealed the actual scope the Mac missed: 3,768 files, of which 301 are >=4MB
+(~29.7 GB total, largest a 3.94 GB .mov). The Mac's upload script skipped every file >=4MB,
+so it could never converge — the entire large-file corpus (the bulk of the gap) was being
+ignored.
+
+Wrote a correct uploader (`upload-quality-final.ps1`): idempotent (skip a file if SharePoint
+already has it at matching size), simple PUT for <4MB and Graph chunked upload sessions for
+>=4MB, `\\?\` long-path handling, Graph token auto-refresh for long runs, an internal time
+budget, and a tailable progress log. Validated the risky path on the single largest file
+(3.94 GB) before the full run. That test also uncovered the second root cause: the RMM agent
+ignores the `timeout` field (caps ~300 s, producing the Mac's zombie "running" commands) and
+honors `timeout_seconds` instead.
+
+Cancelled the hung command `9e0fcfe8`, dispatched the full uploader with `timeout_seconds:
+10800`, and monitored to completion in the background. Result: 1,137 uploaded (283 large +
+854 small), 2,628 already-matching, 3 errors. The idempotent size-check also silently
+repaired ~700 files that existed but had a mismatched size (partial/corrupt residue from the
+earlier failed attempts).
+
+Verified the final state with a full Datto-vs-SharePoint diff: 0 Datto files missing from
+SharePoint (all 3,768 present), and exactly 4 differences — all live current work edited today
+by named client staff. Mike chose to preserve them. Documented the outcome in an authoritative
+completion doc, retired the contradictory continuation doc, logged the two lessons to
+errorlog, and saved two reference memories.
+
+## Key Decisions
+
+- Trusted live Graph/RMM enumeration over the committed docs — the docs contradicted each
+  other and were hours stale. Ground truth drove every decision.
+- Did not undo the Mac's wipe-and-reupload (already executed before pickup); finished the
+  rebuild additively (upload-only). Performed no deletions.
+- Fixed the >=4MB gap with proper chunked upload sessions rather than continuing to skip large
+  files (the actual reason the job never finished).
+- Validated the chunked-upload/long-path/Content-Range path on one real 3.94 GB file before
+  committing to a ~30 GB run.
+- Preserved the 4 live-work files (1 new xlsx + 3 open/locked docs edited today by staff)
+  rather than forcing a byte-identical match — overwriting would have destroyed today's work.
+  Confirmed with Mike.
+- Kept the client secret out of the repo (scripts carry `SECRET_PLACEHOLDER`; injected at
+  dispatch time from the vault into a scratch payload that was deleted).
+
+## Problems Encountered
+
+- **Upload could never reach 3,768:** prior script skipped all 301 files >=4MB (~29.7 GB).
+  Fixed with Graph chunked upload sessions (10 MB chunks, `Content-Range`).
+- **Zombie/hung RMM commands:** agent ignores `timeout`, capping at ~300 s. Fixed by using
+  `timeout_seconds` (both fields sent). Cancelled the stuck `9e0fcfe8` first.
+- **Recursive `/children` count timed out** (one HTTP call per folder). Switched to the Graph
+  `/root/delta` endpoint — whole drive in ~9 paged calls.
+- **`requests` not installed** on GURU-5070 Python. Rewrote checks with stdlib `urllib`.
+- **Long paths (>260) in Datto tree:** used the `\\?\` prefix for `[IO.File]` reads.
+- **3 upload errors (423 Locked / 409 Conflict):** files were locked because staff had them
+  open — live current work, correctly left untouched.
+- **SharePoint 3,769 vs Datto 3,768 (one extra):** diffed to `Temperature Excursion Log.xlsx`,
+  created today by client staff; preserved, not deleted.
+
+## Configuration Changes
+
+Created:
+- `clients/birth-biologic/scripts/enumerate-datto.ps1` — Datto enumeration + manifest writer.
+- `clients/birth-biologic/scripts/upload-quality-final.ps1` — idempotent chunked uploader
+  (carries `SECRET_PLACEHOLDER`, not a real secret).
+- `clients/birth-biologic/docs/migration/2026-06-30-quality-sync-COMPLETE.md` — authoritative
+  final record.
+- `.claude/memory/reference_gururmm_command_timeout_seconds.md`
+- `.claude/memory/reference_sharepoint_graph_large_file_upload.md`
+
+Modified:
+- `clients/birth-biologic/CONTINUE-QUALITY-SYNC.md` — replaced with a RESOLVED pointer.
+- `.claude/memory/MEMORY.md` — two new reference index lines.
+- `errorlog.md` — one `--friction` (timeout_seconds) + one `--correction` (>=4MB skip).
+
+## Credentials & Secrets
+
+- No new credentials created or discovered. Used existing vault entries only.
+- Graph app (client-credentials): vault `msp-tools/computerguru-tenant-admin`,
+  `credentials.client_secret`. Tenant `19a568e8-9e88-413b-9341-cbc224b39145`, client
+  `709e6eed-0711-4875-9c44-2d3518c47063`.
+- GuruRMM API: via `.claude/scripts/rmm-auth.sh` (vault
+  `infrastructure/gururmm-server.sops.yaml`).
+- Note: dispatching PowerShell that embeds the Graph secret leaves that secret in the RMM
+  command-history record (same as the prior Mac scripts). Acceptable here (internal RMM), but
+  a future hardening would have the agent fetch the secret itself.
+
+## Infrastructure & Servers
+
+- Agent: ACG-DWP-X-BB, id `a4524e85-8a07-45d0-91b1-51ce7e2ca74a`.
+- Datto source: `C:\Users\Public\Desktop\Datto Workplace Server Projects\Quality Department`
+  (3,768 files, 301 >=4MB, ~29.7 GB).
+- SharePoint drive: `b!F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9`.
+- RMM API (internal): `http://172.16.3.30:3001` (JWT via rmm-auth.sh).
+- VM work files: `C:\Windows\Temp\quality-manifest.txt`, `C:\Windows\Temp\quality-upload.log`.
+
+## Commands & Outputs
+
+- Count via delta: `GET /drives/{drive}/root/delta?$select=...&$top=500` — final live count 3,769.
+- Dispatch: `POST {RMM}/api/agents/{id}/command` with
+  `{command_type:"powershell", command:<script>, timeout_seconds:10800}`.
+- Poll: `GET {RMM}/api/commands/{id}` (status completed/failed/cancelled).
+- Cancel: `POST {RMM}/api/commands/{id}/cancel`.
+- Full run result: `up=1137 skip=2628 err=3 big=283 total=3768 reachedEnd=True`.
+- Errors: `409 Conflict` DEV39.Exhibit B.docx; `423 Locked` on both `...Current.docx` files.
+- Diff: 0 Datto files missing from SP; 1 extra in SP
+  (`LOGS/Equipment/3. Temp Excursions/Temperature Excursion Log.xlsx`, created 2026-06-30 by
+  Mary Ster, edited by Kristin Steen).
+
+## Pending / Incomplete Tasks
+
+- None functional. Migration complete and verified.
+- Optional future: parallel-stream uploader (multiple concurrent files + ~60 MiB chunks +
+  `Expect100Continue=$false`) to beat the ~40 Mbps single-session SharePoint Online ceiling on
+  large migrations.
+- Optional cleanup: remove `C:\Windows\Temp\quality-manifest.txt` / `quality-upload.log` from
+  ACG-DWP-X-BB (harmless, no secrets).
+
+## Reference Information
+
+- Authoritative doc: `clients/birth-biologic/docs/migration/2026-06-30-quality-sync-COMPLETE.md`
+- Command IDs: hung/cancelled `9e0fcfe8-0619-4a39-bd9c-6f5fd75c9b55`; full run
+  `4c978424-03cf-401c-805a-45162ff52be2`; big-file test
+  `fb3a6c4b-7158-4b77-9988-4326503753d8`.
+- Memories: `gururmm-command-timeout-seconds`, `sharepoint-graph-large-file-upload`.