diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index ce791afb..3499981a 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -32,6 +32,8 @@ - [Gitea Internal API Access](reference_gitea_internal.md) — git.azcomputerguru.com is NOT behind Cloudflare — it's the office Cox IP NAT'd to NPM (openresty) on Jupiter. Prefer internal 172.16.3.20:3000 for reliability (bypasses NPM SSL-renewal reload blips). - [Gitea git-op latency](reference_gitea_git_op_latency.md) — SSH (.20:2222) is SLOWEST (~1.5s); internal HTTP+token ~0.55s; SOPS lookup only ~0.33s. Don't switch to SSH for speed. Gitea SSH is .20:2222 (API ssh_url .21 is wrong). - [GuruRMM technical reference](reference_gururmm.md) — Server (172.16.3.30) layout + downloads dir `/var/www/gururmm/downloads` + `.channel` sidecar rollout control (stable/beta) + privileged server access via the server's OWN root RMM agent (hostname `gururmm`, no SSH needed; plink fallback) + API + `context=user_session` (WTS impersonation) + build-pipeline vendoring at `deploy/build-pipeline/` + Linux agent systemd sandbox trap. +- [GuruRMM command timeout_seconds](reference_gururmm_command_timeout_seconds.md) — agent command dispatch honors `timeout_seconds`, NOT `timeout`; long jobs die ~300s / go zombie (`running`, empty stdout) otherwise. Cost Birth Biologic a full day. +- [SharePoint Graph large-file upload](reference_sharepoint_graph_large_file_upload.md) — <4MB simple PUT, >=4MB MUST use chunked upload session (Content-Range); `\\?\` long paths; idempotent size-check; verify counts via /root/delta; single stream ~40Mbps (SPO throttle). - [RMM agent update model](rmm-agent-update-model.md) — Agent updates are server-PUSH on heartbeat (no self-poll); available versions = filesystem scan needing a `.sha256`; promote flips `.channel` sidecars beta→stable globally. Two stranders: beta-first freezes stable until an explicit promote; agents older than ~0.6.50 re-enroll with a NEW device_id/agent row when updated. - [GuruRMM physical server storage](gururmm-physical-server-storage.md) — New box 172.16.1.231 (temp IP→will be .30), Ubuntu 26.04, ssh key `gururmm-physical`/alias `gururmm-new`. SSD (915G root) = HOT (PG default tablespace + WAL + builds); HDD ext4 at `/data` = COLD (`gururmm_cold` PG tablespace for aged `agent_logs` partitions + downloads + backups + archive). The #3 retention answer. - [Trebesch DESKTOP-QNP3ON5 shell replacement](reference_trebesch_qnp3on5.md) — AT Trebesch box runs an Explorer shell replacement; explorer.exe owner check returns blank — use Win32_ComputerSystem.UserName. GuruRMM SWIFT-LION-2892. diff --git a/.claude/memory/reference_gururmm_command_timeout_seconds.md b/.claude/memory/reference_gururmm_command_timeout_seconds.md new file mode 100644 index 00000000..59930f66 --- /dev/null +++ b/.claude/memory/reference_gururmm_command_timeout_seconds.md @@ -0,0 +1,20 @@ +--- +name: gururmm-command-timeout-seconds +description: GuruRMM agent command dispatch honors timeout_seconds, NOT timeout — long jobs die at ~300s otherwise +metadata: + type: reference +--- + +When dispatching a command to a GuruRMM agent (`POST {RMM}/api/agents/{id}/command`), the +execution timeout is controlled by **`timeout_seconds`**, not `timeout`. Passing only +`timeout` leaves the agent at its default cap (~300s): long-running commands get killed and +often surface as a zombie `status: running` with empty stdout that never terminates. + +For multi-minute/multi-hour work (large uploads, big enumerations) set `timeout_seconds` high +(e.g. 10800) — send both fields to be safe. Poll `GET {RMM}/api/commands/{command_id}` for +`status` in {completed, failed, cancelled}; cancel a stuck one with +`POST {RMM}/api/commands/{id}/cancel`. + +Cost the fleet a full day of failed Birth Biologic SharePoint uploads (Mac session) before +this was spotted. See [[reference_sharepoint_graph_large_file_upload]]. Auth helper: +`.claude/scripts/rmm-auth.sh` (internal `172.16.3.30:3001`). diff --git a/.claude/memory/reference_sharepoint_graph_large_file_upload.md b/.claude/memory/reference_sharepoint_graph_large_file_upload.md new file mode 100644 index 00000000..fa039791 --- /dev/null +++ b/.claude/memory/reference_sharepoint_graph_large_file_upload.md @@ -0,0 +1,27 @@ +--- +name: sharepoint-graph-large-file-upload +description: Uploading files to SharePoint via Graph — simple PUT <4MB, chunked upload session >=4MB; verify counts via delta +metadata: + type: reference +--- + +Pushing a folder tree into a SharePoint doc library via Microsoft Graph (app-only): + +- **<4MB:** simple `PUT /drives/{drive}/root:/{path}:/content`. +- **>=4MB:** MUST use an **upload session** — `POST .../root:/{path}:/createUploadSession` + then `PUT` the file in chunks (multiple of 320 KiB; 10 MB works) with a + `Content-Range: bytes {start}-{end}/{total}` header. In PowerShell 5.1 + `Invoke-RestMethod -Headers @{ 'Content-Range'=... }` handles this fine. A naive script that + only does <4MB PUTs will silently skip every large file and never reach the target count. +- **Long Windows paths (>260):** prefix the local path with `\\?\` for `[IO.File]` reads. +- **Idempotent sync:** existence-check each file (`GET root:/{path}?$select=size`) and skip if + size matches — this also catches/repairs partial-upload residue from earlier failed runs. +- **Throughput:** a single sequential upload stream to SharePoint Online plateaus ~40 Mbps + regardless of link speed (per-session SPO throttle + PS5.1 HTTP stack + Expect100Continue). + For speed use parallel file streams + larger chunks + `Expect100Continue=$false`. +- **Verify total file count** with the Graph **delta** endpoint + (`/drives/{drive}/root/delta`) — whole-drive enumeration in a few paged calls, far cheaper + than recursive `/children`. + +Proven end-to-end on Birth Biologic Quality Systems (3,768 files, 301 >=4MB, ~29.7 GB; +largest 3.94 GB). Dispatched via GuruRMM — see [[gururmm-command-timeout-seconds]]. diff --git a/clients/birth-biologic/CONTINUE-QUALITY-SYNC.md b/clients/birth-biologic/CONTINUE-QUALITY-SYNC.md index 275ab6b2..daaba368 100644 --- a/clients/birth-biologic/CONTINUE-QUALITY-SYNC.md +++ b/clients/birth-biologic/CONTINUE-QUALITY-SYNC.md @@ -1,271 +1,12 @@ -# Birth Biologic Quality Department Sync - CONTINUATION INSTRUCTIONS +# Birth Biologic Quality Department Sync — RESOLVED -**Date:** 2026-06-30 -**Status:** IN PROGRESS - Upload script running -**Client:** Birth Biologic -**Task:** Sync SharePoint Quality Systems Department to match Datto exactly (3,768 files) +**Status:** COMPLETE (2026-06-30, GURU-5070). This continuation file is obsolete — do NOT +act on the instructions that used to be here. ---- +Authoritative record: `docs/migration/2026-06-30-quality-sync-COMPLETE.md`. -## CURRENT STATE - -**SharePoint Status:** -- Current file count: 3,249 files -- Target file count: 3,768 files (from Datto) -- Gap: 519 files remaining - -**Active RMM Command:** -- Command ID: `9e0fcfe8-0619-4a39-bd9c-6f5fd75c9b55` -- Agent: ACG-DWP-X-BB (a4524e85-8a07-45d0-91b1-51ce7e2ca74a) -- Status: Running (as of last check) -- Purpose: Upload all 3,768 files from Datto to SharePoint via Graph API -- Drive ID concatenation workaround: `"b" + "!" + "..."` to avoid PowerShell escaping - -**Background Monitor:** -- Task ID: b24474c (monitoring upload every minute for 20 minutes) - ---- - -## WHAT HAPPENED - -1. **Initial Approach:** Tried OneDrive sync by robocopy to local OneDrive folder - - Result: Synced 3,249 files then STALLED (no progress for 35+ minutes) - -2. **Switched to Direct Graph API Upload:** - - Multiple attempts failed due to PowerShell escaping the `!` in drive ID - - Drive ID: `b!F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9` - - Problem: PowerShell kept converting `b!` to `b\!` causing HTTP 400 errors - - Solution: Concatenate at runtime: `$driveId = "b" + "!" + "F8Bz..."` - -3. **Current Upload:** - - Command dispatched successfully with drive ID concatenation - - Script has been running but showing no output yet (may still be scanning files) - ---- - -## CREDENTIALS - -**Graph API (from vault):** -- Path: `msp-tools/computerguru-tenant-admin` -- Tenant ID: 19a568e8-9e88-413b-9341-cbc224b39145 -- Client ID: 709e6eed-0711-4875-9c44-2d3518c47063 -- Client Secret: (in vault at `credentials.client_secret`) - -**GuruRMM:** -- Vault path: `infrastructure/gururmm-server.sops.yaml` -- Agent ID: a4524e85-8a07-45d0-91b1-51ce7e2ca74a (ACG-DWP-X-BB) - -**SharePoint Drive ID:** -- `b!F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9` - ---- - -## NEXT STEPS TO CONTINUE - -### Option 1: Check if current upload completed - -```bash -# Authenticate to RMM -eval "$(bash .claude/scripts/rmm-auth.sh)" - -# Check upload status -curl -s "$RMM/api/commands/9e0fcfe8-0619-4a39-bd9c-6f5fd75c9b55" \ - -H "Authorization: Bearer $TOKEN" > /tmp/upload-status.json - -python3 -c " -import json -with open('/tmp/upload-status.json') as f: - data = json.load(f) -print(f\"Status: {data.get('status')}\") -print(f\"Exit code: {data.get('exit_code')}\") -stdout = data.get('stdout', '') or '' -if len(stdout) > 0: - print('\n--- Last 30 lines ---') - for line in stdout.split('\n')[-30:]: - print(line) -" - -# Then verify SharePoint count -python3 clients/birth-biologic/scripts/check-quality-status.py -``` - -### Option 2: If upload failed, use the working PowerShell script - -The correct script is in: `clients/birth-biologic/scripts/upload-final-working.ps1` - -Run via RMM: -```bash -eval "$(bash .claude/scripts/rmm-auth.sh)" -AGENT_ID="a4524e85-8a07-45d0-91b1-51ce7e2ca74a" -CLIENT_SECRET=$(bash .claude/scripts/vault.sh get-field msp-tools/computerguru-tenant-admin credentials.client_secret) - -# Script content with proper drive ID concatenation -SCRIPT='... (see upload-final-working.ps1) ...' - -PAYLOAD=$(jq -n --arg cmd "$SCRIPT" '{command_type: "powershell", command: $cmd, timeout_seconds: 1800}') -curl -s -X POST "$RMM/api/agents/$AGENT_ID/command" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d "$PAYLOAD" | jq -r '.command_id' -``` - ---- - -## FILES CREATED - -**Scripts:** -- `clients/birth-biologic/scripts/check-quality-status.py` - Check SharePoint file count -- `clients/birth-biologic/scripts/upload-datto-to-sharepoint.ps1` - Base upload script -- `clients/birth-biologic/scripts/reset-quality-exact.py` - Initial reset script (used) -- `clients/birth-biologic/scripts/exact-sync-quality.py` - Robocopy approach -- `clients/birth-biologic/scripts/sync-quality-simple.py` - Earlier attempt -- `clients/birth-biologic/scripts/finish-upload.py` - Bulk upload attempt -- `clients/birth-biologic/scripts/upload-remaining-files.py` - Remaining files upload - -**Todo List:** -1. [completed] Delete ALL files from SharePoint Quality Systems Department -2. [in_progress] Copy ALL files from Datto to SharePoint exactly as they exist -3. [pending] Verify SharePoint has exactly 3768 files matching Datto - ---- - -## DATTO SOURCE PATH - -**On ACG-DWP-X-BB:** -``` -C:\Users\Public\Desktop\Datto Workplace Server Projects\Quality Department -``` - -**Total files:** 3,768 files (verified via Get-ChildItem -Recurse) - -**Files >4MB:** ~63 files (these are being skipped - need separate large file upload later) - ---- - -## VERIFICATION COMMAND - -Once upload completes: - -```python -python3 clients/birth-biologic/scripts/check-quality-status.py -``` - -Expected output: -``` -SharePoint: 3768 files -Datto: 3768 files -Gap: 0 files - -[OK] MATCH - SharePoint has exactly 3768 files -``` - ---- - -## TROUBLESHOOTING - -**If upload shows 0 uploaded, 3467 errors:** -- This means drive ID was escaped wrong (b\! instead of b!) -- Solution: Use string concatenation `"b" + "!" + "..."` - -**If upload hangs with no output:** -- PowerShell may have syntax error -- Check command_text field to verify script sent correctly - -**If you need to cancel running command:** -```bash -curl -s -X POST "$RMM/api/commands/9e0fcfe8-0619-4a39-bd9c-6f5fd75c9b55/cancel" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" -``` - ---- - -## WORKING UPLOAD SCRIPT TEMPLATE - -Save this as `upload-final-working.ps1`: - -```powershell -$source = "C:\Users\Public\Desktop\Datto Workplace Server Projects\Quality Department" -$driveId = "b" + "!" + "F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9" -$tenantId = "19a568e8-9e88-413b-9341-cbc224b39145" -$clientId = "709e6eed-0711-4875-9c44-2d3518c47063" -$clientSecret = "GET_FROM_VAULT" - -Write-Host "Getting Graph API token..." -$tokenBody = @{ - client_id = $clientId - client_secret = $clientSecret - scope = "https://graph.microsoft.com/.default" - grant_type = "client_credentials" -} - -$tokenResponse = Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token" -Body $tokenBody -$token = $tokenResponse.access_token -Write-Host "[OK] Token acquired" - -Write-Host "Scanning Datto files..." -$files = Get-ChildItem $source -Recurse -File -ErrorAction SilentlyContinue -Write-Host "[OK] Found $($files.Count) files" - -Write-Host "Uploading files..." -$uploaded = 0 -$errors = 0 - -foreach ($file in $files) { - $relativePath = $file.FullName.Substring($source.Length + 1) - $uploadPath = $relativePath.Replace("\", "/") - - try { - if ($file.Length -lt 4MB) { - $uploadUrl = "https://graph.microsoft.com/v1.0/drives/$driveId/root:/$uploadPath" + ":/content" - $headers = @{ - "Authorization" = "Bearer $token" - "Content-Type" = "application/octet-stream" - } - - $fileBytes = [System.IO.File]::ReadAllBytes($file.FullName) - Invoke-RestMethod -Method Put -Uri $uploadUrl -Headers $headers -Body $fileBytes -UseBasicParsing | Out-Null - - $uploaded++ - if ($uploaded % 100 -eq 0) { - Write-Host " Uploaded $uploaded files..." - } - } - } catch { - $errors++ - } -} - -Write-Host "" -Write-Host "Upload Complete" -Write-Host "Uploaded: $uploaded" -Write-Host "Errors: $errors" -Write-Host "Total: $($files.Count)" -``` - ---- - -## SESSION NOTES - -- OneDrive sync is unreliable for large migrations (stalled at 86%) -- Direct Graph API upload is the correct approach -- PowerShell exclamation mark escaping is a major gotcha -- Use string concatenation to avoid escape issues -- Files >4MB need separate upload session logic (not implemented yet) -- There are ~63 large files that will need separate handling - -**Client was promised this yesterday - NOW VERY ANGRY** - ---- - -## QUICK RESUME CHECKLIST - -1. [ ] Check if command 9e0fcfe8 completed -2. [ ] Verify SharePoint file count (should be 3,768) -3. [ ] If not complete, dispatch new upload with working script -4. [ ] Monitor for completion (15-20 minutes) -5. [ ] Verify final count matches Datto -6. [ ] Handle large files (>4MB) if needed -7. [ ] Document completion - -**END OF CONTINUATION INSTRUCTIONS** +Summary: all 3,768 Datto files are present in SharePoint (verified via Graph delta). The +only differences are 4 files that are live current work edited today by client staff +(1 new xlsx + 3 open docs), intentionally preserved per Mike. The prior approach failed all +day because its script skipped every file >=4MB (301 files, ~29.7 GB) and used the RMM +`timeout` field (capped ~300s) instead of `timeout_seconds`. diff --git a/clients/birth-biologic/docs/migration/2026-06-30-quality-sync-COMPLETE.md b/clients/birth-biologic/docs/migration/2026-06-30-quality-sync-COMPLETE.md new file mode 100644 index 00000000..a1613abc --- /dev/null +++ b/clients/birth-biologic/docs/migration/2026-06-30-quality-sync-COMPLETE.md @@ -0,0 +1,70 @@ +# Birth Biologic — Quality Systems Department -> Datto sync — COMPLETE + +**Date:** 2026-06-30 +**Completed by:** Mike Swanson / claude-main (GURU-5070) +**Status:** COMPLETE — verified against live Graph enumeration + +## Outcome + +Every file in the Datto source is now present in SharePoint. + +| | Datto (source) | SharePoint (final) | +|---|---:|---:| +| Files | 3,768 | 3,769 | +| Datto files missing from SharePoint | — | **0** | + +All 301 large files (>=4MB, ~29.7 GB total, largest a 3.94 GB .mov) were uploaded via +Graph chunked upload sessions. The idempotent size-check pass also detected and re-uploaded +~700 files that existed in SharePoint with a mismatched size (partial/corrupt residue from +the earlier failed OneDrive/PUT attempts). + +## The 4 intentional differences (live current work — PRESERVED per Mike, 2026-06-30) + +These are the "recently modified, do not clobber" carve-out. Confirmed as live edits made +*today* by named client staff, so they were left intact rather than forced to match Datto: + +| File | Datto size | SharePoint size | Note | +|---|---:|---:|---| +| `LOGS/Equipment/3. Temp Excursions/Temperature Excursion Log.xlsx` | not present | 40,253 | Created 2026-06-30 by Mary Ster, edited by Kristin Steen | +| `LOGS/Equipment/2. Validation List/QSP-200.003.A Validation List.Current.docx` | 41,908 | 51,510 | Locked/open, edited today | +| `LOGS/Equipment/1.Equipment List/QSP-200.001.C Equipment List.Current.docx` | 47,947 | 57,545 | Locked/open, edited today | +| `LOGS/Quality Assurance Reporting Log/Deviations/2024/DEV39.Exhibit B.docx` | 15,525 | 22,437 | In use, edited today | + +The three docs surfaced as upload "errors" (HTTP 423 Locked / 409 Conflict) precisely +*because* staff had them open — SharePoint's lock protected live work. That is the correct +result, not a failure. + +## What actually fixed it (root cause of the all-day Mac failures) + +1. **The prior upload script skipped every file >=4MB.** Datto has 301 such files (~29.7 GB). + The old approach could never reach 3,768 no matter how many times it ran. Fix: proper Graph + **upload sessions** (10 MB chunks, `Content-Range`) for large files. +2. **RMM agent ignores the `timeout` field; it honors `timeout_seconds`.** Commands sent with + `timeout` were capped at ~300 s, so long uploads died / went zombie ("running", no output). + Using `timeout_seconds` allowed multi-minute/multi-hour commands to run to completion. +3. **Long paths (>260 chars) in the Datto tree** were handled with the `\\?\` prefix for file + reads. + +## Method + +- `clients/birth-biologic/scripts/enumerate-datto.ps1` — enumerate Datto, write a + `relpath|size` manifest to `C:\Windows\Temp\quality-manifest.txt` on ACG-DWP-X-BB. +- `clients/birth-biologic/scripts/upload-quality-final.ps1` — idempotent uploader: for each + Datto file, skip if SharePoint already has it at matching size; else upload (simple PUT + <4MB, chunked upload session >=4MB). Long-path safe, refreshes the Graph token on long + runs, internal time budget + progress log (`C:\Windows\Temp\quality-upload.log`). +- Ground truth verified from this machine via the Graph **delta** endpoint (whole-drive + enumeration; far fewer round-trips than recursive `children` calls). + +## Agent / drive + +- Agent: ACG-DWP-X-BB (`a4524e85-8a07-45d0-91b1-51ce7e2ca74a`) +- Datto source: `C:\Users\Public\Desktop\Datto Workplace Server Projects\Quality Department` +- SharePoint drive: `b!F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9` +- Graph app: `msp-tools/computerguru-tenant-admin` (tenant 19a568e8-...) + +## Note on the earlier docs + +The earlier `2026-06-30-quality-sync-to-datto.md` (surgical 5-file delete) and +`CONTINUE-QUALITY-SYNC.md` (wipe + re-upload, in progress) describe superseded intermediate +states. This file is the authoritative final record. diff --git a/clients/birth-biologic/scripts/enumerate-datto.ps1 b/clients/birth-biologic/scripts/enumerate-datto.ps1 new file mode 100644 index 00000000..d519dbf9 --- /dev/null +++ b/clients/birth-biologic/scripts/enumerate-datto.ps1 @@ -0,0 +1,12 @@ +$ErrorActionPreference = 'SilentlyContinue' +$source = 'C:\Users\Public\Desktop\Datto Workplace Server Projects\Quality Department' +$files = Get-ChildItem -LiteralPath $source -Recurse -File +$big = @($files | Where-Object { $_.Length -ge 4MB }) +$bytes = ($files | Measure-Object Length -Sum).Sum +$manifest = $files | ForEach-Object { $_.FullName.Substring($source.Length + 1).Replace('\','/') + '|' + $_.Length } +$manifest | Out-File -FilePath 'C:\Windows\Temp\quality-manifest.txt' -Encoding UTF8 +Write-Host "COUNT=$($files.Count)" +Write-Host "BIG4MB=$($big.Count)" +Write-Host "BYTES=$bytes" +Write-Host "MANIFEST_LINES=$($manifest.Count)" +Write-Host "MANIFEST_PATH=C:\Windows\Temp\quality-manifest.txt" diff --git a/clients/birth-biologic/scripts/upload-quality-final.ps1 b/clients/birth-biologic/scripts/upload-quality-final.ps1 new file mode 100644 index 00000000..01a63b08 --- /dev/null +++ b/clients/birth-biologic/scripts/upload-quality-final.ps1 @@ -0,0 +1,101 @@ +# Birth Biologic - Quality Systems Department: converge SharePoint to Datto (3768 files) +# Idempotent: skips files already present with matching size. Chunked upload sessions for >=4MB. +# Long-path safe (\\?\). Refreshes Graph token on long runs. Writes progress to a tailable log. +$ErrorActionPreference = 'Stop' + +$source = 'C:\Users\Public\Desktop\Datto Workplace Server Projects\Quality Department' +$driveId = 'b!F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9' +$tenantId = '19a568e8-9e88-413b-9341-cbc224b39145' +$clientId = '709e6eed-0711-4875-9c44-2d3518c47063' +$clientSecret = 'SECRET_PLACEHOLDER' +$logPath = 'C:\Windows\Temp\quality-upload.log' +$manifestPath = 'C:\Windows\Temp\quality-manifest.txt' + +"=== upload start ===" | Set-Content -LiteralPath $logPath -Encoding UTF8 +function Log($m){ $line = (Get-Date -Format 'HH:mm:ss') + ' ' + $m; Write-Host $line; Add-Content -LiteralPath $logPath -Value $line } + +$script:token = $null +$script:tokenAt = $null +function Get-Token { + $body = @{ client_id=$clientId; client_secret=$clientSecret; scope='https://graph.microsoft.com/.default'; grant_type='client_credentials' } + $r = Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token" -Body $body + $script:token = $r.access_token + $script:tokenAt = Get-Date +} +function Fresh-Token { + if ($null -eq $script:token -or ((Get-Date) - $script:tokenAt).TotalMinutes -gt 40) { Get-Token } + return $script:token +} +function Enc($rel){ ($rel.Split('/') | ForEach-Object { [uri]::EscapeDataString($_) }) -join '/' } +function Long($p){ if ($p.StartsWith('\\?\')) { $p } else { '\\?\' + $p } } + +Get-Token +Log "Token acquired." + +$lines = Get-Content -LiteralPath $manifestPath | Where-Object { $_ -and $_.Contains('|') } +$items = foreach($l in $lines){ + $idx = $l.LastIndexOf('|') + [pscustomobject]@{ rel = $l.Substring(0,$idx); size = [int64]$l.Substring($idx+1) } +} +# small files first so the visible count climbs fast, then the big ones +$items = $items | Sort-Object size +Log ("Manifest items: " + $items.Count) + +# Internal time budget: exit cleanly before any agent-side cap; re-dispatch resumes (idempotent). +$deadline = (Get-Date).AddSeconds(9600) +$timedOut = $false + +$uploaded=0; $skipped=0; $errors=0; $bigUp=0; $i=0 +foreach($it in $items){ + $i++ + if ((Get-Date) -gt $deadline){ $timedOut = $true; Log "TIME budget reached at item $i; stopping pass cleanly."; break } + $rel = $it.rel; $size = $it.size; $enc = Enc $rel + $tok = Fresh-Token + $h = @{ Authorization = "Bearer $tok" } + + $exists=$false + try { + $meta = Invoke-RestMethod -Method Get -Uri "https://graph.microsoft.com/v1.0/drives/$driveId/root:/$enc" -Headers $h + if ([int64]$meta.size -eq $size) { $exists=$true } + } catch { $exists=$false } + if ($exists){ $skipped++; if($i % 250 -eq 0){ Log " $i/$($items.Count) up=$uploaded skip=$skipped err=$errors big=$bigUp" }; continue } + + $full = Long (Join-Path $source ($rel.Replace('/','\'))) + try { + if ($size -lt 4194304){ + $bytes = [System.IO.File]::ReadAllBytes($full) + $hh = @{ Authorization = "Bearer $tok"; 'Content-Type'='application/octet-stream' } + Invoke-RestMethod -Method Put -Uri "https://graph.microsoft.com/v1.0/drives/$driveId/root:/$enc`:/content" -Headers $hh -Body $bytes -UseBasicParsing | Out-Null + $uploaded++ + } else { + $sessUri = "https://graph.microsoft.com/v1.0/drives/$driveId/root:/$enc`:/createUploadSession" + $sessBody = @{ item = @{ '@microsoft.graph.conflictBehavior'='replace' } } | ConvertTo-Json + $sess = Invoke-RestMethod -Method Post -Uri $sessUri -Headers @{ Authorization="Bearer $tok"; 'Content-Type'='application/json' } -Body $sessBody + $upUrl = $sess.uploadUrl + $fs = [System.IO.File]::Open($full,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::Read) + try { + $chunk = 10485760 # 10 MB (multiple of 320 KiB) + $buf = New-Object byte[] $chunk + $pos = [int64]0 + while ($pos -lt $size){ + $read = $fs.Read($buf,0,$chunk) + if ($read -le 0){ break } + if ($read -eq $chunk){ $payload = $buf } else { $payload = New-Object byte[] $read; [Array]::Copy($buf,$payload,$read) } + $end = $pos + $read - 1 + $ch = @{ 'Content-Range' = "bytes $pos-$end/$size" } + Invoke-RestMethod -Method Put -Uri $upUrl -Headers $ch -Body $payload | Out-Null + $pos += $read + } + } finally { $fs.Close() } + $uploaded++; $bigUp++ + Log (" BIG ok " + [math]::Round($size/1MB,1) + "MB: $rel") + } + } catch { + $errors++ + Log " ERROR: $rel :: $($_.Exception.Message)" + } + if($i % 100 -eq 0){ Log " $i/$($items.Count) up=$uploaded skip=$skipped err=$errors big=$bigUp" } +} +$allDone = (-not $timedOut) +Log "DONE up=$uploaded skip=$skipped err=$errors big=$bigUp total=$($items.Count) reachedEnd=$allDone" +Write-Host "RESULT up=$uploaded skip=$skipped err=$errors big=$bigUp total=$($items.Count) reachedEnd=$allDone" diff --git a/clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md b/clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md new file mode 100644 index 00000000..ed4ff2eb --- /dev/null +++ b/clients/birth-biologic/session-logs/2026-06/2026-06-30-mike-birthbio-quality-sync-complete.md @@ -0,0 +1,141 @@ +# Birth Biologic — Quality Systems Department -> Datto SharePoint sync (COMPLETED) + +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Session Summary + +Picked up the Birth Biologic Quality Systems Department -> Datto SharePoint migration after a +Mac session (Mikes-MacBook-Air) worked it all day and failed to complete. The goal: make the +SharePoint "Quality Systems Department" document library contain every file from the Datto +Workplace source (3,768 files) on agent ACG-DWP-X-BB. Started by loading the pulled-in +continuation docs and reconciling two contradictory records — an earlier one claiming a +surgical 5-file delete "COMPLETE," and a later `CONTINUE-QUALITY-SYNC.md` describing a +wipe-and-reupload stuck at 3,249/3,768. + +Rather than trust the stale docs, pulled live ground truth via the Graph delta endpoint: +SharePoint held 3,337 files (only 17 >=4MB), gap of 431. Re-enumerated the Datto source via +RMM, which revealed the actual scope the Mac missed: 3,768 files, of which 301 are >=4MB +(~29.7 GB total, largest a 3.94 GB .mov). The Mac's upload script skipped every file >=4MB, +so it could never converge — the entire large-file corpus (the bulk of the gap) was being +ignored. + +Wrote a correct uploader (`upload-quality-final.ps1`): idempotent (skip a file if SharePoint +already has it at matching size), simple PUT for <4MB and Graph chunked upload sessions for +>=4MB, `\\?\` long-path handling, Graph token auto-refresh for long runs, an internal time +budget, and a tailable progress log. Validated the risky path on the single largest file +(3.94 GB) before the full run. That test also uncovered the second root cause: the RMM agent +ignores the `timeout` field (caps ~300 s, producing the Mac's zombie "running" commands) and +honors `timeout_seconds` instead. + +Cancelled the hung command `9e0fcfe8`, dispatched the full uploader with `timeout_seconds: +10800`, and monitored to completion in the background. Result: 1,137 uploaded (283 large + +854 small), 2,628 already-matching, 3 errors. The idempotent size-check also silently +repaired ~700 files that existed but had a mismatched size (partial/corrupt residue from the +earlier failed attempts). + +Verified the final state with a full Datto-vs-SharePoint diff: 0 Datto files missing from +SharePoint (all 3,768 present), and exactly 4 differences — all live current work edited today +by named client staff. Mike chose to preserve them. Documented the outcome in an authoritative +completion doc, retired the contradictory continuation doc, logged the two lessons to +errorlog, and saved two reference memories. + +## Key Decisions + +- Trusted live Graph/RMM enumeration over the committed docs — the docs contradicted each + other and were hours stale. Ground truth drove every decision. +- Did not undo the Mac's wipe-and-reupload (already executed before pickup); finished the + rebuild additively (upload-only). Performed no deletions. +- Fixed the >=4MB gap with proper chunked upload sessions rather than continuing to skip large + files (the actual reason the job never finished). +- Validated the chunked-upload/long-path/Content-Range path on one real 3.94 GB file before + committing to a ~30 GB run. +- Preserved the 4 live-work files (1 new xlsx + 3 open/locked docs edited today by staff) + rather than forcing a byte-identical match — overwriting would have destroyed today's work. + Confirmed with Mike. +- Kept the client secret out of the repo (scripts carry `SECRET_PLACEHOLDER`; injected at + dispatch time from the vault into a scratch payload that was deleted). + +## Problems Encountered + +- **Upload could never reach 3,768:** prior script skipped all 301 files >=4MB (~29.7 GB). + Fixed with Graph chunked upload sessions (10 MB chunks, `Content-Range`). +- **Zombie/hung RMM commands:** agent ignores `timeout`, capping at ~300 s. Fixed by using + `timeout_seconds` (both fields sent). Cancelled the stuck `9e0fcfe8` first. +- **Recursive `/children` count timed out** (one HTTP call per folder). Switched to the Graph + `/root/delta` endpoint — whole drive in ~9 paged calls. +- **`requests` not installed** on GURU-5070 Python. Rewrote checks with stdlib `urllib`. +- **Long paths (>260) in Datto tree:** used the `\\?\` prefix for `[IO.File]` reads. +- **3 upload errors (423 Locked / 409 Conflict):** files were locked because staff had them + open — live current work, correctly left untouched. +- **SharePoint 3,769 vs Datto 3,768 (one extra):** diffed to `Temperature Excursion Log.xlsx`, + created today by client staff; preserved, not deleted. + +## Configuration Changes + +Created: +- `clients/birth-biologic/scripts/enumerate-datto.ps1` — Datto enumeration + manifest writer. +- `clients/birth-biologic/scripts/upload-quality-final.ps1` — idempotent chunked uploader + (carries `SECRET_PLACEHOLDER`, not a real secret). +- `clients/birth-biologic/docs/migration/2026-06-30-quality-sync-COMPLETE.md` — authoritative + final record. +- `.claude/memory/reference_gururmm_command_timeout_seconds.md` +- `.claude/memory/reference_sharepoint_graph_large_file_upload.md` + +Modified: +- `clients/birth-biologic/CONTINUE-QUALITY-SYNC.md` — replaced with a RESOLVED pointer. +- `.claude/memory/MEMORY.md` — two new reference index lines. +- `errorlog.md` — one `--friction` (timeout_seconds) + one `--correction` (>=4MB skip). + +## Credentials & Secrets + +- No new credentials created or discovered. Used existing vault entries only. +- Graph app (client-credentials): vault `msp-tools/computerguru-tenant-admin`, + `credentials.client_secret`. Tenant `19a568e8-9e88-413b-9341-cbc224b39145`, client + `709e6eed-0711-4875-9c44-2d3518c47063`. +- GuruRMM API: via `.claude/scripts/rmm-auth.sh` (vault + `infrastructure/gururmm-server.sops.yaml`). +- Note: dispatching PowerShell that embeds the Graph secret leaves that secret in the RMM + command-history record (same as the prior Mac scripts). Acceptable here (internal RMM), but + a future hardening would have the agent fetch the secret itself. + +## Infrastructure & Servers + +- Agent: ACG-DWP-X-BB, id `a4524e85-8a07-45d0-91b1-51ce7e2ca74a`. +- Datto source: `C:\Users\Public\Desktop\Datto Workplace Server Projects\Quality Department` + (3,768 files, 301 >=4MB, ~29.7 GB). +- SharePoint drive: `b!F8BzMb1YakCIWCyWlmczb09LHqtxDxVMpLT6kAwYmsM7NUY4oPLSRq7ng3tJq-E9`. +- RMM API (internal): `http://172.16.3.30:3001` (JWT via rmm-auth.sh). +- VM work files: `C:\Windows\Temp\quality-manifest.txt`, `C:\Windows\Temp\quality-upload.log`. + +## Commands & Outputs + +- Count via delta: `GET /drives/{drive}/root/delta?$select=...&$top=500` — final live count 3,769. +- Dispatch: `POST {RMM}/api/agents/{id}/command` with + `{command_type:"powershell", command: