sync: auto-sync from GURU-5070 at 2026-07-03 13:18:27

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-03 13:18:27
This commit is contained in:
2026-07-03 13:19:15 -07:00
parent 41c12a934f
commit 78f794a924
6 changed files with 275 additions and 1 deletions

View File

@@ -1 +1 @@
eyJ0eXAiOiJKV1QiLCJub25jZSI6IlhjWmZZT3hzX0lySmZ6a3E5dnozYzhrSGZqM003c2N3UDZwLWFQUTBfaGciLCJhbGciOiJSUzI1NiIsIng1dCI6ImFGa21LVkZjLTRXVjZzWENCdk5aa1hJNTA1WSIsImtpZCI6ImFGa21LVkZjLTRXVjZzWENCdk5aa1hJNTA1WSJ9.eyJhdWQiOiJodHRwczovL2dyYXBoLm1pY3Jvc29mdC5jb20iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xOWE1NjhlOC05ZTg4LTQxM2ItOTM0MS1jYmMyMjRiMzkxNDUvIiwiaWF0IjoxNzgzMDM5NDc0LCJuYmYiOjE3ODMwMzk0NzQsImV4cCI6MTc4MzA0MzM3NCwiYWNycyI6WyJwZmRyIl0sImFpbyI6ImsyRmdZTGg5Y083a1hPUElXdVk3ZGFHQmJSRzEvbjlrMlc1TnVGQ2E2ZmN0dE5qd3htWUEiLCJhcHBfZGlzcGxheW5hbWUiOiJDb21wdXRlckd1cnUgLSBUZW5hbnQgQWRtaW4iLCJhcHBpZCI6IjcwOWU2ZWVkLTA3MTEtNDg3NS05YzQ0LTJkMzUxOGM0NzA2MyIsImFwcGlkYWNyIjoiMiIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzE5YTU2OGU4LTllODgtNDEzYi05MzQxLWNiYzIyNGIzOTE0NS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjdhMTk5YjExLTk3ZmItNGU2NS05MTdkLWY4ZDI5YTUzYmE0OSIsInJoIjoiMS5BVVlBNkdpbEdZaWVPMEdUUWN2Q0pMT1JSUU1BQUFBQUFBQUF3QUFBQUFBQUFBQUFBQUJHQUEuIiwicm9sZXMiOlsiUG9saWN5LlJlYWRXcml0ZS5Db25kaXRpb25hbEFjY2VzcyIsIlVzZXIuUmVhZFdyaXRlLkFsbCIsIlNlY3VyaXR5RXZlbnRzLlJlYWQuQWxsIiwiVXNlckF1dGhlbnRpY2F0aW9uTWV0aG9kLlJlYWRXcml0ZS5BbGwiLCJBcHBsaWNhdGlvbi5SZWFkV3JpdGUuQWxsIiwiRGlyZWN0b3J5LlJlYWRXcml0ZS5BbGwiLCJTaXRlcy5SZWFkV3JpdGUuQWxsIiwiQXBwUm9sZUFzc2lnbm1lbnQuUmVhZFdyaXRlLkFsbCIsIlJvbGVNYW5hZ2VtZW50LlJlYWRXcml0ZS5EaXJlY3RvcnkiLCJQb2xpY3kuUmVhZC5BbGwiLCJTaXRlcy5GdWxsQ29udHJvbC5BbGwiXSwic3ViIjoiN2ExOTliMTEtOTdmYi00ZTY1LTkxN2QtZjhkMjlhNTNiYTQ5IiwidGVuYW50X3JlZ2lvbl9zY29wZSI6Ik5BIiwidGlkIjoiMTlhNTY4ZTgtOWU4OC00MTNiLTkzNDEtY2JjMjI0YjM5MTQ1IiwidXRpIjoiMEl3N1NVWTRua0NibU9kVzZ1VW5BQSIsInZlciI6IjEuMCIsIndpZHMiOlsiYjFiZTFjM2UtYjY1ZC00ZjE5LTg0MjctZjZmYTBkOTdmZWI5IiwiMDk5N2ExZDAtMGQxZC00YWNiLWI0MDgtZDVjYTczMTIxZTkwIl0sInhtc19hY2QiOjE3NzY3MDg0MDEsInhtc19hY3RfZmN0IjoiMyA5IiwieG1zX2Z0ZCI6Ill6QjIwNkktSVJiOXBPTkN1WkZIZFpzR2VVaThJekpBZlVVMVQ2NUxxM1VCZFhOM1pYTjBNeTFrYzIxeiIsInhtc19pZHJlbCI6IjQgNyIsInhtc19wZnRleHAiOjE3ODMxMjk3NzQsInhtc19yZCI6IjAuNDJMbFlCSmlkQk1TNFdBVkVwQnpTenkzb0xIRlo3ZFh0YW5YTE5OY0lSRU9UaUdCdEk5ZVV3LWVldWs1UWV3NWMtTjJsczlDSWh3Y1FnTE1EQkJ3QUVvTGlYQndDd2xzZVNEN05kemY3dVFpU2FrV241YXRBZ0EiLCJ4bXNfc3ViX2ZjdCI6IjMgOSIsInhtc190Y2R0IjoxNTgxMDIyOTI3LCJ4bXNfdG50X2ZjdCI6IjMgNCJ9.nz5EdcWvEBK4R-D61jyT72d7Bah5gPGJxMfNfocee6DoMZK1rzOMEPwisIR98OD1aq7C3hn9FzB4bn4SCbHdXo5abETX2xn1Tr9hFCuJrXESC0fMbssfIjzV4bO-xVqQk-ylQAax0vaH05SvL2SMqAZYaM86e-K9IIgKc_IYjzVfvjuKbVyWw2s4NeBki4gwAoesPp1DS6K_RDEDpov_x0B6PULGDbQYzwe1YriA1kmnfYP_56YJqLCHMBFH7FeXkDeRpAVSMqWeZOALoA1EYaX1InF22t1alblT59m6rEN6C_Xo39xzwUKhisHPIcRZLNyS2d7NP3X2vtXd44n8FQ
eyJ0eXAiOiJKV1QiLCJub25jZSI6IlNzRUhBcVJSVzVzOFYwTzBaVVBUdnZGZjRnVXl2YmRvNjFiVklhYjJNbDgiLCJhbGciOiJSUzI1NiIsIng1dCI6ImFGa21LVkZjLTRXVjZzWENCdk5aa1hJNTA1WSIsImtpZCI6ImFGa21LVkZjLTRXVjZzWENCdk5aa1hJNTA1WSJ9.eyJhdWQiOiJodHRwczovL2dyYXBoLm1pY3Jvc29mdC5jb20iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xOWE1NjhlOC05ZTg4LTQxM2ItOTM0MS1jYmMyMjRiMzkxNDUvIiwiaWF0IjoxNzgzMDQ4NDQwLCJuYmYiOjE3ODMwNDg0NDAsImV4cCI6MTc4MzA1MjM0MCwiYWNycyI6WyJwZmRyIl0sImFpbyI6ImsyRmdZQWhtcnMxdW5yZlo2TmVTNUxmbDNzNkZNNXA0YnNoN1RuNGhmc1hJZkhWYjkzMEEiLCJhcHBfZGlzcGxheW5hbWUiOiJDb21wdXRlckd1cnUgLSBUZW5hbnQgQWRtaW4iLCJhcHBpZCI6IjcwOWU2ZWVkLTA3MTEtNDg3NS05YzQ0LTJkMzUxOGM0NzA2MyIsImFwcGlkYWNyIjoiMiIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzE5YTU2OGU4LTllODgtNDEzYi05MzQxLWNiYzIyNGIzOTE0NS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjdhMTk5YjExLTk3ZmItNGU2NS05MTdkLWY4ZDI5YTUzYmE0OSIsInJoIjoiMS5BVVlBNkdpbEdZaWVPMEdUUWN2Q0pMT1JSUU1BQUFBQUFBQUF3QUFBQUFBQUFBQUFBQUJHQUEuIiwicm9sZXMiOlsiUG9saWN5LlJlYWRXcml0ZS5Db25kaXRpb25hbEFjY2VzcyIsIlVzZXIuUmVhZFdyaXRlLkFsbCIsIlNlY3VyaXR5RXZlbnRzLlJlYWQuQWxsIiwiVXNlckF1dGhlbnRpY2F0aW9uTWV0aG9kLlJlYWRXcml0ZS5BbGwiLCJBcHBsaWNhdGlvbi5SZWFkV3JpdGUuQWxsIiwiRGlyZWN0b3J5LlJlYWRXcml0ZS5BbGwiLCJTaXRlcy5SZWFkV3JpdGUuQWxsIiwiQXBwUm9sZUFzc2lnbm1lbnQuUmVhZFdyaXRlLkFsbCIsIlJvbGVNYW5hZ2VtZW50LlJlYWRXcml0ZS5EaXJlY3RvcnkiLCJQb2xpY3kuUmVhZC5BbGwiLCJTaXRlcy5GdWxsQ29udHJvbC5BbGwiXSwic3ViIjoiN2ExOTliMTEtOTdmYi00ZTY1LTkxN2QtZjhkMjlhNTNiYTQ5IiwidGVuYW50X3JlZ2lvbl9zY29wZSI6Ik5BIiwidGlkIjoiMTlhNTY4ZTgtOWU4OC00MTNiLTkzNDEtY2JjMjI0YjM5MTQ1IiwidXRpIjoiRFE2UE4wWlRqMEdSMDhWeEhhUWJBQSIsInZlciI6IjEuMCIsIndpZHMiOlsiYjFiZTFjM2UtYjY1ZC00ZjE5LTg0MjctZjZmYTBkOTdmZWI5IiwiMDk5N2ExZDAtMGQxZC00YWNiLWI0MDgtZDVjYTczMTIxZTkwIl0sInhtc19hY2QiOjE3NzY3MDg0MDEsInhtc19hY3RfZmN0IjoiOSAzIiwieG1zX2Z0ZCI6IjZuYTJJRXF0MWliaVd5emFrbzdzM0VHbEdnbm1JRlBUSjdFWUZTcEZPcjBCZFhOemIzVjBhQzFrYzIxeiIsInhtc19pZHJlbCI6IjEwIDciLCJ4bXNfcGZ0ZXhwIjoxNzgzMTM4NzQwLCJ4bXNfcmQiOiIwLjQyTGxZQkppZEJNUzRXQVZFcEJ6U3p5M29MSEZaN2RYdGFuWExOTmNJUkVPVGlHQnRJOWVVdy1lZXVrNVFldzVjLU4ybHM5Q0lod2NRZ0xNREJCd0FFb0xpWEJ3Q3dsc2VTRDdOZHpmN3VRaVNha1duNWF0QWdBIiwieG1zX3N1Yl9mY3QiOiIzIDkiLCJ4bXNfdGNkdCI6MTU4MTAyMjkyNywieG1zX3RudF9mY3QiOiIzIDE0In0.Rfz02nCkcm5iJ19EQVEFhZEcBf_xdd6kXTck-paz5UVgRhSWE8TDPfF9eliB38OZ59bRJdkY6TdtUNrlAAHhJxHs8Hxq6vyYJwKAhzfD-XYntD00Mas1yEe1pobc4vjLPrNYl5ie2KIZvSpmvUc97Lvf5juRueRbvUh9k9QgVwKfGBCHVXirBv9Bekhx4XUraskE-UvROmDvf5mi1DtkskMIC4-Xutz6gFTbJDj-uwTtI2vZa9dQMsVJfMpmq6pWD5lNrIm1tk6LMMIBVA00h3ue5hIt7v5IggvFFqk2JlacMnElQPNkgpwaMJTdcwrVEuonVZuKpv3iYZJU8MtCcg

1
.claude/MEMORY.md Normal file
View File

@@ -0,0 +1 @@
- [PS5.1 -Headers $H empty-auth quirk](.claude/memory/ps51_invoke_restmethod_headers_variable.md) - build Graph auth headers inline per call under RMM powershell -File; capture Graph error bodies

View File

@@ -0,0 +1,25 @@
---
name: ps51-invoke-restmethod-headers-variable
description: PS5.1 via RMM powershell -File - Invoke-RestMethod -Headers $H (script-scope hashtable) sent an EMPTY Authorization header; build headers inline per call instead
metadata:
type: reference
---
On ACG-DWP-X-BB (WS2019, PowerShell 5.1, script executed via GuruRMM agent as
`powershell -NoProfile -File script.ps1`), passing a script-scope hashtable variable to
`Invoke-RestMethod -Headers $H` (where `$H=@{Authorization="Bearer $tok"}`) resulted in
Graph rejecting every call with `InvalidAuthenticationToken / "Access token is empty"`
even though the token variable was verifiably populated (fingerprint printed correctly)
and the same token + same URL worked with an inline-built header from the same machine.
**Fix that works:** build the header at each call site — e.g.
`function Hdr { return @{Authorization=("Bearer " + $script:tok)} }` then
`Invoke-RestMethod -Uri $u -Headers (Hdr)`. Inline `@{Authorization=('Bearer '+$t)}` also works.
Diagnosed 2026-07-02 during the BirthBio Datto-vs-SharePoint reconciliation after two
full failed runs (masqueraded first as token expiry, then as clock skew — the real tell
was the Graph error body "Access token is empty", captured only after adding response-body
extraction to the retry helper). Always capture the Graph error BODY, not just the
exception message: "(401) Unauthorized" alone cost three debugging cycles.
Related: [[gururmm-command-timeout-seconds]], [[sharepoint-graph-large-file-upload]]

1
.runner-tok Normal file
View File

@@ -0,0 +1 @@
eyJ0eXAiOiJKV1QiLCJub25jZSI6ImhsZElMaDBmNm5TMkV1bnhIam1xZjNLUEFUVzlEZUpwVEhRR2FCVFptYm8iLCJhbGciOiJSUzI1NiIsIng1dCI6ImFGa21LVkZjLTRXVjZzWENCdk5aa1hJNTA1WSIsImtpZCI6ImFGa21LVkZjLTRXVjZzWENCdk5aa1hJNTA1WSJ9.eyJhdWQiOiJodHRwczovL2dyYXBoLm1pY3Jvc29mdC5jb20iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xOWE1NjhlOC05ZTg4LTQxM2ItOTM0MS1jYmMyMjRiMzkxNDUvIiwiaWF0IjoxNzgzMDQ4Mjc0LCJuYmYiOjE3ODMwNDgyNzQsImV4cCI6MTc4MzA1MjE3NCwiYWNycyI6WyJwZmRyIl0sImFpbyI6IkFTUUEyLzhjQUFBQUtzcXJxZTRkYVZNbWMvSEk0R2pGK3hxdkVzdkRpWTJnTzVWSlE4aytUSTA9IiwiYXBwX2Rpc3BsYXluYW1lIjoiQ29tcHV0ZXJHdXJ1IC0gVGVuYW50IEFkbWluIiwiYXBwaWQiOiI3MDllNmVlZC0wNzExLTQ4NzUtOWM0NC0yZDM1MThjNDcwNjMiLCJhcHBpZGFjciI6IjIiLCJpZHAiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xOWE1NjhlOC05ZTg4LTQxM2ItOTM0MS1jYmMyMjRiMzkxNDUvIiwiaWR0eXAiOiJhcHAiLCJvaWQiOiI3YTE5OWIxMS05N2ZiLTRlNjUtOTE3ZC1mOGQyOWE1M2JhNDkiLCJyaCI6IjEuQVVZQTZHaWxHWWllTzBHVFFjdkNKTE9SUlFNQUFBQUFBQUFBd0FBQUFBQUFBQUFBQUFCR0FBLiIsInJvbGVzIjpbIlBvbGljeS5SZWFkV3JpdGUuQ29uZGl0aW9uYWxBY2Nlc3MiLCJVc2VyLlJlYWRXcml0ZS5BbGwiLCJTZWN1cml0eUV2ZW50cy5SZWFkLkFsbCIsIlVzZXJBdXRoZW50aWNhdGlvbk1ldGhvZC5SZWFkV3JpdGUuQWxsIiwiQXBwbGljYXRpb24uUmVhZFdyaXRlLkFsbCIsIkRpcmVjdG9yeS5SZWFkV3JpdGUuQWxsIiwiU2l0ZXMuUmVhZFdyaXRlLkFsbCIsIkFwcFJvbGVBc3NpZ25tZW50LlJlYWRXcml0ZS5BbGwiLCJSb2xlTWFuYWdlbWVudC5SZWFkV3JpdGUuRGlyZWN0b3J5IiwiUG9saWN5LlJlYWQuQWxsIiwiU2l0ZXMuRnVsbENvbnRyb2wuQWxsIl0sInN1YiI6IjdhMTk5YjExLTk3ZmItNGU2NS05MTdkLWY4ZDI5YTUzYmE0OSIsInRlbmFudF9yZWdpb25fc2NvcGUiOiJOQSIsInRpZCI6IjE5YTU2OGU4LTllODgtNDEzYi05MzQxLWNiYzIyNGIzOTE0NSIsInV0aSI6Ik9TMlRQV1I3Y1VtYmpIUWFXRkVkQUEiLCJ2ZXIiOiIxLjAiLCJ3aWRzIjpbImIxYmUxYzNlLWI2NWQtNGYxOS04NDI3LWY2ZmEwZDk3ZmViOSIsIjA5OTdhMWQwLTBkMWQtNGFjYi1iNDA4LWQ1Y2E3MzEyMWU5MCJdLCJ4bXNfYWNkIjoxNzc2NzA4NDAxLCJ4bXNfYWN0X2ZjdCI6IjkgMyIsInhtc19mdGQiOiJkRm1iM1Fqa3oxMjBKZFJHbjNfNnVzajZDZHdwNi1hcW5BcDFXT21iZENFQmRYTmxZWE4wTFdSemJYTSIsInhtc19pZHJlbCI6IjcgMTYiLCJ4bXNfcGZ0ZXhwIjoxNzgzMTM4NTc0LCJ4bXNfcmQiOiIwLjQyTGxZQkppZEJNUzRXQVZFcEJ6U3p5M29MSEZaN2RYdGFuWExOTmNJUkVPVGlHQnRJOWVVdy1lZXVrNVFldzVjLU4ybHM5Q0lod2NRZ0xNREJCd0FFb0xpWEJ3Q3dsc2VTRDdOZHpmN3VRaVNha1duNWF0QWdBIiwieG1zX3N1Yl9mY3QiOiIzIDkiLCJ4bXNfdGNkdCI6MTU4MTAyMjkyNywieG1zX3RudF9mY3QiOiIzIDgifQ.B5m5DUvFOW5_ob86hgh6nF7al3Y56hClOdscolCMtvGPoN1OKxilEeU1yy7hHUG67NGH7mUXa4MYkXul5Xv42jvHYMvcI_t8zQY68Wm4td30nIgXFhvVfyT8eV8Dhs6EF2x6HOkyF58XZe9tI837eQmo2mAdSoJZbJF-bDcKmHjTQWJZAcycXlRr5JKUWq4QVXYUmhihpOHt9n4eVJAbuHF1sokDc8XfysRp93sa72Fz4FEhaM8ejv3oNj0l7AE3UmIfL4Z2FT5qglaJmKzELXV-dOAwyt_LpYjoxcCoVdzrm7OGKW38CbjaJKNIXs1bHbmxuNMkup3tpdt8c6CV9g

View File

@@ -0,0 +1,55 @@
# Datto -> SharePoint FULL reconciliation - COMPLETE - 2026-07-02/03
## Outcome
**557 missing files recovered; all 5 site mappings independently verified at zero genuinely-missing.**
| Site | Datto files | Missing found | Recovered | Verified residual |
|---|---|---|---|---|
| Donor Services | 57,329 | 74 | 73 | 1 = intentional ghost-skip ("26-02328 - Copy.pdf", same-size renamed sibling in SP) |
| Admin | 6,301 | 472 | 468 + 4 (long-path retry) | 0 |
| Birth Biologic Activity Reports | 1 | 1 | 1 | 0 |
| Quality (QSD) | 3,768 | 11 | 11 | 0 |
| Supply Management | 160 | 0 | - | 0 |
sp-only counts (files in SP not in Datto: staff's new post-migration work) and size-diffs
(docx uniformly larger in SP = SharePoint Office property promotion; content-harmless) are
expected and documented per-site in C:\GuruCompare\ on ACG-DWP-X-BB.
## Root cause (answers "how was it missing after the sync to 0")
The 6/27 delta upload used simple Graph PUTs which **silently skip files >=4MB**; the
"reconciled to 0" claim trusted the uploader's own success reporting. Quality got the
chunked-upload fix 6/30; Donor Services and Admin were never re-verified. Admin was worst
(472 files, incl. >260-char long-path victims in consultant folder trees). Staff had begun
manually recreating missing donor PDFs (e.g. 26-04085.pdf, Mary Ster, 7/2).
## Method (repeatable)
`C:\GuruCompare\bb-fix-all.ps1` on ACG-DWP-X-BB (source in ClaudeTools scratch/session log):
per-site full inventory diff (path+size, case-insensitive) + rail-guarded fix:
- chunked upload sessions for >=4MB, simple PUT below
- conflictBehavior=fail (never overwrites SP)
- same-size-in-folder ghost detection (never resurrects renamed files)
- \\?\ long-path file opens; FileShare::ReadWrite (Datto service can stay running)
- fail-hard SP enumeration w/ 429 retry; sanity-abort if SP looks empty; per-site fresh token
Run via GuruRMM per site: `powershell -File C:\GuruCompare\bb-fix-all.ps1 -Site <n>`
(fresh Graph token to C:\GuruCompare\tok.txt first). Logs: C:\GuruCompare\<site>-fix-log.txt.
## Gotchas hit (memorialized)
- PS5.1 under RMM powershell -File: `-Headers $H` script-scope hashtable sent EMPTY
Authorization ("Access token is empty") - build headers inline. Memory: ps51-invoke-restmethod-headers-variable.
- Two failed runs masqueraded as token expiry / clock skew; capturing the Graph error BODY
was what cracked it.
- ACG-DWP-X-BB clock was NTP-corrected 8:03 PM 7/2 (had drifted while parked).
## Open items
- OneDrive placeholder mirror on ACG-DWP-X-BB for standing local comparisons: policies ready
to push; needs interactive OneDrive sign-in as sysadmin@birthbiologic.com via ScreenConnect.
- Datto Workplace service on the VM left RUNNING (catch-net for the still-active
donor-scan-to-Datto workflow; 2 scans from 6/29 + HR file were among the recovered).
Decide: keep as ongoing catch-net vs re-freeze.
- Client comms: staff can stop manually recreating missing April/May donor records.

View File

@@ -0,0 +1,192 @@
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
## Session Summary
Long multi-thread session. Three bodies of work: (1) Peaceful Spirit infrastructure — bringing a new
NW-site domain controller online and cleaning up the dead PST-SERVER2; (2) Peaceful Spirit data
recovery — the 2025 server-crash corrupted-file investigation, recovering four payroll spreadsheets
and emailing them to the client; (3) the dominant thread — a deep GuruRMM VSS/CrowdStrike redesign
that evolved from "internalize the agent's VSS ops to native COM" into a full architectural pivot
("agent = policy configurator, not shadow operator") with two new spec folders and a validated
native-COM code path.
Peaceful Spirit: a new physical server (shipped as PST-SERVER01) was enrolled in GuruRMM, renamed to
PST-DC-NW pre-domain-join (via RMM), then joined + promoted DC/GC/DNS into the NW site — all via RMM.
Before promotion, the dead PST-SERVER2 (hardware died ~2026-06-14) was metadata-cleaned from AD
(DFSR member/connection objects, NTDS/config server object, DC computer account, 18 stale DNS
records); PST-DFS was re-added with PST-DC-NW as the C:\Shares receiver (initial ~265 GB sync
started). Data recovery: read Mara's Claude-analysis docx, inventoried PST-SERVER D: (the old crashed
server's drive), scanned the live G:\Shares for Mike's `[C]`-prefixed corrupted files (5,044 files),
then content-fingerprinted the carved `D:\Unknown folder` output to positively identify four
MPEG-corrupted 2024/2025 payroll xlsx (incl. the `IC Payments 2-12 to 2-26-25` file Mara chased for a
year), which were emailed to Mara from mike@azcomputerguru.com.
GuruRMM VSS: the session opened investigating a CrowdStrike Falcon T1490 "Inhibit System Recovery"
detection on NEPTUNE — the GuruRMM agent's scheduled VSS pass, running `powershell -ExecutionPolicy
Bypass` shadow-copy ops, matched the ransomware TTP and Falcon blocked it. This drove: an audit of all
agent shell-outs (Grok + agy reviewed), an adversarial pass (agy) that corrected the plan, a
`vss-native-com` spec, and actual implementation — native VSS provision (validated on NEPTUNE) and a
hand-rolled `IVssBackupComponents` create/delete (built from the SDK vsbackup.h, compiled + create
runtime-validated). Runtime testing then revealed the actual T1490 trigger was the PRUNE/DELETE (all
explicit deletes are Falcon-blocked; cap-driven eviction rotates shadows fine), prompting the pivot:
drop the scheduled prune, configure the OS-native shadow-copy system instead. A second spec,
`vss-policy-config`, captures that; multi-AI (Grok + Gemini) confirmed a uniform native-COM-create
approach works on all SKUs including workstations, and added the `MaxShadowCopies` count governor.
Task 1 of that spec (the blocking spike) was completed on GURU-5070 (Win11 Pro client): MaxShadowCopies
FIFO-evicts (does not block), and native COM create works on the client SKU.
Also rewired the `agy` skill from the dead Google `gemini` npm CLI (which failed with
`throwIneligibleOrProjectIdError`) to the new Antigravity `agy` binary, and repointed grok's xsearch
fallback to it.
## Key Decisions
- **PST-DC-NW rename before domain join.** Renaming a promoted DC is messy (DNS/SPN/DFSR cleanup), so
the rename ran first, on the workgroup box, via RMM.
- **PST-SERVER2 got a full AD metadata cleanup, not just power-off.** It died past cases where you can
resume replication; leaving stale DC metadata is what caused the June tombstone mess. Removed via
direct AD object deletion (Remove-DfsrMember fails on a dead member — "network path not found").
- **Data recovery: identify carved files by content, not by name.** The crash detached filenames from
data blocks; carved output has bracket names. Fingerprinted sharedStrings + exact byte-size +
in-workbook date serials to positively match. Did NOT place recovered files into the live HIPAA
share — emailed to Mara for her to verify.
- **VSS: use native VSS COM API `IVssBackupComponents`, NOT WMI** (adversarial-pass finding) — WMI
`Win32_ShadowCopy` still runs in wmiprvse.exe and trips T1490; only the native requestor reads as
legitimate backup software.
- **Concurrency: dedicated COM-owning OS thread + MPSC + timeout, NOT tokio::spawn_blocking** — a hung
COM/WMI call on the shared blocking pool would deadlock the agent.
- **THE PIVOT: agent = configurator, not operator.** Runtime testing proved the T1490 trigger is the
DELETE/prune (Falcon blocks ALL explicit shadow deletes), while cap-driven kernel eviction rotates
shadows fine. So drop the scheduled prune (retention = cap), configure the OS-native create schedule,
and the agent just sets policy + reports status. This removed most of the create/delete-internalization
work from the hot path.
- **Workstations: uniform native-COM create on all SKUs** (multi-AI validated). `vssadmin create shadow`
is Server-only, but `IVssBackupComponents::DoSnapshotSet` and volsnap cap-eviction are NOT SKU-gated —
only the CLI is. The scheduled-task action is the signed agent's native create on every SKU.
- **Two retention governors, both FIFO, zero deletes:** shadow-storage MaxSize (native COM) +
MaxShadowCopies count (registry). Grok caught the second one; Task 1 confirmed it FIFO-evicts.
- **Do NOT ship the Falcon IOA exclusion** (`powershell.exe`+`gururmm_*.ps1`) — adversarial review
flagged it as an RCE backdoor (SystemTemp is world-writable-ish). Stopgap = pause the VSS schedule.
- **Remote build/verify loop:** this box (GURU-5070) can't `cargo check` the agent (no MSVC build
tools); push branch to internal Gitea over HTTP, `cargo check`/`build` on the Pluto build host.
- **Binary transfer to RMM-managed boxes: use the RMM server's `/downloads` path** (NEPTUNE/GURU-5070
reach `rmm.azcomputerguru.com/downloads` over HTTPS) — peer-to-peer (Pluto:8099) is network-segmented.
## Problems Encountered
- **Falcon T1490 blocked the agent's VSS pass on NEPTUNE.** Root cause = the temp-ps1 + ExecutionPolicy
Bypass shadow-op pattern (ransomware TTP). Immediate: disabled the `GuruRMM-VSS-Snapshot` scheduled
task (stopgap). Real fix = the redesign.
- **`IVssBackupComponents` is absent from windows-rs 0.58** (its header vsbackup.h is outside
win32metadata). Resolved by hand-declaring the interface: pulled the exact 40-slot vtable order + IID
+ method signatures from the SDK `vsbackup.h` on Pluto, built it with `windows_core::imp::define_interface!`
+ a manual vtable (usize placeholders for the 33 unused slots) + `vssapi.dll` FFI.
- **Link error: `VssFreeSnapshotProperties` not in vssapi.lib** (LNK2019). Resolved by freeing the
VSS_SNAPSHOT_PROP string members directly via `CoTaskMemFree` (what the function does internally).
- **`AddDiffArea`/`ChangeDiffAreaMaximumSize` take `*const u16`, not `PCWSTR`** — caught by the Pluto
cargo check (a check I can't run locally). Fixed.
- **RMM inline-JSON dispatch mangles backslash payloads** (recurring) — used the ps-encoded script-file
path and forward slashes throughout.
- **Native delete "succeeded" but didn't remove the shadow.** Investigation: the shadow was a normal
ClientAccessible shadow, and vssadmin + WMI `.Delete()` ALSO failed — because Falcon blocks all
explicit shadow deletes (T1490). This was the pivotal finding driving the configurator redesign.
- **agy/Gemini CLI dead** (`throwIneligibleOrProjectIdError`) — the old npm gemini CLI needs a
GOOGLE_CLOUD_PROJECT the account can't supply. Resolved by rewiring the skill to the Antigravity
`agy` binary (own auth, no project ID).
- **Gitea push over the Cloudflare https remote fails auth** — pushed to the internal HTTP Gitea
(`http://172.16.3.20:3000`) with URL-encoded vault creds (password has URL-breaking chars).
- **One transient native-create slowness on GURU-5070** (>150s once, then <75s) — not a systematic
hang; the scheduled create action needs a sane timeout/retry.
## Configuration Changes
Peaceful Spirit / infra (via RMM, on PST-SERVER `87293069-...` and PST-DC-NW `f60e9820-...`):
- PST-SERVER01 renamed to PST-DC-NW; domain-joined PEACEFULSPIRIT.local; promoted DC/GC/DNS (site NW);
static 192.168.1.5; timezone AZ. PST-SERVER2 AD/DNS/DFSR metadata removed. PST-DFS re-added with
PST-DC-NW C:\Shares receiver. DSRM password vaulted.
- `GuruRMM-VSS-Snapshot` scheduled task DISABLED on NEPTUNE (`b3a9b454-...`) — T1490 stopgap.
Repo (guru-rmm submodule, branch `feat/vss-native-com`):
- `agent/src/vss_com.rs` (new) — native COM VSS worker: dedicated COM thread + MPSC; `provision` (cap),
`create`/`delete` (hand-rolled IVssBackupComponents).
- `agent/src/vss.rs` — provision/create/delete branched native(modern)/PS(legacy); added top-level
`provision_storage` wrapper.
- `agent/src/main.rs` — added `mod vss_com`; hidden diagnostic verbs `vss-provision-test`,
`vss-roundtrip-test`.
- `agent/Cargo.toml` — +windows features `Win32_Storage_Vss`, `Win32_System_Com`; +`windows-core` dep.
- `specs/vss-native-com/` (4 files) — the internalization spec (superseded-in-part by the pivot).
- `specs/vss-policy-config/` (4 files) — the configurator redesign spec (Task 1 DONE).
- `docs/RMM_THOUGHTS.md` — internalize-VSS entry, shell-out audit, adversarial-pass corrections, the
configurator PIVOT.
- `specs/crowdstrike-falcon/{plan,references}.md` — T1490 rollout dependency (pause schedule, not exclusion).
Repo (ClaudeTools main):
- `.claude/skills/agy/scripts/ask-agy.sh` (new) — Antigravity CLI wrapper; `ask-gemini.sh` now a shim.
- `.claude/skills/agy/SKILL.md` — rewired to agy.
- `.claude/skills/grok/scripts/ask-grok.sh` — xsearch fallback repointed gemini->agy.
- `.claude/identity.json` (local, gitignored) — added `agy` block, retired `gemini`.
- `wiki/clients/peaceful-spirit.md` — extensive updates (PST-DC-NW, metadata cleanup, 2025 crash &
corruption section, recovery results).
## Credentials & Secrets
- **PST-DC-NW DSRM password** — created + vaulted at `clients/peaceful-spirit/dc-nw` (SOPS vault).
- **PEACEFULSPIRIT\sysadmin (DA)** — read from vault `clients/peaceful-spirit/server` for the
join/promotion; used via RMM Invoke-Command (FQDN). Not rotated (RMM internal).
- **GURU-5070 finding (NOT vaulted — flag for follow-up):** `~/.gemini/antigravity-cli/settings.json`
contains a live 1Password service-account token (`OP_SERVICE_ACCOUNT_TOKEN=ops_eyJ...`) in its
`permissions.allow` list, in plaintext. Should be vaulted + scrubbed. (The adjacent
`GEMINI_API_KEY=AIzaSyDummyKey...` is a dummy.)
- Gitea push creds: vault `services/gitea.sops.yaml` (credentials.username/password); push via internal
HTTP `172.16.3.20:3000`.
## Infrastructure & Servers
- **PST-DC-NW** (Peaceful Spirit NW DC) — 192.168.1.5, Win Server 2019, RMM agent `f60e9820-4a00-4598-83f7-c14085db5768`, site "North West".
- **PST-SERVER** (CC DC) — 192.168.0.2, RMM agent `87293069-33b6-45e8-a68f-6811216cdb96`. D: = old crashed server's drive (recovery corpus; FROZEN, no cleanup). G:\Shares = live data.
- **PST-SERVER2** — DEAD (~2026-06-14), metadata cleaned 2026-07-03.
- **NEPTUNE** (ACG internal) — 172.16.3.11 / WAN 67.206.163.122, Win Server 2022, RMM agent `b3a9b454-86eb-491c-ac67-c1f98987d8dc`, CrowdStrike Falcon present. Runs cbb.exe (MSP360, native VSS backup). Leftover test shadow `{8FEFDAE3-A002-439F-B362-92FC4B9CCDAD}` (Falcon-blocked delete; will rotate).
- **GURU-5070** (this box) — Win11 Pro (ProductType=1), RMM agent `819df0c8-4824-4424-b55a-2c5cb4d6ca39`, site "Mike's Car", Falcon present. Leftover test shadow `{87AFEA99-3336-4F85-A9D9-0E3F2F00D51B}`.
- **Pluto** build host — `Administrator@172.16.3.36` (SSH key), C:\gururmm checkout, has MSVC + Rust (stable + 1.77 legacy) + Windows SDK. Used for `cargo check`/`build`.
- **RMM server** — `guru@172.16.3.30` (Ubuntu 22.04, SSH key), serves `rmm.azcomputerguru.com/downloads` from `/var/www/gururmm/downloads`. Used to stage test binaries for RMM-managed boxes.
- **Internal Gitea** — `http://172.16.3.20:3000` (HTTP, not the Cloudflare https remote).
- **B2 (Backblaze)** — 15 buckets; generic `MSPBackups20200311` holds other clients only (no pre-crash Peaceful Spirit data); ACG-PST plan created on crash day (no pre-incident copy).
## Commands & Outputs
- Falcon efficacy (Datto EDR): 500 detections / 90 days fleet-wide = ZERO T1490/shadow detections from native backup binaries.
- MaxShadowCopies FIFO test (GURU-5070): `MaxShadowCopies=3`, WMI Create -> oldest evicted, count stayed 3. `[RESULT] FIFO-EVICT`.
- Native COM create on client (GURU-5070): `[OK] created shadow_id={87AFEA99...} device=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7`.
- Native provision runtime (NEPTUNE): `ChangeDiffAreaMaximumSize` moved C: cap 279GB(15%)->298GB(16%), restored.
- Pluto build: STABLE_RC=0, LEGACY_RC=0 (both variants); Task 4 fresh binary sha256 `1368eba8...`.
- Push pattern: `git push http://<user>:<urlenc-pw>@172.16.3.20:3000/azcomputerguru/gururmm.git feat/vss-native-com`.
## Pending / Incomplete Tasks
- **vss-policy-config Task 2+** (unblocked): build the `vss-create` verb + scheduled-task registration
(uniform native create, all SKUs) + set both governors (cap via COM, MaxShadowCopies via registry) +
every-N-hours schedule support; retire the legacy `GuruRMM-VSS-Snapshot` task on migration; drop the
scheduled prune; update status/compliance. Then build both variants on Pluto + runtime-test on a
Server AND a Win10/11 Pro box; merge (fleet migration).
- **Vault the leaked 1Password token** in GURU-5070's antigravity settings.json + scrub it.
- **Leftover test shadows:** `{8FEFDAE3}` (NEPTUNE), `{87AFEA99}` (GURU-5070) — Falcon-blocked delete;
will rotate via governors. Optional cleanup.
- **Peaceful Spirit open items** (in the wiki): PST-DC-NW DFS-R initial ~265 GB sync -> then Gate 4
(share C:\Shares, add folder target + 2nd namespace root); deletion recovery ~3,342 genuine files
(awaiting Mike/Mara go); 5th corrupted payroll file `IC Payments 5-13 to 5-27-25` unmatched.
- **EV code-signing + AV-vendor whitelisting** — prerequisite the adversarial review flagged for any
on-demand delete on Falcon hosts.
- Parked agent shell-out internalizations: `users.rs` (T1098, native NetUser*), `inventory.rs`.
## Reference Information
- Branch: `feat/vss-native-com` (guru-rmm submodule). Key commits: `35491a4` (vss-policy-config spec),
`d04a2d8` (Task 1 done), `2314ad55` (agy rewire), `07c8dc0` (every-N-hours).
- IVssBackupComponents IID: `665c1d5f-c218-414d-a05d-7fef5f9d5c86`. VSS SW provider: `b5946137-7b9f-4925-af80-51abd60b20d5`. CLSID_VssSnapshotMgmt: `0b5a2c52-3eb9-470a-96e2-6c6d4570e40f`.
- MaxShadowCopies: `HKLM\SYSTEM\CurrentControlSet\Services\VSS\Settings\MaxShadowCopies` (DWORD, default 64, max 512).
- SDK header (Pluto): `C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\um\vsbackup.h`.
- Recovered payroll files emailed to Mara (info@bestmassageintucson.com) from mike@azcomputerguru.com.
- Specs: `projects/msp-tools/guru-rmm/specs/{vss-native-com,vss-policy-config}/`.