claudetools/session-logs/2026-07/2026-07-03-mike-gururmm-vss-configurator-and-pst-recovery.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Long multi-thread session. Three bodies of work: (1) Peaceful Spirit infrastructure — bringing a new NW-site domain controller online and cleaning up the dead PST-SERVER2; (2) Peaceful Spirit data recovery — the 2025 server-crash corrupted-file investigation, recovering four payroll spreadsheets and emailing them to the client; (3) the dominant thread — a deep GuruRMM VSS/CrowdStrike redesign that evolved from "internalize the agent's VSS ops to native COM" into a full architectural pivot ("agent = policy configurator, not shadow operator") with two new spec folders and a validated native-COM code path.

Peaceful Spirit: a new physical server (shipped as PST-SERVER01) was enrolled in GuruRMM, renamed to PST-DC-NW pre-domain-join (via RMM), then joined + promoted DC/GC/DNS into the NW site — all via RMM. Before promotion, the dead PST-SERVER2 (hardware died ~2026-06-14) was metadata-cleaned from AD (DFSR member/connection objects, NTDS/config server object, DC computer account, 18 stale DNS records); PST-DFS was re-added with PST-DC-NW as the C:\Shares receiver (initial ~265 GB sync started). Data recovery: read Mara's Claude-analysis docx, inventoried PST-SERVER D: (the old crashed server's drive), scanned the live G:\Shares for Mike's [C]-prefixed corrupted files (5,044 files), then content-fingerprinted the carved D:\Unknown folder output to positively identify four MPEG-corrupted 2024/2025 payroll xlsx (incl. the IC Payments 2-12 to 2-26-25 file Mara chased for a year), which were emailed to Mara from mike@azcomputerguru.com.

GuruRMM VSS: the session opened investigating a CrowdStrike Falcon T1490 "Inhibit System Recovery" detection on NEPTUNE — the GuruRMM agent's scheduled VSS pass, running powershell -ExecutionPolicy Bypass shadow-copy ops, matched the ransomware TTP and Falcon blocked it. This drove: an audit of all agent shell-outs (Grok + agy reviewed), an adversarial pass (agy) that corrected the plan, a vss-native-com spec, and actual implementation — native VSS provision (validated on NEPTUNE) and a hand-rolled IVssBackupComponents create/delete (built from the SDK vsbackup.h, compiled + create runtime-validated). Runtime testing then revealed the actual T1490 trigger was the PRUNE/DELETE (all explicit deletes are Falcon-blocked; cap-driven eviction rotates shadows fine), prompting the pivot: drop the scheduled prune, configure the OS-native shadow-copy system instead. A second spec, vss-policy-config, captures that; multi-AI (Grok + Gemini) confirmed a uniform native-COM-create approach works on all SKUs including workstations, and added the MaxShadowCopies count governor. Task 1 of that spec (the blocking spike) was completed on GURU-5070 (Win11 Pro client): MaxShadowCopies FIFO-evicts (does not block), and native COM create works on the client SKU.

Also rewired the agy skill from the dead Google gemini npm CLI (which failed with throwIneligibleOrProjectIdError) to the new Antigravity agy binary, and repointed grok's xsearch fallback to it.

Key Decisions

• PST-DC-NW rename before domain join. Renaming a promoted DC is messy (DNS/SPN/DFSR cleanup), so the rename ran first, on the workgroup box, via RMM.
• PST-SERVER2 got a full AD metadata cleanup, not just power-off. It died past cases where you can resume replication; leaving stale DC metadata is what caused the June tombstone mess. Removed via direct AD object deletion (Remove-DfsrMember fails on a dead member — "network path not found").
• Data recovery: identify carved files by content, not by name. The crash detached filenames from data blocks; carved output has bracket names. Fingerprinted sharedStrings + exact byte-size + in-workbook date serials to positively match. Did NOT place recovered files into the live HIPAA share — emailed to Mara for her to verify.
• VSS: use native VSS COM API IVssBackupComponents, NOT WMI (adversarial-pass finding) — WMI Win32_ShadowCopy still runs in wmiprvse.exe and trips T1490; only the native requestor reads as legitimate backup software.
• Concurrency: dedicated COM-owning OS thread + MPSC + timeout, NOT tokio::spawn_blocking — a hung COM/WMI call on the shared blocking pool would deadlock the agent.
• THE PIVOT: agent = configurator, not operator. Runtime testing proved the T1490 trigger is the DELETE/prune (Falcon blocks ALL explicit shadow deletes), while cap-driven kernel eviction rotates shadows fine. So drop the scheduled prune (retention = cap), configure the OS-native create schedule, and the agent just sets policy + reports status. This removed most of the create/delete-internalization work from the hot path.
• Workstations: uniform native-COM create on all SKUs (multi-AI validated). vssadmin create shadow is Server-only, but IVssBackupComponents::DoSnapshotSet and volsnap cap-eviction are NOT SKU-gated — only the CLI is. The scheduled-task action is the signed agent's native create on every SKU.
• Two retention governors, both FIFO, zero deletes: shadow-storage MaxSize (native COM) + MaxShadowCopies count (registry). Grok caught the second one; Task 1 confirmed it FIFO-evicts.
• Do NOT ship the Falcon IOA exclusion (powershell.exe+gururmm_*.ps1) — adversarial review flagged it as an RCE backdoor (SystemTemp is world-writable-ish). Stopgap = pause the VSS schedule.
• Remote build/verify loop: this box (GURU-5070) can't cargo check the agent (no MSVC build tools); push branch to internal Gitea over HTTP, cargo check/build on the Pluto build host.
• Binary transfer to RMM-managed boxes: use the RMM server's /downloads path (NEPTUNE/GURU-5070 reach rmm.azcomputerguru.com/downloads over HTTPS) — peer-to-peer (Pluto:8099) is network-segmented.

Problems Encountered

• Falcon T1490 blocked the agent's VSS pass on NEPTUNE. Root cause = the temp-ps1 + ExecutionPolicy Bypass shadow-op pattern (ransomware TTP). Immediate: disabled the GuruRMM-VSS-Snapshot scheduled task (stopgap). Real fix = the redesign.
• IVssBackupComponents is absent from windows-rs 0.58 (its header vsbackup.h is outside win32metadata). Resolved by hand-declaring the interface: pulled the exact 40-slot vtable order + IID
  • method signatures from the SDK vsbackup.h on Pluto, built it with windows_core::imp::define_interface!
  • a manual vtable (usize placeholders for the 33 unused slots) + vssapi.dll FFI.
• Link error: VssFreeSnapshotProperties not in vssapi.lib (LNK2019). Resolved by freeing the VSS_SNAPSHOT_PROP string members directly via CoTaskMemFree (what the function does internally).
• AddDiffArea/ChangeDiffAreaMaximumSize take *const u16, not PCWSTR — caught by the Pluto cargo check (a check I can't run locally). Fixed.
• RMM inline-JSON dispatch mangles backslash payloads (recurring) — used the ps-encoded script-file path and forward slashes throughout.
• Native delete "succeeded" but didn't remove the shadow. Investigation: the shadow was a normal ClientAccessible shadow, and vssadmin + WMI .Delete() ALSO failed — because Falcon blocks all explicit shadow deletes (T1490). This was the pivotal finding driving the configurator redesign.
• agy/Gemini CLI dead (throwIneligibleOrProjectIdError) — the old npm gemini CLI needs a GOOGLE_CLOUD_PROJECT the account can't supply. Resolved by rewiring the skill to the Antigravity agy binary (own auth, no project ID).
• Gitea push over the Cloudflare https remote fails auth — pushed to the internal HTTP Gitea (http://172.16.3.20:3000) with URL-encoded vault creds (password has URL-breaking chars).
• One transient native-create slowness on GURU-5070 (>150s once, then <75s) — not a systematic hang; the scheduled create action needs a sane timeout/retry.

Configuration Changes

Peaceful Spirit / infra (via RMM, on PST-SERVER 87293069-... and PST-DC-NW f60e9820-...):

• PST-SERVER01 renamed to PST-DC-NW; domain-joined PEACEFULSPIRIT.local; promoted DC/GC/DNS (site NW); static 192.168.1.5; timezone AZ. PST-SERVER2 AD/DNS/DFSR metadata removed. PST-DFS re-added with PST-DC-NW C:\Shares receiver. DSRM password vaulted.
• GuruRMM-VSS-Snapshot scheduled task DISABLED on NEPTUNE (b3a9b454-...) — T1490 stopgap.

Repo (guru-rmm submodule, branch feat/vss-native-com):

• agent/src/vss_com.rs (new) — native COM VSS worker: dedicated COM thread + MPSC; provision (cap), create/delete (hand-rolled IVssBackupComponents).
• agent/src/vss.rs — provision/create/delete branched native(modern)/PS(legacy); added top-level provision_storage wrapper.
• agent/src/main.rs — added mod vss_com; hidden diagnostic verbs vss-provision-test, vss-roundtrip-test.
• agent/Cargo.toml — +windows features Win32_Storage_Vss, Win32_System_Com; +windows-core dep.
• specs/vss-native-com/ (4 files) — the internalization spec (superseded-in-part by the pivot).
• specs/vss-policy-config/ (4 files) — the configurator redesign spec (Task 1 DONE).
• docs/RMM_THOUGHTS.md — internalize-VSS entry, shell-out audit, adversarial-pass corrections, the configurator PIVOT.
• specs/crowdstrike-falcon/{plan,references}.md — T1490 rollout dependency (pause schedule, not exclusion).

Repo (ClaudeTools main):

• .claude/skills/agy/scripts/ask-agy.sh (new) — Antigravity CLI wrapper; ask-gemini.sh now a shim.
• .claude/skills/agy/SKILL.md — rewired to agy.
• .claude/skills/grok/scripts/ask-grok.sh — xsearch fallback repointed gemini->agy.
• .claude/identity.json (local, gitignored) — added agy block, retired gemini.
• wiki/clients/peaceful-spirit.md — extensive updates (PST-DC-NW, metadata cleanup, 2025 crash & corruption section, recovery results).

Credentials & Secrets

• PST-DC-NW DSRM password — created + vaulted at clients/peaceful-spirit/dc-nw (SOPS vault).
• PEACEFULSPIRIT\sysadmin (DA) — read from vault clients/peaceful-spirit/server for the join/promotion; used via RMM Invoke-Command (FQDN). Not rotated (RMM internal).
• GURU-5070 finding (NOT vaulted — flag for follow-up): ~/.gemini/antigravity-cli/settings.json contains a live 1Password service-account token (OP_SERVICE_ACCOUNT_TOKEN=ops_eyJ...) in its permissions.allow list, in plaintext. Should be vaulted + scrubbed. (The adjacent GEMINI_API_KEY=AIzaSyDummyKey... is a dummy.)
• Gitea push creds: vault services/gitea.sops.yaml (credentials.username/password); push via internal HTTP 172.16.3.20:3000.

Infrastructure & Servers

• PST-DC-NW (Peaceful Spirit NW DC) — 192.168.1.5, Win Server 2019, RMM agent f60e9820-4a00-4598-83f7-c14085db5768, site "North West".
• PST-SERVER (CC DC) — 192.168.0.2, RMM agent 87293069-33b6-45e8-a68f-6811216cdb96. D: = old crashed server's drive (recovery corpus; FROZEN, no cleanup). G:\Shares = live data.
• PST-SERVER2 — DEAD (~2026-06-14), metadata cleaned 2026-07-03.
• NEPTUNE (ACG internal) — 172.16.3.11 / WAN 67.206.163.122, Win Server 2022, RMM agent b3a9b454-86eb-491c-ac67-c1f98987d8dc, CrowdStrike Falcon present. Runs cbb.exe (MSP360, native VSS backup). Leftover test shadow {8FEFDAE3-A002-439F-B362-92FC4B9CCDAD} (Falcon-blocked delete; will rotate).
• GURU-5070 (this box) — Win11 Pro (ProductType=1), RMM agent 819df0c8-4824-4424-b55a-2c5cb4d6ca39, site "Mike's Car", Falcon present. Leftover test shadow {87AFEA99-3336-4F85-A9D9-0E3F2F00D51B}.
• Pluto build host — Administrator@172.16.3.36 (SSH key), C:\gururmm checkout, has MSVC + Rust (stable + 1.77 legacy) + Windows SDK. Used for cargo check/build.
• RMM server — guru@172.16.3.30 (Ubuntu 22.04, SSH key), serves rmm.azcomputerguru.com/downloads from /var/www/gururmm/downloads. Used to stage test binaries for RMM-managed boxes.
• Internal Gitea — http://172.16.3.20:3000 (HTTP, not the Cloudflare https remote).
• B2 (Backblaze) — 15 buckets; generic MSPBackups20200311 holds other clients only (no pre-crash Peaceful Spirit data); ACG-PST plan created on crash day (no pre-incident copy).

Commands & Outputs

• Falcon efficacy (Datto EDR): 500 detections / 90 days fleet-wide = ZERO T1490/shadow detections from native backup binaries.
• MaxShadowCopies FIFO test (GURU-5070): MaxShadowCopies=3, WMI Create -> oldest evicted, count stayed 3. [RESULT] FIFO-EVICT.
• Native COM create on client (GURU-5070): [OK] created shadow_id={87AFEA99...} device=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7.
• Native provision runtime (NEPTUNE): ChangeDiffAreaMaximumSize moved C: cap 279GB(15%)->298GB(16%), restored.
• Pluto build: STABLE_RC=0, LEGACY_RC=0 (both variants); Task 4 fresh binary sha256 1368eba8....
• Push pattern: git push http://<user>:<urlenc-pw>@172.16.3.20:3000/azcomputerguru/gururmm.git feat/vss-native-com.

Pending / Incomplete Tasks

• vss-policy-config Task 2+ (unblocked): build the vss-create verb + scheduled-task registration (uniform native create, all SKUs) + set both governors (cap via COM, MaxShadowCopies via registry) + every-N-hours schedule support; retire the legacy GuruRMM-VSS-Snapshot task on migration; drop the scheduled prune; update status/compliance. Then build both variants on Pluto + runtime-test on a Server AND a Win10/11 Pro box; merge (fleet migration).
• Vault the leaked 1Password token in GURU-5070's antigravity settings.json + scrub it.
• Leftover test shadows: {8FEFDAE3} (NEPTUNE), {87AFEA99} (GURU-5070) — Falcon-blocked delete; will rotate via governors. Optional cleanup.
• Peaceful Spirit open items (in the wiki): PST-DC-NW DFS-R initial ~265 GB sync -> then Gate 4 (share C:\Shares, add folder target + 2nd namespace root); deletion recovery ~3,342 genuine files (awaiting Mike/Mara go); 5th corrupted payroll file IC Payments 5-13 to 5-27-25 unmatched.
• EV code-signing + AV-vendor whitelisting — prerequisite the adversarial review flagged for any on-demand delete on Falcon hosts.
• Parked agent shell-out internalizations: users.rs (T1098, native NetUser*), inventory.rs.

Reference Information

• Branch: feat/vss-native-com (guru-rmm submodule). Key commits: 35491a4 (vss-policy-config spec), d04a2d8 (Task 1 done), 2314ad55 (agy rewire), 07c8dc0 (every-N-hours).
• IVssBackupComponents IID: 665c1d5f-c218-414d-a05d-7fef5f9d5c86. VSS SW provider: b5946137-7b9f-4925-af80-51abd60b20d5. CLSID_VssSnapshotMgmt: 0b5a2c52-3eb9-470a-96e2-6c6d4570e40f.
• MaxShadowCopies: HKLM\SYSTEM\CurrentControlSet\Services\VSS\Settings\MaxShadowCopies (DWORD, default 64, max 512).
• SDK header (Pluto): C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\um\vsbackup.h.
• Recovered payroll files emailed to Mara (info@bestmassageintucson.com) from mike@azcomputerguru.com.
• Specs: projects/msp-tools/guru-rmm/specs/{vss-native-com,vss-policy-config}/.