sync: auto-sync from GURU-5070 at 2026-07-03 13:18:27
Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-03 13:18:27
This commit is contained in:
@@ -0,0 +1,192 @@
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** GURU-5070
|
||||
- **Role:** admin
|
||||
|
||||
## Session Summary
|
||||
|
||||
Long multi-thread session. Three bodies of work: (1) Peaceful Spirit infrastructure — bringing a new
|
||||
NW-site domain controller online and cleaning up the dead PST-SERVER2; (2) Peaceful Spirit data
|
||||
recovery — the 2025 server-crash corrupted-file investigation, recovering four payroll spreadsheets
|
||||
and emailing them to the client; (3) the dominant thread — a deep GuruRMM VSS/CrowdStrike redesign
|
||||
that evolved from "internalize the agent's VSS ops to native COM" into a full architectural pivot
|
||||
("agent = policy configurator, not shadow operator") with two new spec folders and a validated
|
||||
native-COM code path.
|
||||
|
||||
Peaceful Spirit: a new physical server (shipped as PST-SERVER01) was enrolled in GuruRMM, renamed to
|
||||
PST-DC-NW pre-domain-join (via RMM), then joined + promoted DC/GC/DNS into the NW site — all via RMM.
|
||||
Before promotion, the dead PST-SERVER2 (hardware died ~2026-06-14) was metadata-cleaned from AD
|
||||
(DFSR member/connection objects, NTDS/config server object, DC computer account, 18 stale DNS
|
||||
records); PST-DFS was re-added with PST-DC-NW as the C:\Shares receiver (initial ~265 GB sync
|
||||
started). Data recovery: read Mara's Claude-analysis docx, inventoried PST-SERVER D: (the old crashed
|
||||
server's drive), scanned the live G:\Shares for Mike's `[C]`-prefixed corrupted files (5,044 files),
|
||||
then content-fingerprinted the carved `D:\Unknown folder` output to positively identify four
|
||||
MPEG-corrupted 2024/2025 payroll xlsx (incl. the `IC Payments 2-12 to 2-26-25` file Mara chased for a
|
||||
year), which were emailed to Mara from mike@azcomputerguru.com.
|
||||
|
||||
GuruRMM VSS: the session opened investigating a CrowdStrike Falcon T1490 "Inhibit System Recovery"
|
||||
detection on NEPTUNE — the GuruRMM agent's scheduled VSS pass, running `powershell -ExecutionPolicy
|
||||
Bypass` shadow-copy ops, matched the ransomware TTP and Falcon blocked it. This drove: an audit of all
|
||||
agent shell-outs (Grok + agy reviewed), an adversarial pass (agy) that corrected the plan, a
|
||||
`vss-native-com` spec, and actual implementation — native VSS provision (validated on NEPTUNE) and a
|
||||
hand-rolled `IVssBackupComponents` create/delete (built from the SDK vsbackup.h, compiled + create
|
||||
runtime-validated). Runtime testing then revealed the actual T1490 trigger was the PRUNE/DELETE (all
|
||||
explicit deletes are Falcon-blocked; cap-driven eviction rotates shadows fine), prompting the pivot:
|
||||
drop the scheduled prune, configure the OS-native shadow-copy system instead. A second spec,
|
||||
`vss-policy-config`, captures that; multi-AI (Grok + Gemini) confirmed a uniform native-COM-create
|
||||
approach works on all SKUs including workstations, and added the `MaxShadowCopies` count governor.
|
||||
Task 1 of that spec (the blocking spike) was completed on GURU-5070 (Win11 Pro client): MaxShadowCopies
|
||||
FIFO-evicts (does not block), and native COM create works on the client SKU.
|
||||
|
||||
Also rewired the `agy` skill from the dead Google `gemini` npm CLI (which failed with
|
||||
`throwIneligibleOrProjectIdError`) to the new Antigravity `agy` binary, and repointed grok's xsearch
|
||||
fallback to it.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **PST-DC-NW rename before domain join.** Renaming a promoted DC is messy (DNS/SPN/DFSR cleanup), so
|
||||
the rename ran first, on the workgroup box, via RMM.
|
||||
- **PST-SERVER2 got a full AD metadata cleanup, not just power-off.** It died past cases where you can
|
||||
resume replication; leaving stale DC metadata is what caused the June tombstone mess. Removed via
|
||||
direct AD object deletion (Remove-DfsrMember fails on a dead member — "network path not found").
|
||||
- **Data recovery: identify carved files by content, not by name.** The crash detached filenames from
|
||||
data blocks; carved output has bracket names. Fingerprinted sharedStrings + exact byte-size +
|
||||
in-workbook date serials to positively match. Did NOT place recovered files into the live HIPAA
|
||||
share — emailed to Mara for her to verify.
|
||||
- **VSS: use native VSS COM API `IVssBackupComponents`, NOT WMI** (adversarial-pass finding) — WMI
|
||||
`Win32_ShadowCopy` still runs in wmiprvse.exe and trips T1490; only the native requestor reads as
|
||||
legitimate backup software.
|
||||
- **Concurrency: dedicated COM-owning OS thread + MPSC + timeout, NOT tokio::spawn_blocking** — a hung
|
||||
COM/WMI call on the shared blocking pool would deadlock the agent.
|
||||
- **THE PIVOT: agent = configurator, not operator.** Runtime testing proved the T1490 trigger is the
|
||||
DELETE/prune (Falcon blocks ALL explicit shadow deletes), while cap-driven kernel eviction rotates
|
||||
shadows fine. So drop the scheduled prune (retention = cap), configure the OS-native create schedule,
|
||||
and the agent just sets policy + reports status. This removed most of the create/delete-internalization
|
||||
work from the hot path.
|
||||
- **Workstations: uniform native-COM create on all SKUs** (multi-AI validated). `vssadmin create shadow`
|
||||
is Server-only, but `IVssBackupComponents::DoSnapshotSet` and volsnap cap-eviction are NOT SKU-gated —
|
||||
only the CLI is. The scheduled-task action is the signed agent's native create on every SKU.
|
||||
- **Two retention governors, both FIFO, zero deletes:** shadow-storage MaxSize (native COM) +
|
||||
MaxShadowCopies count (registry). Grok caught the second one; Task 1 confirmed it FIFO-evicts.
|
||||
- **Do NOT ship the Falcon IOA exclusion** (`powershell.exe`+`gururmm_*.ps1`) — adversarial review
|
||||
flagged it as an RCE backdoor (SystemTemp is world-writable-ish). Stopgap = pause the VSS schedule.
|
||||
- **Remote build/verify loop:** this box (GURU-5070) can't `cargo check` the agent (no MSVC build
|
||||
tools); push branch to internal Gitea over HTTP, `cargo check`/`build` on the Pluto build host.
|
||||
- **Binary transfer to RMM-managed boxes: use the RMM server's `/downloads` path** (NEPTUNE/GURU-5070
|
||||
reach `rmm.azcomputerguru.com/downloads` over HTTPS) — peer-to-peer (Pluto:8099) is network-segmented.
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **Falcon T1490 blocked the agent's VSS pass on NEPTUNE.** Root cause = the temp-ps1 + ExecutionPolicy
|
||||
Bypass shadow-op pattern (ransomware TTP). Immediate: disabled the `GuruRMM-VSS-Snapshot` scheduled
|
||||
task (stopgap). Real fix = the redesign.
|
||||
- **`IVssBackupComponents` is absent from windows-rs 0.58** (its header vsbackup.h is outside
|
||||
win32metadata). Resolved by hand-declaring the interface: pulled the exact 40-slot vtable order + IID
|
||||
+ method signatures from the SDK `vsbackup.h` on Pluto, built it with `windows_core::imp::define_interface!`
|
||||
+ a manual vtable (usize placeholders for the 33 unused slots) + `vssapi.dll` FFI.
|
||||
- **Link error: `VssFreeSnapshotProperties` not in vssapi.lib** (LNK2019). Resolved by freeing the
|
||||
VSS_SNAPSHOT_PROP string members directly via `CoTaskMemFree` (what the function does internally).
|
||||
- **`AddDiffArea`/`ChangeDiffAreaMaximumSize` take `*const u16`, not `PCWSTR`** — caught by the Pluto
|
||||
cargo check (a check I can't run locally). Fixed.
|
||||
- **RMM inline-JSON dispatch mangles backslash payloads** (recurring) — used the ps-encoded script-file
|
||||
path and forward slashes throughout.
|
||||
- **Native delete "succeeded" but didn't remove the shadow.** Investigation: the shadow was a normal
|
||||
ClientAccessible shadow, and vssadmin + WMI `.Delete()` ALSO failed — because Falcon blocks all
|
||||
explicit shadow deletes (T1490). This was the pivotal finding driving the configurator redesign.
|
||||
- **agy/Gemini CLI dead** (`throwIneligibleOrProjectIdError`) — the old npm gemini CLI needs a
|
||||
GOOGLE_CLOUD_PROJECT the account can't supply. Resolved by rewiring the skill to the Antigravity
|
||||
`agy` binary (own auth, no project ID).
|
||||
- **Gitea push over the Cloudflare https remote fails auth** — pushed to the internal HTTP Gitea
|
||||
(`http://172.16.3.20:3000`) with URL-encoded vault creds (password has URL-breaking chars).
|
||||
- **One transient native-create slowness on GURU-5070** (>150s once, then <75s) — not a systematic
|
||||
hang; the scheduled create action needs a sane timeout/retry.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
Peaceful Spirit / infra (via RMM, on PST-SERVER `87293069-...` and PST-DC-NW `f60e9820-...`):
|
||||
- PST-SERVER01 renamed to PST-DC-NW; domain-joined PEACEFULSPIRIT.local; promoted DC/GC/DNS (site NW);
|
||||
static 192.168.1.5; timezone AZ. PST-SERVER2 AD/DNS/DFSR metadata removed. PST-DFS re-added with
|
||||
PST-DC-NW C:\Shares receiver. DSRM password vaulted.
|
||||
- `GuruRMM-VSS-Snapshot` scheduled task DISABLED on NEPTUNE (`b3a9b454-...`) — T1490 stopgap.
|
||||
|
||||
Repo (guru-rmm submodule, branch `feat/vss-native-com`):
|
||||
- `agent/src/vss_com.rs` (new) — native COM VSS worker: dedicated COM thread + MPSC; `provision` (cap),
|
||||
`create`/`delete` (hand-rolled IVssBackupComponents).
|
||||
- `agent/src/vss.rs` — provision/create/delete branched native(modern)/PS(legacy); added top-level
|
||||
`provision_storage` wrapper.
|
||||
- `agent/src/main.rs` — added `mod vss_com`; hidden diagnostic verbs `vss-provision-test`,
|
||||
`vss-roundtrip-test`.
|
||||
- `agent/Cargo.toml` — +windows features `Win32_Storage_Vss`, `Win32_System_Com`; +`windows-core` dep.
|
||||
- `specs/vss-native-com/` (4 files) — the internalization spec (superseded-in-part by the pivot).
|
||||
- `specs/vss-policy-config/` (4 files) — the configurator redesign spec (Task 1 DONE).
|
||||
- `docs/RMM_THOUGHTS.md` — internalize-VSS entry, shell-out audit, adversarial-pass corrections, the
|
||||
configurator PIVOT.
|
||||
- `specs/crowdstrike-falcon/{plan,references}.md` — T1490 rollout dependency (pause schedule, not exclusion).
|
||||
|
||||
Repo (ClaudeTools main):
|
||||
- `.claude/skills/agy/scripts/ask-agy.sh` (new) — Antigravity CLI wrapper; `ask-gemini.sh` now a shim.
|
||||
- `.claude/skills/agy/SKILL.md` — rewired to agy.
|
||||
- `.claude/skills/grok/scripts/ask-grok.sh` — xsearch fallback repointed gemini->agy.
|
||||
- `.claude/identity.json` (local, gitignored) — added `agy` block, retired `gemini`.
|
||||
- `wiki/clients/peaceful-spirit.md` — extensive updates (PST-DC-NW, metadata cleanup, 2025 crash &
|
||||
corruption section, recovery results).
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- **PST-DC-NW DSRM password** — created + vaulted at `clients/peaceful-spirit/dc-nw` (SOPS vault).
|
||||
- **PEACEFULSPIRIT\sysadmin (DA)** — read from vault `clients/peaceful-spirit/server` for the
|
||||
join/promotion; used via RMM Invoke-Command (FQDN). Not rotated (RMM internal).
|
||||
- **GURU-5070 finding (NOT vaulted — flag for follow-up):** `~/.gemini/antigravity-cli/settings.json`
|
||||
contains a live 1Password service-account token (`OP_SERVICE_ACCOUNT_TOKEN=ops_eyJ...`) in its
|
||||
`permissions.allow` list, in plaintext. Should be vaulted + scrubbed. (The adjacent
|
||||
`GEMINI_API_KEY=AIzaSyDummyKey...` is a dummy.)
|
||||
- Gitea push creds: vault `services/gitea.sops.yaml` (credentials.username/password); push via internal
|
||||
HTTP `172.16.3.20:3000`.
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **PST-DC-NW** (Peaceful Spirit NW DC) — 192.168.1.5, Win Server 2019, RMM agent `f60e9820-4a00-4598-83f7-c14085db5768`, site "North West".
|
||||
- **PST-SERVER** (CC DC) — 192.168.0.2, RMM agent `87293069-33b6-45e8-a68f-6811216cdb96`. D: = old crashed server's drive (recovery corpus; FROZEN, no cleanup). G:\Shares = live data.
|
||||
- **PST-SERVER2** — DEAD (~2026-06-14), metadata cleaned 2026-07-03.
|
||||
- **NEPTUNE** (ACG internal) — 172.16.3.11 / WAN 67.206.163.122, Win Server 2022, RMM agent `b3a9b454-86eb-491c-ac67-c1f98987d8dc`, CrowdStrike Falcon present. Runs cbb.exe (MSP360, native VSS backup). Leftover test shadow `{8FEFDAE3-A002-439F-B362-92FC4B9CCDAD}` (Falcon-blocked delete; will rotate).
|
||||
- **GURU-5070** (this box) — Win11 Pro (ProductType=1), RMM agent `819df0c8-4824-4424-b55a-2c5cb4d6ca39`, site "Mike's Car", Falcon present. Leftover test shadow `{87AFEA99-3336-4F85-A9D9-0E3F2F00D51B}`.
|
||||
- **Pluto** build host — `Administrator@172.16.3.36` (SSH key), C:\gururmm checkout, has MSVC + Rust (stable + 1.77 legacy) + Windows SDK. Used for `cargo check`/`build`.
|
||||
- **RMM server** — `guru@172.16.3.30` (Ubuntu 22.04, SSH key), serves `rmm.azcomputerguru.com/downloads` from `/var/www/gururmm/downloads`. Used to stage test binaries for RMM-managed boxes.
|
||||
- **Internal Gitea** — `http://172.16.3.20:3000` (HTTP, not the Cloudflare https remote).
|
||||
- **B2 (Backblaze)** — 15 buckets; generic `MSPBackups20200311` holds other clients only (no pre-crash Peaceful Spirit data); ACG-PST plan created on crash day (no pre-incident copy).
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
- Falcon efficacy (Datto EDR): 500 detections / 90 days fleet-wide = ZERO T1490/shadow detections from native backup binaries.
|
||||
- MaxShadowCopies FIFO test (GURU-5070): `MaxShadowCopies=3`, WMI Create -> oldest evicted, count stayed 3. `[RESULT] FIFO-EVICT`.
|
||||
- Native COM create on client (GURU-5070): `[OK] created shadow_id={87AFEA99...} device=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7`.
|
||||
- Native provision runtime (NEPTUNE): `ChangeDiffAreaMaximumSize` moved C: cap 279GB(15%)->298GB(16%), restored.
|
||||
- Pluto build: STABLE_RC=0, LEGACY_RC=0 (both variants); Task 4 fresh binary sha256 `1368eba8...`.
|
||||
- Push pattern: `git push http://<user>:<urlenc-pw>@172.16.3.20:3000/azcomputerguru/gururmm.git feat/vss-native-com`.
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **vss-policy-config Task 2+** (unblocked): build the `vss-create` verb + scheduled-task registration
|
||||
(uniform native create, all SKUs) + set both governors (cap via COM, MaxShadowCopies via registry) +
|
||||
every-N-hours schedule support; retire the legacy `GuruRMM-VSS-Snapshot` task on migration; drop the
|
||||
scheduled prune; update status/compliance. Then build both variants on Pluto + runtime-test on a
|
||||
Server AND a Win10/11 Pro box; merge (fleet migration).
|
||||
- **Vault the leaked 1Password token** in GURU-5070's antigravity settings.json + scrub it.
|
||||
- **Leftover test shadows:** `{8FEFDAE3}` (NEPTUNE), `{87AFEA99}` (GURU-5070) — Falcon-blocked delete;
|
||||
will rotate via governors. Optional cleanup.
|
||||
- **Peaceful Spirit open items** (in the wiki): PST-DC-NW DFS-R initial ~265 GB sync -> then Gate 4
|
||||
(share C:\Shares, add folder target + 2nd namespace root); deletion recovery ~3,342 genuine files
|
||||
(awaiting Mike/Mara go); 5th corrupted payroll file `IC Payments 5-13 to 5-27-25` unmatched.
|
||||
- **EV code-signing + AV-vendor whitelisting** — prerequisite the adversarial review flagged for any
|
||||
on-demand delete on Falcon hosts.
|
||||
- Parked agent shell-out internalizations: `users.rs` (T1098, native NetUser*), `inventory.rs`.
|
||||
|
||||
## Reference Information
|
||||
|
||||
- Branch: `feat/vss-native-com` (guru-rmm submodule). Key commits: `35491a4` (vss-policy-config spec),
|
||||
`d04a2d8` (Task 1 done), `2314ad55` (agy rewire), `07c8dc0` (every-N-hours).
|
||||
- IVssBackupComponents IID: `665c1d5f-c218-414d-a05d-7fef5f9d5c86`. VSS SW provider: `b5946137-7b9f-4925-af80-51abd60b20d5`. CLSID_VssSnapshotMgmt: `0b5a2c52-3eb9-470a-96e2-6c6d4570e40f`.
|
||||
- MaxShadowCopies: `HKLM\SYSTEM\CurrentControlSet\Services\VSS\Settings\MaxShadowCopies` (DWORD, default 64, max 512).
|
||||
- SDK header (Pluto): `C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\um\vsbackup.h`.
|
||||
- Recovered payroll files emailed to Mara (info@bestmassageintucson.com) from mike@azcomputerguru.com.
|
||||
- Specs: `projects/msp-tools/guru-rmm/specs/{vss-native-com,vss-policy-config}/`.
|
||||
Reference in New Issue
Block a user