diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index d75c76ee..99dd545d 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -223,3 +223,4 @@ - [Backup targets never guessed](feedback_backup_targets_never_guessed.md) — billed-vs-running mismatches are billing questions for Mike/Winter; never infer/deploy backup targets - [Fire #dev-alerts for client/RMM work](feedback_fire_dev_alerts_for_client_work.md) — Howard+Mike watch #dev-alerts Discord for live visibility into what techs/sessions touch. Fire post-bot-alert.sh ([DEV]/[RMM]/[DEPLOY]/[GURURMM] prefix -> #dev-alerts) at the START of any RMM/Dev or active-client-machine action. No CLAUDE.md rule yet — this memory is the rule. - [M365 app: SharePoint needs CERT](reference_m365_app_sharepoint_rest_vs_graph.md) — SP app-only works via the CERT (get-token.sh sharepoint); the secret gives "Unsupported app only token". Graph app-only uses the secret. +- [EDR auto-isolates GuruRMM PowerShell](project_edr_rmm_autoisolation_fp.md) — Datto "Exfiltration Over HTTP" rule auto-isolates benign GuruRMM scripts fleet-wide (vwp-qbs 4h outage); watcher + narrow suppression deployed, policy fix pending Mike diff --git a/.claude/memory/project_edr_rmm_autoisolation_fp.md b/.claude/memory/project_edr_rmm_autoisolation_fp.md new file mode 100644 index 00000000..b94edf1a --- /dev/null +++ b/.claude/memory/project_edr_rmm_autoisolation_fp.md @@ -0,0 +1,24 @@ +--- +name: project_edr_rmm_autoisolation_fp +description: Datto EDR "Exfiltration Over HTTP" rule auto-isolates benign GuruRMM PowerShell fleet-wide; watcher + narrow suppression deployed, policy fix pending Mike +metadata: + type: project +--- + +Incident 2026-07-07: **vwp-qbs** (Valley Wide Plastering QuickBooks server) was auto-isolated by Datto EDR — Mike lost ~4h before realizing EDR did it. No notification fired (EDR alert delivery was never wired to any channel). + +**Root cause:** Datto EDR rule **"Exfiltration Over HTTP Protocol" (MITRE T1041, HIGH)** carries an automated response of **`kill-process` + `isolate-host`**. It fires on *signed* PowerShell doing an outbound HTTPS POST — which our **GuruRMM automation** does routinely. Process tree on the alerts: `gururmm-agent.exe -> cmd.exe -> powershell.exe`. Confirmed benign both times (vwp-qbs = a vSphere VM-power-on script; tps-tina = a network/connectivity diagnostic). + +**This is SYSTEMIC, not a one-off.** The same rule fired + attempted auto-isolate on **tps-tina** (The Prairie Schooner) the same day — different GuruRMM script, different client. Per-command-line suppression is whack-a-mole. Note: the isolate-host response only actually cuts a host off where its endpoint policy has isolation enabled (vwp-qbs got isolated; tps-tina fired the same alert but `extensionSuccess=None`, stayed online). + +**What's deployed (done):** +- Narrow suppression rule (Datto EDR, id `e4dd55bf`) for VWP's exact script: matches Process Command Line + Grand Parent = `gururmm-agent.exe`. Does NOT blind other PowerShell. API schema for suppression is now in `.claude/skills/datto-edr/references/api-reference.md`. +- **EDR isolation watcher**: `.claude/scripts/edr-isolation-watch.sh` + `register-edr-watcher.ps1`, registered as Windows Scheduled Task **"ClaudeTools - EDR Isolation Watcher"** on Howard-Home (runs every 10 min, posts new auto-isolations to #dev-alerts — see [[feedback_fire_dev_alerts_for_client_work]]). Plain script, zero tokens. State file `.edr-watch-state.json` (gitignored) dedupes; seeded with the vwp-qbs + tps-tina event ids so it stays silent for the already-triaged ones. Retire when GuruRMM EDR webhook (Feature 6) lands. + +**Pending Mike decisions (do NOT act without him):** +1. Auto-isolate vs **alert-only** on behavioral rules — the real fix (the isolation, not the alert, caused the outage). +2. Whether to allow a small curated "GuruRMM-origin + this-rule OK" exception set — but NOT a blanket `gururmm-agent.exe` whitelist (RMM = top MSP attack path; blinding EDR there is dangerous). +3. **Fix the RMM scripts** to stop looking like exfil (drop `-EncodedCommand`, least-priv accounts). +4. Hygiene: VWP's script had **ESXi root creds hardcoded** (base64, cert-validation disabled) — already in vault `clients/vwp/esxi.sops.yaml`; move to a least-priv service account + vault injection. + +See [[reference_datto_edr_detection_behavior]] (behavioral rules DO fire on signed PowerShell, unlike reputation rules). diff --git a/.claude/memory/reference_datto_edr_detection_behavior.md b/.claude/memory/reference_datto_edr_detection_behavior.md index e8c0b558..a75add06 100644 --- a/.claude/memory/reference_datto_edr_detection_behavior.md +++ b/.claude/memory/reference_datto_edr_detection_behavior.md @@ -14,6 +14,8 @@ Verified live 2026-06-25/26 on RMM-TEST-MACHINE (EDR agent `b98b3ba0-...`, group **Datto EDR is reputation-based, NOT structural.** A synthetic "looks suspicious" artifact (Run-key/scheduled task launching hidden `-EncodedCommand` powershell) is collected by the forensic scan but scored BENIGN → no alert (powershell.exe is signed/clean). To get an EDR detection you need a real reputation hit: wire a known-bad file as the **executable of an autostart** (Run-key/scheduled task) so the survey collects + hashes it. EICAR-as-autostart works → high-sev `rule` alert. A loose file on disk is NOT scanned by the EDR forensic survey (it only walks execution/persistence artifacts). +**BUT behavioral rules DO fire on signed PowerShell** (unlike the reputation scoring above). Rules like **"Exfiltration Over HTTP Protocol" (T1041)** key on *runtime behavior* (outbound HTTP from powershell), not file reputation, so they alert HIGH on clean signed `powershell.exe` — and can carry an automated `isolate-host`/`kill-process` response. This routinely false-positives on GuruRMM automation (`gururmm-agent.exe -> cmd.exe -> powershell`). See [[project_edr_rmm_autoisolation_fp]]. + **AV-suppression gotchas (to isolate EDR on an endpoint):** - Datto AV is **tamper-protected**: `Stop-Service EndpointProtectionService2 -Force` is refused ("cannot be stopped"); can't disable from the endpoint. Supported path = console policy (AV disabled / path-exclusion) — console-only, like policy assignment. - Disabling Datto AV in the policy **uninstalls** the AV component on the box (services `EndpointProtectionService`/`...2` go absent; `HUNTAgent` EDR stays). Platform `dattoAvEnabled` flips to False at the console first; the on-box apply lags a few minutes. diff --git a/.claude/scripts/edr-isolation-watch.sh b/.claude/scripts/edr-isolation-watch.sh index d2c8d460..2a28078c 100644 --- a/.claude/scripts/edr-isolation-watch.sh +++ b/.claude/scripts/edr-isolation-watch.sh @@ -17,7 +17,7 @@ set -uo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" EDR="bash $ROOT/.claude/scripts/py.sh $ROOT/.claude/skills/datto-edr/scripts/edr.py" DM="bash $ROOT/.claude/scripts/discord-dm.sh" -export STATE="$ROOT/.claude/scripts/.edr-watch-state.json" +export STATE="${STATE:-$ROOT/.claude/scripts/.edr-watch-state.json}" TARGET="dev" # #dev-alerts = Mike + Howard, private # Pull last 24h of detections (list includes responseData) and emit any NEW @@ -61,6 +61,10 @@ while IFS='|' read -r aid host org sev rule ts; do - Time: $ts - This machine was cut off the network by an EDR automated response. Verify it is intended before restoring. - Console: https://azcomp4587.infocyte.com (alert id: $aid)" + if [ "${DRY_RUN:-0}" = "1" ]; then + printf '[DRY_RUN] would post to #%s:\n%s\n\n' "$TARGET" "$msg" + continue + fi if ! printf '%s' "$msg" | $DM "$TARGET" 2>/dev/null; then bash "$ROOT/.claude/scripts/log-skill-error.sh" "edr-isolation-watch" "discord post failed for $host ($aid)" 2>/dev/null || true fi diff --git a/.claude/scripts/register-edr-watcher.ps1 b/.claude/scripts/register-edr-watcher.ps1 new file mode 100644 index 00000000..24d068aa --- /dev/null +++ b/.claude/scripts/register-edr-watcher.ps1 @@ -0,0 +1,45 @@ +# register-edr-watcher.ps1 +# Register the "ClaudeTools - EDR Isolation Watcher" scheduled task on this Windows +# machine (the same box that runs GPS-RMM-AutoEnroll et al). The task runs +# edr-isolation-watch.sh every 10 minutes, which polls Datto EDR for hosts auto- +# isolated by an EDR response and posts each NEW one to #dev-alerts (Mike + Howard). +# +# Interim alerting until the GuruRMM EDR webhook integration (Feature 6) lands. +# Idempotent: -Force replaces any existing task with the same name. +# +# Run from an ordinary (non-admin) PowerShell: +# powershell -ExecutionPolicy Bypass -File C:\claudetools\.claude\scripts\register-edr-watcher.ps1 + +$ErrorActionPreference = "Stop" +$TaskName = "ClaudeTools - EDR Isolation Watcher" + +# Bash launcher (mirrors the GPS-RMM-AutoEnroll task action) +$BashExe = "C:\Program Files\Git\bin\bash.exe" +if (-not (Test-Path $BashExe)) { Write-Host "[ERROR] Git bash not found at $BashExe" -ForegroundColor Red; exit 1 } + +$ScriptPosix = "/c/claudetools/.claude/scripts/edr-isolation-watch.sh" +$ScriptWin = "C:\claudetools\.claude\scripts\edr-isolation-watch.sh" +if (-not (Test-Path $ScriptWin)) { Write-Host "[ERROR] Watcher not found at $ScriptWin" -ForegroundColor Red; exit 1 } + +$Action = New-ScheduledTaskAction -Execute $BashExe -Argument "-lc $ScriptPosix" + +# Every 10 minutes, indefinitely (10-year duration avoids the MaxValue quirk). +$Trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).Date ` + -RepetitionInterval (New-TimeSpan -Minutes 10) ` + -RepetitionDuration (New-TimeSpan -Days 3650) + +# Run as the current user, only when logged on (needs the interactive vault/env, +# same posture as the GrepAI watcher). No admin required. +$Principal = New-ScheduledTaskPrincipal -UserId "$env:USERDOMAIN\$env:USERNAME" -LogonType Interactive -RunLevel Limited + +$Settings = New-ScheduledTaskSettingsSet ` + -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries ` + -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 5) ` + -MultipleInstances IgnoreNew + +Register-ScheduledTask -TaskName $TaskName -Action $Action -Trigger $Trigger ` + -Principal $Principal -Settings $Settings ` + -Description "Polls Datto EDR every 10 min for auto-isolated hosts; posts new ones to #dev-alerts. Interim until GuruRMM EDR webhook (Feature 6)." -Force | Out-Null + +Write-Host "[OK] Registered scheduled task: $TaskName (every 10 min)" -ForegroundColor Green +Get-ScheduledTask -TaskName $TaskName | Select-Object TaskName, State | Format-Table -AutoSize