From 766c7fa54d0ad10fca415a368f4818066321b984 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Tue, 7 Jul 2026 10:27:17 -0700 Subject: [PATCH] sync: auto-sync from GURU-5070 at 2026-07-07 10:26:45 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-07 10:26:45 --- .claude/CLAUDE.md | 6 + .claude/memory/MEMORY.md | 1 + .../feedback_simplest_solution_first.md | 20 +++ ...26-07-07-mike-vwp-qbs-network-isolation.md | 158 ++++++++++++++++++ errorlog.md | 6 + wiki/clients/valleywide.md | 19 ++- 6 files changed, 207 insertions(+), 3 deletions(-) create mode 100644 .claude/memory/feedback_simplest_solution_first.md create mode 100644 clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-qbs-network-isolation.md diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 3a47b327..230df219 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -30,6 +30,12 @@ failures, production risk). Bump one tier for: security, auth, credential, migra production, data-loss. Detail: EXTENDED + `.claude/OLLAMA.md`. ## Key rules (always) +- **Simplest solution FIRST.** Start with the least complex thing that could answer the + problem, confirm it works, and only THEN graduate to more advanced/complicated approaches + as the simple one proves insufficient. Don't reach for SSH/plink/multi-hop tooling when a + one-shot query (a single DNS lookup, one API call, one command) would answer it. E.g. "what + IP does the UDM think X is?" = `nslookup X `, not shelling into the device. Escalate + deliberately, not by default. - **NO EMOJIS.** Use ASCII markers: `[OK]` `[ERROR]` `[WARNING]` `[INFO]` `[CRITICAL]`. - **Skill-first — if a skill/command covers the task, USE IT; never hand-roll the API.** When a request maps to an installed skill or slash-command, INVOKE THAT SKILL instead of diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index c6300815..d75c76ee 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -67,6 +67,7 @@ - [Mike — font preference](user_font_preference.md) — Mike prefers Lucida Console for monospace UI. ## Feedback +- [Simplest solution first](feedback_simplest_solution_first.md) — Start with the least-complex thing that could answer the problem (one DNS lookup / one API call / one command), confirm it works, THEN graduate to advanced/complicated as needed. Don't reach for SSH/plink/multi-hop when a one-shot query answers it. Now a CLAUDE.md core rule. - [Report times in Arizona time](feedback_timezone_arizona_reporting.md) — All user-facing times in America/Phoenix (MST, no DST), labeled AZ; convert from UTC. Set by Winter 2026-07-02. - [RMM dashboard: beta before main](rmm-dashboard-beta-before-main.md) — GuruRMM website/dashboard changes land on beta (rmm-beta.azcomputerguru.com) and get tested BEFORE pushing/merging to main, unless told otherwise. Pipeline auto-builds beta FROM main, so don't merge to get on beta — build the feature branch's dashboard and rsync to /var/www/gururmm/dashboard-beta via a git worktree. Promote beta→prod with promote-dashboard.sh --confirm. - [Prefer SSH over RMM](feedback_prefer_ssh_over_rmm.md) — When a target has SSH (key auth) and the task is easier over it, default to `scp script + ssh run` (system OpenSSH); RMM runs as SYSTEM + hits the server-side timeout reaper. Reserve RMM as fallback when SSH/VPN is down. diff --git a/.claude/memory/feedback_simplest_solution_first.md b/.claude/memory/feedback_simplest_solution_first.md new file mode 100644 index 00000000..d6c0c297 --- /dev/null +++ b/.claude/memory/feedback_simplest_solution_first.md @@ -0,0 +1,20 @@ +--- +name: feedback_simplest_solution_first +description: Always try the simplest solution to a problem first; escalate to complex only as needed +metadata: + type: feedback +--- + +Always START with the simplest solution that could solve the problem. Confirm it works, then +graduate to more advanced/complicated approaches only as the simple one proves insufficient. + +**Why:** Mike watched a "ask the UDM what IP it thinks VWP-QBS is" turn balloon into vault +reads, plink host-key pinning, and password sanity checks — when the clean answer was a single +`nslookup vwp-qbs.vwp.us 172.16.9.1` from a machine already on the VPN. The heavy path burned +tokens and his patience ("incapable of doing things cleanly like a week ago"). Now a standing +CLAUDE.md core rule. + +**How to apply:** Before picking tooling, ask "what is the least-complex thing that answers +this?" — one DNS lookup, one API call, one command. Do that first and observe the result. +Only reach for SSH/plink/double-hops/multi-step orchestration when the simple probe genuinely +can't answer. Escalate deliberately, not by default. See [[feedback_verify_live_before_acting]]. diff --git a/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-qbs-network-isolation.md b/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-qbs-network-isolation.md new file mode 100644 index 00000000..6be75ccf --- /dev/null +++ b/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-qbs-network-isolation.md @@ -0,0 +1,158 @@ +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Session Summary + +VWP-QBS (physical Dell PowerEdge R640, QuickBooks + RDS host) came back from an iDRAC +force-reboot with completely broken networking. The OS was up but every off-subnet ping — +gateway, 8.8.8.8, and even the server's own IP — returned `PING: transmit failed. General +failure.`, while `127.0.0.1` pinged fine, the box still pulled a DHCP lease (172.16.9.163), +and DNS resolved. Prior remediation on the box (netsh int ip reset, adapter remove/re-add, +manual IP) had not helped. + +Diagnosis proceeded simplest-first. Loopback working ruled out a dead stack; own-IP failing +while loopback worked pointed at the WFP/filter layer rather than the NIC or routing (an +own-IP ping loops back above the miniport). Firewall fully off did not restore it, which +eliminated the Windows Firewall (MpsSvc) and implicated a third-party WFP provider. BFE and +the firewall service were confirmed running. `netsh wfp show filters` / `show state` revealed +the culprit: an **`Infocyte blocking traffic`** block-all filter plus a set of `Infocyte Allow +Inbound to ` allow filters (Infocyte/AWS/Google/Cloudflare cloud IPs and DHCP `*:68`) — +i.e. the host was stuck in **Datto EDR / Infocyte network isolation (containment)**. The EDR +had been removed via its own uninstaller, but a vendor bug left the isolation WFP filters +behind. This explained every symptom: `*:68` allowed → DHCP worked; everything else blocked → +general failure; loopback on a different WFP path → survived. + +The fix required deleting the orphaned WFP filters. netsh cannot delete WFP filters, and the +box had no network, so WFPExplorer (portable, x64) was packaged into an ISO on GURU-5070 and +mounted on VWP-QBS via iDRAC virtual media. Run elevated, its Filters view was used to delete +all `Infocyte` filters. Connectivity returned the instant `Infocyte blocking traffic` was +removed. The Infocyte sub-layer/provider would not delete (boot-time pinned) and were left in +place — harmless once their filters are gone. + +Post-fix verification confirmed VWP-QBS back on its correct address **172.16.9.169** (ping +replies; AD DNS on DC1 and the UDM both resolve `vwp-qbs.vwp.us` → .169; the transient DHCP +.163 released). The Valleywide wiki was updated with the incident, a history row, and the +now-verified DRAC IP. Work was billed 3.0h remote to Syncro ticket #32507 (invoice #67994, +prepaid block 29.5 → 26.5). A new CLAUDE.md core rule ("simplest solution first") was added +after an earlier over-engineered UDM DNS lookup. + +## Key Decisions + +- **Diagnosed WFP layer via the own-IP-vs-loopback split.** Own-IP "General failure" while + 127.0.0.1 works isolates the fault to the interface-bound WFP filter path, not the NIC, + route, or core stack — this drove the whole diagnosis and avoided chasing the QLogic driver. +- **Firewall-off as the decisive bisect.** Turning the Windows Firewall fully off and still + seeing the failure proved the block was a third-party WFP provider, not MpsSvc. +- **Delivered WFPExplorer via iDRAC virtual media ISO.** The box had no network, so an ISO + mounted through the DRAC was the only way to get a deletion tool onto the isolated host. +- **Left the Infocyte sub-layer/provider in place.** They are boot-time pinned and refused + deletion; with their filters gone they are inert, so surgery stopped at the filters. +- **Billed as regular remote, not emergency.** Mike specified "remote"; regular debits 3.0 + prepaid hrs vs 4.5 for emergency. Kept regular per his word. + +## Problems Encountered + +- **"General failure" on all off-subnet traffic incl. own IP** → root cause was orphaned + Datto EDR / Infocyte WFP isolation filters surviving the EDR uninstall. Removed the + `Infocyte blocking traffic` filter (and the Infocyte allow filters) with WFPExplorer. +- **No native way to delete WFP filters** (netsh has no delete; `advfirewall reset` only + touches MpsSvc filters) → used WFPExplorer. +- **Isolated box, no network, for tool delivery** → packaged WFPExplorer into an ISO and + mounted via iDRAC virtual media. +- **Sub-layer/provider would not delete** (boot-time pinned) → left in place; harmless. +- **Self-inflicted: wrote scratch to `/tmp` inside the Syncro invoice block** → the + `block-tmp-path` PreToolUse hook blocked the entire Bash call, so no invoice ran; re-ran + without the `/tmp` write. Logged `--friction` (ref `feedback_tmp_path_windows`). No duplicate + created because the hook blocks pre-execution. +- **Earlier over-engineering: a UDM DNS lookup escalated to plink/vault/host-key work** → + Mike corrected it; added the "simplest solution first" CLAUDE.md rule + feedback memory. + +## Configuration Changes + +- **VWP-QBS (172.16.9.169):** deleted orphaned Infocyte WFP filters (`Infocyte blocking + traffic` + `Infocyte Allow Inbound to ` set) via WFPExplorer. Windows Firewall + re-enabled (`netsh advfirewall set allprofiles state on`). Winsock/int-ip/advfirewall were + reset during diagnosis. Datto RMM agent (v4.4.11915) still installed — removal not decided. +- **`.claude/CLAUDE.md`** — added core "Simplest solution FIRST" rule to Key rules. +- **`.claude/memory/feedback_simplest_solution_first.md`** — new feedback memory + MEMORY.md + index line. +- **`wiki/clients/valleywide.md`** — new Security Posture incident (2026-07-07 Infocyte + isolation), history row, fixed the DRAC-IP flag (now 192.168.3.189), bumped last_compiled + + covered range to 2026-07-07. +- **`errorlog.md`** — three entries this session: two `--correction` (over-engineered UDM + lookup; wrong "EDR not cleanly uninstalled" assumption) and one `--friction` (`/tmp` in the + Syncro block). +- **Scratchpad:** `make-wfp-iso.ps1` (IMAPI2 ISO builder) — session scratch, not committed. + +## Credentials & Secrets + +No new credentials created or discovered this session. Referenced existing vault entries: +- `clients/vwp/quickbooks-server-idrac` — VWP-QBS iDRAC root (192.168.3.189). +- `clients/vwp/udm` — VWP UDM root (172.16.9.1); password rejected on a plink attempt this + session (not needed after pivoting to a local `nslookup` against the UDM). + +## Infrastructure & Servers + +- **VWP-QBS** — physical Dell PowerEdge R640, Service Tag **C84TTQ2**. OS IP **172.16.9.169** + (`vwp-qbs.vwp.us`), NIC1 = QLogic BCM57800 10GbE (ifIndex 9, MAC D0-94-66-01-13-9F). + **iDRAC 192.168.3.189** (VLAN 99 mgmt). Server 2022. QuickBooks + RDS license host. +- **DNS:** AD DNS on VWP-DC1 (172.16.9.2) and the UDM (172.16.9.1) both resolve + `vwp-qbs.vwp.us` → 172.16.9.169. Transient DHCP address during the outage was 172.16.9.163 + (now released). +- **UDM** — 172.16.9.1 (gateway/DNS). VPN pool 192.168.4.0/24. +- iDRAC/BMC Redfish must be queried from a Server 2019 box (DC1) via curl — Server 2025 strict + TLS cannot negotiate the BMC's older ciphers. + +## Commands & Outputs + +``` +# Signature of orphaned EDR isolation: +ping 127.0.0.1 -> Reply (core stack OK) +ping 172.16.9.169 -> General failure (own IP, via WFP interface layer) +netsh advfirewall set allprofiles state off; ping 172.16.9.169 -> still General failure + (rules out MpsSvc; third-party WFP provider) + +# Confirm: +netsh wfp show filters file=C:\filters.xml +netsh wfp show state file=C:\wfp.xml +Select-String C:\wfp.xml -Pattern 'Infocyte' + -> Infocyte blocking traffic + -> Infocyte Allow Inbound to 54.187.170.202:* (and AWS/Google/CF + *:68 DHCP) + +# Fix: WFPExplorer (elevated) -> Filters -> delete all Infocyte filters. Reboot. Firewall on. + +# Verify address: +ping 172.16.9.169 -> Reply TTL 127 +nslookup vwp-qbs.vwp.us 172.16.9.2 -> 172.16.9.169 (AD DNS) +nslookup vwp-qbs.vwp.us 172.16.9.1 -> 172.16.9.169 (UDM) +``` + +ISO build (GURU-5070): IMAPI2 `MsftFileSystemImage` via `make-wfp-iso.ps1` → +`C:\Users\guru\Downloads\WFPExp.iso` (2,695,168 bytes), mounted on VWP-QBS via iDRAC virtual +media. + +## Pending / Incomplete Tasks + +- **Datto RMM agent (v4.4.11915) still installed on VWP-QBS** — decide whether to remove it + too (it is a separate agent from the removed EDR; likely innocent, but if pulling Datto off + the box, remove it as well). +- **Confirm VWP-QBS NIC config** — static .169 vs DHCP reservation; the netsh resets + + adapter re-add can wipe a static config. `netsh int ip show config "NIC1"`. +- **Datto EDR uninstall bug** — the uninstaller left isolation WFP filters behind; note the + pattern for any other Datto EDR removals on the fleet. + +## Reference Information + +- Syncro ticket **#32507** (id 113550350) — "Emergency - ESXi Host Thermal Shutdown - Server + Room Cooling Failure"; billed 3.0h remote, comment id 422566623, line id 43168141. +- Syncro **invoice #67994** (id 1650970956), total $0.00 (prepaid); VWP block 29.5 → 26.5. +- Customer: Valley Wide Plastering Inc (VWP), Syncro id 31694734. +- WFPExplorer (Pavel Yosifovich) — used to delete WFP filters; source repo needs build, or use + the prebuilt release exe (`WFPExp.exe`). +- Reusable signature: **own-IP "General failure" + loopback and DHCP working = orphaned host + isolation WFP filter from a removed/broken EDR** (Infocyte/Datto, SentinelOne, CrowdStrike, + Defender for Endpoint). Confirm with `netsh wfp show filters`; remove with WFPExplorer. +- Related earlier log: `2026-07-07-mike-vwp-infra-esxi-qbs.md` (ESXi license recovery + QBS + force-reboot that preceded this). diff --git a/errorlog.md b/errorlog.md index 57a403a7..8d6f4f4d 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-07 | GURU-5070 | bash/env | [friction] appended a scratch write to /tmp/_vwpinv inside the syncro invoice block; PreToolUse block-tmp-path hook blocked the ENTIRE Bash call so no invoice ran - had to re-run. Never write scratch to /tmp on Windows; use a repo-relative path or skip it [ctx: ref=feedback_tmp_path_windows] + +2026-07-07 | GURU-5070 | troubleshooting/assumption | [correction] repeatedly asserted Datto EDR was not cleanly uninstalled and pushed reinstall-to-uninstall; user had already removed it via its own uninstaller. A persistent WFP block filter can survive a clean uninstall (vendor bug) - frame it that way, don't blame the operator's removal + +2026-07-07 | GURU-5070 | approach/complexity | [correction] escalated straight to plink/vault/host-key pinning to query UDM DNS; simplest solution was a single nslookup against the UDM [ctx: ref=feedback_simplest_solution_first] + 2026-07-07 | GURU-BEAST-ROG | rmm/ilo-redfish | [friction] embedded double-quotes in curl.exe -d JSON body stripped by PowerShell native arg passing (MalformedJSON from iLO); fix = write body to temp file via [char]34, --data-binary @file [ctx: ref=feedback_windows_quote_stripping] 2026-07-07 | Howard-Home | rmm/ps-encoded | [friction] ps-encoded.sh fails 'iconv: command not found' on HOWARD-HOME Git Bash -> encoded dispatch unavailable; fell back to inline jq --arg dispatch (safe for quote-free payloads) [ctx: ref=reference_windows_edition_upgrade_rmm host=HOWARD-HOME] diff --git a/wiki/clients/valleywide.md b/wiki/clients/valleywide.md index aae50863..a26c458c 100644 --- a/wiki/clients/valleywide.md +++ b/wiki/clients/valleywide.md @@ -2,7 +2,7 @@ type: client name: valleywide display_name: Valley Wide Plastering -last_compiled: 2026-06-23 +last_compiled: 2026-07-07 compiled_by: GURU-5070/claude-main sources: - clients/valleywide/README.md @@ -271,6 +271,18 @@ RDWeb (`https://VWP-QBS/RDWeb/Pages/login.aspx`) was publicly exposed via UDM po Power outage caused HP ProLiant NVRAM corruption (BIOS/iLO factory reset). VWP-QBS Dell had a boot retry loop (resolved via DRAC). XenServer was offline. All recovered onsite. **Root cause: no UPS on HP server.** +### 2026-07-07: VWP-QBS Stuck in Orphaned EDR Network Isolation (~3 hr remote) + +After VWP-QBS was force-rebooted via iDRAC (192.168.3.189) following a hung OS, it came up but **all off-subnet networking failed** — `ping` to its own IP, the gateway, and 8.8.8.8 all returned **"PING: transmit failed. General failure."** Yet `127.0.0.1` pinged fine, it still pulled a DHCP lease (172.16.9.163), and DNS resolved. + +**Root cause:** the removed **Datto EDR (Infocyte engine)** left the host in **orphaned WFP network isolation / containment.** A persistent `Infocyte blocking traffic` WFP filter (block-all) plus an allowlist of Infocyte cloud IPs and DHCP (`*:68`) survived the EDR uninstall — which is why DHCP worked but nothing else did. The EDR was removed via its own uninstaller; a vendor bug left the isolation filters behind. `netsh advfirewall set allprofiles state off`, `netsh winsock reset` / `int ip reset` / `advfirewall reset`, adapter remove/re-add, and reboots all did nothing because the block lived under **Infocyte's own WFP provider/sub-layer, below the Windows Firewall (MpsSvc)**. + +**Diagnosis:** `netsh wfp show filters file=C:\filters.xml` / `show state` — grep for `Infocyte`, revealing `Infocyte blocking traffic` + the `Infocyte Allow Inbound to ` allowlist. + +**Fix:** ran **WFPExplorer** as admin (delivered to the isolated box as an ISO via **DRAC virtual media** — no network), **Filters** view → deleted all `Infocyte` filters. Connectivity restored the instant `Infocyte blocking traffic` was removed. The Infocyte **sub-layer/provider would not delete** (boot-time pinned) — harmless once its filters are gone. Verified back on **172.16.9.169** (AD DNS on DC1 + UDM both agree; stale .163 released). Firewall re-enabled. + +**Reusable signature (fleet-wide):** **`ping` to own IP = "General failure" while `127.0.0.1` works AND DHCP still works** → suspect an **orphaned host-isolation WFP filter from a removed/broken EDR** (Infocyte/Datto, SentinelOne, CrowdStrike, Defender for Endpoint all isolate this way). Firewall-off and stack resets won't clear it. Confirm with `netsh wfp show filters`; remove the vendor's filters with **WFPExplorer** (run elevated). **Note:** Datto RMM agent (v4.4.11915) is still installed on VWP-QBS — decide whether to pull it too. + --- ## History Highlights @@ -290,17 +302,18 @@ Power outage caused HP ProLiant NVRAM corruption (BIOS/iLO factory reset). VWP-Q | 2026-06-13 | **Syncro** and **Datto RMM Agent** deployment GPOs disabled (`AllSettingsDisabled`, flags=3) via LDAP on VWP_ADSRVR. Existing agents not yet uninstalled — awaiting direction. | | 2026-06-15 | **VWP-FILES scan-to-folder fix.** Copier scan-to-`\\192.168.0.20` broke after the 2026-06-13 cutover — root cause was the dual-homed server having a default gateway only on the 172.16.9.132 NIC, so replies on the .20 NIC to off-subnet clients were dropped (not a VLAN-routing limit; the UDM routes all VLANs). Fix: single-homed VWP-FILES on 192.168.0.20 (gw 192.168.0.1) by disconnecting the .132 vNIC host-side via `Disconnect-VMNetworkAdapter` on VWP-HYPERV1 (in-guest change dropped the RMM agent + auto-rolled-back). .132 vNIC left disconnected (reversible), not removed. Scanner = Brother MFC-L3780CDW (vault `clients/vwp/brother-mfc-l3780cdw`). | | 2026-06-23 | **SMB1 enabled on VWP-FILES** to restore G:/Orders access for the legacy Windows XP app VM (V-XP) after the 6/13 migration — Server 2019 defaults SMB1-off; XP speaks only SMB1. Diagnosed via GuruRMM (payroll desktop G: was fine; Orders runs on V-XP). Ticket #32448, 1.5 hr emergency block deduction (prepay 19.0→17.5 at close). | +| 2026-07-07 | **VWP-QBS stuck in orphaned Datto EDR / Infocyte network isolation** after an iDRAC force-reboot. All off-subnet traffic "General failure" (own IP too) while loopback + DHCP worked — a persistent `Infocyte blocking traffic` WFP filter survived the EDR uninstall. Cleared via WFPExplorer (delivered by DRAC virtual media ISO) → deleted the Infocyte filters; back on 172.16.9.169. **~3 hr remote.** See Security Posture 2026-07-07. | --- ## Compilation Notes -**Date range covered:** 2026-04-13 through 2026-06-23. +**Date range covered:** 2026-04-13 through 2026-07-07. **Items flagged [unverified]:** - M365 MFA and mail flow configuration — never investigated - HP iLO credentials post factory-reset — accessible 2026-05-12 so credentials were re-established; confirm vault entry -- DRAC IP for VWP-QBS — functional but undocumented +- DRAC IP for VWP-QBS — **documented 2026-07-07: iDRAC 192.168.3.189** (R640, Service Tag C84TTQ2); vault `clients/vwp/quickbooks-server-idrac`. Reach via curl from a 2019 box (DC1) — Server 2025 TLS can't negotiate the BMC's older ciphers. - Yealink provisioning status — 11 phones pending as of 2026-04-22; no follow-up confirmed - RDS CAL grace period — may have expired - AD replication of GPO `flags=3` changes to VWP-DC1 — ADWS not reachable over SSH from ADSRVR; normal replication expected but not spot-checked