sync: auto-sync from GURU-5070 at 2026-07-07 10:26:45

Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-07 10:26:45
2026-07-07 10:27:17 -07:00
parent ff6546bee8
commit 766c7fa54d
6 changed files with 207 additions and 3 deletions
--- a/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-qbs-network-isolation.md
+++ b/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-qbs-network-isolation.md
@@ -0,0 +1,158 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+VWP-QBS (physical Dell PowerEdge R640, QuickBooks + RDS host) came back from an iDRAC
+force-reboot with completely broken networking. The OS was up but every off-subnet ping —
+gateway, 8.8.8.8, and even the server's own IP — returned `PING: transmit failed. General
+failure.`, while `127.0.0.1` pinged fine, the box still pulled a DHCP lease (172.16.9.163),
+and DNS resolved. Prior remediation on the box (netsh int ip reset, adapter remove/re-add,
+manual IP) had not helped.
+
+Diagnosis proceeded simplest-first. Loopback working ruled out a dead stack; own-IP failing
+while loopback worked pointed at the WFP/filter layer rather than the NIC or routing (an
+own-IP ping loops back above the miniport). Firewall fully off did not restore it, which
+eliminated the Windows Firewall (MpsSvc) and implicated a third-party WFP provider. BFE and
+the firewall service were confirmed running. `netsh wfp show filters` / `show state` revealed
+the culprit: an **`Infocyte blocking traffic`** block-all filter plus a set of `Infocyte Allow
+Inbound to <IP>` allow filters (Infocyte/AWS/Google/Cloudflare cloud IPs and DHCP `*:68`) —
+i.e. the host was stuck in **Datto EDR / Infocyte network isolation (containment)**. The EDR
+had been removed via its own uninstaller, but a vendor bug left the isolation WFP filters
+behind. This explained every symptom: `*:68` allowed → DHCP worked; everything else blocked →
+general failure; loopback on a different WFP path → survived.
+
+The fix required deleting the orphaned WFP filters. netsh cannot delete WFP filters, and the
+box had no network, so WFPExplorer (portable, x64) was packaged into an ISO on GURU-5070 and
+mounted on VWP-QBS via iDRAC virtual media. Run elevated, its Filters view was used to delete
+all `Infocyte` filters. Connectivity returned the instant `Infocyte blocking traffic` was
+removed. The Infocyte sub-layer/provider would not delete (boot-time pinned) and were left in
+place — harmless once their filters are gone.
+
+Post-fix verification confirmed VWP-QBS back on its correct address **172.16.9.169** (ping
+replies; AD DNS on DC1 and the UDM both resolve `vwp-qbs.vwp.us` → .169; the transient DHCP
+.163 released). The Valleywide wiki was updated with the incident, a history row, and the
+now-verified DRAC IP. Work was billed 3.0h remote to Syncro ticket #32507 (invoice #67994,
+prepaid block 29.5 → 26.5). A new CLAUDE.md core rule ("simplest solution first") was added
+after an earlier over-engineered UDM DNS lookup.
+
+## Key Decisions
+
+- **Diagnosed WFP layer via the own-IP-vs-loopback split.** Own-IP "General failure" while
+  127.0.0.1 works isolates the fault to the interface-bound WFP filter path, not the NIC,
+  route, or core stack — this drove the whole diagnosis and avoided chasing the QLogic driver.
+- **Firewall-off as the decisive bisect.** Turning the Windows Firewall fully off and still
+  seeing the failure proved the block was a third-party WFP provider, not MpsSvc.
+- **Delivered WFPExplorer via iDRAC virtual media ISO.** The box had no network, so an ISO
+  mounted through the DRAC was the only way to get a deletion tool onto the isolated host.
+- **Left the Infocyte sub-layer/provider in place.** They are boot-time pinned and refused
+  deletion; with their filters gone they are inert, so surgery stopped at the filters.
+- **Billed as regular remote, not emergency.** Mike specified "remote"; regular debits 3.0
+  prepaid hrs vs 4.5 for emergency. Kept regular per his word.
+
+## Problems Encountered
+
+- **"General failure" on all off-subnet traffic incl. own IP** → root cause was orphaned
+  Datto EDR / Infocyte WFP isolation filters surviving the EDR uninstall. Removed the
+  `Infocyte blocking traffic` filter (and the Infocyte allow filters) with WFPExplorer.
+- **No native way to delete WFP filters** (netsh has no delete; `advfirewall reset` only
+  touches MpsSvc filters) → used WFPExplorer.
+- **Isolated box, no network, for tool delivery** → packaged WFPExplorer into an ISO and
+  mounted via iDRAC virtual media.
+- **Sub-layer/provider would not delete** (boot-time pinned) → left in place; harmless.
+- **Self-inflicted: wrote scratch to `/tmp` inside the Syncro invoice block** → the
+  `block-tmp-path` PreToolUse hook blocked the entire Bash call, so no invoice ran; re-ran
+  without the `/tmp` write. Logged `--friction` (ref `feedback_tmp_path_windows`). No duplicate
+  created because the hook blocks pre-execution.
+- **Earlier over-engineering: a UDM DNS lookup escalated to plink/vault/host-key work** →
+  Mike corrected it; added the "simplest solution first" CLAUDE.md rule + feedback memory.
+
+## Configuration Changes
+
+- **VWP-QBS (172.16.9.169):** deleted orphaned Infocyte WFP filters (`Infocyte blocking
+  traffic` + `Infocyte Allow Inbound to <IP>` set) via WFPExplorer. Windows Firewall
+  re-enabled (`netsh advfirewall set allprofiles state on`). Winsock/int-ip/advfirewall were
+  reset during diagnosis. Datto RMM agent (v4.4.11915) still installed — removal not decided.
+- **`.claude/CLAUDE.md`** — added core "Simplest solution FIRST" rule to Key rules.
+- **`.claude/memory/feedback_simplest_solution_first.md`** — new feedback memory + MEMORY.md
+  index line.
+- **`wiki/clients/valleywide.md`** — new Security Posture incident (2026-07-07 Infocyte
+  isolation), history row, fixed the DRAC-IP flag (now 192.168.3.189), bumped last_compiled +
+  covered range to 2026-07-07.
+- **`errorlog.md`** — three entries this session: two `--correction` (over-engineered UDM
+  lookup; wrong "EDR not cleanly uninstalled" assumption) and one `--friction` (`/tmp` in the
+  Syncro block).
+- **Scratchpad:** `make-wfp-iso.ps1` (IMAPI2 ISO builder) — session scratch, not committed.
+
+## Credentials & Secrets
+
+No new credentials created or discovered this session. Referenced existing vault entries:
+- `clients/vwp/quickbooks-server-idrac` — VWP-QBS iDRAC root (192.168.3.189).
+- `clients/vwp/udm` — VWP UDM root (172.16.9.1); password rejected on a plink attempt this
+  session (not needed after pivoting to a local `nslookup` against the UDM).
+
+## Infrastructure & Servers
+
+- **VWP-QBS** — physical Dell PowerEdge R640, Service Tag **C84TTQ2**. OS IP **172.16.9.169**
+  (`vwp-qbs.vwp.us`), NIC1 = QLogic BCM57800 10GbE (ifIndex 9, MAC D0-94-66-01-13-9F).
+  **iDRAC 192.168.3.189** (VLAN 99 mgmt). Server 2022. QuickBooks + RDS license host.
+- **DNS:** AD DNS on VWP-DC1 (172.16.9.2) and the UDM (172.16.9.1) both resolve
+  `vwp-qbs.vwp.us` → 172.16.9.169. Transient DHCP address during the outage was 172.16.9.163
+  (now released).
+- **UDM** — 172.16.9.1 (gateway/DNS). VPN pool 192.168.4.0/24.
+- iDRAC/BMC Redfish must be queried from a Server 2019 box (DC1) via curl — Server 2025 strict
+  TLS cannot negotiate the BMC's older ciphers.
+
+## Commands & Outputs
+
+```
+# Signature of orphaned EDR isolation:
+ping 127.0.0.1          -> Reply           (core stack OK)
+ping 172.16.9.169       -> General failure (own IP, via WFP interface layer)
+netsh advfirewall set allprofiles state off; ping 172.16.9.169 -> still General failure
+                        (rules out MpsSvc; third-party WFP provider)
+
+# Confirm:
+netsh wfp show filters file=C:\filters.xml
+netsh wfp show state   file=C:\wfp.xml
+Select-String C:\wfp.xml -Pattern 'Infocyte'
+  -> <name>Infocyte blocking traffic</name>
+  -> <name>Infocyte Allow Inbound to 54.187.170.202:*</name>  (and AWS/Google/CF + *:68 DHCP)
+
+# Fix: WFPExplorer (elevated) -> Filters -> delete all Infocyte filters. Reboot. Firewall on.
+
+# Verify address:
+ping 172.16.9.169                 -> Reply TTL 127
+nslookup vwp-qbs.vwp.us 172.16.9.2 -> 172.16.9.169   (AD DNS)
+nslookup vwp-qbs.vwp.us 172.16.9.1 -> 172.16.9.169   (UDM)
+```
+
+ISO build (GURU-5070): IMAPI2 `MsftFileSystemImage` via `make-wfp-iso.ps1` →
+`C:\Users\guru\Downloads\WFPExp.iso` (2,695,168 bytes), mounted on VWP-QBS via iDRAC virtual
+media.
+
+## Pending / Incomplete Tasks
+
+- **Datto RMM agent (v4.4.11915) still installed on VWP-QBS** — decide whether to remove it
+  too (it is a separate agent from the removed EDR; likely innocent, but if pulling Datto off
+  the box, remove it as well).
+- **Confirm VWP-QBS NIC config** — static .169 vs DHCP reservation; the netsh resets +
+  adapter re-add can wipe a static config. `netsh int ip show config "NIC1"`.
+- **Datto EDR uninstall bug** — the uninstaller left isolation WFP filters behind; note the
+  pattern for any other Datto EDR removals on the fleet.
+
+## Reference Information
+
+- Syncro ticket **#32507** (id 113550350) — "Emergency - ESXi Host Thermal Shutdown - Server
+  Room Cooling Failure"; billed 3.0h remote, comment id 422566623, line id 43168141.
+- Syncro **invoice #67994** (id 1650970956), total $0.00 (prepaid); VWP block 29.5 → 26.5.
+- Customer: Valley Wide Plastering Inc (VWP), Syncro id 31694734.
+- WFPExplorer (Pavel Yosifovich) — used to delete WFP filters; source repo needs build, or use
+  the prebuilt release exe (`WFPExp.exe`).
+- Reusable signature: **own-IP "General failure" + loopback and DHCP working = orphaned host
+  isolation WFP filter from a removed/broken EDR** (Infocyte/Datto, SentinelOne, CrowdStrike,
+  Defender for Endpoint). Confirm with `netsh wfp show filters`; remove with WFPExplorer.
+- Related earlier log: `2026-07-07-mike-vwp-infra-esxi-qbs.md` (ESXi license recovery + QBS
+  force-reboot that preceded this).