claudetools/clients/valleywide/session-logs/2026-07/2026-07-07-mike-vwp-qbs-network-isolation.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

VWP-QBS (physical Dell PowerEdge R640, QuickBooks + RDS host) came back from an iDRAC force-reboot with completely broken networking. The OS was up but every off-subnet ping — gateway, 8.8.8.8, and even the server's own IP — returned PING: transmit failed. General failure., while 127.0.0.1 pinged fine, the box still pulled a DHCP lease (172.16.9.163), and DNS resolved. Prior remediation on the box (netsh int ip reset, adapter remove/re-add, manual IP) had not helped.

Diagnosis proceeded simplest-first. Loopback working ruled out a dead stack; own-IP failing while loopback worked pointed at the WFP/filter layer rather than the NIC or routing (an own-IP ping loops back above the miniport). Firewall fully off did not restore it, which eliminated the Windows Firewall (MpsSvc) and implicated a third-party WFP provider. BFE and the firewall service were confirmed running. netsh wfp show filters / show state revealed the culprit: an Infocyte blocking traffic block-all filter plus a set of Infocyte Allow Inbound to <IP> allow filters (Infocyte/AWS/Google/Cloudflare cloud IPs and DHCP *:68) — i.e. the host was stuck in Datto EDR / Infocyte network isolation (containment). The EDR had been removed via its own uninstaller, but a vendor bug left the isolation WFP filters behind. This explained every symptom: *:68 allowed → DHCP worked; everything else blocked → general failure; loopback on a different WFP path → survived.

The fix required deleting the orphaned WFP filters. netsh cannot delete WFP filters, and the box had no network, so WFPExplorer (portable, x64) was packaged into an ISO on GURU-5070 and mounted on VWP-QBS via iDRAC virtual media. Run elevated, its Filters view was used to delete all Infocyte filters. Connectivity returned the instant Infocyte blocking traffic was removed. The Infocyte sub-layer/provider would not delete (boot-time pinned) and were left in place — harmless once their filters are gone.

Post-fix verification confirmed VWP-QBS back on its correct address 172.16.9.169 (ping replies; AD DNS on DC1 and the UDM both resolve vwp-qbs.vwp.us → .169; the transient DHCP .163 released). The Valleywide wiki was updated with the incident, a history row, and the now-verified DRAC IP. Work was billed 3.0h remote to Syncro ticket #32507 (invoice #67994, prepaid block 29.5 → 26.5). A new CLAUDE.md core rule ("simplest solution first") was added after an earlier over-engineered UDM DNS lookup.

Key Decisions

• Diagnosed WFP layer via the own-IP-vs-loopback split. Own-IP "General failure" while 127.0.0.1 works isolates the fault to the interface-bound WFP filter path, not the NIC, route, or core stack — this drove the whole diagnosis and avoided chasing the QLogic driver.
• Firewall-off as the decisive bisect. Turning the Windows Firewall fully off and still seeing the failure proved the block was a third-party WFP provider, not MpsSvc.
• Delivered WFPExplorer via iDRAC virtual media ISO. The box had no network, so an ISO mounted through the DRAC was the only way to get a deletion tool onto the isolated host.
• Left the Infocyte sub-layer/provider in place. They are boot-time pinned and refused deletion; with their filters gone they are inert, so surgery stopped at the filters.
• Billed as regular remote, not emergency. Mike specified "remote"; regular debits 3.0 prepaid hrs vs 4.5 for emergency. Kept regular per his word.

Problems Encountered

• "General failure" on all off-subnet traffic incl. own IP → root cause was orphaned Datto EDR / Infocyte WFP isolation filters surviving the EDR uninstall. Removed the Infocyte blocking traffic filter (and the Infocyte allow filters) with WFPExplorer.
• No native way to delete WFP filters (netsh has no delete; advfirewall reset only touches MpsSvc filters) → used WFPExplorer.
• Isolated box, no network, for tool delivery → packaged WFPExplorer into an ISO and mounted via iDRAC virtual media.
• Sub-layer/provider would not delete (boot-time pinned) → left in place; harmless.
• Self-inflicted: wrote scratch to /tmp inside the Syncro invoice block → the block-tmp-path PreToolUse hook blocked the entire Bash call, so no invoice ran; re-ran without the /tmp write. Logged --friction (ref feedback_tmp_path_windows). No duplicate created because the hook blocks pre-execution.
• Earlier over-engineering: a UDM DNS lookup escalated to plink/vault/host-key work → Mike corrected it; added the "simplest solution first" CLAUDE.md rule + feedback memory.

Configuration Changes

• VWP-QBS (172.16.9.169): deleted orphaned Infocyte WFP filters (Infocyte blocking traffic + Infocyte Allow Inbound to <IP> set) via WFPExplorer. Windows Firewall re-enabled (netsh advfirewall set allprofiles state on). Winsock/int-ip/advfirewall were reset during diagnosis. Datto RMM agent (v4.4.11915) still installed — removal not decided.
• .claude/CLAUDE.md — added core "Simplest solution FIRST" rule to Key rules.
• .claude/memory/feedback_simplest_solution_first.md — new feedback memory + MEMORY.md index line.
• wiki/clients/valleywide.md — new Security Posture incident (2026-07-07 Infocyte isolation), history row, fixed the DRAC-IP flag (now 192.168.3.189), bumped last_compiled + covered range to 2026-07-07.
• errorlog.md — three entries this session: two --correction (over-engineered UDM lookup; wrong "EDR not cleanly uninstalled" assumption) and one --friction (/tmp in the Syncro block).
• Scratchpad: make-wfp-iso.ps1 (IMAPI2 ISO builder) — session scratch, not committed.

Credentials & Secrets

No new credentials created or discovered this session. Referenced existing vault entries:

• clients/vwp/quickbooks-server-idrac — VWP-QBS iDRAC root (192.168.3.189).
• clients/vwp/udm — VWP UDM root (172.16.9.1); password rejected on a plink attempt this session (not needed after pivoting to a local nslookup against the UDM).

Infrastructure & Servers

• VWP-QBS — physical Dell PowerEdge R640, Service Tag C84TTQ2. OS IP 172.16.9.169 (vwp-qbs.vwp.us), NIC1 = QLogic BCM57800 10GbE (ifIndex 9, MAC D0-94-66-01-13-9F). iDRAC 192.168.3.189 (VLAN 99 mgmt). Server 2022. QuickBooks + RDS license host.
• DNS: AD DNS on VWP-DC1 (172.16.9.2) and the UDM (172.16.9.1) both resolve vwp-qbs.vwp.us → 172.16.9.169. Transient DHCP address during the outage was 172.16.9.163 (now released).
• UDM — 172.16.9.1 (gateway/DNS). VPN pool 192.168.4.0/24.
• iDRAC/BMC Redfish must be queried from a Server 2019 box (DC1) via curl — Server 2025 strict TLS cannot negotiate the BMC's older ciphers.

Commands & Outputs

# Signature of orphaned EDR isolation:
ping 127.0.0.1          -> Reply           (core stack OK)
ping 172.16.9.169       -> General failure (own IP, via WFP interface layer)
netsh advfirewall set allprofiles state off; ping 172.16.9.169 -> still General failure
                        (rules out MpsSvc; third-party WFP provider)

# Confirm:
netsh wfp show filters file=C:\filters.xml
netsh wfp show state   file=C:\wfp.xml
Select-String C:\wfp.xml -Pattern 'Infocyte'
  -> <name>Infocyte blocking traffic</name>
  -> <name>Infocyte Allow Inbound to 54.187.170.202:*</name>  (and AWS/Google/CF + *:68 DHCP)

# Fix: WFPExplorer (elevated) -> Filters -> delete all Infocyte filters. Reboot. Firewall on.

# Verify address:
ping 172.16.9.169                 -> Reply TTL 127
nslookup vwp-qbs.vwp.us 172.16.9.2 -> 172.16.9.169   (AD DNS)
nslookup vwp-qbs.vwp.us 172.16.9.1 -> 172.16.9.169   (UDM)

ISO build (GURU-5070): IMAPI2 MsftFileSystemImage via make-wfp-iso.ps1 → C:\Users\guru\Downloads\WFPExp.iso (2,695,168 bytes), mounted on VWP-QBS via iDRAC virtual media.

Pending / Incomplete Tasks

• Datto RMM agent (v4.4.11915) still installed on VWP-QBS — decide whether to remove it too (it is a separate agent from the removed EDR; likely innocent, but if pulling Datto off the box, remove it as well).
• Confirm VWP-QBS NIC config — static .169 vs DHCP reservation; the netsh resets + adapter re-add can wipe a static config. netsh int ip show config "NIC1".
• Datto EDR uninstall bug — the uninstaller left isolation WFP filters behind; note the pattern for any other Datto EDR removals on the fleet.

Reference Information

• Syncro ticket #32507 (id 113550350) — "Emergency - ESXi Host Thermal Shutdown - Server Room Cooling Failure"; billed 3.0h remote, comment id 422566623, line id 43168141.
• Syncro invoice #67994 (id 1650970956), total $0.00 (prepaid); VWP block 29.5 → 26.5.
• Customer: Valley Wide Plastering Inc (VWP), Syncro id 31694734.
• WFPExplorer (Pavel Yosifovich) — used to delete WFP filters; source repo needs build, or use the prebuilt release exe (WFPExp.exe).
• Reusable signature: own-IP "General failure" + loopback and DHCP working = orphaned host isolation WFP filter from a removed/broken EDR (Infocyte/Datto, SentinelOne, CrowdStrike, Defender for Endpoint). Confirm with netsh wfp show filters; remove with WFPExplorer.
• Related earlier log: 2026-07-07-mike-vwp-infra-esxi-qbs.md (ESXi license recovery + QBS force-reboot that preceded this).