## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

VWP-QBS (physical Dell PowerEdge R640, QuickBooks + RDS host) came back from an iDRAC
force-reboot with completely broken networking. The OS was up but every off-subnet ping —
gateway, 8.8.8.8, and even the server's own IP — returned `PING: transmit failed. General
failure.`, while `127.0.0.1` pinged fine, the box still pulled a DHCP lease (172.16.9.163),
and DNS resolved. Prior remediation on the box (netsh int ip reset, adapter remove/re-add,
manual IP) had not helped.

Diagnosis proceeded simplest-first. Loopback working ruled out a dead stack; own-IP failing
while loopback worked pointed at the WFP/filter layer rather than the NIC or routing (an
own-IP ping loops back above the miniport). Firewall fully off did not restore it, which
eliminated the Windows Firewall (MpsSvc) and implicated a third-party WFP provider. BFE and
the firewall service were confirmed running. `netsh wfp show filters` / `show state` revealed
the culprit: an **`Infocyte blocking traffic`** block-all filter plus a set of `Infocyte Allow
Inbound to <IP>` allow filters (Infocyte/AWS/Google/Cloudflare cloud IPs and DHCP `*:68`) —
i.e. the host was stuck in **Datto EDR / Infocyte network isolation (containment)**. The EDR
had been removed via its own uninstaller, but a vendor bug left the isolation WFP filters
behind. This explained every symptom: `*:68` allowed → DHCP worked; everything else blocked →
general failure; loopback on a different WFP path → survived.

The fix required deleting the orphaned WFP filters. netsh cannot delete WFP filters, and the
box had no network, so WFPExplorer (portable, x64) was packaged into an ISO on GURU-5070 and
mounted on VWP-QBS via iDRAC virtual media. Run elevated, its Filters view was used to delete
all `Infocyte` filters. Connectivity returned the instant `Infocyte blocking traffic` was
removed. The Infocyte sub-layer/provider would not delete (boot-time pinned) and were left in
place — harmless once their filters are gone.

Post-fix verification confirmed VWP-QBS back on its correct address **172.16.9.169** (ping
replies; AD DNS on DC1 and the UDM both resolve `vwp-qbs.vwp.us` → .169; the transient DHCP
.163 released). The Valleywide wiki was updated with the incident, a history row, and the
now-verified DRAC IP. Work was billed 3.0h remote to Syncro ticket #32507 (invoice #67994,
prepaid block 29.5 → 26.5). A new CLAUDE.md core rule ("simplest solution first") was added
after an earlier over-engineered UDM DNS lookup.

## Key Decisions

- **Diagnosed WFP layer via the own-IP-vs-loopback split.** Own-IP "General failure" while
  127.0.0.1 works isolates the fault to the interface-bound WFP filter path, not the NIC,
  route, or core stack — this drove the whole diagnosis and avoided chasing the QLogic driver.
- **Firewall-off as the decisive bisect.** Turning the Windows Firewall fully off and still
  seeing the failure proved the block was a third-party WFP provider, not MpsSvc.
- **Delivered WFPExplorer via iDRAC virtual media ISO.** The box had no network, so an ISO
  mounted through the DRAC was the only way to get a deletion tool onto the isolated host.
- **Left the Infocyte sub-layer/provider in place.** They are boot-time pinned and refused
  deletion; with their filters gone they are inert, so surgery stopped at the filters.
- **Billed as regular remote, not emergency.** Mike specified "remote"; regular debits 3.0
  prepaid hrs vs 4.5 for emergency. Kept regular per his word.

## Problems Encountered

- **"General failure" on all off-subnet traffic incl. own IP** → root cause was orphaned
  Datto EDR / Infocyte WFP isolation filters surviving the EDR uninstall. Removed the
  `Infocyte blocking traffic` filter (and the Infocyte allow filters) with WFPExplorer.
- **No native way to delete WFP filters** (netsh has no delete; `advfirewall reset` only
  touches MpsSvc filters) → used WFPExplorer.
- **Isolated box, no network, for tool delivery** → packaged WFPExplorer into an ISO and
  mounted via iDRAC virtual media.
- **Sub-layer/provider would not delete** (boot-time pinned) → left in place; harmless.
- **Self-inflicted: wrote scratch to `/tmp` inside the Syncro invoice block** → the
  `block-tmp-path` PreToolUse hook blocked the entire Bash call, so no invoice ran; re-ran
  without the `/tmp` write. Logged `--friction` (ref `feedback_tmp_path_windows`). No duplicate
  created because the hook blocks pre-execution.
- **Earlier over-engineering: a UDM DNS lookup escalated to plink/vault/host-key work** →
  Mike corrected it; added the "simplest solution first" CLAUDE.md rule + feedback memory.

## Configuration Changes

- **VWP-QBS (172.16.9.169):** deleted orphaned Infocyte WFP filters (`Infocyte blocking
  traffic` + `Infocyte Allow Inbound to <IP>` set) via WFPExplorer. Windows Firewall
  re-enabled (`netsh advfirewall set allprofiles state on`). Winsock/int-ip/advfirewall were
  reset during diagnosis. Datto RMM agent (v4.4.11915) still installed — removal not decided.
- **`.claude/CLAUDE.md`** — added core "Simplest solution FIRST" rule to Key rules.
- **`.claude/memory/feedback_simplest_solution_first.md`** — new feedback memory + MEMORY.md
  index line.
- **`wiki/clients/valleywide.md`** — new Security Posture incident (2026-07-07 Infocyte
  isolation), history row, fixed the DRAC-IP flag (now 192.168.3.189), bumped last_compiled +
  covered range to 2026-07-07.
- **`errorlog.md`** — three entries this session: two `--correction` (over-engineered UDM
  lookup; wrong "EDR not cleanly uninstalled" assumption) and one `--friction` (`/tmp` in the
  Syncro block).
- **Scratchpad:** `make-wfp-iso.ps1` (IMAPI2 ISO builder) — session scratch, not committed.

## Credentials & Secrets

No new credentials created or discovered this session. Referenced existing vault entries:
- `clients/vwp/quickbooks-server-idrac` — VWP-QBS iDRAC root (192.168.3.189).
- `clients/vwp/udm` — VWP UDM root (172.16.9.1); password rejected on a plink attempt this
  session (not needed after pivoting to a local `nslookup` against the UDM).

## Infrastructure & Servers

- **VWP-QBS** — physical Dell PowerEdge R640, Service Tag **C84TTQ2**. OS IP **172.16.9.169**
  (`vwp-qbs.vwp.us`), NIC1 = QLogic BCM57800 10GbE (ifIndex 9, MAC D0-94-66-01-13-9F).
  **iDRAC 192.168.3.189** (VLAN 99 mgmt). Server 2022. QuickBooks + RDS license host.
- **DNS:** AD DNS on VWP-DC1 (172.16.9.2) and the UDM (172.16.9.1) both resolve
  `vwp-qbs.vwp.us` → 172.16.9.169. Transient DHCP address during the outage was 172.16.9.163
  (now released).
- **UDM** — 172.16.9.1 (gateway/DNS). VPN pool 192.168.4.0/24.
- iDRAC/BMC Redfish must be queried from a Server 2019 box (DC1) via curl — Server 2025 strict
  TLS cannot negotiate the BMC's older ciphers.

## Commands & Outputs

```
# Signature of orphaned EDR isolation:
ping 127.0.0.1          -> Reply           (core stack OK)
ping 172.16.9.169       -> General failure (own IP, via WFP interface layer)
netsh advfirewall set allprofiles state off; ping 172.16.9.169 -> still General failure
                        (rules out MpsSvc; third-party WFP provider)

# Confirm:
netsh wfp show filters file=C:\filters.xml
netsh wfp show state   file=C:\wfp.xml
Select-String C:\wfp.xml -Pattern 'Infocyte'
  -> <name>Infocyte blocking traffic</name>
  -> <name>Infocyte Allow Inbound to 54.187.170.202:*</name>  (and AWS/Google/CF + *:68 DHCP)

# Fix: WFPExplorer (elevated) -> Filters -> delete all Infocyte filters. Reboot. Firewall on.

# Verify address:
ping 172.16.9.169                 -> Reply TTL 127
nslookup vwp-qbs.vwp.us 172.16.9.2 -> 172.16.9.169   (AD DNS)
nslookup vwp-qbs.vwp.us 172.16.9.1 -> 172.16.9.169   (UDM)
```

ISO build (GURU-5070): IMAPI2 `MsftFileSystemImage` via `make-wfp-iso.ps1` →
`C:\Users\guru\Downloads\WFPExp.iso` (2,695,168 bytes), mounted on VWP-QBS via iDRAC virtual
media.

## Pending / Incomplete Tasks

- **Datto RMM agent (v4.4.11915) still installed on VWP-QBS** — decide whether to remove it
  too (it is a separate agent from the removed EDR; likely innocent, but if pulling Datto off
  the box, remove it as well).
- **Confirm VWP-QBS NIC config** — static .169 vs DHCP reservation; the netsh resets +
  adapter re-add can wipe a static config. `netsh int ip show config "NIC1"`.
- **Datto EDR uninstall bug** — the uninstaller left isolation WFP filters behind; note the
  pattern for any other Datto EDR removals on the fleet.

## Reference Information

- Syncro ticket **#32507** (id 113550350) — "Emergency - ESXi Host Thermal Shutdown - Server
  Room Cooling Failure"; billed 3.0h remote, comment id 422566623, line id 43168141.
- Syncro **invoice #67994** (id 1650970956), total $0.00 (prepaid); VWP block 29.5 → 26.5.
- Customer: Valley Wide Plastering Inc (VWP), Syncro id 31694734.
- WFPExplorer (Pavel Yosifovich) — used to delete WFP filters; source repo needs build, or use
  the prebuilt release exe (`WFPExp.exe`).
- Reusable signature: **own-IP "General failure" + loopback and DHCP working = orphaned host
  isolation WFP filter from a removed/broken EDR** (Infocyte/Datto, SentinelOne, CrowdStrike,
  Defender for Endpoint). Confirm with `netsh wfp show filters`; remove with WFPExplorer.
- Related earlier log: `2026-07-07-mike-vwp-infra-esxi-qbs.md` (ESXi license recovery + QBS
  force-reboot that preceded this).
