From f356b034735871cb79528c3a9c0b0029f050540e Mon Sep 17 00:00:00 2001 From: Howard Enos Date: Mon, 20 Jul 2026 14:23:16 -0700 Subject: [PATCH] sync: auto-sync from HOWARD-HOME at 2026-07-20 14:22:49 Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-07-20 14:22:49 --- ...0-howard-maintenance-pc-icons-homedepot.md | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/clients/cascades/session-logs/2026-07/2026-07-20-howard-maintenance-pc-icons-homedepot.md b/clients/cascades/session-logs/2026-07/2026-07-20-howard-maintenance-pc-icons-homedepot.md index ee457ba5..4cedf0ec 100644 --- a/clients/cascades/session-logs/2026-07/2026-07-20-howard-maintenance-pc-icons-homedepot.md +++ b/clients/cascades/session-logs/2026-07/2026-07-20-howard-maintenance-pc-icons-homedepot.md @@ -49,3 +49,90 @@ Investigated two issues on MAINTENANCE-PC at Cascades of Tucson, reported by the - Akamai error reference: #18.aef7d517.1784559975.f84fc2a - RMM commands dispatched: 85fc1844 (reboot/update diag), e5d3cab3 (network diag), 5ec44cf1 (Chrome extension enum) + +--- + +## Update: 14:21 PT -- BSOD investigation, AV cleanup, account fixes + +### Session Summary + +Continued MAINTENANCE-PC investigation after the initial icons + Home Depot work. GuruRMM had detected a kernel crash: `DRIVER_IRQL_NOT_LESS_OR_EQUAL (0xd1)` with dump `C:\Windows\Minidump\071926-13531-01.dmp`. The BSOD at 4:57 PM on 7/19 was the Event 6008 "unexpected shutdown" that triggered the reboot cascade and icon rearrangement. + +Investigated the BSOD and found 4 crashes in 3 months (two 0xd1, two 0xef CRITICAL_PROCESS_DIED). The two 0xd1 crashes shared an identical fault offset (`7ab2`), confirming the same driver and code path. The machine had Malwarebytes (full install, v5.6.1.257) running alongside Datto AV + Windows Defender. Initially misidentified several kernel drivers signed by "Avira Operations GmbH" (rtp1, rtp2, rtp_elam, BdNet, BdSentry, netprotection_network_filter, netprotection_network_filter2) as orphan drivers from a previous Avira/Bitdefender install. Disabled and deleted them from System32\drivers via Safe Mode. This broke Datto AV -- those drivers are part of the Bitdefender Endpoint Protection SDK that Datto AV uses under the hood. The driver source files exist in `C:\Program Files\infocyte\agent\dattoav\Endpoint Protection SDK\drivers\`. + +Restored the drivers from the Datto AV SDK backup directory and re-enabled services, then reinstalled Datto EDR over the existing installation using the deploy command (`Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey 6qw68y2rwl`). Final verification showed HUNTAgent running, all Datto AV drivers loaded (rtp1, rtp2, BdNet, BdSentry running), and Datto AV registered in WMI. Malwarebytes fully removed (the actual BSOD root cause -- its kernel drivers conflicting with Datto AV's Bitdefender drivers at DISPATCH_LEVEL IRQL). + +Also fixed several account issues: looked up John Trozzi's AD account on CS-SERVER (never logged in, password never set), discovered "John Trozzi" local account was disabled since 2023, set localadmin password to `r3tr0gradE99#`, disabled casadmin201, unhid localadmin from the login screen (was hidden via `SpecialAccounts\UserList` registry key = 0). Vaulted localadmin credentials. + +### Key Decisions + +- The BSOD root cause was Malwarebytes kernel drivers (MbamChameleon.sys, mbamswissarmy.sys, mbae.sys + network filter driver oem26.inf) conflicting with Datto AV's Bitdefender SDK drivers at DISPATCH_LEVEL IRQL in the NDIS network stack. Removing Malwarebytes was the correct fix. +- Initially misidentified Datto AV's Bitdefender-signed drivers as orphan Avira leftovers. CRITICAL LESSON: Datto AV uses the Bitdefender Endpoint Protection SDK; its drivers are signed by "Avira Operations GmbH" (same Gen Digital corporate family). Always check `C:\Program Files\infocyte\agent\dattoav\Endpoint Protection SDK\drivers\` before assuming BD/Avira drivers are orphans. +- Rebooted the machine into Safe Mode (`safeboot minimal`) to bypass driver tamper protection -- but MAINTENANCE-PC is Wi-Fi only (TP-Link USB adapter on Wi-Fi 2), so Safe Mode killed all network connectivity and the RMM agent couldn't reach back. Required an on-site visit. Should have checked the network adapter type before choosing Safe Mode. +- Reinstalled Datto EDR over the existing broken installation rather than attempting a full uninstall (which requires a console-issued uninstall token). The over-top install repaired the broken driver state. + +### Problems Encountered + +- **Avira/BD driver tamper protection:** Even as SYSTEM with SeTakeOwnershipPrivilege, could not modify the rtp1/rtp2/rtp_elam registry keys -- Avira's tamper protection locked the ACLs at the kernel level while the drivers were loaded. Required Safe Mode to bypass. +- **Safe Mode on Wi-Fi-only device:** `bcdedit /set safeboot minimal` killed connectivity (Wi-Fi drivers don't load in Safe Mode). Had to drive to Cascades to run the cleanup commands locally. Logged as friction in errorlog.md. +- **Misidentified Datto AV drivers as orphans:** Disabled and deleted rtp1/rtp2/rtp_elam/BdNet/BdSentry/netprotection_network_filter* thinking they were leftover Avira drivers. They belonged to Datto AV. Restored from the SDK backup directory and reinstalled EDR. Logged as correction in errorlog.md and saved as feedback memory. +- **Windows Startup Repair triggered:** After the Safe Mode driver deletion, Windows detected the missing drivers and ran Startup Repair on next boot. Completed normally. +- **Datto EDR uninstall requires token:** `agent.exe --uninstall` returned "Uninstall token not approved." Worked around by installing over the existing broken installation. +- **UAC prompt defaulting to casadmin201:** The localadmin account was hidden via `SpecialAccounts\UserList` registry key (value 0). Set to 1 to make it visible on the login screen. + +### Configuration Changes + +- Malwarebytes v5.6.1.257 fully uninstalled (Howard did GUI uninstall on-site) +- localadmin password set to `r3tr0gradE99#` on MAINTENANCE-PC +- casadmin201 account disabled on MAINTENANCE-PC +- localadmin unhidden from login screen (SpecialAccounts\UserList = 1) +- Datto AV drivers restored from SDK directory to System32\drivers +- Datto EDR reinstalled over existing installation +- Cleanup script `remove-avira-drivers.bat` placed on localadmin desktop (can be deleted) +- Accidental registry key `HKLM\SYSTEM\CurrentControlSet\Services\rtp` deleted +- Vault entry created: `clients/cascades-tucson/maintenance-pc-localadmin.sops.yaml` +- Memory file created: `feedback_datto_av_bitdefender_drivers.md` + +### Credentials & Secrets + +- **MAINTENANCE-PC localadmin:** password `r3tr0gradE99#` (same as CS-SERVER sysadmin) + - Vault: `clients/cascades-tucson/maintenance-pc-localadmin.sops.yaml` +- **John Trozzi AD:** `John.Trozzi@cascadestucson.com`, OU=Maintenance, password never set, never logged in +- **Datto EDR deploy key (Cascades):** RegKey `6qw68y2rwl`, group `1dbd2b02-f7df-45d0-a7f2-18667f48447f` + +### Infrastructure & Servers + +- **MAINTENANCE-PC** (Cascades of Tucson) + - GuruRMM agent: `2fbda0d2-efd7-4389-9ac5-3f89db467c1f` + - Datto EDR agent: `9047be1a-b38c-47df-ad99-55e730a84580` + - Datto EDR org: `2d5ea96e-3228-461b-9c60-13ae464b61d8` (Cascades of Tucson) + - Model: Lenovo F0EW005TUS (IdeaCentre), BIOS O4SKT25A (2020-12-20) + - Network: Wi-Fi only (TP-Link USB adapter, Realtek driver 1030.52.1216.2025), IP 10.0.20.96 + - Also has built-in Realtek 8822CE Wi-Fi (not primary) and Realtek PCIe GbE (no cable) + - Local accounts: localadmin (admin, active), Bruce Miller (active), casadmin201 (disabled), John Trozzi (disabled), nick (disabled) +- **CS-SERVER** (Cascades of Tucson, GuruRMM agent: `c39f1de7-d5b6-45ae-b132-e06977ab1713`) -- used for AD lookups + +### Commands & Outputs + +- BSOD history: 4 crashes in 3 months (0xd1 on 7/19 + 5/26, 0xef on 5/30 + 5/1) +- Malwarebytes uninstall: GUI uninstall by Howard on-site +- Safe Mode cleanup (on-site): `reg add` for rtp1/rtp2/rtp_elam Start=4, `del` for .sys files +- EDR reinstall: `Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey 6qw68y2rwl` +- Discord DM sent to Howard with Safe Mode cleanup commands (message_id 1528815475566444685) + +### Pending / Incomplete Tasks + +- Delete `remove-avira-drivers.bat` from localadmin desktop (cleanup script no longer needed) +- Monitor for BSODs over the next few weeks -- with Malwarebytes gone, the driver conflict should be resolved +- Datto RMM service (CagService) was shown as Stopped/Disabled earlier -- may need investigation (this is the OLD Datto RMM, not GuruRMM) +- EndpointProtectionService (Datto AV engine) was Stopped after reinstall -- may need a reboot to fully start +- Defender RTP is off (expected with Datto AV active, but verify Datto AV is actually scanning) +- John Trozzi AD account has never been used (password never set) -- determine if he needs domain login or if local-only is intentional + +### Reference Information + +- Datto AV driver source: `C:\Program Files\infocyte\agent\dattoav\Endpoint Protection SDK\drivers\x64\` +- Minidump: `C:\Windows\Minidump\071926-13531-01.dmp` (0xd1 BSOD) +- BSOD report IDs: 1eeb0df0 (7/19), ee4b925b (5/26), dc5325cd (5/30), 2cebdc31 (5/1) +- RMM commands: a87eb58e (BSOD diag), e13065a8 (security inventory), 962f9b8f (desktop script), 7aa2be12 (EDR reinstall), 3e5925d5 (final verification) +- Errorlog entries: Safe Mode Wi-Fi friction, Datto AV driver misidentification correction