sync: auto-sync from HOWARD-HOME at 2026-07-20 14:22:49
Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-07-20 14:22:49
This commit is contained in:
@@ -49,3 +49,90 @@ Investigated two issues on MAINTENANCE-PC at Cascades of Tucson, reported by the
|
||||
|
||||
- Akamai error reference: #18.aef7d517.1784559975.f84fc2a
|
||||
- RMM commands dispatched: 85fc1844 (reboot/update diag), e5d3cab3 (network diag), 5ec44cf1 (Chrome extension enum)
|
||||
|
||||
---
|
||||
|
||||
## Update: 14:21 PT -- BSOD investigation, AV cleanup, account fixes
|
||||
|
||||
### Session Summary
|
||||
|
||||
Continued MAINTENANCE-PC investigation after the initial icons + Home Depot work. GuruRMM had detected a kernel crash: `DRIVER_IRQL_NOT_LESS_OR_EQUAL (0xd1)` with dump `C:\Windows\Minidump\071926-13531-01.dmp`. The BSOD at 4:57 PM on 7/19 was the Event 6008 "unexpected shutdown" that triggered the reboot cascade and icon rearrangement.
|
||||
|
||||
Investigated the BSOD and found 4 crashes in 3 months (two 0xd1, two 0xef CRITICAL_PROCESS_DIED). The two 0xd1 crashes shared an identical fault offset (`7ab2`), confirming the same driver and code path. The machine had Malwarebytes (full install, v5.6.1.257) running alongside Datto AV + Windows Defender. Initially misidentified several kernel drivers signed by "Avira Operations GmbH" (rtp1, rtp2, rtp_elam, BdNet, BdSentry, netprotection_network_filter, netprotection_network_filter2) as orphan drivers from a previous Avira/Bitdefender install. Disabled and deleted them from System32\drivers via Safe Mode. This broke Datto AV -- those drivers are part of the Bitdefender Endpoint Protection SDK that Datto AV uses under the hood. The driver source files exist in `C:\Program Files\infocyte\agent\dattoav\Endpoint Protection SDK\drivers\`.
|
||||
|
||||
Restored the drivers from the Datto AV SDK backup directory and re-enabled services, then reinstalled Datto EDR over the existing installation using the deploy command (`Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey 6qw68y2rwl`). Final verification showed HUNTAgent running, all Datto AV drivers loaded (rtp1, rtp2, BdNet, BdSentry running), and Datto AV registered in WMI. Malwarebytes fully removed (the actual BSOD root cause -- its kernel drivers conflicting with Datto AV's Bitdefender drivers at DISPATCH_LEVEL IRQL).
|
||||
|
||||
Also fixed several account issues: looked up John Trozzi's AD account on CS-SERVER (never logged in, password never set), discovered "John Trozzi" local account was disabled since 2023, set localadmin password to `r3tr0gradE99#`, disabled casadmin201, unhid localadmin from the login screen (was hidden via `SpecialAccounts\UserList` registry key = 0). Vaulted localadmin credentials.
|
||||
|
||||
### Key Decisions
|
||||
|
||||
- The BSOD root cause was Malwarebytes kernel drivers (MbamChameleon.sys, mbamswissarmy.sys, mbae.sys + network filter driver oem26.inf) conflicting with Datto AV's Bitdefender SDK drivers at DISPATCH_LEVEL IRQL in the NDIS network stack. Removing Malwarebytes was the correct fix.
|
||||
- Initially misidentified Datto AV's Bitdefender-signed drivers as orphan Avira leftovers. CRITICAL LESSON: Datto AV uses the Bitdefender Endpoint Protection SDK; its drivers are signed by "Avira Operations GmbH" (same Gen Digital corporate family). Always check `C:\Program Files\infocyte\agent\dattoav\Endpoint Protection SDK\drivers\` before assuming BD/Avira drivers are orphans.
|
||||
- Rebooted the machine into Safe Mode (`safeboot minimal`) to bypass driver tamper protection -- but MAINTENANCE-PC is Wi-Fi only (TP-Link USB adapter on Wi-Fi 2), so Safe Mode killed all network connectivity and the RMM agent couldn't reach back. Required an on-site visit. Should have checked the network adapter type before choosing Safe Mode.
|
||||
- Reinstalled Datto EDR over the existing broken installation rather than attempting a full uninstall (which requires a console-issued uninstall token). The over-top install repaired the broken driver state.
|
||||
|
||||
### Problems Encountered
|
||||
|
||||
- **Avira/BD driver tamper protection:** Even as SYSTEM with SeTakeOwnershipPrivilege, could not modify the rtp1/rtp2/rtp_elam registry keys -- Avira's tamper protection locked the ACLs at the kernel level while the drivers were loaded. Required Safe Mode to bypass.
|
||||
- **Safe Mode on Wi-Fi-only device:** `bcdedit /set safeboot minimal` killed connectivity (Wi-Fi drivers don't load in Safe Mode). Had to drive to Cascades to run the cleanup commands locally. Logged as friction in errorlog.md.
|
||||
- **Misidentified Datto AV drivers as orphans:** Disabled and deleted rtp1/rtp2/rtp_elam/BdNet/BdSentry/netprotection_network_filter* thinking they were leftover Avira drivers. They belonged to Datto AV. Restored from the SDK backup directory and reinstalled EDR. Logged as correction in errorlog.md and saved as feedback memory.
|
||||
- **Windows Startup Repair triggered:** After the Safe Mode driver deletion, Windows detected the missing drivers and ran Startup Repair on next boot. Completed normally.
|
||||
- **Datto EDR uninstall requires token:** `agent.exe --uninstall` returned "Uninstall token not approved." Worked around by installing over the existing broken installation.
|
||||
- **UAC prompt defaulting to casadmin201:** The localadmin account was hidden via `SpecialAccounts\UserList` registry key (value 0). Set to 1 to make it visible on the login screen.
|
||||
|
||||
### Configuration Changes
|
||||
|
||||
- Malwarebytes v5.6.1.257 fully uninstalled (Howard did GUI uninstall on-site)
|
||||
- localadmin password set to `r3tr0gradE99#` on MAINTENANCE-PC
|
||||
- casadmin201 account disabled on MAINTENANCE-PC
|
||||
- localadmin unhidden from login screen (SpecialAccounts\UserList = 1)
|
||||
- Datto AV drivers restored from SDK directory to System32\drivers
|
||||
- Datto EDR reinstalled over existing installation
|
||||
- Cleanup script `remove-avira-drivers.bat` placed on localadmin desktop (can be deleted)
|
||||
- Accidental registry key `HKLM\SYSTEM\CurrentControlSet\Services\rtp` deleted
|
||||
- Vault entry created: `clients/cascades-tucson/maintenance-pc-localadmin.sops.yaml`
|
||||
- Memory file created: `feedback_datto_av_bitdefender_drivers.md`
|
||||
|
||||
### Credentials & Secrets
|
||||
|
||||
- **MAINTENANCE-PC localadmin:** password `r3tr0gradE99#` (same as CS-SERVER sysadmin)
|
||||
- Vault: `clients/cascades-tucson/maintenance-pc-localadmin.sops.yaml`
|
||||
- **John Trozzi AD:** `John.Trozzi@cascadestucson.com`, OU=Maintenance, password never set, never logged in
|
||||
- **Datto EDR deploy key (Cascades):** RegKey `6qw68y2rwl`, group `1dbd2b02-f7df-45d0-a7f2-18667f48447f`
|
||||
|
||||
### Infrastructure & Servers
|
||||
|
||||
- **MAINTENANCE-PC** (Cascades of Tucson)
|
||||
- GuruRMM agent: `2fbda0d2-efd7-4389-9ac5-3f89db467c1f`
|
||||
- Datto EDR agent: `9047be1a-b38c-47df-ad99-55e730a84580`
|
||||
- Datto EDR org: `2d5ea96e-3228-461b-9c60-13ae464b61d8` (Cascades of Tucson)
|
||||
- Model: Lenovo F0EW005TUS (IdeaCentre), BIOS O4SKT25A (2020-12-20)
|
||||
- Network: Wi-Fi only (TP-Link USB adapter, Realtek driver 1030.52.1216.2025), IP 10.0.20.96
|
||||
- Also has built-in Realtek 8822CE Wi-Fi (not primary) and Realtek PCIe GbE (no cable)
|
||||
- Local accounts: localadmin (admin, active), Bruce Miller (active), casadmin201 (disabled), John Trozzi (disabled), nick (disabled)
|
||||
- **CS-SERVER** (Cascades of Tucson, GuruRMM agent: `c39f1de7-d5b6-45ae-b132-e06977ab1713`) -- used for AD lookups
|
||||
|
||||
### Commands & Outputs
|
||||
|
||||
- BSOD history: 4 crashes in 3 months (0xd1 on 7/19 + 5/26, 0xef on 5/30 + 5/1)
|
||||
- Malwarebytes uninstall: GUI uninstall by Howard on-site
|
||||
- Safe Mode cleanup (on-site): `reg add` for rtp1/rtp2/rtp_elam Start=4, `del` for .sys files
|
||||
- EDR reinstall: `Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey 6qw68y2rwl`
|
||||
- Discord DM sent to Howard with Safe Mode cleanup commands (message_id 1528815475566444685)
|
||||
|
||||
### Pending / Incomplete Tasks
|
||||
|
||||
- Delete `remove-avira-drivers.bat` from localadmin desktop (cleanup script no longer needed)
|
||||
- Monitor for BSODs over the next few weeks -- with Malwarebytes gone, the driver conflict should be resolved
|
||||
- Datto RMM service (CagService) was shown as Stopped/Disabled earlier -- may need investigation (this is the OLD Datto RMM, not GuruRMM)
|
||||
- EndpointProtectionService (Datto AV engine) was Stopped after reinstall -- may need a reboot to fully start
|
||||
- Defender RTP is off (expected with Datto AV active, but verify Datto AV is actually scanning)
|
||||
- John Trozzi AD account has never been used (password never set) -- determine if he needs domain login or if local-only is intentional
|
||||
|
||||
### Reference Information
|
||||
|
||||
- Datto AV driver source: `C:\Program Files\infocyte\agent\dattoav\Endpoint Protection SDK\drivers\x64\`
|
||||
- Minidump: `C:\Windows\Minidump\071926-13531-01.dmp` (0xd1 BSOD)
|
||||
- BSOD report IDs: 1eeb0df0 (7/19), ee4b925b (5/26), dc5325cd (5/30), 2cebdc31 (5/1)
|
||||
- RMM commands: a87eb58e (BSOD diag), e13065a8 (security inventory), 962f9b8f (desktop script), 7aa2be12 (EDR reinstall), 3e5925d5 (final verification)
|
||||
- Errorlog entries: Safe Mode Wi-Fi friction, Datto AV driver misidentification correction
|
||||
|
||||
Reference in New Issue
Block a user