claudetools/clients/cascades-tucson/session-logs/2026-06/2026-06-26-howard-cs-server-datto-removal-smb-rootcause.md

User

• User: Howard Enos (howard)
• Machine: Howard-Home
• Role: tech

CS-SERVER: full Datto stack removal + SMB error-67 root-cause reframe

Session Summary

Continuation of the Cascades NAS->CS-SERVER migration / CS-SERVER SMB "error 67" investigation. Two outcomes this session: (1) completely removed the Datto software stack from CS-SERVER, and (2) disproved the working theory that the AV was causing the SMB failure and narrowed the real cause.

The AV on CS-SERVER was misidentified in the prior session as GravityZone Bitdefender. Investigation showed it is actually DattoAV — the "Endpoint Protection SDK" (v1.0.2510.6851) under C:\Program Files\infocyte\agent\dattoav, managed by the Datto RMM agent (CagService/CentraStage) and Datto EDR Agent (HUNTAgent / Infocyte HUNT, v3.17.1.5552). DattoAV uses the Bitdefender engine + Avira "Sentry" driver, which is why the kernel minifilters were named BdSentry ("Avira Sentry Driver"), rtp1, rtp2. This is why removing the box from the GravityZone console did nothing — GravityZone never managed it.

Removed all three Datto components in order: endpointprotection.exe uninstallSdk (cleared rtp1/rtp2/BdSentry registry keys + unloaded the minifilters with no reboot), Datto RMM via CentraStage\uninst.exe /VERYSILENT (clean, exit 0), then the Datto EDR agent. The EDR agent's GUI uninstaller crashed headless (wgpu surface panic) and agent.exe --uninstall --no-gui was blocked by uninstall-protection ("token required"). Stopped + disabled HUNTAgent (held across reboot — no tamper watchdog), rebooted to clear the in-memory state, then — since the tamper drivers were already gone — force-removed the orphaned EDR agent: killed processes, sc delete HUNTAgent, removed C:\Program Files\infocyte, deleted the uninstall registry entries. CS-SERVER is now 100% Datto-free (no services, dirs, registry entries, or drivers).

Critically, SMB error 67 persisted unchanged after the full AV removal + reboot, so the AV was NOT the cause. Deep diagnosis then revealed the SMB server is actually healthy for domain-joined (Kerberos) clients — Get-SmbSession shows 7+ live SMB 3.1.1 sessions with open files (lauren.hasselman, Sharon.Edwards, Crystal.Rodriguez, Megan.Hiatt, Ashley.Jensen, chris.knight) established after the reboot. The failure is confined to workgroup/NTLM and loopback connections: Karen's workgroup PC (DESKTOP-LPOPV30) gets error 67 to every share (IPC$, C$, a brand-new test share) even in her real user session with correct credentials, and Get-SmbConnection shows the SMB session never even negotiates (no SMBServer or SMBClient event is logged on either side — the failure is pre-SMB-negotiate, BAD_NET_NAME). NTLM restriction is NOT set on the server (checked RestrictReceivingNTLMTraffic, LmCompatibilityLevel, NTLM Operational log — all clear). The Datto EDR uninstall token Howard found is moot for CS-SERVER (already removed) but useful for other Cascades agents.

Key Decisions

• Removed the entire Datto stack (AV + RMM + EDR) per Howard's "uninstall all the datto software, we can install again if needed" — GuruRMM is a separate product, so removing Datto RMM did not sever our control channel.
• Force-removed the orphaned EDR agent rather than chase the uninstall-protection token: the kernel tamper drivers (BdSentry/rtp*) were already removed by uninstallSdk, so nothing was actively protecting the files; sc delete + dir/reg removal was clean and reboot-safe.
• Stopped treating the AV as the SMB root cause once error 67 reproduced identically after full removal + reboot, AND once 7 live domain SMB 3.1.1 sessions proved the server engine is healthy.
• Did NOT make further SMB-server config changes (e.g., disabling multichannel) unilaterally on the production DC — paused to get Howard's direction.

Problems Encountered

• AV misidentified as GravityZone Bitdefender -> actually DattoAV (Endpoint Protection SDK) via the Datto/Infocyte agent. Logged as a --correction in errorlog.
• Datto EDR GUI uninstaller crashed headless (wgpu "Invalid surface" panic) under SYSTEM -> used --no-gui, which then hit uninstall-protection ("token required").
• EDR uninstall-protection token unavailable -> CS-SERVER is not enrolled in the Datto EDR tenant (azcomp4587) anymore (tenant-wide hostname search returns empty), so no token can be issued for it. Force-removal used instead.
• rtp1/rtp2 reverted Start=4->1 and reloaded after the first reboot -> the EDR agent re-armed them on boot; resolved by removing the managing agents, not just the drivers.
• shutdown /a/RPC-dependent ops returned 1717/1702 during the SMB-broken window -> switched to local-only checks.
• Disproved my own NTLM-restriction theory and the prior CSC-ENT-vs-CSCNet theory with evidence (NTLM policy unset; domain clients on the same 10.0.20.x subnet as Karen work).

Configuration Changes

• CS-SERVER (192.168.2.248): removed DattoAV / Endpoint Protection SDK, Datto RMM (CentraStage, CagService), Datto EDR Agent (HUNTAgent). Deleted services, removed C:\Program Files\infocyte and C:\Program Files (x86)\CentraStage, removed all related Uninstall registry keys, kernel drivers rtp1/rtp2/BdSentry gone. Rebooted ~06:43 MST.
• Prior-session NullSessionShares REG type repair (REG_SZ->REG_MULTI_SZ) confirmed still good (Get-SmbServerConfiguration works).
• Created + removed a throwaway test share ztest$ (C:\ztest) during diagnosis.
• No NTFS/share ACL changes. No multichannel change (deferred for Howard's call).

Credentials & Secrets

• Datto EDR (azcomp4587) org-level uninstall-protection token: viewable in console at https://azcomp4587.infocyte.com/organizations/2d5ea96e-3228-461b-9c60-13ae464b61d8 (unmask button). NOT YET VAULTED (value not captured) — vault into msp-tools/datto-edr.sops.yaml if Howard pastes it; useful for removing other Cascades EDR agents.
• CS-SERVER orphaned EDR agent config (now removed): api-url https://azcomp4587.infocyte.com:443, server-key b7cmnghlgh.
• Karen domain pwd still vaulted: clients/cascades-tucson/karen-rossini.sops.yaml.

Infrastructure & Servers

• CS-SERVER: DC cascades.local/CASCADES, IPs 192.168.2.248 (Ethernet) + 192.168.2.254 (Hyper-V vSwitch). GuruRMM agent c39f1de7-d5b6-45ae-b132-e06977ab1713. SMB 3.1.1, signing required, RejectUnencryptedAccess=True, EnableMultiChannel=True (advertises .248, .254, IPv6 ULAs fde4::/fd8f::). TCP 445 listens on :: (dual-stack) owned by PID 4 (kernel). Booted 06:43 MST 2026-06-26.
• Live SMB clients on CS-SERVER (post-reboot, proof server is healthy): lauren.hasselman (10.0.20.235), Sharon.Edwards (192.168.3.133), Crystal.Rodriguez (10.0.20.205, 7 opens), Megan.Hiatt (10.0.20.202), Ashley.Jensen (192.168.3.37), chris.knight + DESKTOP-N5G1ROO$ (10.0.20.183).
• Karen's PC: DESKTOP-LPOPV30 (WORKGROUP), agent ad725bb2-d8cb-4a83-8203-6f7e9c906b29, logged on locally as desktop-lpopv30\karen rossini. Multi-homed mess: only 10.0.20.100 (Wi-Fi 2) has a real IP; 4 other adapters on APIPA 169.254.x (Ethernet, Wi-Fi, 2x LAC*). cmdkey has Domain:CS-SERVER -> CASCADES\karen.rossini and Domain:CASCADESDS.
• Meredith: ASSISTMAN-PC cf86fa5e-96a2-494d-9cb1-8be22a518ad0 (on 192.168.2.x / CSC ENT, on-link with CS-SERVER; reportedly can access shares).
• Datto EDR tenant azcomp4587: Cascades org 2d5ea96e-3228-461b-9c60-13ae464b61d8, site 1dbd2b02-f7df-45d0-a7f2-18667f48447f (33 agents, NO server enrolled).

Commands & Outputs

• endpointprotection.exe uninstallSdk -> rtp1/rtp2/BdSentry reg keys GONE, filters unloaded (no reboot). exit 2 (worked despite nonzero).
• CentraStage\uninst.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART -> exit 0, clean.
• agent.exe --uninstall --no-gui -> "Uninstall protection is enabled: token required".
• Force removal: kill agent/system-tray, sc.exe delete HUNTAgent (SUCCESS), Remove-Item C:\Program Files\infocyte -Recurse -Force (clean), reg Uninstall keys removed.
• Post-removal verify: NO datto/edr uninstall entries; HUNTAgent/CagService deleted; infocyte
  • CentraStage dirs False; rtp1/rtp2/BdSentry keys False; no filters.
• SMB after full removal+reboot: net use \\localhost\IPC$ -> error 67; New-SmbMapping \\localhost\{IPC$,C$,ztest$} -> "network name cannot be found"; NO SMBServer/SMBClient events for the attempts. Get-SmbSession -> 7 live domain SMB3.1.1 sessions.
• Karen user-session: whoami=desktop-lpopv30\karen rossini; net use \\CS-SERVER\Server /user:CASCADES\karen.rossini <correct pw> -> error 67; Get-SmbConnection -> none.
• NTLM policy: RestrictReceivingNTLMTraffic unset, LmCompatibilityLevel unset, NTLM Operational log empty. NoLmHash=1, RestrictNullSessAccess=1, restrictanonymous=1 (normal).

Pending / Incomplete Tasks

• SMB error 67 root cause (UNRESOLVED): server healthy for domain/Kerberos clients; fails for Karen's workgroup PC + loopback with BAD_NET_NAME before SMB negotiate (no events). Next discriminators: (a) fresh connect from Meredith (workgroup, on-link 192.168.2.x) — works? -> isolates Karen-machine vs workgroup-general; (b) Karen's APIPA adapter cleanup; (c) test whether SMB multichannel advertising unreachable interfaces (.254 vSwitch + IPv6 ULAs) breaks routed/workgroup new-connection setup -> candidate fix: unbind File&Printer from .254 / disable multichannel (prod DC — needs care, domain clients currently fine); (d) consider domain-joining Karen (then Kerberos like working clients).
• Vault the Datto EDR org uninstall token if Howard captures it.
• Resume Karen ALDocs repoint (\\CS-SERVER\Server\ALDocs) via drive-map once access works.
• Correct memory project_cascades_network_segments.md (AV was DattoAV; SMB cause is NOT the AV; server healthy for domain clients).

Reference Information

• DattoAV = Datto EDR's Endpoint Protection SDK (Bitdefender engine + Avira Sentry). NOT GravityZone. Managed by Datto RMM (CentraStage/CagService) + Datto EDR (HUNTAgent/Infocyte).
• Datto EDR tenant: azcomp4587.infocyte.com; skill /datto-edr; vault msp-tools/datto-edr.sops.yaml. Cascades org 2d5ea96e-3228-461b-9c60-13ae464b61d8.
• GuruRMM agent CS-SERVER c39f1de7-d5b6-45ae-b132-e06977ab1713.
• drive-map skill: .claude/skills/drive-map/.