sync: auto-sync from HOWARD-HOME at 2026-06-26 07:19:00
Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-26 07:19:00
This commit is contained in:
@@ -0,0 +1,153 @@
|
||||
## User
|
||||
- **User:** Howard Enos (howard)
|
||||
- **Machine:** Howard-Home
|
||||
- **Role:** tech
|
||||
|
||||
# CS-SERVER: full Datto stack removal + SMB error-67 root-cause reframe
|
||||
|
||||
## Session Summary
|
||||
|
||||
Continuation of the Cascades NAS->CS-SERVER migration / CS-SERVER SMB "error 67"
|
||||
investigation. Two outcomes this session: (1) **completely removed the Datto software
|
||||
stack from CS-SERVER**, and (2) **disproved the working theory** that the AV was causing
|
||||
the SMB failure and narrowed the real cause.
|
||||
|
||||
The AV on CS-SERVER was misidentified in the prior session as GravityZone Bitdefender.
|
||||
Investigation showed it is actually **DattoAV** — the "Endpoint Protection SDK"
|
||||
(v1.0.2510.6851) under `C:\Program Files\infocyte\agent\dattoav`, managed by the **Datto
|
||||
RMM agent (CagService/CentraStage)** and **Datto EDR Agent (HUNTAgent / Infocyte HUNT,
|
||||
v3.17.1.5552)**. DattoAV uses the Bitdefender engine + Avira "Sentry" driver, which is why
|
||||
the kernel minifilters were named `BdSentry` ("Avira Sentry Driver"), `rtp1`, `rtp2`. This
|
||||
is why removing the box from the GravityZone console did nothing — GravityZone never
|
||||
managed it.
|
||||
|
||||
Removed all three Datto components in order: `endpointprotection.exe uninstallSdk` (cleared
|
||||
`rtp1`/`rtp2`/`BdSentry` registry keys + unloaded the minifilters with no reboot), Datto RMM
|
||||
via `CentraStage\uninst.exe /VERYSILENT` (clean, exit 0), then the Datto EDR agent. The EDR
|
||||
agent's GUI uninstaller crashed headless (wgpu surface panic) and `agent.exe --uninstall
|
||||
--no-gui` was blocked by uninstall-protection ("token required"). Stopped + disabled
|
||||
HUNTAgent (held across reboot — no tamper watchdog), rebooted to clear the in-memory state,
|
||||
then — since the tamper drivers were already gone — **force-removed** the orphaned EDR
|
||||
agent: killed processes, `sc delete HUNTAgent`, removed `C:\Program Files\infocyte`, deleted
|
||||
the uninstall registry entries. CS-SERVER is now 100% Datto-free (no services, dirs,
|
||||
registry entries, or drivers).
|
||||
|
||||
Critically, **SMB error 67 persisted unchanged after the full AV removal + reboot**, so the
|
||||
AV was NOT the cause. Deep diagnosis then revealed the SMB server is actually **healthy for
|
||||
domain-joined (Kerberos) clients** — `Get-SmbSession` shows 7+ live SMB 3.1.1 sessions with
|
||||
open files (lauren.hasselman, Sharon.Edwards, Crystal.Rodriguez, Megan.Hiatt, Ashley.Jensen,
|
||||
chris.knight) established after the reboot. The failure is confined to **workgroup/NTLM and
|
||||
loopback** connections: Karen's workgroup PC (DESKTOP-LPOPV30) gets error 67 to every share
|
||||
(IPC$, C$, a brand-new test share) even in her real user session with correct credentials,
|
||||
and `Get-SmbConnection` shows the SMB session never even negotiates (no SMBServer or
|
||||
SMBClient event is logged on either side — the failure is pre-SMB-negotiate, BAD_NET_NAME).
|
||||
NTLM restriction is NOT set on the server (checked `RestrictReceivingNTLMTraffic`,
|
||||
`LmCompatibilityLevel`, NTLM Operational log — all clear). The Datto EDR uninstall token
|
||||
Howard found is moot for CS-SERVER (already removed) but useful for other Cascades agents.
|
||||
|
||||
## Key Decisions
|
||||
- Removed the entire Datto stack (AV + RMM + EDR) per Howard's "uninstall all the datto
|
||||
software, we can install again if needed" — GuruRMM is a separate product, so removing
|
||||
Datto RMM did not sever our control channel.
|
||||
- Force-removed the orphaned EDR agent rather than chase the uninstall-protection token:
|
||||
the kernel tamper drivers (BdSentry/rtp*) were already removed by `uninstallSdk`, so
|
||||
nothing was actively protecting the files; `sc delete` + dir/reg removal was clean and
|
||||
reboot-safe.
|
||||
- Stopped treating the AV as the SMB root cause once error 67 reproduced identically after
|
||||
full removal + reboot, AND once 7 live domain SMB 3.1.1 sessions proved the server engine
|
||||
is healthy.
|
||||
- Did NOT make further SMB-server config changes (e.g., disabling multichannel) unilaterally
|
||||
on the production DC — paused to get Howard's direction.
|
||||
|
||||
## Problems Encountered
|
||||
- **AV misidentified as GravityZone Bitdefender** -> actually DattoAV (Endpoint Protection
|
||||
SDK) via the Datto/Infocyte agent. Logged as a `--correction` in errorlog.
|
||||
- **Datto EDR GUI uninstaller crashed headless** (wgpu "Invalid surface" panic) under SYSTEM
|
||||
-> used `--no-gui`, which then hit uninstall-protection ("token required").
|
||||
- **EDR uninstall-protection token unavailable** -> CS-SERVER is not enrolled in the Datto
|
||||
EDR tenant (azcomp4587) anymore (tenant-wide hostname search returns empty), so no token
|
||||
can be issued for it. Force-removal used instead.
|
||||
- **rtp1/rtp2 reverted Start=4->1 and reloaded after the first reboot** -> the EDR agent
|
||||
re-armed them on boot; resolved by removing the managing agents, not just the drivers.
|
||||
- **`shutdown /a`/RPC-dependent ops returned 1717/1702** during the SMB-broken window ->
|
||||
switched to local-only checks.
|
||||
- **Disproved my own NTLM-restriction theory and the prior CSC-ENT-vs-CSCNet theory** with
|
||||
evidence (NTLM policy unset; domain clients on the same 10.0.20.x subnet as Karen work).
|
||||
|
||||
## Configuration Changes
|
||||
- CS-SERVER (192.168.2.248): **removed** DattoAV / Endpoint Protection SDK, Datto RMM
|
||||
(CentraStage, `CagService`), Datto EDR Agent (`HUNTAgent`). Deleted services, removed
|
||||
`C:\Program Files\infocyte` and `C:\Program Files (x86)\CentraStage`, removed all related
|
||||
Uninstall registry keys, kernel drivers `rtp1`/`rtp2`/`BdSentry` gone. Rebooted ~06:43 MST.
|
||||
- Prior-session NullSessionShares REG type repair (REG_SZ->REG_MULTI_SZ) confirmed still
|
||||
good (`Get-SmbServerConfiguration` works).
|
||||
- Created + removed a throwaway test share `ztest$` (C:\ztest) during diagnosis.
|
||||
- No NTFS/share ACL changes. No multichannel change (deferred for Howard's call).
|
||||
|
||||
## Credentials & Secrets
|
||||
- Datto EDR (azcomp4587) org-level uninstall-protection token: viewable in console at
|
||||
`https://azcomp4587.infocyte.com/organizations/2d5ea96e-3228-461b-9c60-13ae464b61d8`
|
||||
(unmask button). NOT YET VAULTED (value not captured) — vault into
|
||||
`msp-tools/datto-edr.sops.yaml` if Howard pastes it; useful for removing other Cascades
|
||||
EDR agents.
|
||||
- CS-SERVER orphaned EDR agent config (now removed): api-url
|
||||
`https://azcomp4587.infocyte.com:443`, server-key `b7cmnghlgh`.
|
||||
- Karen domain pwd still vaulted: `clients/cascades-tucson/karen-rossini.sops.yaml`.
|
||||
|
||||
## Infrastructure & Servers
|
||||
- CS-SERVER: DC `cascades.local`/`CASCADES`, IPs 192.168.2.248 (Ethernet) + 192.168.2.254
|
||||
(Hyper-V vSwitch). GuruRMM agent `c39f1de7-d5b6-45ae-b132-e06977ab1713`. SMB 3.1.1, signing
|
||||
required, RejectUnencryptedAccess=True, EnableMultiChannel=True (advertises .248, .254, IPv6
|
||||
ULAs fde4::/fd8f::). TCP 445 listens on `::` (dual-stack) owned by PID 4 (kernel). Booted
|
||||
06:43 MST 2026-06-26.
|
||||
- Live SMB clients on CS-SERVER (post-reboot, proof server is healthy): lauren.hasselman
|
||||
(10.0.20.235), Sharon.Edwards (192.168.3.133), Crystal.Rodriguez (10.0.20.205, 7 opens),
|
||||
Megan.Hiatt (10.0.20.202), Ashley.Jensen (192.168.3.37), chris.knight + DESKTOP-N5G1ROO$
|
||||
(10.0.20.183).
|
||||
- Karen's PC: DESKTOP-LPOPV30 (WORKGROUP), agent `ad725bb2-d8cb-4a83-8203-6f7e9c906b29`,
|
||||
logged on locally as `desktop-lpopv30\karen rossini`. Multi-homed mess: only 10.0.20.100
|
||||
(Wi-Fi 2) has a real IP; 4 other adapters on APIPA 169.254.x (Ethernet, Wi-Fi, 2x LAC*).
|
||||
cmdkey has `Domain:CS-SERVER -> CASCADES\karen.rossini` and `Domain:CASCADESDS`.
|
||||
- Meredith: ASSISTMAN-PC `cf86fa5e-96a2-494d-9cb1-8be22a518ad0` (on 192.168.2.x / CSC ENT,
|
||||
on-link with CS-SERVER; reportedly can access shares).
|
||||
- Datto EDR tenant azcomp4587: Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`, site
|
||||
`1dbd2b02-f7df-45d0-a7f2-18667f48447f` (33 agents, NO server enrolled).
|
||||
|
||||
## Commands & Outputs
|
||||
- `endpointprotection.exe uninstallSdk` -> rtp1/rtp2/BdSentry reg keys GONE, filters unloaded
|
||||
(no reboot). exit 2 (worked despite nonzero).
|
||||
- `CentraStage\uninst.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART` -> exit 0, clean.
|
||||
- `agent.exe --uninstall --no-gui` -> "Uninstall protection is enabled: token required".
|
||||
- Force removal: kill agent/system-tray, `sc.exe delete HUNTAgent` (SUCCESS), `Remove-Item
|
||||
C:\Program Files\infocyte -Recurse -Force` (clean), reg Uninstall keys removed.
|
||||
- Post-removal verify: NO datto/edr uninstall entries; HUNTAgent/CagService deleted; infocyte
|
||||
+ CentraStage dirs False; rtp1/rtp2/BdSentry keys False; no filters.
|
||||
- SMB after full removal+reboot: `net use \\localhost\IPC$` -> error 67; `New-SmbMapping
|
||||
\\localhost\{IPC$,C$,ztest$}` -> "network name cannot be found"; NO SMBServer/SMBClient
|
||||
events for the attempts. `Get-SmbSession` -> 7 live domain SMB3.1.1 sessions.
|
||||
- Karen user-session: `whoami`=desktop-lpopv30\karen rossini; `net use \\CS-SERVER\Server
|
||||
/user:CASCADES\karen.rossini <correct pw>` -> error 67; `Get-SmbConnection` -> none.
|
||||
- NTLM policy: RestrictReceivingNTLMTraffic unset, LmCompatibilityLevel unset, NTLM
|
||||
Operational log empty. NoLmHash=1, RestrictNullSessAccess=1, restrictanonymous=1 (normal).
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
- **SMB error 67 root cause (UNRESOLVED):** server healthy for domain/Kerberos clients;
|
||||
fails for Karen's workgroup PC + loopback with BAD_NET_NAME *before* SMB negotiate (no
|
||||
events). Next discriminators: (a) fresh connect from Meredith (workgroup, on-link
|
||||
192.168.2.x) — works? -> isolates Karen-machine vs workgroup-general; (b) Karen's APIPA
|
||||
adapter cleanup; (c) test whether SMB multichannel advertising unreachable interfaces
|
||||
(.254 vSwitch + IPv6 ULAs) breaks routed/workgroup new-connection setup -> candidate fix:
|
||||
unbind File&Printer from .254 / disable multichannel (prod DC — needs care, domain clients
|
||||
currently fine); (d) consider domain-joining Karen (then Kerberos like working clients).
|
||||
- Vault the Datto EDR org uninstall token if Howard captures it.
|
||||
- Resume Karen ALDocs repoint (`\\CS-SERVER\Server\ALDocs`) via drive-map once access works.
|
||||
- Correct memory `project_cascades_network_segments.md` (AV was DattoAV; SMB cause is NOT the
|
||||
AV; server healthy for domain clients).
|
||||
|
||||
## Reference Information
|
||||
- DattoAV = Datto EDR's Endpoint Protection SDK (Bitdefender engine + Avira Sentry). NOT
|
||||
GravityZone. Managed by Datto RMM (CentraStage/CagService) + Datto EDR (HUNTAgent/Infocyte).
|
||||
- Datto EDR tenant: azcomp4587.infocyte.com; skill `/datto-edr`; vault
|
||||
`msp-tools/datto-edr.sops.yaml`. Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`.
|
||||
- GuruRMM agent CS-SERVER `c39f1de7-d5b6-45ae-b132-e06977ab1713`.
|
||||
- drive-map skill: `.claude/skills/drive-map/`.
|
||||
Reference in New Issue
Block a user