From 270e294938801491e9a9946a340ab31c0f8ca5d7 Mon Sep 17 00:00:00 2001 From: Howard Enos Date: Fri, 26 Jun 2026 07:19:29 -0700 Subject: [PATCH] sync: auto-sync from HOWARD-HOME at 2026-06-26 07:19:00 Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-26 07:19:00 --- .../project_cascades_network_segments.md | 40 +++-- ...d-cs-server-datto-removal-smb-rootcause.md | 153 ++++++++++++++++++ errorlog.md | 6 + 3 files changed, 185 insertions(+), 14 deletions(-) create mode 100644 clients/cascades-tucson/session-logs/2026-06/2026-06-26-howard-cs-server-datto-removal-smb-rootcause.md diff --git a/.claude/memory/project_cascades_network_segments.md b/.claude/memory/project_cascades_network_segments.md index bb71ab48..1a794b1c 100644 --- a/.claude/memory/project_cascades_network_segments.md +++ b/.claude/memory/project_cascades_network_segments.md @@ -20,21 +20,33 @@ in the /22). Wi-Fi SSID for corporate = **`CSCNet`**. pfSense `192.168.0.1`. CSCNet PSK + CSC ENT PSK vaulted (`clients/cascades-tucson/wifi-cscnet.sops.yaml`, `wifi-csc-ent.sops.yaml`). -**CORRECTION (don't trust the first theory): the SMB problem is NOT CSC-ENT-vs-CSCNet.** Verified -2026-06-26: fresh SMB connections to `\\cs-server\` fail with **System error 67 -(BAD_NETWORK_NAME), even `IPC$`**, from CSC ENT AND CSCNet alike (Meredith on CSC ENT, Crystal + -Karen on CSCNet all hit it). Only **pre-existing persistent mapped drives** work, and even those are -**intermittent** (Meredith's `Y: \\cs-server\Server` read True then False minutes later). Lower layers -are fine everywhere (ping, nbtstat=CS-SERVER, TCP 445/139). So it's a **CS-SERVER-side SMB -instability**, not the client network. +**CORRECTION x2 (2026-06-26): the SMB problem is NOT CSC-ENT-vs-CSCNet AND NOT the AV.** After a +FULL removal of the Datto stack from CS-SERVER (DattoAV SDK + Datto RMM + Datto EDR — see below) and +a reboot, **error 67 persisted unchanged** — so the AV minifilters (`rtp1`/`rtp2`/`BdSentry`) were a +RED HERRING for SMB. More importantly, the SMB **server is HEALTHY for domain/Kerberos clients**: +`Get-SmbSession` shows 7+ live SMB 3.1.1 sessions with open files (lauren, Sharon, Crystal @10.0.20.205, +Megan, Ashley, chris) established AFTER the reboot. The failure is confined to **workgroup/NTLM + +loopback**: Karen (DESKTOP-LPOPV30, workgroup) gets error 67 to EVERY share (IPC$, C$, a brand-new +test share) even in her real user session with CORRECT creds, and `Get-SmbConnection`=none (the SMB +session never negotiates; NO SMBServer/SMBClient event is logged -> failure is **pre-SMB-negotiate, +BAD_NET_NAME**, not auth). NTLM restriction is NOT set (`RestrictReceivingNTLMTraffic`, +`LmCompatibilityLevel`, NTLM Operational log all clear). `Get-SmbServerConfiguration` now works +(NullSessionShares REG type was repaired prior session). -**Root-cause leads (CS-SERVER):** `Get-SmbServerConfiguration` throws *"Data of this type is not -supported"* (degraded SMB config subsystem); SMB **multichannel advertises multiple interfaces** — -`192.168.2.248` (Ethernet), `192.168.2.254` (Hyper-V vSwitch), and IPv6 ULAs (`fde4::…`,`fd8f::…`). -Clients negotiating channels to unreachable interfaces is the prime suspect for flaky/failed new -sessions. SMB signing required (it's a DC). Likely fixes (NEED approval, prod DC, 13 live sessions): -disable SMB multichannel / unbind File&Printer sharing from `.254`+IPv6 so it serves only on `.248`; -and/or restart LanmanServer (or reboot) to rebuild the degraded config. +**The AV was DattoAV, not GravityZone Bitdefender.** It was the Datto EDR "Endpoint Protection SDK" +under `C:\Program Files\infocyte\agent\dattoav`, managed by Datto RMM (CentraStage/CagService) + Datto +EDR Agent (HUNTAgent/Infocyte HUNT). That is why removing from the GravityZone console did nothing. +ALL Datto software was removed from CS-SERVER 2026-06-26 (services deleted, dirs/registry/drivers gone). +Tenant azcomp4587; Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`; CS-SERVER was already de-enrolled +from EDR. Use the `/datto-edr` skill + `msp-tools/datto-edr.sops.yaml`. + +**Open SMB-67 leads (server fine for domain clients):** SMB multichannel advertises `.248` (Ethernet), +`.254` (Hyper-V vSwitch, often unreachable from routed clients), and IPv6 ULAs (`fde4::`,`fd8f::`) — +a routed WORKGROUP client (Karen @10.0.20.100, 4 of 5 adapters on APIPA 169.254) negotiating channels +to unreachable interfaces is a candidate. Next: fresh connect from Meredith (workgroup, ON-LINK +192.168.2.x) to isolate Karen-machine vs workgroup-general; clean Karen's APIPA adapters; consider +unbinding File&Printer from `.254`/IPv6 (prod DC, domain clients currently fine — needs care); or +domain-join Karen (Kerberos works for everyone who is joined). **Two more migration blockers found:** (1) **CSCNet is WPA3-SAE** — older adapters (e.g. Intel AC 3165 on Meredith's ASSISTMAN-PC) **cannot join it**, so "move everyone to CSCNet" is blocked by hardware. diff --git a/clients/cascades-tucson/session-logs/2026-06/2026-06-26-howard-cs-server-datto-removal-smb-rootcause.md b/clients/cascades-tucson/session-logs/2026-06/2026-06-26-howard-cs-server-datto-removal-smb-rootcause.md new file mode 100644 index 00000000..d96c0861 --- /dev/null +++ b/clients/cascades-tucson/session-logs/2026-06/2026-06-26-howard-cs-server-datto-removal-smb-rootcause.md @@ -0,0 +1,153 @@ +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +# CS-SERVER: full Datto stack removal + SMB error-67 root-cause reframe + +## Session Summary + +Continuation of the Cascades NAS->CS-SERVER migration / CS-SERVER SMB "error 67" +investigation. Two outcomes this session: (1) **completely removed the Datto software +stack from CS-SERVER**, and (2) **disproved the working theory** that the AV was causing +the SMB failure and narrowed the real cause. + +The AV on CS-SERVER was misidentified in the prior session as GravityZone Bitdefender. +Investigation showed it is actually **DattoAV** — the "Endpoint Protection SDK" +(v1.0.2510.6851) under `C:\Program Files\infocyte\agent\dattoav`, managed by the **Datto +RMM agent (CagService/CentraStage)** and **Datto EDR Agent (HUNTAgent / Infocyte HUNT, +v3.17.1.5552)**. DattoAV uses the Bitdefender engine + Avira "Sentry" driver, which is why +the kernel minifilters were named `BdSentry` ("Avira Sentry Driver"), `rtp1`, `rtp2`. This +is why removing the box from the GravityZone console did nothing — GravityZone never +managed it. + +Removed all three Datto components in order: `endpointprotection.exe uninstallSdk` (cleared +`rtp1`/`rtp2`/`BdSentry` registry keys + unloaded the minifilters with no reboot), Datto RMM +via `CentraStage\uninst.exe /VERYSILENT` (clean, exit 0), then the Datto EDR agent. The EDR +agent's GUI uninstaller crashed headless (wgpu surface panic) and `agent.exe --uninstall +--no-gui` was blocked by uninstall-protection ("token required"). Stopped + disabled +HUNTAgent (held across reboot — no tamper watchdog), rebooted to clear the in-memory state, +then — since the tamper drivers were already gone — **force-removed** the orphaned EDR +agent: killed processes, `sc delete HUNTAgent`, removed `C:\Program Files\infocyte`, deleted +the uninstall registry entries. CS-SERVER is now 100% Datto-free (no services, dirs, +registry entries, or drivers). + +Critically, **SMB error 67 persisted unchanged after the full AV removal + reboot**, so the +AV was NOT the cause. Deep diagnosis then revealed the SMB server is actually **healthy for +domain-joined (Kerberos) clients** — `Get-SmbSession` shows 7+ live SMB 3.1.1 sessions with +open files (lauren.hasselman, Sharon.Edwards, Crystal.Rodriguez, Megan.Hiatt, Ashley.Jensen, +chris.knight) established after the reboot. The failure is confined to **workgroup/NTLM and +loopback** connections: Karen's workgroup PC (DESKTOP-LPOPV30) gets error 67 to every share +(IPC$, C$, a brand-new test share) even in her real user session with correct credentials, +and `Get-SmbConnection` shows the SMB session never even negotiates (no SMBServer or +SMBClient event is logged on either side — the failure is pre-SMB-negotiate, BAD_NET_NAME). +NTLM restriction is NOT set on the server (checked `RestrictReceivingNTLMTraffic`, +`LmCompatibilityLevel`, NTLM Operational log — all clear). The Datto EDR uninstall token +Howard found is moot for CS-SERVER (already removed) but useful for other Cascades agents. + +## Key Decisions +- Removed the entire Datto stack (AV + RMM + EDR) per Howard's "uninstall all the datto + software, we can install again if needed" — GuruRMM is a separate product, so removing + Datto RMM did not sever our control channel. +- Force-removed the orphaned EDR agent rather than chase the uninstall-protection token: + the kernel tamper drivers (BdSentry/rtp*) were already removed by `uninstallSdk`, so + nothing was actively protecting the files; `sc delete` + dir/reg removal was clean and + reboot-safe. +- Stopped treating the AV as the SMB root cause once error 67 reproduced identically after + full removal + reboot, AND once 7 live domain SMB 3.1.1 sessions proved the server engine + is healthy. +- Did NOT make further SMB-server config changes (e.g., disabling multichannel) unilaterally + on the production DC — paused to get Howard's direction. + +## Problems Encountered +- **AV misidentified as GravityZone Bitdefender** -> actually DattoAV (Endpoint Protection + SDK) via the Datto/Infocyte agent. Logged as a `--correction` in errorlog. +- **Datto EDR GUI uninstaller crashed headless** (wgpu "Invalid surface" panic) under SYSTEM + -> used `--no-gui`, which then hit uninstall-protection ("token required"). +- **EDR uninstall-protection token unavailable** -> CS-SERVER is not enrolled in the Datto + EDR tenant (azcomp4587) anymore (tenant-wide hostname search returns empty), so no token + can be issued for it. Force-removal used instead. +- **rtp1/rtp2 reverted Start=4->1 and reloaded after the first reboot** -> the EDR agent + re-armed them on boot; resolved by removing the managing agents, not just the drivers. +- **`shutdown /a`/RPC-dependent ops returned 1717/1702** during the SMB-broken window -> + switched to local-only checks. +- **Disproved my own NTLM-restriction theory and the prior CSC-ENT-vs-CSCNet theory** with + evidence (NTLM policy unset; domain clients on the same 10.0.20.x subnet as Karen work). + +## Configuration Changes +- CS-SERVER (192.168.2.248): **removed** DattoAV / Endpoint Protection SDK, Datto RMM + (CentraStage, `CagService`), Datto EDR Agent (`HUNTAgent`). Deleted services, removed + `C:\Program Files\infocyte` and `C:\Program Files (x86)\CentraStage`, removed all related + Uninstall registry keys, kernel drivers `rtp1`/`rtp2`/`BdSentry` gone. Rebooted ~06:43 MST. +- Prior-session NullSessionShares REG type repair (REG_SZ->REG_MULTI_SZ) confirmed still + good (`Get-SmbServerConfiguration` works). +- Created + removed a throwaway test share `ztest$` (C:\ztest) during diagnosis. +- No NTFS/share ACL changes. No multichannel change (deferred for Howard's call). + +## Credentials & Secrets +- Datto EDR (azcomp4587) org-level uninstall-protection token: viewable in console at + `https://azcomp4587.infocyte.com/organizations/2d5ea96e-3228-461b-9c60-13ae464b61d8` + (unmask button). NOT YET VAULTED (value not captured) — vault into + `msp-tools/datto-edr.sops.yaml` if Howard pastes it; useful for removing other Cascades + EDR agents. +- CS-SERVER orphaned EDR agent config (now removed): api-url + `https://azcomp4587.infocyte.com:443`, server-key `b7cmnghlgh`. +- Karen domain pwd still vaulted: `clients/cascades-tucson/karen-rossini.sops.yaml`. + +## Infrastructure & Servers +- CS-SERVER: DC `cascades.local`/`CASCADES`, IPs 192.168.2.248 (Ethernet) + 192.168.2.254 + (Hyper-V vSwitch). GuruRMM agent `c39f1de7-d5b6-45ae-b132-e06977ab1713`. SMB 3.1.1, signing + required, RejectUnencryptedAccess=True, EnableMultiChannel=True (advertises .248, .254, IPv6 + ULAs fde4::/fd8f::). TCP 445 listens on `::` (dual-stack) owned by PID 4 (kernel). Booted + 06:43 MST 2026-06-26. +- Live SMB clients on CS-SERVER (post-reboot, proof server is healthy): lauren.hasselman + (10.0.20.235), Sharon.Edwards (192.168.3.133), Crystal.Rodriguez (10.0.20.205, 7 opens), + Megan.Hiatt (10.0.20.202), Ashley.Jensen (192.168.3.37), chris.knight + DESKTOP-N5G1ROO$ + (10.0.20.183). +- Karen's PC: DESKTOP-LPOPV30 (WORKGROUP), agent `ad725bb2-d8cb-4a83-8203-6f7e9c906b29`, + logged on locally as `desktop-lpopv30\karen rossini`. Multi-homed mess: only 10.0.20.100 + (Wi-Fi 2) has a real IP; 4 other adapters on APIPA 169.254.x (Ethernet, Wi-Fi, 2x LAC*). + cmdkey has `Domain:CS-SERVER -> CASCADES\karen.rossini` and `Domain:CASCADESDS`. +- Meredith: ASSISTMAN-PC `cf86fa5e-96a2-494d-9cb1-8be22a518ad0` (on 192.168.2.x / CSC ENT, + on-link with CS-SERVER; reportedly can access shares). +- Datto EDR tenant azcomp4587: Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`, site + `1dbd2b02-f7df-45d0-a7f2-18667f48447f` (33 agents, NO server enrolled). + +## Commands & Outputs +- `endpointprotection.exe uninstallSdk` -> rtp1/rtp2/BdSentry reg keys GONE, filters unloaded + (no reboot). exit 2 (worked despite nonzero). +- `CentraStage\uninst.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART` -> exit 0, clean. +- `agent.exe --uninstall --no-gui` -> "Uninstall protection is enabled: token required". +- Force removal: kill agent/system-tray, `sc.exe delete HUNTAgent` (SUCCESS), `Remove-Item + C:\Program Files\infocyte -Recurse -Force` (clean), reg Uninstall keys removed. +- Post-removal verify: NO datto/edr uninstall entries; HUNTAgent/CagService deleted; infocyte + + CentraStage dirs False; rtp1/rtp2/BdSentry keys False; no filters. +- SMB after full removal+reboot: `net use \\localhost\IPC$` -> error 67; `New-SmbMapping + \\localhost\{IPC$,C$,ztest$}` -> "network name cannot be found"; NO SMBServer/SMBClient + events for the attempts. `Get-SmbSession` -> 7 live domain SMB3.1.1 sessions. +- Karen user-session: `whoami`=desktop-lpopv30\karen rossini; `net use \\CS-SERVER\Server + /user:CASCADES\karen.rossini ` -> error 67; `Get-SmbConnection` -> none. +- NTLM policy: RestrictReceivingNTLMTraffic unset, LmCompatibilityLevel unset, NTLM + Operational log empty. NoLmHash=1, RestrictNullSessAccess=1, restrictanonymous=1 (normal). + +## Pending / Incomplete Tasks +- **SMB error 67 root cause (UNRESOLVED):** server healthy for domain/Kerberos clients; + fails for Karen's workgroup PC + loopback with BAD_NET_NAME *before* SMB negotiate (no + events). Next discriminators: (a) fresh connect from Meredith (workgroup, on-link + 192.168.2.x) — works? -> isolates Karen-machine vs workgroup-general; (b) Karen's APIPA + adapter cleanup; (c) test whether SMB multichannel advertising unreachable interfaces + (.254 vSwitch + IPv6 ULAs) breaks routed/workgroup new-connection setup -> candidate fix: + unbind File&Printer from .254 / disable multichannel (prod DC — needs care, domain clients + currently fine); (d) consider domain-joining Karen (then Kerberos like working clients). +- Vault the Datto EDR org uninstall token if Howard captures it. +- Resume Karen ALDocs repoint (`\\CS-SERVER\Server\ALDocs`) via drive-map once access works. +- Correct memory `project_cascades_network_segments.md` (AV was DattoAV; SMB cause is NOT the + AV; server healthy for domain clients). + +## Reference Information +- DattoAV = Datto EDR's Endpoint Protection SDK (Bitdefender engine + Avira Sentry). NOT + GravityZone. Managed by Datto RMM (CentraStage/CagService) + Datto EDR (HUNTAgent/Infocyte). +- Datto EDR tenant: azcomp4587.infocyte.com; skill `/datto-edr`; vault + `msp-tools/datto-edr.sops.yaml`. Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`. +- GuruRMM agent CS-SERVER `c39f1de7-d5b6-45ae-b132-e06977ab1713`. +- drive-map skill: `.claude/skills/drive-map/`. diff --git a/errorlog.md b/errorlog.md index 12d06209..78c01d00 100644 --- a/errorlog.md +++ b/errorlog.md @@ -29,6 +29,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · 2026-06-26 | GURU-5070 | remediation-tool | onboard-tenant: failed to acquire Tenant Admin token [ctx: tenant=19a568e8-9e88-413b-9341-cbc224b39145 exit=3] +2026-06-26 | Howard-Home | rmm/cascades-cs-server | [correction] assumed CS-SERVER AV was GravityZone Bitdefender (used bitdefender skill, chased tamper password); actually DattoAV / Endpoint Protection SDK at C:Program Filesinfocytegentdattoav, managed by Datto RMM (CagService) + HUNTAgent. GravityZone removal did nothing; rtp1/rtp2 minifilters re-arm on boot (reverted Start=4->1). rtp1 attached to DeviceNamedPipe = breaks SMB IPC$/RPC 1702. [ctx: host=CS-SERVER client=cascades] + +2026-06-26 | Howard-Home | post-bot-alert | Discord POST failed (non-200/unreachable) [ctx: channel=#dev-alerts http=400 resp={"message": "The request body contains invalid JSON.", "code": 50109}] + +2026-06-26 | Howard-Home | post-bot-alert | Discord POST failed (non-200/unreachable) [ctx: channel=#dev-alerts http=400 resp={"message": "The request body contains invalid JSON.", "code": 50109}] + 2026-06-26 | Howard-Home | drive-map | drive-map verify failed on DESKTOP-LPOPV30 [ctx: cmd=e932bc94-0557-4913-a0b1-c97c1aa5da26] 2026-06-26 | Howard-Home | drive-map | drive-map verify failed on DESKTOP-LPOPV30 [ctx: cmd=18fec38b-8fae-4a1b-a3d8-5b90b124dbc2]