sync: auto-sync from HOWARD-HOME at 2026-06-26 07:19:00

Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-26 07:19:00
2026-06-26 07:19:29 -07:00
parent 15f2d03d7b
commit 270e294938
3 changed files with 185 additions and 14 deletions
--- a/.claude/memory/project_cascades_network_segments.md
+++ b/.claude/memory/project_cascades_network_segments.md
@@ -20,21 +20,33 @@ in the /22). Wi-Fi SSID for corporate = **`CSCNet`**.
 pfSense `192.168.0.1`. CSCNet PSK + CSC ENT PSK vaulted (`clients/cascades-tucson/wifi-cscnet.sops.yaml`,
 `wifi-csc-ent.sops.yaml`).

-**CORRECTION (don't trust the first theory): the SMB problem is NOT CSC-ENT-vs-CSCNet.** Verified
-2026-06-26: fresh SMB connections to `\\cs-server\<share>` fail with **System error 67
-(BAD_NETWORK_NAME), even `IPC$`**, from CSC ENT AND CSCNet alike (Meredith on CSC ENT, Crystal +
-Karen on CSCNet all hit it). Only **pre-existing persistent mapped drives** work, and even those are
-**intermittent** (Meredith's `Y: \\cs-server\Server` read True then False minutes later). Lower layers
-are fine everywhere (ping, nbtstat=CS-SERVER, TCP 445/139). So it's a **CS-SERVER-side SMB
-instability**, not the client network.
+**CORRECTION x2 (2026-06-26): the SMB problem is NOT CSC-ENT-vs-CSCNet AND NOT the AV.** After a
+FULL removal of the Datto stack from CS-SERVER (DattoAV SDK + Datto RMM + Datto EDR — see below) and
+a reboot, **error 67 persisted unchanged** — so the AV minifilters (`rtp1`/`rtp2`/`BdSentry`) were a
+RED HERRING for SMB. More importantly, the SMB **server is HEALTHY for domain/Kerberos clients**:
+`Get-SmbSession` shows 7+ live SMB 3.1.1 sessions with open files (lauren, Sharon, Crystal @10.0.20.205,
+Megan, Ashley, chris) established AFTER the reboot. The failure is confined to **workgroup/NTLM +
+loopback**: Karen (DESKTOP-LPOPV30, workgroup) gets error 67 to EVERY share (IPC$, C$, a brand-new
+test share) even in her real user session with CORRECT creds, and `Get-SmbConnection`=none (the SMB
+session never negotiates; NO SMBServer/SMBClient event is logged -> failure is **pre-SMB-negotiate,
+BAD_NET_NAME**, not auth). NTLM restriction is NOT set (`RestrictReceivingNTLMTraffic`,
+`LmCompatibilityLevel`, NTLM Operational log all clear). `Get-SmbServerConfiguration` now works
+(NullSessionShares REG type was repaired prior session).

-**Root-cause leads (CS-SERVER):** `Get-SmbServerConfiguration` throws *"Data of this type is not
-supported"* (degraded SMB config subsystem); SMB **multichannel advertises multiple interfaces** —
-`192.168.2.248` (Ethernet), `192.168.2.254` (Hyper-V vSwitch), and IPv6 ULAs (`fde4::…`,`fd8f::…`).
-Clients negotiating channels to unreachable interfaces is the prime suspect for flaky/failed new
-sessions. SMB signing required (it's a DC). Likely fixes (NEED approval, prod DC, 13 live sessions):
-disable SMB multichannel / unbind File&Printer sharing from `.254`+IPv6 so it serves only on `.248`;
-and/or restart LanmanServer (or reboot) to rebuild the degraded config.
+**The AV was DattoAV, not GravityZone Bitdefender.** It was the Datto EDR "Endpoint Protection SDK"
+under `C:\Program Files\infocyte\agent\dattoav`, managed by Datto RMM (CentraStage/CagService) + Datto
+EDR Agent (HUNTAgent/Infocyte HUNT). That is why removing from the GravityZone console did nothing.
+ALL Datto software was removed from CS-SERVER 2026-06-26 (services deleted, dirs/registry/drivers gone).
+Tenant azcomp4587; Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`; CS-SERVER was already de-enrolled
+from EDR. Use the `/datto-edr` skill + `msp-tools/datto-edr.sops.yaml`.
+
+**Open SMB-67 leads (server fine for domain clients):** SMB multichannel advertises `.248` (Ethernet),
+`.254` (Hyper-V vSwitch, often unreachable from routed clients), and IPv6 ULAs (`fde4::`,`fd8f::`) —
+a routed WORKGROUP client (Karen @10.0.20.100, 4 of 5 adapters on APIPA 169.254) negotiating channels
+to unreachable interfaces is a candidate. Next: fresh connect from Meredith (workgroup, ON-LINK
+192.168.2.x) to isolate Karen-machine vs workgroup-general; clean Karen's APIPA adapters; consider
+unbinding File&Printer from `.254`/IPv6 (prod DC, domain clients currently fine — needs care); or
+domain-join Karen (Kerberos works for everyone who is joined).

 **Two more migration blockers found:** (1) **CSCNet is WPA3-SAE** — older adapters (e.g. Intel AC 3165
 on Meredith's ASSISTMAN-PC) **cannot join it**, so "move everyone to CSCNet" is blocked by hardware.
--- a/clients/cascades-tucson/session-logs/2026-06/2026-06-26-howard-cs-server-datto-removal-smb-rootcause.md
+++ b/clients/cascades-tucson/session-logs/2026-06/2026-06-26-howard-cs-server-datto-removal-smb-rootcause.md
@@ -0,0 +1,153 @@
+## User
+- **User:** Howard Enos (howard)
+- **Machine:** Howard-Home
+- **Role:** tech
+
+# CS-SERVER: full Datto stack removal + SMB error-67 root-cause reframe
+
+## Session Summary
+
+Continuation of the Cascades NAS->CS-SERVER migration / CS-SERVER SMB "error 67"
+investigation. Two outcomes this session: (1) **completely removed the Datto software
+stack from CS-SERVER**, and (2) **disproved the working theory** that the AV was causing
+the SMB failure and narrowed the real cause.
+
+The AV on CS-SERVER was misidentified in the prior session as GravityZone Bitdefender.
+Investigation showed it is actually **DattoAV** — the "Endpoint Protection SDK"
+(v1.0.2510.6851) under `C:\Program Files\infocyte\agent\dattoav`, managed by the **Datto
+RMM agent (CagService/CentraStage)** and **Datto EDR Agent (HUNTAgent / Infocyte HUNT,
+v3.17.1.5552)**. DattoAV uses the Bitdefender engine + Avira "Sentry" driver, which is why
+the kernel minifilters were named `BdSentry` ("Avira Sentry Driver"), `rtp1`, `rtp2`. This
+is why removing the box from the GravityZone console did nothing — GravityZone never
+managed it.
+
+Removed all three Datto components in order: `endpointprotection.exe uninstallSdk` (cleared
+`rtp1`/`rtp2`/`BdSentry` registry keys + unloaded the minifilters with no reboot), Datto RMM
+via `CentraStage\uninst.exe /VERYSILENT` (clean, exit 0), then the Datto EDR agent. The EDR
+agent's GUI uninstaller crashed headless (wgpu surface panic) and `agent.exe --uninstall
+--no-gui` was blocked by uninstall-protection ("token required"). Stopped + disabled
+HUNTAgent (held across reboot — no tamper watchdog), rebooted to clear the in-memory state,
+then — since the tamper drivers were already gone — **force-removed** the orphaned EDR
+agent: killed processes, `sc delete HUNTAgent`, removed `C:\Program Files\infocyte`, deleted
+the uninstall registry entries. CS-SERVER is now 100% Datto-free (no services, dirs,
+registry entries, or drivers).
+
+Critically, **SMB error 67 persisted unchanged after the full AV removal + reboot**, so the
+AV was NOT the cause. Deep diagnosis then revealed the SMB server is actually **healthy for
+domain-joined (Kerberos) clients** — `Get-SmbSession` shows 7+ live SMB 3.1.1 sessions with
+open files (lauren.hasselman, Sharon.Edwards, Crystal.Rodriguez, Megan.Hiatt, Ashley.Jensen,
+chris.knight) established after the reboot. The failure is confined to **workgroup/NTLM and
+loopback** connections: Karen's workgroup PC (DESKTOP-LPOPV30) gets error 67 to every share
+(IPC$, C$, a brand-new test share) even in her real user session with correct credentials,
+and `Get-SmbConnection` shows the SMB session never even negotiates (no SMBServer or
+SMBClient event is logged on either side — the failure is pre-SMB-negotiate, BAD_NET_NAME).
+NTLM restriction is NOT set on the server (checked `RestrictReceivingNTLMTraffic`,
+`LmCompatibilityLevel`, NTLM Operational log — all clear). The Datto EDR uninstall token
+Howard found is moot for CS-SERVER (already removed) but useful for other Cascades agents.
+
+## Key Decisions
+- Removed the entire Datto stack (AV + RMM + EDR) per Howard's "uninstall all the datto
+  software, we can install again if needed" — GuruRMM is a separate product, so removing
+  Datto RMM did not sever our control channel.
+- Force-removed the orphaned EDR agent rather than chase the uninstall-protection token:
+  the kernel tamper drivers (BdSentry/rtp*) were already removed by `uninstallSdk`, so
+  nothing was actively protecting the files; `sc delete` + dir/reg removal was clean and
+  reboot-safe.
+- Stopped treating the AV as the SMB root cause once error 67 reproduced identically after
+  full removal + reboot, AND once 7 live domain SMB 3.1.1 sessions proved the server engine
+  is healthy.
+- Did NOT make further SMB-server config changes (e.g., disabling multichannel) unilaterally
+  on the production DC — paused to get Howard's direction.
+
+## Problems Encountered
+- **AV misidentified as GravityZone Bitdefender** -> actually DattoAV (Endpoint Protection
+  SDK) via the Datto/Infocyte agent. Logged as a `--correction` in errorlog.
+- **Datto EDR GUI uninstaller crashed headless** (wgpu "Invalid surface" panic) under SYSTEM
+  -> used `--no-gui`, which then hit uninstall-protection ("token required").
+- **EDR uninstall-protection token unavailable** -> CS-SERVER is not enrolled in the Datto
+  EDR tenant (azcomp4587) anymore (tenant-wide hostname search returns empty), so no token
+  can be issued for it. Force-removal used instead.
+- **rtp1/rtp2 reverted Start=4->1 and reloaded after the first reboot** -> the EDR agent
+  re-armed them on boot; resolved by removing the managing agents, not just the drivers.
+- **`shutdown /a`/RPC-dependent ops returned 1717/1702** during the SMB-broken window ->
+  switched to local-only checks.
+- **Disproved my own NTLM-restriction theory and the prior CSC-ENT-vs-CSCNet theory** with
+  evidence (NTLM policy unset; domain clients on the same 10.0.20.x subnet as Karen work).
+
+## Configuration Changes
+- CS-SERVER (192.168.2.248): **removed** DattoAV / Endpoint Protection SDK, Datto RMM
+  (CentraStage, `CagService`), Datto EDR Agent (`HUNTAgent`). Deleted services, removed
+  `C:\Program Files\infocyte` and `C:\Program Files (x86)\CentraStage`, removed all related
+  Uninstall registry keys, kernel drivers `rtp1`/`rtp2`/`BdSentry` gone. Rebooted ~06:43 MST.
+- Prior-session NullSessionShares REG type repair (REG_SZ->REG_MULTI_SZ) confirmed still
+  good (`Get-SmbServerConfiguration` works).
+- Created + removed a throwaway test share `ztest$` (C:\ztest) during diagnosis.
+- No NTFS/share ACL changes. No multichannel change (deferred for Howard's call).
+
+## Credentials & Secrets
+- Datto EDR (azcomp4587) org-level uninstall-protection token: viewable in console at
+  `https://azcomp4587.infocyte.com/organizations/2d5ea96e-3228-461b-9c60-13ae464b61d8`
+  (unmask button). NOT YET VAULTED (value not captured) — vault into
+  `msp-tools/datto-edr.sops.yaml` if Howard pastes it; useful for removing other Cascades
+  EDR agents.
+- CS-SERVER orphaned EDR agent config (now removed): api-url
+  `https://azcomp4587.infocyte.com:443`, server-key `b7cmnghlgh`.
+- Karen domain pwd still vaulted: `clients/cascades-tucson/karen-rossini.sops.yaml`.
+
+## Infrastructure & Servers
+- CS-SERVER: DC `cascades.local`/`CASCADES`, IPs 192.168.2.248 (Ethernet) + 192.168.2.254
+  (Hyper-V vSwitch). GuruRMM agent `c39f1de7-d5b6-45ae-b132-e06977ab1713`. SMB 3.1.1, signing
+  required, RejectUnencryptedAccess=True, EnableMultiChannel=True (advertises .248, .254, IPv6
+  ULAs fde4::/fd8f::). TCP 445 listens on `::` (dual-stack) owned by PID 4 (kernel). Booted
+  06:43 MST 2026-06-26.
+- Live SMB clients on CS-SERVER (post-reboot, proof server is healthy): lauren.hasselman
+  (10.0.20.235), Sharon.Edwards (192.168.3.133), Crystal.Rodriguez (10.0.20.205, 7 opens),
+  Megan.Hiatt (10.0.20.202), Ashley.Jensen (192.168.3.37), chris.knight + DESKTOP-N5G1ROO$
+  (10.0.20.183).
+- Karen's PC: DESKTOP-LPOPV30 (WORKGROUP), agent `ad725bb2-d8cb-4a83-8203-6f7e9c906b29`,
+  logged on locally as `desktop-lpopv30\karen rossini`. Multi-homed mess: only 10.0.20.100
+  (Wi-Fi 2) has a real IP; 4 other adapters on APIPA 169.254.x (Ethernet, Wi-Fi, 2x LAC*).
+  cmdkey has `Domain:CS-SERVER -> CASCADES\karen.rossini` and `Domain:CASCADESDS`.
+- Meredith: ASSISTMAN-PC `cf86fa5e-96a2-494d-9cb1-8be22a518ad0` (on 192.168.2.x / CSC ENT,
+  on-link with CS-SERVER; reportedly can access shares).
+- Datto EDR tenant azcomp4587: Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`, site
+  `1dbd2b02-f7df-45d0-a7f2-18667f48447f` (33 agents, NO server enrolled).
+
+## Commands & Outputs
+- `endpointprotection.exe uninstallSdk` -> rtp1/rtp2/BdSentry reg keys GONE, filters unloaded
+  (no reboot). exit 2 (worked despite nonzero).
+- `CentraStage\uninst.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART` -> exit 0, clean.
+- `agent.exe --uninstall --no-gui` -> "Uninstall protection is enabled: token required".
+- Force removal: kill agent/system-tray, `sc.exe delete HUNTAgent` (SUCCESS), `Remove-Item
+  C:\Program Files\infocyte -Recurse -Force` (clean), reg Uninstall keys removed.
+- Post-removal verify: NO datto/edr uninstall entries; HUNTAgent/CagService deleted; infocyte
+  + CentraStage dirs False; rtp1/rtp2/BdSentry keys False; no filters.
+- SMB after full removal+reboot: `net use \\localhost\IPC$` -> error 67; `New-SmbMapping
+  \\localhost\{IPC$,C$,ztest$}` -> "network name cannot be found"; NO SMBServer/SMBClient
+  events for the attempts. `Get-SmbSession` -> 7 live domain SMB3.1.1 sessions.
+- Karen user-session: `whoami`=desktop-lpopv30\karen rossini; `net use \\CS-SERVER\Server
+  /user:CASCADES\karen.rossini <correct pw>` -> error 67; `Get-SmbConnection` -> none.
+- NTLM policy: RestrictReceivingNTLMTraffic unset, LmCompatibilityLevel unset, NTLM
+  Operational log empty. NoLmHash=1, RestrictNullSessAccess=1, restrictanonymous=1 (normal).
+
+## Pending / Incomplete Tasks
+- **SMB error 67 root cause (UNRESOLVED):** server healthy for domain/Kerberos clients;
+  fails for Karen's workgroup PC + loopback with BAD_NET_NAME *before* SMB negotiate (no
+  events). Next discriminators: (a) fresh connect from Meredith (workgroup, on-link
+  192.168.2.x) — works? -> isolates Karen-machine vs workgroup-general; (b) Karen's APIPA
+  adapter cleanup; (c) test whether SMB multichannel advertising unreachable interfaces
+  (.254 vSwitch + IPv6 ULAs) breaks routed/workgroup new-connection setup -> candidate fix:
+  unbind File&Printer from .254 / disable multichannel (prod DC — needs care, domain clients
+  currently fine); (d) consider domain-joining Karen (then Kerberos like working clients).
+- Vault the Datto EDR org uninstall token if Howard captures it.
+- Resume Karen ALDocs repoint (`\\CS-SERVER\Server\ALDocs`) via drive-map once access works.
+- Correct memory `project_cascades_network_segments.md` (AV was DattoAV; SMB cause is NOT the
+  AV; server healthy for domain clients).
+
+## Reference Information
+- DattoAV = Datto EDR's Endpoint Protection SDK (Bitdefender engine + Avira Sentry). NOT
+  GravityZone. Managed by Datto RMM (CentraStage/CagService) + Datto EDR (HUNTAgent/Infocyte).
+- Datto EDR tenant: azcomp4587.infocyte.com; skill `/datto-edr`; vault
+  `msp-tools/datto-edr.sops.yaml`. Cascades org `2d5ea96e-3228-461b-9c60-13ae464b61d8`.
+- GuruRMM agent CS-SERVER `c39f1de7-d5b6-45ae-b132-e06977ab1713`.
+- drive-map skill: `.claude/skills/drive-map/`.
--- a/errorlog.md
+++ b/errorlog.md
@@ -29,6 +29,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·

 2026-06-26 | GURU-5070 | remediation-tool | onboard-tenant: failed to acquire Tenant Admin token [ctx: tenant=19a568e8-9e88-413b-9341-cbc224b39145 exit=3]

+2026-06-26 | Howard-Home | rmm/cascades-cs-server | [correction] assumed CS-SERVER AV was GravityZone Bitdefender (used bitdefender skill, chased tamper password); actually DattoAV / Endpoint Protection SDK at C:Program Filesinfocytegentdattoav, managed by Datto RMM (CagService) + HUNTAgent. GravityZone removal did nothing; rtp1/rtp2 minifilters re-arm on boot (reverted Start=4->1). rtp1 attached to DeviceNamedPipe = breaks SMB IPC$/RPC 1702. [ctx: host=CS-SERVER client=cascades]
+
+2026-06-26 | Howard-Home | post-bot-alert | Discord POST failed (non-200/unreachable) [ctx: channel=#dev-alerts http=400 resp={"message": "The request body contains invalid JSON.", "code": 50109}]
+
+2026-06-26 | Howard-Home | post-bot-alert | Discord POST failed (non-200/unreachable) [ctx: channel=#dev-alerts http=400 resp={"message": "The request body contains invalid JSON.", "code": 50109}]
+
 2026-06-26 | Howard-Home | drive-map | drive-map verify failed on DESKTOP-LPOPV30 [ctx: cmd=e932bc94-0557-4913-a0b1-c97c1aa5da26]

 2026-06-26 | Howard-Home | drive-map | drive-map verify failed on DESKTOP-LPOPV30 [ctx: cmd=18fec38b-8fae-4a1b-a3d8-5b90b124dbc2]