diff --git a/clients/desert-rv-center/session-logs/2026-07/2026-07-08-howard-becky-zerotier-rdp-restore.md b/clients/desert-rv-center/session-logs/2026-07/2026-07-08-howard-becky-zerotier-rdp-restore.md new file mode 100644 index 00000000..4c12957d --- /dev/null +++ b/clients/desert-rv-center/session-logs/2026-07/2026-07-08-howard-becky-zerotier-rdp-restore.md @@ -0,0 +1,75 @@ +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Alicia at Desert RV Center (DESKTOP-LBO396F) could not RDP into DRV-BECKY over ZeroTier. Screenshots showed the RDP client reaching ZeroTier IP 172.29.196.107 and failing with error 0x204 ("Remote Desktop can't connect to the remote computer"), which is a transport/port reachability failure rather than an authentication problem. Neither DRV-BECKY nor DESKTOP-LBO396F was enrolled in GuruRMM at session start, so direct probing was not initially possible. + +Diagnosis started by using DESKTOP-AK2KPOB — the one Desert RV machine already in RMM, physically on-site on the 10.0.0.0/24 LAN and domain-joined to DESERTRV.local — as a probe point. AK2KPOB is NOT on the ZeroTier network, so its inability to reach the ZT IP was expected and non-diagnostic. However, BECKY resolved on the LAN to 10.0.0.165 (ARP MAC 00-15-5D = Microsoft Hyper-V, so BECKY is a Hyper-V VM). A direct LAN port test from AK2KPOB proved BECKY was healthy: TCP 3389 (RDP) returned True on the LAN, while ICMP and 445 were firewall-blocked. This confirmed BECKY was powered on and RDP was working — the break was entirely in the ZeroTier transport. + +Remote silent push of the RMM agent to BECKY was not possible: BECKY's host firewall exposes only 3389; WinRM (5985), RPC (135), and SMB (445/139) were all closed. DESKTOP-LBO396F was off-net entirely (remote, not on this LAN, and the operator's machine is not a ZeroTier member), so there was no route to push to it either. While options were being presented, the user installed the RMM agent on both machines via ScreenConnect. Both then enrolled and came online (DRV-BECKY id 921b569e..., DESKTOP-LBO396F id 0a3f6f6e...). + +With an agent on BECKY, the root cause was found immediately: the ZeroTierOneService was Stopped (StartType Automatic). The ZeroTier adapter still held 172.29.196.107, so the IP Alicia was dialing was correct — but with the service down nothing was listening on the tunnel, producing 0x204. Starting the service brought the node ONLINE and it rejoined the "Desert RV" network (nwid 35c192ce9b730d6e, status OK). An initial end-to-end test from LBO396F failed because the direct ZeroTier path had not yet established; a retry ~1 min later succeeded (DIRECT peer path, 11ms latency, TCP 3389 to 172.29.196.107 = True). The user confirmed RDP login worked. The service was then configured to auto-restart on failure to prevent recurrence, BECKY's login password was vaulted, and a warranty ticket (#32514) was created and invoiced at 0.5h / $0.00. + +## Key Decisions + +- Used DESKTOP-AK2KPOB (existing on-site RMM agent) as a LAN probe to isolate BECKY's health from the ZeroTier transport, rather than assuming BECKY was down. This proved RDP was fine on the LAN and pointed squarely at ZeroTier. +- Interpreted error 0x204 as a reachability/port failure (not credentials), so the provided BECKY password was not the relevant lever — did not chase auth. +- Did not attempt a remote silent push to BECKY once port testing showed only 3389 open (all admin channels firewalled) — correctly identified push as infeasible before wasting cycles on PsExec/WinRM. +- On the first failed end-to-end test (both nodes ONLINE, firewall/profile ruled out), attributed it to ZeroTier direct-path warm-up and retried rather than mis-diagnosing a firewall block. The retry confirmed this. +- Added service recovery (auto-restart on failure) as a durable safeguard so a future ZeroTier service stop self-heals. +- Billed as 0.5h warranty (product 1049360, Exempt Labor, $0.00) per user instruction. + +## Problems Encountered + +- Neither target machine was in RMM at start — resolved when the user pushed agents via ScreenConnect mid-session; diagnosis then completed via RMM directly on BECKY. +- BECKY host firewall allows only RDP (3389); WinRM/RPC/SMB closed — ruled out all remote silent-install methods. Documented as the reason push was infeasible. +- First end-to-end RDP test from LBO396F failed (TcpTestSucceeded False) seconds after BECKY came online — cause was ZeroTier direct-path establishment delay, not a real fault; retry succeeded. +- Multiple RMM diagnostic commands timed out when using Test-NetConnection against multiple closed ports (each closed-port probe blocks ~20s) — switched to a fast .NET TcpClient socket test with a 2s timeout. + +## Configuration Changes + +- **DRV-BECKY:** started `ZeroTierOneService` (was Stopped, StartType Automatic → now Running). +- **DRV-BECKY:** set service recovery via `sc.exe failure` — restart on failure at 5s/10s/30s, reset period 86400s; `failureflag` set to 1. +- **Vault (new file):** `clients/desert-rv-center/drv-becky-rdp.sops.yaml` — BECKY RDP login credential (committed + pushed to vault repo). +- **Both machines** enrolled in GuruRMM (by user via ScreenConnect) under Desert RV Center / Main. + +## Credentials & Secrets + +- **DRV-BECKY local Windows / RDP login:** password `Token1225*` (username not captured — login screen only showed the password field; noted as UNKNOWN in the vault entry for later update). + - Vaulted at: `clients/desert-rv-center/drv-becky-rdp.sops.yaml` (decrypt round-trip verified). +- **GuruRMM Desert RV site enrollment** (pre-existing): `clients/desert-rv-center/gururmm-site-main.sops.yaml` — site_code WEST-VALLEY-6327, site_id 852f4419-890d-4832-b2b0-8f34fac88be3. + +## Infrastructure & Servers + +- **DRV-BECKY:** Hyper-V VM (ARP MAC prefix 00-15-5D), domain DESERTRV.local, LAN IP 10.0.0.165, ZeroTier IP 172.29.196.107 (network "Desert RV" nwid 35c192ce9b730d6e). ZeroTier node addr 855a7db4f0, ZeroTier v1.14.0. Host firewall: only 3389 open (WinRM 5985 / RPC 135 / SMB 445,139 closed; ICMP blocked). RDP enabled (fDenyTSConnections=0), NLA default. ~8GB RAM VM. +- **DESKTOP-LBO396F** (Alicia's remote machine): ZeroTier node addr 25b90fd31e, v1.16.1, ZeroTier IP 172.29.254.82/16 on network "Desert RV", status OK/ONLINE. Uses a "Remote" RDP desktop shortcut pointing to 172.29.196.107. +- **DESKTOP-AK2KPOB:** on-site Desert RV desktop, domain-joined DESERTRV.local, LAN 10.0.0.189, GuruRMM agent id 8605d665-f598-4d8a-8ea0-c75b57491a75. NOT a ZeroTier member. Used as LAN probe. +- **GuruRMM agent IDs:** DRV-BECKY 921b569e-9807-4396-8e6c-53f3d1410442; DESKTOP-LBO396F 0a3f6f6e-18be-483a-b422-888c350fc2a7. +- LAN DNS server observed: 10.0.0.182 (LAPTOP-HVKG0GQM). + +## Commands & Outputs + +- RMM probe from AK2KPOB — LAN reachability to BECKY: + - `Test-NetConnection 10.0.0.165 -Port 3389` → TcpTestSucceeded: True (RDP up on LAN) + - `-Port 445` → False; ping → False (ICMP blocked) +- BECKY ZeroTier state (before fix): `Get-Service ZeroTierOneService` → Stopped/Automatic; `zerotier-cli info` → connection failed (service down); adapter still had 172.29.196.107. +- Fix: `Set-Service ZeroTierOneService -StartupType Automatic; Start-Service ZeroTierOneService` → Running; `zerotier-cli info` → `855a7db4f0 1.14.0 ONLINE`; `listnetworks` → `35c192ce9b730d6e Desert RV ... OK PRIVATE ... 172.29.196.107/16`. +- Recovery config: `sc.exe failure ZeroTierOneService reset= 86400 actions= restart/5000/restart/10000/restart/30000` → SUCCESS; `sc.exe failureflag ZeroTierOneService 1` → SUCCESS. +- End-to-end from LBO396F (after warm-up): `Test-NetConnection 172.29.196.107 -Port 3389` → TcpTestSucceeded: True; peers showed BECKY 855a7db4f0 DIRECT lat 11. +- Diagnostic gotcha: multi-port `Test-NetConnection` to closed ports timed out the RMM command (each ~20s); replaced with `New-Object Net.Sockets.TcpClient` + `BeginConnect`/`WaitOne(2000)`. + +## Pending / Incomplete Tasks + +- **BECKY slowness** (user-reported, not yet investigated): 8GB RAM VM, ~3GB used, uptime ~3.5 days, one svchost (PID 1756) with high cumulative CPU. Offer stands to diagnose driver (Windows Update / runaway service / disk/SMART). +- **BECKY admin username unknown** — vault entry `drv-becky-rdp.sops.yaml` has password only; update `credentials.username` once confirmed. +- Consider verifying LBO396F ZeroTier service is also set to auto-start/recover (not changed this session). + +## Reference Information + +- **Syncro ticket #32514** (id 113625645), Desert RV Center (customer 19357852): "Remote access to DRV-BECKY failed (RDP error 0x204) - ZeroTier service restored". Billed 0.5h warranty (product 1049360, Exempt Labor) = $0.00. Invoice id 1650984937. Status Invoiced. Link: https://computerguru.syncromsp.com/tickets/113625645 +- GuruRMM installer (Desert RV, site WEST-VALLEY-6327): https://rmm.azcomputerguru.com/install/WEST-VALLEY-6327 +- ZeroTier network "Desert RV": nwid 35c192ce9b730d6e (PRIVATE, 172.29.0.0/16). +- RMM commands dispatched from AK2KPOB and both new agents; bot alerts posted to #dev-alerts (RMM) and #bot-alerts (Syncro).