claudetools/projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-edr-mass-deploy-imc-printserver.md

User

• User: Howard Enos (howard)
• Machine: Howard-Home
• Role: tech

Session Summary

Continued the GPS->GuruRMM coverage audit. The session had two arcs: (1) closing out the two straggler machines from the Monday queue (CP-QB, IMC-PRINTSERVER), then (2) executing the full Bitdefender->Datto EDR AV switch across every reachable GPS-client machine.

CP-QB (Curtis Plumbing) was fixed and enrolled: its WMI class store was corrupt (Win32_Processor = Invalid class, broken since ~2022), which killed the universal installer's architecture-detection call and also froze its ScreenConnect inventory. mofcomp cimwin32.mof restored the class; the installer then ran clean and the agent enrolled and was reassigned to Curtis Plumbing. IMC-PRINTSERVER was diagnosed but left blocked: the GuruRMM agent was manually installed over the LAN (PowerShell is dead on the box, SMB hangs, the SC extension eats chained/multi-line commands, so the binary was served over HTTP from IMC1 and installed with plain cmd), but the box drops ALL outbound TLS (even google.com), so the installed agent cannot reach the server. DM'd Mike to ask whether that lockdown is by design or should be cleared.

The main work was the AV switch. Built tools/edr-deploy.py — a throttled deployer that recomputes coverage from a live Datto EDR agent dump each run, targets ONLINE Windows RMM agents of GPS clients that have a vaulted registration key and are not already in EDR, and pushes the official Install-EDR one-liner through the RMM one machine at a time with a 20s gap (retry-on-5xx, poll-to-completion). Glaz-Tech excluded by design. Run 1 hit 79 machines across 24 clients: 74 installed cleanly. Follow-up runs cleared the rest except two legacy boxes.

The five apparent failures were run to ground individually. Install-EDR is silent in non-interactive mode (every status/error line is gated behind if ($Interactive)), so a "completed / blank output" RMM result is meaningless — the visible "Installed RTS agent" text actually comes from agent.windows.exe itself. Investigating: DForth-Shipp (Win11) and SS-SP1 (Server 2016) already had huntAgent running (false negatives); AD1 (Server 2016) had a broken partial install (stale stopped service, no binary) that Install-EDR silently skipped, fixed with -Force; AMT-PC (Win7 SP1 / PowerShell 2.0) and SRV02 (Server 2008 R2) are genuine legacy failures where the modern agent binary crashes or no-ops on execution. Also deployed EDR to DESKTOP-PL2RCGL (Robyn Pittman, the 7/4 mid-delivery retry) once it came online. Datto EDR authorized-agent count rose 228 -> 302.

Key Decisions

• Deferred Bitdefender REMOVAL entirely — EDR-first, per-client, only after that client's EDR is confirmed ACTIVE. Nothing removed this session.
• Kept the deploy strictly throttled (sequential, 20s gaps, single command in flight) per Howard's "go slow, don't hammer the RMM server" instruction.
• Built a reusable coverage-recomputing tool rather than a one-shot script, so re-runs are idempotent and safe (targets only genuinely-missing online machines each time).
• Classified AMT-PC and SRV02 as legacy-OS blocked rather than continuing to brute-force installs on incompatible boxes; flagged for prerequisites/onsite.
• Created an onsite-check list (onsite-check-list.md) for machines/clients with no SC, no Syncro checkin, and no Bitdefender — to be handled at the next site visit, not chased remotely (Howard's rule).
• Added Horseshoe's existing EDR key to the vault rather than minting a new one (org+group already existed; only the vault entry was missing).

Problems Encountered

• CP-QB universal installer failed: corrupt WMI (Win32_Processor Invalid class). Fixed via mofcomp %windir%\system32\wbem\cimwin32.mof (verifyrepository reported CONSISTENT — it does not catch a missing class).
• IMC-PRINTSERVER cannot connect: drops all outbound TLS (Cloudflare hosts, rmm-api direct IP, and google.com — ClientHello then silence). SC survives because its relay is not TLS. Agent installed but dormant. Escalated to Mike (by-design vs fix). No reinstall needed once TLS is unblocked.
• Five EDR "failures" were mostly false negatives. Root-caused via the fact that Install-EDR is silent unless -Interactive; used -Interactive *> file + read-back and direct service/binary checks. Two real legacy blocks remain (AMT-PC Win7/PS2, SRV02 2008 R2).
• AD1 had a stale huntAgent service with no binary; Install-EDR's "already installed" early-return silently skipped it. -Force reinstalled successfully.
• edr.py needed DATTO_EDR_TOKEN exported from the vault (msp-tools/datto-edr.sops.yaml field credentials.api_token) — the repo skill copy path differs from the user-home copy.

Configuration Changes

• Created projects/gps-rmm-audit/tools/edr-deploy.py — throttled BD->EDR deployer.
• Created projects/gps-rmm-audit/onsite-check-list.md — no-remote-channel machines/clients.
• Created projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-cpqb-fixed-imcprintserver-tls-blocked.md (earlier in the session; CP-QB + IMC-PRINTSERVER detail).
• Appended to projects/gps-rmm-audit/tracker.md — CP-QB/IMC-PRINTSERVER outcome, onsite group pointer, and the full AV-switch deploy section.
• Vault: added Horseshoe Management: jthm976pwo to msp-tools/datto-edr-regkeys.sops.yaml.
• Deploy artifacts (gitignored/scratch): edr-deploy-live.log, edr-deploy-run3.log, edr-deploy-results.md, .edr-agents-now.json under projects/gps-rmm-audit/.

Credentials & Secrets

• Datto EDR per-client registration keys — vault msp-tools/datto-edr-regkeys.sops.yaml (field map under credentials:). Used verbatim as Install-EDR -RegKey. Newly added this session: Horseshoe Management = jthm976pwo (key already existed in Datto; only the vault entry was missing). Keys used directly this session incl. Arizona Medical Transit arizonam01, Dataforth 27lzj6npdb, Robyn Pittman pittman001.
• Datto EDR API token — vault msp-tools/datto-edr.sops.yaml field credentials.api_token (export as DATTO_EDR_TOKEN for edr.py).
• ScreenConnect API secret — vault msp-tools/screenconnect.sops.yaml field credentials.api_secret.
• GuruRMM staging site key DARK-STORM-3150 — vault infrastructure/gururmm-staging-site.sops.yaml.

Infrastructure & Servers

• GuruRMM API: http://172.16.3.30:3001 (via rmm-auth.sh); dashboard rmm.azcomputerguru.com; agent WS wss://rmm-api.azcomputerguru.com/ws (rmm-api resolves to direct IP 72.194.62.10, NOT Cloudflare). GuruRMM root Linux agent id 9b92b187-98c7-41b0-9e97-1698d263c42d (hostname gururmm) used to read the server's own nginx access log.
• Datto EDR: https://azcomp4587.infocyte.com; agent binary https://s3.us-east-2.amazonaws.com/infocyte-support/executables/agent.windows64.exe (v3.17.1.5409). Authorized agents 228 -> 302 this session.
• IMC-PRINTSERVER: Win10 22H2, 192.168.0.79/24, gw 192.168.0.254, DNS/DHCP = IMC1 (192.168.0.2), MAC A0-1E-0B-11-60-1D. SC session ccbd66a7-f94a-4d87-a18b-5f81a30784d0. IMC1 RMM agent id 88cbf7c0-abfa-4f12-846c-96274f718bff.
• Straggler agent ids: AMT-PC 2471f636-c73d-4d90-aabf-7da42a58917d; SRV02 4628e665-ac21-4ff7-bde2-c649682f6241; AD1 bf7bc5ee-4167-4a62-912a-c88b11a5943d; DForth-Shipp 9974ca3c-02d0-4632-8532-508d34344b8c; SS-SP1 0db6ed6b-48a6-4dcd-9366-e9a67eefa3d2; DESKTOP-PL2RCGL b12cd995-016e-4eff-bdcb-1033aa9f3e78.
• Horseshoe EDR: org e40037bf-37ee-4767-9e36-889253df03de, group 49d5fbf2-f377-4ede-9e4c-1c8655939b63.

Commands & Outputs

• EDR deploy (throttled): RMM=.. TOK=.. python projects/gps-rmm-audit/tools/edr-deploy.py --dry | --live [--gap 20].
• Official install one-liner (per box, via RMM powershell elevated): [System.Net.ServicePointManager]::SecurityProtocol=[Enum]::ToObject([System.Net.SecurityProtocolType],3072); (new-object Net.WebClient).DownloadString("https://raw.githubusercontent.com/Infocyte/PowershellTools/master/AgentDeployment/install_huntagent.ps1") | iex; Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey <key>.
• True-state check on a box: Get-Service huntAgent + Test-Path "C:\Program Files\infocyte\agent\agent.exe". Verbose install: add -Interactive *> C:\Windows\Temp\x.log then read the file (Install-EDR is silent otherwise).
• -Force reinstalls over a stale huntAgent service (fixed AD1: log "[Success] Installation Succeeded! Agent associated to https://azcomp4587.infocyte.com").
• AMT-PC diagnosis: Get-CimInstance "not recognized" => PowerShell 2.0 => legacy Win7; agent.windows64.exe downloads (38472512 bytes) but crashes the process tree on exec.

Pending / Incomplete Tasks

• IMC-PRINTSERVER: awaiting Mike's ruling (TLS lockdown by design vs fix). Agent installed + dormant; self-enrolls when TLS egress is unblocked. If by design, tag onsite-check.
• Two legacy EDR blocks: AMT-PC (Win7 SP1/PS2), SRV02 (Server 2008 R2) — need OS prereqs (SHA-2 servicing, WMF/.NET, VC++ runtime) or onsite/legacy-agent handling.
• Verify SS-SP1 (Safesite) reports into the EDR console — huntAgent runs locally but not visible under "sp1"; likely registered under its real hostname or awaiting approval.
• Bitdefender REMOVAL not started — per-client, after EDR confirmed ACTIVE. Glaztech stays BD.
• Confirm avInstalled enables via org policy for the newly-deployed agents (stale-field quirk — read the org policy, not the raw field).
• Autoenroll (Mon 06:00) will surface more offline GPS machines; re-run edr-deploy.py to sweep them (now includes Horseshoe with the vaulted key).
• Onsite-check list clients to locate/confirm-unmanaged: Little Hearts Little Hands, Residential & Renovation Engineering, Janet Altschuler, Business Services of Tucson, Marty Ryan.

Reference Information

• EDR install script (readable): raw.githubusercontent.com/Infocyte/PowershellTools/master/AgentDeployment/install_huntagent.ps1 — Install-EDR early-returns: line 63 (no URL), 69 (not admin), 95 (huntAgent service exists, skip unless -Force). All status/error output gated behind if ($Interactive).
• Tool: projects/gps-rmm-audit/tools/edr-deploy.py. Results: edr-deploy-results.md, edr-deploy-live.log, edr-deploy-run3.log.
• Deploy recipe + EDR skill: .claude/skills/datto-edr/ (edr.py deploy-cmd --group|--regkey, create-group, mint-key). URL must be full -URL "https://azcomp4587.infocyte.com", not -InstanceName (loose .com regex bug).
• Prior session log (this session, earlier): 2026-07-05-howard-cpqb-fixed-imcprintserver-tls-blocked.md.