164 lines
10 KiB
Markdown
164 lines
10 KiB
Markdown
## User
|
|
- **User:** Howard Enos (howard)
|
|
- **Machine:** Howard-Home
|
|
- **Role:** tech
|
|
|
|
## Session Summary
|
|
|
|
Continued the GPS->GuruRMM coverage audit. The session had two arcs: (1) closing out the
|
|
two straggler machines from the Monday queue (CP-QB, IMC-PRINTSERVER), then (2) executing
|
|
the full Bitdefender->Datto EDR AV switch across every reachable GPS-client machine.
|
|
|
|
CP-QB (Curtis Plumbing) was fixed and enrolled: its WMI class store was corrupt
|
|
(`Win32_Processor` = Invalid class, broken since ~2022), which killed the universal
|
|
installer's architecture-detection call and also froze its ScreenConnect inventory.
|
|
`mofcomp cimwin32.mof` restored the class; the installer then ran clean and the agent
|
|
enrolled and was reassigned to Curtis Plumbing. IMC-PRINTSERVER was diagnosed but left
|
|
blocked: the GuruRMM agent was manually installed over the LAN (PowerShell is dead on the
|
|
box, SMB hangs, the SC extension eats chained/multi-line commands, so the binary was served
|
|
over HTTP from IMC1 and installed with plain cmd), but the box drops ALL outbound TLS (even
|
|
google.com), so the installed agent cannot reach the server. DM'd Mike to ask whether that
|
|
lockdown is by design or should be cleared.
|
|
|
|
The main work was the AV switch. Built `tools/edr-deploy.py` — a throttled deployer that
|
|
recomputes coverage from a live Datto EDR agent dump each run, targets ONLINE Windows RMM
|
|
agents of GPS clients that have a vaulted registration key and are not already in EDR, and
|
|
pushes the official Install-EDR one-liner through the RMM one machine at a time with a 20s
|
|
gap (retry-on-5xx, poll-to-completion). Glaz-Tech excluded by design. Run 1 hit 79 machines
|
|
across 24 clients: 74 installed cleanly. Follow-up runs cleared the rest except two legacy
|
|
boxes.
|
|
|
|
The five apparent failures were run to ground individually. Install-EDR is silent in
|
|
non-interactive mode (every status/error line is gated behind `if ($Interactive)`), so a
|
|
"completed / blank output" RMM result is meaningless — the visible "Installed RTS agent"
|
|
text actually comes from `agent.windows.exe` itself. Investigating: DForth-Shipp (Win11)
|
|
and SS-SP1 (Server 2016) already had huntAgent running (false negatives); AD1 (Server 2016)
|
|
had a broken partial install (stale stopped service, no binary) that Install-EDR silently
|
|
skipped, fixed with `-Force`; AMT-PC (Win7 SP1 / PowerShell 2.0) and SRV02 (Server 2008 R2)
|
|
are genuine legacy failures where the modern agent binary crashes or no-ops on execution.
|
|
Also deployed EDR to DESKTOP-PL2RCGL (Robyn Pittman, the 7/4 mid-delivery retry) once it
|
|
came online. Datto EDR authorized-agent count rose 228 -> 302.
|
|
|
|
## Key Decisions
|
|
|
|
- Deferred Bitdefender REMOVAL entirely — EDR-first, per-client, only after that client's
|
|
EDR is confirmed ACTIVE. Nothing removed this session.
|
|
- Kept the deploy strictly throttled (sequential, 20s gaps, single command in flight) per
|
|
Howard's "go slow, don't hammer the RMM server" instruction.
|
|
- Built a reusable coverage-recomputing tool rather than a one-shot script, so re-runs are
|
|
idempotent and safe (targets only genuinely-missing online machines each time).
|
|
- Classified AMT-PC and SRV02 as legacy-OS blocked rather than continuing to brute-force
|
|
installs on incompatible boxes; flagged for prerequisites/onsite.
|
|
- Created an onsite-check list (`onsite-check-list.md`) for machines/clients with no SC, no
|
|
Syncro checkin, and no Bitdefender — to be handled at the next site visit, not chased
|
|
remotely (Howard's rule).
|
|
- Added Horseshoe's existing EDR key to the vault rather than minting a new one (org+group
|
|
already existed; only the vault entry was missing).
|
|
|
|
## Problems Encountered
|
|
|
|
- CP-QB universal installer failed: corrupt WMI (`Win32_Processor` Invalid class). Fixed via
|
|
`mofcomp %windir%\system32\wbem\cimwin32.mof` (verifyrepository reported CONSISTENT — it
|
|
does not catch a missing class).
|
|
- IMC-PRINTSERVER cannot connect: drops all outbound TLS (Cloudflare hosts, rmm-api direct
|
|
IP, and google.com — ClientHello then silence). SC survives because its relay is not TLS.
|
|
Agent installed but dormant. Escalated to Mike (by-design vs fix). No reinstall needed once
|
|
TLS is unblocked.
|
|
- Five EDR "failures" were mostly false negatives. Root-caused via the fact that Install-EDR
|
|
is silent unless `-Interactive`; used `-Interactive *> file` + read-back and direct
|
|
service/binary checks. Two real legacy blocks remain (AMT-PC Win7/PS2, SRV02 2008 R2).
|
|
- AD1 had a stale huntAgent service with no binary; Install-EDR's "already installed"
|
|
early-return silently skipped it. `-Force` reinstalled successfully.
|
|
- edr.py needed `DATTO_EDR_TOKEN` exported from the vault (`msp-tools/datto-edr.sops.yaml`
|
|
field `credentials.api_token`) — the repo skill copy path differs from the user-home copy.
|
|
|
|
## Configuration Changes
|
|
|
|
- Created `projects/gps-rmm-audit/tools/edr-deploy.py` — throttled BD->EDR deployer.
|
|
- Created `projects/gps-rmm-audit/onsite-check-list.md` — no-remote-channel machines/clients.
|
|
- Created `projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-cpqb-fixed-imcprintserver-tls-blocked.md`
|
|
(earlier in the session; CP-QB + IMC-PRINTSERVER detail).
|
|
- Appended to `projects/gps-rmm-audit/tracker.md` — CP-QB/IMC-PRINTSERVER outcome, onsite
|
|
group pointer, and the full AV-switch deploy section.
|
|
- Vault: added `Horseshoe Management: jthm976pwo` to `msp-tools/datto-edr-regkeys.sops.yaml`.
|
|
- Deploy artifacts (gitignored/scratch): `edr-deploy-live.log`, `edr-deploy-run3.log`,
|
|
`edr-deploy-results.md`, `.edr-agents-now.json` under `projects/gps-rmm-audit/`.
|
|
|
|
## Credentials & Secrets
|
|
|
|
- Datto EDR per-client registration keys — vault `msp-tools/datto-edr-regkeys.sops.yaml`
|
|
(field map under `credentials:`). Used verbatim as `Install-EDR -RegKey`. Newly added this
|
|
session: **Horseshoe Management = jthm976pwo** (key already existed in Datto; only the vault
|
|
entry was missing). Keys used directly this session incl. Arizona Medical Transit
|
|
`arizonam01`, Dataforth `27lzj6npdb`, Robyn Pittman `pittman001`.
|
|
- Datto EDR API token — vault `msp-tools/datto-edr.sops.yaml` field `credentials.api_token`
|
|
(export as `DATTO_EDR_TOKEN` for `edr.py`).
|
|
- ScreenConnect API secret — vault `msp-tools/screenconnect.sops.yaml` field
|
|
`credentials.api_secret`.
|
|
- GuruRMM staging site key `DARK-STORM-3150` — vault
|
|
`infrastructure/gururmm-staging-site.sops.yaml`.
|
|
|
|
## Infrastructure & Servers
|
|
|
|
- GuruRMM API: `http://172.16.3.30:3001` (via `rmm-auth.sh`); dashboard rmm.azcomputerguru.com;
|
|
agent WS `wss://rmm-api.azcomputerguru.com/ws` (rmm-api resolves to direct IP 72.194.62.10,
|
|
NOT Cloudflare). GuruRMM root Linux agent id `9b92b187-98c7-41b0-9e97-1698d263c42d`
|
|
(hostname `gururmm`) used to read the server's own nginx access log.
|
|
- Datto EDR: `https://azcomp4587.infocyte.com`; agent binary
|
|
`https://s3.us-east-2.amazonaws.com/infocyte-support/executables/agent.windows64.exe`
|
|
(v3.17.1.5409). Authorized agents 228 -> 302 this session.
|
|
- IMC-PRINTSERVER: Win10 22H2, 192.168.0.79/24, gw 192.168.0.254, DNS/DHCP = IMC1
|
|
(192.168.0.2), MAC A0-1E-0B-11-60-1D. SC session ccbd66a7-f94a-4d87-a18b-5f81a30784d0.
|
|
IMC1 RMM agent id 88cbf7c0-abfa-4f12-846c-96274f718bff.
|
|
- Straggler agent ids: AMT-PC 2471f636-c73d-4d90-aabf-7da42a58917d; SRV02
|
|
4628e665-ac21-4ff7-bde2-c649682f6241; AD1 bf7bc5ee-4167-4a62-912a-c88b11a5943d;
|
|
DForth-Shipp 9974ca3c-02d0-4632-8532-508d34344b8c; SS-SP1
|
|
0db6ed6b-48a6-4dcd-9366-e9a67eefa3d2; DESKTOP-PL2RCGL b12cd995-016e-4eff-bdcb-1033aa9f3e78.
|
|
- Horseshoe EDR: org e40037bf-37ee-4767-9e36-889253df03de, group
|
|
49d5fbf2-f377-4ede-9e4c-1c8655939b63.
|
|
|
|
## Commands & Outputs
|
|
|
|
- EDR deploy (throttled): `RMM=.. TOK=.. python projects/gps-rmm-audit/tools/edr-deploy.py
|
|
--dry | --live [--gap 20]`.
|
|
- Official install one-liner (per box, via RMM powershell elevated):
|
|
`[System.Net.ServicePointManager]::SecurityProtocol=[Enum]::ToObject([System.Net.SecurityProtocolType],3072);
|
|
(new-object Net.WebClient).DownloadString("https://raw.githubusercontent.com/Infocyte/PowershellTools/master/AgentDeployment/install_huntagent.ps1")
|
|
| iex; Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey <key>`.
|
|
- True-state check on a box: `Get-Service huntAgent` + `Test-Path "C:\Program
|
|
Files\infocyte\agent\agent.exe"`. Verbose install: add `-Interactive *> C:\Windows\Temp\x.log`
|
|
then read the file (Install-EDR is silent otherwise).
|
|
- `-Force` reinstalls over a stale huntAgent service (fixed AD1: log "[Success] Installation
|
|
Succeeded! Agent associated to https://azcomp4587.infocyte.com").
|
|
- AMT-PC diagnosis: `Get-CimInstance` "not recognized" => PowerShell 2.0 => legacy Win7;
|
|
agent.windows64.exe downloads (38472512 bytes) but crashes the process tree on exec.
|
|
|
|
## Pending / Incomplete Tasks
|
|
|
|
- IMC-PRINTSERVER: awaiting Mike's ruling (TLS lockdown by design vs fix). Agent installed +
|
|
dormant; self-enrolls when TLS egress is unblocked. If by design, tag onsite-check.
|
|
- Two legacy EDR blocks: AMT-PC (Win7 SP1/PS2), SRV02 (Server 2008 R2) — need OS prereqs
|
|
(SHA-2 servicing, WMF/.NET, VC++ runtime) or onsite/legacy-agent handling.
|
|
- Verify SS-SP1 (Safesite) reports into the EDR console — huntAgent runs locally but not
|
|
visible under "sp1"; likely registered under its real hostname or awaiting approval.
|
|
- Bitdefender REMOVAL not started — per-client, after EDR confirmed ACTIVE. Glaztech stays BD.
|
|
- Confirm `avInstalled` enables via org policy for the newly-deployed agents (stale-field
|
|
quirk — read the org policy, not the raw field).
|
|
- Autoenroll (Mon 06:00) will surface more offline GPS machines; re-run edr-deploy.py to
|
|
sweep them (now includes Horseshoe with the vaulted key).
|
|
- Onsite-check list clients to locate/confirm-unmanaged: Little Hearts Little Hands,
|
|
Residential & Renovation Engineering, Janet Altschuler, Business Services of Tucson,
|
|
Marty Ryan.
|
|
|
|
## Reference Information
|
|
|
|
- EDR install script (readable): raw.githubusercontent.com/Infocyte/PowershellTools/master/AgentDeployment/install_huntagent.ps1
|
|
— Install-EDR early-returns: line 63 (no URL), 69 (not admin), 95 (huntAgent service exists,
|
|
skip unless -Force). All status/error output gated behind `if ($Interactive)`.
|
|
- Tool: `projects/gps-rmm-audit/tools/edr-deploy.py`. Results: `edr-deploy-results.md`,
|
|
`edr-deploy-live.log`, `edr-deploy-run3.log`.
|
|
- Deploy recipe + EDR skill: `.claude/skills/datto-edr/` (`edr.py deploy-cmd --group|--regkey`,
|
|
`create-group`, `mint-key`). URL must be full `-URL "https://azcomp4587.infocyte.com"`, not
|
|
`-InstanceName` (loose `.com` regex bug).
|
|
- Prior session log (this session, earlier): 2026-07-05-howard-cpqb-fixed-imcprintserver-tls-blocked.md.
|