From 563ff9e8fab3a017adac68f01f5596ff3f9e3ca2 Mon Sep 17 00:00:00 2001 From: Howard Enos Date: Thu, 25 Jun 2026 21:22:22 -0700 Subject: [PATCH] sync: auto-sync from HOWARD-HOME at 2026-06-25 21:21:56 Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-25 21:21:56 --- .bd2_ids.txt | 7 + .bd2_res.txt | 0 .../feedback_windows_pro_upgrade_billing.md | 17 ++- .watch_targets.py | 5 + .../docs/REMAINING-WORK-PLAN.md | 19 ++- ...-howard-edr-rollout-bitdefender-removal.md | 140 ++++++++++++++++++ errorlog.md | 6 + 7 files changed, 189 insertions(+), 5 deletions(-) create mode 100644 .bd2_ids.txt create mode 100644 .bd2_res.txt create mode 100644 .watch_targets.py create mode 100644 clients/cascades-tucson/session-logs/2026-06/2026-06-25-howard-edr-rollout-bitdefender-removal.md diff --git a/.bd2_ids.txt b/.bd2_ids.txt new file mode 100644 index 00000000..bd103932 --- /dev/null +++ b/.bd2_ids.txt @@ -0,0 +1,7 @@ +f269118d-93cf-48a3-96ae-5a9acaf154c9 BD:DESKTOP-KQSL232 +811e345f-ba2f-45a4-bd6e-4dd1ccf37793 BD:DESKTOP-MD6UQI3 +455d1284-febc-4280-b4df-f44be051120e BD:DESKTOP-TRCIEJA +3baffe0b-f17a-46b6-a418-60cf3802c11b BD:SALES4-PC +b0cb408c-a699-4737-a772-525503e2dab1 BD:Laptop4 +a4623704-ec6e-4855-8a37-e463734f4293 INSTALL:DESKTOP-F94M8UT +d1806aa3-9711-4f54-9a41-fd334efd8fd3 INSTALL:NurseAssist diff --git a/.bd2_res.txt b/.bd2_res.txt new file mode 100644 index 00000000..e69de29b diff --git a/.claude/memory/feedback_windows_pro_upgrade_billing.md b/.claude/memory/feedback_windows_pro_upgrade_billing.md index dbcd51df..e5138d46 100644 --- a/.claude/memory/feedback_windows_pro_upgrade_billing.md +++ b/.claude/memory/feedback_windows_pro_upgrade_billing.md @@ -20,6 +20,19 @@ Windows Home to Pro, invoice that customer $99 per machine — and name the mach - **Track activations** — each consumes one MAK count; watch the remaining allocation (exceeding it means contacting Microsoft to raise the count). -**Pending application:** the 5 Cascades machines upgraded 2026-06-24 ~6PM (SALES4-PC, NURSEASSIST, -MDIRECTOR-PC, LAPTOP-8P7HDSEI, MEMRECEPT-PC) = **5 x $99 = $495 to invoice Cascades** once activated. +**Note:** the ACG MAK is actually a **Windows Pro for Workstations** MAK — `slmgr /ipk ` retargets +the edition to `ProfessionalWorkstation` (a higher SKU than plain Pro, but fine for domain join). Flow that +works remotely as SYSTEM via RMM: `changepk.exe /productkey VK7JG-NPHTM-C97JM-9MPGT-3V66T` (generic Pro key +flips Core->Professional, no auto-reboot in SYSTEM context) -> **reboot once** to sync registry+licensing -> +`slmgr /ipk ` -> `/ato` (retry `/ato` if it hits a transient `0x8004FE92`). Some machines self-activate +Pro for FREE via a built-in digital entitlement after the edition flip — those consume NO MAK count and are +NOT billed. + +**Cascades application status (2026-06-25, done remotely):** +- **MEMRECEPT-PC + LAPTOP-8P7HDSEI** — activated via the MAK. **INVOICED 2026-06-25** on Syncro ticket **#32466** + (invoice #1650806091): 2 x $99 "Windows Pro Upgrade" lines (machine named on each) + 1.0h remote labor (drawn + from Cascades' prepay block, $0) = $215.23 w/ AZ tax. Created a reusable taxable **"Windows Pro Upgrade"** + Syncro product (id 23571919, $99) for this — use it for future Home->Pro upgrade billing. +- **MDIRECTOR-PC** — self-activated FREE via digital entitlement; NO MAK, NO charge. +- **NurseAssist** (offline; possible dupe of `Assistnurse-pc`) + **SALES4-PC** (Tamra departing, bypassed) — not done. See [[feedback_verify_live_before_acting]] for the verify-before-billing discipline. diff --git a/.watch_targets.py b/.watch_targets.py new file mode 100644 index 00000000..a6ef6804 --- /dev/null +++ b/.watch_targets.py @@ -0,0 +1,5 @@ +import json,sys +d=json.load(sys.stdin) +t={'DESKTOP-F94M8UT','NurseAssist','DESKTOP-KQSL232','DESKTOP-MD6UQI3','DESKTOP-TRCIEJA','SALES4-PC','Laptop4'} +on=[a['hostname'] for a in d if a.get('hostname') in t and a.get('status')=='online'] +print(','.join(sorted(on))) diff --git a/clients/cascades-tucson/docs/REMAINING-WORK-PLAN.md b/clients/cascades-tucson/docs/REMAINING-WORK-PLAN.md index eb252c2a..9487d81e 100644 --- a/clients/cascades-tucson/docs/REMAINING-WORK-PLAN.md +++ b/clients/cascades-tucson/docs/REMAINING-WORK-PLAN.md @@ -86,9 +86,22 @@ retire per-PC Synology Drive Client. LAPTOP-8P7HDSEI, MDIRECTOR-PC, MEMRECEPT-PC, NurseAssist, SALES4-PC. **Howard handling the Home->Pro upgrades himself, ONSITE** (decision 2026-06-25). - *2026-06-25 live re-check: the 6PM cron `ad0a56a9` never completed — all 5 still `EditionID=Core` - (Home), Licensed on Home keys, none half-upgraded. Remote job abandoned; Howard doing them onsite. - Next step for these 5 = domain-join once they read `EditionID=Professional`. ProductName reads - "Windows 10 Home" even on the Win11 boxes (stale registry string) — trust EditionID, not ProductName.* + (Home), Licensed on Home keys, none half-upgraded. ProductName reads "Windows 10 Home" even on the + Win11 boxes (stale registry string) — trust EditionID, not ProductName.* + - **DONE 2026-06-25 (~8:45 PM, remotely via RMM, no users logged in):** the 3 online Home boxes + upgraded Home->Pro. Process: `changepk.exe /productkey ` flips Core->Professional + (as SYSTEM it does NOT auto-reboot; registry vs licensing go out of sync — **reboot once to finalize**), + then activate. Results: + - **MDIRECTOR-PC** -> Professional, **self-activated FREE via a built-in Pro digital entitlement** + (no MAK used, no charge). READY to domain-join. + - **MEMRECEPT-PC** + **LAPTOP-8P7HDSEI** -> activated with the ACG MAK + (`infrastructure/windows-pro-mak`). NOTE: the MAK is a **Pro for Workstations** MAK — `/ipk` retargets + the edition to `ProfessionalWorkstation` (higher SKU, fine for domain join), `/dli` = Licensed, + VOLUME_MAK channel. **2 MAK counts consumed -> bill 2x $99 = $198 to Cascades** (line items name each + machine). MEMRECEPT needed an `/ato` retry (first attempt hit transient `0x8004FE92`). + - **Still pending:** NurseAssist (OFFLINE — and flagged as a possible dupe of `Assistnurse-pc`, verify + before upgrading) and SALES4-PC (bypassed — Tamra departing, repurpose TBD). + - Next step for the 3 upgraded boxes = **domain-join** (they now read `EditionID=Professional`/PfW). - **OneDrive KFM ON** (unlink before folder-redirect GPO): LAPTOP-8P7HDSEI, NurseAssist. - **Pending reboots + KFM unlinks: held for onsite** (Howard) — disruptive to clear remotely. - **LAPTOP-DRQ5L558** is off the Cascades network (8.8.8.8/1.1.1.1 DNS, no DC reachability) — diff --git a/clients/cascades-tucson/session-logs/2026-06/2026-06-25-howard-edr-rollout-bitdefender-removal.md b/clients/cascades-tucson/session-logs/2026-06/2026-06-25-howard-edr-rollout-bitdefender-removal.md new file mode 100644 index 00000000..2b90bc19 --- /dev/null +++ b/clients/cascades-tucson/session-logs/2026-06/2026-06-25-howard-edr-rollout-bitdefender-removal.md @@ -0,0 +1,140 @@ +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Audited Datto EDR coverage across all Cascades of Tucson devices in GuruRMM, reconciled it +against the Datto EDR (Infocyte/azcomp4587) agent inventory, checked every reachable device +for Bitdefender, then deployed EDR to the gaps and oversaw Bitdefender removal where it was +still active. Driven by the migration off Syncro-deployed Bitdefender onto Datto EDR/AV. + +Reconciliation: GuruRMM had 33 Cascades devices; Datto EDR had 27 agents (org +`2d5ea96e-3228-461b-9c60-13ae464b61d8`). Matching normalized hostnames found 8 RMM devices +with no EDR agent. A per-device Bitdefender sweep (services + uninstall registry + install +dir) over the 27 online machines found one machine with FULL active Bitdefender +(RECEPTIONIST-PC, both of its two physical boxes), six with only an orphaned +`C:\Program Files\Bitdefender` folder (BD already uninstalled, remnant dir), and the rest +clean. Six offline machines could not be checked. + +Deployment: pushed the Datto EDR agent to the 6 online, Bitdefender-clean, no-EDR machines +via the GuruRMM `/rmm` install one-liner with the existing Cascades registration key +`6qw68y2rwl`. All 6 installed (exit 0) and enrolled into the Cascades EDR org (count 27->33). + +Bitdefender removal: RECEPTIONIST-PC is two distinct physical boxes sharing a hostname +(serials MJ0KQH4R and MJ0KQHNP), both Syncro-deployed BEST 8.26.6.644 on policy "GPS Default" +with anti-tampering on and NO uninstall password. The GravityZone API cannot uninstall +(createUninstallTask is dead in this API version) and masks the uninstall password +(`passwordConfig.value` returns ""); no console creds were available locally (SOPS has only the +API key; op CLI not installed). Howard ran the GravityZone console "Uninstall client" task on +both boxes; verified BD fully removed on both (services gone, dir gone, app entry gone, no +reboot needed) while the EDR agent stayed healthy. The EDR check during removal exposed that +only ONE of the two physical RECEPTIONIST-PC boxes actually had EDR (the hostname-dedup had +masked the other's gap); installed EDR on the second box (Cascades EDR 33->34, two +`receptionist-pc` entries). + +Cleanup + remaining: deleted the 6 orphaned Bitdefender folders (safety-checked: skip if any BD +service/app present); queued BD-aware EDR installs to the 2 offline no-EDR machines +(DESKTOP-F94M8UT, NurseAssist) and BD-checks to the 5 remaining offline has-EDR machines; all +run on reconnect. Howard ran a wake command but no targets reconnected during the session. A +background watcher (`bfm81iqdz`) was left polling GuruRMM to process machines as they wake. + +## Key Decisions + +- Reconciled by normalized hostname across two systems of record (GuruRMM = "all devices", + Datto EDR = "has agent") rather than trusting either alone; this surfaced both the 8 missing-EDR + devices and (via serial check) the duplicate-hostname masking on RECEPTIONIST-PC. +- Used the existing Cascades registration key `6qw68y2rwl` (target group + `1dbd2b02-f7df-45d0-a7f2-18667f48447f`) so new agents land in the correct org/group; did not mint a new key. +- Refused to brute-force tamper-protected Bitdefender from the endpoint; recommended (and Howard + used) the GravityZone console "Uninstall client" task as the clean, server-side, deregistering path. +- Made the queued offline EDR installs BD-aware (skip if active BD services found) so they never + stack EDR/Datto-AV on top of an active Bitdefender when the machine reconnects. +- Made the leftover-folder deletion safety-checked (only delete `C:\Program Files\Bitdefender` + when no BD service/app is present). +- Left a background watcher instead of busy-polling, since woken machines were not reconnecting. + +## Problems Encountered + +- BD-check result file mis-parsed: hostnames carried an embedded CR (`\r`) from a Windows + CRLF round-trip (`python print` -> file -> bash `read`), and Python universal-newline mode split + lines at the CR, collapsing dict keys. Fixed by reading bytes and stripping `\r` before splitting. +- `/tmp` read-back mismatch (Git-Bash vs Python) recurred; switched to repo-relative scratch files. +- `edr.py agent <8-char-id>` returned HTTP 500 (API needs full UUID); resolved EDR agent ids by + client-side prefix match over the full 216-agent list. +- GravityZone API could neither uninstall nor reveal the uninstall password (createUninstallTask + dead; passwordConfig value masked); resolved via the console uninstall task (Howard). +- Discovered RECEPTIONIST-PC is two physical machines sharing a hostname; only one had EDR. The + dedup-by-hostname in the reconciliation had hidden the second box's gap. Caught it during BD-removal + verification and installed EDR on the second box. +- cwd drift: a prior `cd` into the skill scripts dir made a later relative `rmm-auth.sh` path fail; + re-ran from repo root. + +## Configuration Changes + +- No repo file changes this session (operational work against GuruRMM, Datto EDR, GravityZone). +- Endpoint changes (Cascades fleet): EDR agent installed on 7 machines; 6 orphaned BD folders + deleted; BD removed from 2 RECEPTIONIST-PC boxes (via GravityZone, Howard-initiated). + +## Credentials & Secrets + +- Datto EDR Cascades registration key used for installs: `6qw68y2rwl` (target group + `1dbd2b02-f7df-45d0-a7f2-18667f48447f`). Other Cascades keys present: `911xpmkfta`, `b7cmnghlgh`. + These are agent enrollment keys (auto-approve into the group), not secrets to vault. +- Datto EDR API token: vault `msp-tools/datto-edr.sops.yaml` credentials.api_token (unchanged). +- GravityZone API key: vault `msp-tools/gravityzone.sops.yaml` (API only; no console login stored — + console uninstall needs a human-held GravityZone console login not in SOPS, and op CLI is not + installed on Howard-Home). +- Bitdefender uninstall password: NONE set on the "GPS Default" policy (confirmed by Howard in console). + +## Infrastructure & Servers + +- GuruRMM API: http://172.16.3.30:3001 (auth via vault infrastructure/gururmm-server.sops.yaml). +- Datto EDR (Infocyte HUNT): https://azcomp4587.infocyte.com ; Cascades org + `2d5ea96e-3228-461b-9c60-13ae464b61d8` (27->34 agents); Cascades target group + `1dbd2b02-f7df-45d0-a7f2-18667f48447f`. +- Bitdefender GravityZone: cloud.gravityzone.bitdefender.com ; Cascades company + `66b0448e1e0441d02508bad8` ; policy "GPS Default" `5c42940b6e16d61a0c8b4568` (antiTampering on, + no uninstall password). RECEPTIONIST-PC GZ endpoints `66b04593e14f46ee79b1c87f`, + `66b045ee2f4dee3f01f54630` ; BEST 8.26.6.644. +- RECEPTIONIST-PC physical boxes: serial MJ0KQH4R (RMM 57f19e17-8792-46cc-b9fd-f1909836cd17, IP + 192.168.3.187) and MJ0KQHNP (RMM 2e8d8b73-82f6-4151-a3ce-879c55de4b82). Both Syncro-managed. + +## Commands & Outputs + +- Cascades RMM devices: `bash .claude/scripts/rmm-search.sh -c cascades --json` (33 devices). +- Cascades EDR agents: `edr.py agents --org 2d5ea96e-... ` (27 -> 34). +- EDR install one-liner (per machine via /rmm): + `(new-object Net.WebClient).DownloadString("https://raw.githubusercontent.com/Infocyte/PowershellTools/master/AgentDeployment/install_huntagent.ps1") | iex; Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey 6qw68y2rwl` + -> "Installed RTS agent to C:\Program Files\infocyte\agent\agent.exe" (exit 0). +- BD detect (per machine): services `^EP(Security|Protected|Update|Redline|Integration)Service$` + + uninstall-registry DisplayName match `Bitdefender|GravityZone` + `Test-Path 'C:\Program Files\Bitdefender'`. +- GravityZone policy uninstall-password field: `gz.py policy 5c42940b6e16d61a0c8b4568 --json` -> + `settings.general.advanced.passwordConfig = {"profile":3,"value":""}` (value always masked by API). + +## Pending / Incomplete Tasks + +- QUEUED (auto-run on reconnect; all 7 still offline at session end): + - EDR install (BD-aware): DESKTOP-F94M8UT (RMM 675311a1-...), NurseAssist (fc88f14b-...). + - BD-check: DESKTOP-KQSL232 (f1674059-...), DESKTOP-MD6UQI3 (99d7c8a7-...), + DESKTOP-TRCIEJA (c9bf1a2d-...), SALES4-PC (975f70d8-...), Laptop4 (7a23fa6c-...). + - Background watcher `bfm81iqdz` polling for reconnects (40 min window). +- laptop3 (EDR agent active 2026-06-26, v5552) has NO matching GuruRMM agent -> install RMM agent + or reconcile hostname (inverse coverage gap). +- Stale EDR agents to confirm/remove: laptop1 (last seen 2026-05-08, v4377), cascades-laptop + (2026-06-23, v5409). +- Confirm Cascades is removed from Syncro's Bitdefender deployment so BD does not redeploy onto the + cleaned machines (Syncro AV management is GUI-only). +- DESKTOP-F94M8UT (last seen 06-23) and DESKTOP-KQSL232 (05-29) look powered-off/off-network; WoL + did not reach them this session. + +## Reference Information + +- Datto EDR skill: `.claude/skills/datto-edr/` ; GravityZone skill: `.claude/skills/bitdefender/` + (gz.py; createUninstallTask is DEAD in this API version -> console-only uninstall). +- Memory: `.claude/memory/reference_datto_edr_detection_behavior.md`. +- Earlier same-day work (datto-edr skill build + AV/EDR detection proof) logged in + `session-logs/2026-06/2026-06-25-howard-datto-edr-skill-and-lifecycle-test.md`. +- Cascades EDR now 34 agents; 8 original gaps -> 7 closed (6 online + RECEPTIONIST box2), 2 queued + (offline), net remaining gap = the 2 offline + laptop3 RMM-side. diff --git a/errorlog.md b/errorlog.md index 46120806..8e4fac40 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · 2026-06-26 | GURU-5070 | remediation-tool | [correction] claimed no tier has mail read/write and reached for an EWS workaround; correct: exchange-op (Exchange Operator app) = Exchange Administrator role + full_access_as_app + Exchange.ManageAsApp = full all-access for ANY mailbox/Exchange op including moving mail [ctx: tenant=tedards.net recurring=true ref=feedback_exchange_op_all_access] +2026-06-26 | Howard-Home | synology/ssh | syno-ssh recipe 'run' failed (rc=255) [ctx: host=192.168.0.120] + +2026-06-26 | Howard-Home | synology/ssh | syno SSH connect/auth failed (rc=255) [ctx: host=192.168.0.120 vp=clients/cascades-tucson/synology-cascadesds.sops.yaml] + +2026-06-26 | Howard-Home | rmm/bash-quoting | [friction] doubled single-quotes ('') inside a single-quoted bash $SCRIPT collapse the string, leaving a PowerShell registry path with a space unquoted (Windows NT) -> read fails, ErrorActionPreference=Stop aborts before changepk. Fix: use double-quotes for paths inside the single-quoted bash heredoc, never doubled single-quotes. [ctx: ref=feedback_windows_quote_stripping] + 2026-06-25 | GURU-5070 | vault/display | [friction] echoing a vault entry, sed line-redaction missed the multi-line JSON private_key (matched 'key:' not 'private_key": "') and printed the full SA private key; when displaying vault entries use vault.sh get-field for named fields or drop the entire credentials: block, never a line-regex over JSON credential blobs 2026-06-26 | GURU-BEAST-ROG | email-investigation | [correction] assumed tedards.net also uses GuruProtect/Inky; correct: only ACG uses Inky for inbound. Tedards routes directly to Exchange Online.