azcomputerguru/claudetools
1
0
Fork 0
Code Issues 58 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
567fd8dd8c1993e18575bbfc2a0b569f4026825e
claudetools/projects/gps-rmm-audit/session-logs/2026-07/2026-07-03-howard-gps-rmm-coverage-audit.md
Howard Enos 567fd8dd8c sync: auto-sync from HOWARD-HOME at 2026-07-04 20:44:56 
Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-04 20:44:56
2026-07-04 20:45:25 -07:00

39 KiB
Raw Blame History

User

  • User: Howard Enos (howard)
  • Machine: Howard-Home
  • Role: tech

Session Summary

Ran a full audit of every Syncro business paying for GPS (Guru Protection Service) and reconciled it against GuruRMM coverage. Started by identifying the 44 GPS clients (40 active, 4 paused) from Syncro recurring schedules, then built a machine-readable reconciliation: Syncro billed GPS device counts (the "should have") vs live GuruRMM /api/agents (the reality), cross-checked against Datto EDR and Bitdefender for AV/backup context. Produced a durable tracker at projects/gps-rmm-audit/tracker.md splitting clients into Bucket A (7 count-match), Bucket B (8 present-but-short), and Bucket C (25 with no RMM org).

Bucket A was verified end-to-end across all layers: machine reconciliation, backups (B2/MSP360), AV (Datto EDR + Bitdefender — coverage is split across both tools), vault entries, and wiki (created the missing Arizona Medical Transit article). Surfaced several findings: Dataforth +8 machines over billed, Cascades duplicate agent + unbilled backup bucket, Len's LAB-SVR offline since Jun 18, AMT billed backup with no B2 destination. Backup billing mismatches were flagged and held for Winter per Howard's instruction (an early DM to Winter was retracted via the bot).

Fixed a data error in RMM: the org "Russo, Steve" was mislabeled — it is Russo Law Firm (Steve Russo owner, Shannon Trionfo main contact); renamed via PUT /api/clients/:id. Captured a standing AV strategy: migrate Bitdefender -> Datto EDR for all clients except Glaztech and Dataforth.

Bucket B reconciled with a RMM-vs-Datto-vs-Bitdefender coverage matrix, cleanly splitting into real deploy gaps (IMC, Safesite, Horseshoe, Grabb, QWM) and billing flags (Jimmy, Stamback, Glaz-Tech). Built and registered a daily Windows scheduled task (GPS-RMM-Progress, 8:07am) that compares live RMM counts to GPS targets and DMs Howard the remaining gaps until complete; verified working (baseline 46/189 devices, 32 clients short).

Executed the IMC deploy as the template. Determined the true fleet (~22 active AD/Bitdefender machines vs 1 in RMM). Remote-exec from the DC failed on Win10/11 client defaults (DCOM firewalled, schtasks version-rejected by Win11, WinRM off, SYSTEM can't create GPOs, SSH route not up). The working channel was ScreenConnect send-command (runs as SYSTEM on the guest). Pushed the site installer to 20 of 21 targets; RMM IMC agents climbed from 1 to 12 (online now, offline queued in SC). Saved the deploy pattern as reusable memory.

Key Decisions

  • Built a machine-readable targets.json rather than parsing the markdown tracker daily (fragile).
  • AV coverage must be checked in BOTH Datto EDR and Bitdefender before declaring a gap — coverage is split (Datto for big clients, Bitdefender for small).
  • Bitdefender inventory is now a discovery source only (which machines exist), not a coverage target, given the Bitdefender->Datto EDR migration (except Glaztech + Dataforth).
  • Daily reminder implemented as a LOCAL Windows scheduled task (has vault/RMM/Discord creds) rather than a cloud routine (which lacks vault access).
  • IMC deploy via ScreenConnect, not GPO — GPO's only advantage (no stored cred) evaporated once SYSTEM proved unable to create GPOs, and ScreenConnect send-command runs as SYSTEM with no creds at all.
  • Any DA-password use through the RMM channel was immediately scrubbed via DELETE /api/commands/:id.

Problems Encountered

  • RMM org "Russo, Steve" mislabeled -> renamed to Russo Law Firm (owner/contact confirmed by Howard).
  • Winter DM sent before Howard said "hold until verified" -> deleted from her DM channel via the bot (HTTP 204).
  • Remote agent deploy from the DC failed across four methods (New-PSDrive, WMI/DCOM, schtasks/S, sc.exe) due to Win11 client firewall/version defaults -> pivoted to ScreenConnect send-command, which worked.
  • iconv absent in Git-Bash -> computed base64 (UTF-16LE) for the encoded command via py -c.
  • schtasks /TR has a 261-char cap -> used bare powershell -enc instead of the full inline command.

Configuration Changes

  • Created projects/gps-rmm-audit/tracker.md — master audit tracker/checklist.
  • Created projects/gps-rmm-audit/targets.json — machine-readable enrollment targets.
  • Created .claude/scripts/gps-rmm-progress-check.sh — daily progress check + Discord DM.
  • Created wiki/clients/arizona-medical-transit.md — new client wiki article.
  • Created .claude/memory/project_av_migration_bitdefender_to_edr.md + .claude/memory/reference_rmm_deploy_via_screenconnect.md (+ MEMORY.md index lines).
  • Registered Windows scheduled task GPS-RMM-Progress (daily 8:07am, Howard-Home).
  • RMM: renamed client b83b0c29-81e6-4ce4-98ec-74639998d2ce "Russo, Steve" -> "Russo Law Firm".
  • RMM: enrolled ~11 new IMC agents (deploy in progress).
  • errorlog.md: one [correction] entry (Russo).

Credentials & Secrets

  • No new credentials created. Used existing vault entries: clients/imc/imc1.sops.yaml (IMC\guru Domain Admin — used transiently for DC deploy attempts, scrubbed from RMM logs), clients/imc/gururmm-site-main.sops.yaml (site code INNER-BRIDGE-8354).
  • Note: IMC1 SSH password-fallback is stored in clients/imc/imc1.sops.yaml (vault) — not reproduced here.

Infrastructure & Servers

  • GuruRMM API: http://172.16.3.30:3001 (auth via vault infrastructure/gururmm-server.sops.yaml).
  • IMC1 = Primary DC, domain IMC.local, 192.168.0.2, Server 2016. GuruRMM client_id 4fb591e7-... / agent 88cbf7c0-abfa-4f12-846c-96274f718bff. IMC RMM site IMCMain / INNER-BRIDGE-8354.
  • IMC reachable only via Tailscale subnet router pfsense-2 (100.119.153.74) advertising 192.168.0.0/24 — route not currently accepted on Howard-Home.
  • ScreenConnect: https://computerguru.screenconnect.com (RESTful API Manager ext) — deploy channel for IMC.
  • Datto EDR orgs: Instrumental Music Center not a distinct org (agents land in Default RMM Org); Dataforth 51, Cascades 34, Safesite 48, Horseshoe 6, Glaztech 5.

Commands & Outputs

  • Deploy one-liner (per client, site-coded): irm 'https://rmm.azcomputerguru.com/install/<SITE-CODE>/windows'|iex — pushed via ScreenConnect send-command as powershell -NoProfile -ExecutionPolicy Bypass -EncodedCommand <base64-UTF16LE>.
  • Daily check: bash .claude/scripts/gps-rmm-progress-check.sh [--dry-run]. Baseline: 46/189 devices in RMM, 32 clients short.
  • RMM command scrub: DELETE /api/commands/:id -> HTTP 204.

Pending / Incomplete Tasks

  • IMC: ~8 offline machines queued in SC (install on reconnect); IMC-L1-GRAPHICS has no SC session (stale, handle separately). Verify final 20/20 via daily check.
  • Bucket B remaining real-deploy clients: Safesite (~6+), Horseshoe (~5-8), Grabb & Durando (~3), Quantum Wealth (~1) — same ScreenConnect channel.
  • Bucket B billing flags (Jimmy, Stamback, Glaz-Tech) — document for Winter/Mike, no action.
  • Bucket C: 25 clients with no RMM org — onboard + deploy.
  • AV migration (Bitdefender -> Datto EDR, except Glaztech + Dataforth) — after RMM everywhere.
  • Backup billing flags (AMT/Cascades/Len's) — send Winter the full verified list once complete.
  • Task #4: confirm Glaz-Tech 159 GPS billing anomaly with Mike.

Reference Information

Update: 17:15 PT — Bucket B enrollment push (ScreenConnect)

Continued the deploy using the proven ScreenConnect send-command channel across the remaining Bucket B short clients. Enrolled everything reachable via SC; flagged the rest for a later come-back pass.

Results (RMM agent count before -> now):

  • Instrumental Music Center: 1 -> 12 (~8 offline queued in SC; IMC-L1-GRAPHICS no SC session)
  • Horseshoe Management: 1 -> 4 (site GOLD-OCEAN-4982; hsm-cathy + desktop-jk4e68n still installing)
  • Quantum Wealth Management: 2 -> 3 (target met; QUANTUMSERVER + DESKTOP-K89A8CF, site GREEN-CLOUD-1199)
  • Safesite: 31 -> 34+ (20 gap machines pushed to the "Unknown" catch-all site LIGHT-CLOUD-3585; 3 had no SC session)
  • Grabb & Durando: 9 -> 9 (real gap ~3; those machines + HOMEPC have no SC session; most BD "gap" entries are stale/duplicate)

Key findings:

  • ScreenConnect coverage VARIES per client — universal on IMC/Horseshoe/QWM/Safesite, sparse on Grabb.
  • Safesite has ~48 real machines in Datto vs 37 billed -> likely under-billed AND under-deployed. Deployed to "Unknown" site because the 3-site split (Bell/Glendale/Unknown) can't be mapped from asset-tag hostnames; needs re-siting.
  • Jimmy Company and Stamback Septic are billing flags (1 / ~2 real machines vs 12 / 8 billed) — nothing to enroll.

Come-back pass items (recorded in tracker.md): Safesite re-siting from Unknown -> Bell/Glendale; Safesite under-billing reconcile; Grabb ~3 no-SC machines + HOMEPC site; IMC-L1-GRAPHICS; billing flags to Winter (Jimmy/Stamback/Glaz-Tech + AMT/Cascades/Len's backups); Bucket C (25 clients) need /rmm onboard before deploy.

Site codes used this session: IMC INNER-BRIDGE-8354; Horseshoe GOLD-OCEAN-4982; QWM GREEN-CLOUD-1199; Safesite Unknown LIGHT-CLOUD-3585 (Bell RED-HAWK-6595, Glendale SWIFT-OCEAN-8321); Grabb Main LIGHT-PEAK-6399 (Bob's House LIGHT-GATE-7086, Jeff's House UPPER-FALCON-8240).

Update: 17:45 PT — Bucket C onboard + deploy

Wrote a reusable helper projects/gps-rmm-audit/tools/bucketc-onboard-deploy.sh (onboard RMM client+site -> vault key -> deploy agent via ScreenConnect to BD-company machines with SC sessions). Ran it across Bucket C.

Onboarded 16 new GuruRMM clients (client+site created, enrollment key vaulted at clients//gururmm-site-main.sops.yaml) and deployed ~44 agents via ScreenConnect: Reliant (CALM-HAWK-3954, 4), Curtis Plumbing (SILVER-WOLF-6785, 4), PUTT Land Surveying (EAST-CASTLE-3313, 3), Prairie Schooner (UPPER-HARBOR-4168, 3), T & C Sorensen (IRON-FORGE-1700, 4), Zeus Nestora (GREEN-TIGER-6194, 3), Brett Interiors (IRON-EAGLE-4784, 4), Bill Tedards (CALM-PEAK-4628, 2 from Datto), Design and Brand Envoys (SOUTH-STAR-8736, 3), Heieck (WILD-MOON-9773, 0 - no SC), Multicultural Counseling (EAST-OCEAN-2818, 3), MVAN (LOWER-FORGE-6736, 1), The Marc Group (SILVER-OCEAN-6422, 2), Mineralogical Record (BLUE-MOON-8542, 5 BD+Datto), Pro-Tech Services (INNER-GATE-4746, 2, pre-existing), Inside Track (CALM-GATE-2273, 1).

Discovery source = Bitdefender company, falling back to Datto EDR where BD is empty (Bill Tedards, Mineralogical extras). Bash tool default timeout is 2 min; raise to up to 600000ms for long batches (the 10-client batch was cut off and finished separately - the helper's existing-client guard made re-runs idempotent).

Not onboarded (no BD/Datto machines found - come-back to locate or confirm unmanaged): Little Hearts, Janet Altschuler, Business Services of Tucson, Andy's Mobile Fuel, Gary A Hartman LLC, Marty Ryan, Residential and Renovation Engineering, Ridgetop Group, Robyn Pittman.

Reliant caveat: its BD company mixes Reliant + Farwest (FW*) + Wilcox (WILCOX*) - only Reliant machines targeted.

Update: 18:20 PT — Syncro authoritative reconciliation + needs-ScreenConnect list

Pivoted the missing-machine discovery to Syncro assets (the authoritative managed-device inventory) instead of Bitdefender/Datto. Every "no-SC" machine is a live Syncro Device asset. Built projects/gps-rmm-audit/tools/needs-sc.py (Python - avoids bash while-read/stdin + Windows cp1252 gotchas) which reconciles Syncro assets vs live GuruRMM agents per GPS client and writes projects/gps-rmm-audit/needs-screenconnect.md.

Result: 114 Syncro-managed machines are not in GuruRMM. List sent to Howard in Discord (3 chunks) and includes each machine's policy_folder_id.

Policy finding: Syncro public REST API does NOT expose policy management (GET /policies = 404); RMM policy/ScreenConnect-install-policy work is Syncro-console-only. Assets carry policy_folder_id (folders are largely per-client, ~37 distinct across the gap). Path forward = in the Syncro console, put the gap devices under the ScreenConnect-install policy folder (or add the GuruRMM one-liner to that policy); Syncro reaches 100% of managed devices, solving the no-SC/no-footprint gap the ScreenConnect-only channel couldn't. Several gap entries are Macs (need macOS agent, not the Windows one).

Also: the Syncro-stored ScreenConnect GUID on assets is stale (send-command HTTP 500 / session lookup null) - that's why by-name SC lookup missed these machines.

Friction logged: curl inside while-read loop steals loop stdin (fix: </dev/null); Python open(...,'w') defaults to cp1252 on Windows (fix: encoding='utf-8').

Update: 18:45 PT — active-only needs-install list + SC policy pushed

Added a recency filter to needs-sc.py using kabuto_information.last_synced_at (updated_at is just the record touch, not the agent check-in). Filtered the missing-machine list to only those seen in Syncro within the last 31 days: 114 -> 64 active (50 excluded as offline >31d). Each entry now shows policy_folder_id + last-seen date + days-ago. Sent the 64-machine active list to Howard in Discord (4 chunks). Regenerate anytime: MAXDAYS= SK=.. RMM=.. TOK=.. py needs-sc.py.

Howard pushed a Syncro policy that installs ScreenConnect on the machines as they come online; some may install already, remainder tracked daily. NOTE: SC install alone does not install GuruRMM - after SC lands, GuruRMM still must be pushed via SC (our channel), unless the Syncro policy also runs the GuruRMM one-liner.

Next step (agreed, not yet done): close the loop - upgrade the daily GPS-RMM-Progress job to auto-push GuruRMM to needs-install machines once they have a live SC session (or add the GuruRMM one-liner to the Syncro SC policy). Billing findings to Winter/Mike remain HELD (do not send yet, per Howard).

Update: 19:30 PT — ScreenConnect cleanup begun (Dataforth Phase A)

Started a ScreenConnect session-hygiene pass on the client machines we added to RMM, beginning with Dataforth. Reverse-engineered the SC custom-property slot mapping from a labeled test session (Howard entered known values on HOWARD-HOME): CP1=Company, CP2=Site, CP3=Department, CP4=Device Type, CP5-7 unused, CP8=Tag. Saved as memory reference_screenconnect_custom_property_slots. NOTE: the SC API hides property LABELS (only values); UpdateSessionCustomProperties REPLACES the whole 8-element array, so always read current values, change only intended slots, write back.

Dataforth findings: Company was a 3-way split (DF - Dataforth 39 / Dataforth Corp 10 / Dataforth Corporation 1); eng-dev-server had a duplicate session; CP2-8 (Site/Dept/DeviceType/Tag) all blank; RMM also lumps all 51 under site D1 (unreliable); D1 and D2 are one flat 192.168.0.0/24 subnet so IP can't distinguish building - needs UniFi switch/AP topology. Dataforth is NOT on the UOS controller but IS a cloud UDM (UDM-Dataforth) reachable via the Site Manager connector.

Phase A executed (all 51 sessions): normalized Company -> 'DF - Dataforth'; set Site D1/D2 on 19 via hostname prefix (safe rule: prefix/suffix -D1-/-D2- only, so AD1/AD2 servers aren't miscategorized) = D1:8, D2:11; 31 non-prefixed remain blank (Phase B via UniFi); 1 pre-existing 'CC' left. eng-dev-server dup flagged for manual console removal (API can't delete sessions); DF-D2-Training-LT has no live SC session. Dev-alert posted.

Next: Phase B Part 1 = Device Type (CP4) for all 51 + Department (CP3) where derivable from hostname/wiki; Part 2 = UniFi (UDM-Dataforth connector) map switch/AP -> D1/D2 for the 31 blanks. Bash 2-min default timeout keeps cutting SC batches - use timeout 600000; SC writes are idempotent so re-runs are safe.

Update: 20:10 PT — Dataforth ScreenConnect Phase B (Device Type, Department, UniFi site map)

Phase B Part 1: set Device Type (CP4) on all 51 (Server/Laptop/Workstation from hostname+os_type) and Department (CP3) on 30 (hostname role tokens: ENGI->Engineering, MFG->Manufacturing, RCVG->Receiving, QC->Quality, AS->Assembly, CONF->Conference, SHIP->Shipping, servers->IT, +wiki person hints). 21 Departments blank (generic DESKTOP-/Station/named boxes) - need Howard input, come-back.

Phase B Part 2 (D1/D2 site via UniFi): Dataforth is cloud UDM-Dataforth (host id D021F96C10F9...:1721816512), reachable via Site Manager connector. Its switches/APs are BUILDING-NAMED (D1 Engineering, D1-Server Room, D2-SMT, D2-Breakroom, etc.). Pulled full stat/device + stat/sta directly (gw-sitemanager 'net raw' caps output at [:5000] - bypass by curling api.ui.com /v1/connector/consoles//proxy/network/api/s/default/stat/{device,sta} with the vaulted key services/unifi-site-manager). Mapped each client's uplink sw_mac/ap_mac -> device name -> building. Set Site on 21 more machines (total 40/51: 19 prefix + 21 UniFi). 9 still blank = offline (not in live UniFi client list): AD1, SAGE-SQL, QCInspection, DFASLB0519, Dforthpc18, LAPTOP-RD47E88A, DESKTOP-AH0SLT7/-FT0T4MK/-SA4N14M.

Dataforth SC remaining come-back: 21 Departments, 9 Sites (offline), Tag (CP8) convention, eng-dev-server duplicate (manual console removal - API can't delete sessions). Then repeat the whole SC cleanup for the other worked clients.

Update: 20:40 PT — Cascades ScreenConnect cleanup + Dataforth RMM site fix

Dataforth RMM sites: created a D2 site (id ed1d28c7-3f22-4578-a3f8-cabe6100382a, code UPPER-FALCON-1529) under client c9242164-8e66-43ba-b9e8-84520d096562 and moved 26 machines off the lumped D1 (POST /api/agents/:id/move {site_id}) using the corrected SC Site (CP2) as source of truth. Now D1:25 / D2:26. 12 unmapped (offline) left on D1 for come-back.

Cascades (next client, single campus, department-focused): normalized Company -> 'Cascades of Tucson' (was Cascades of Tucson 24 / Cascades 6 / blank 1); set Device Type on all (matched Cascades' existing 'Desktop'/'Laptop'/'Server' vocab, NOT Dataforth's 'Workstation'); set Department on 24 from hostname roles + the WIKI person->machine map (wiki/clients/cascades-tucson.md maps e.g. Ashley Jensen->Accounting DESKTOP-U2DHAP0, Chris Knight->Accounting DESKTOP-N5G1ROO, Shelby Trozzi->MemCare MDIRECTOR-PC, Sharon Edwards->Life Enrichment DESKTOP-DLTAGOI, caregiver laptops Laptop2/DRQ5L558/E0STJJE8->Nursing); fixed 'Accouting' typo. 9 departments still unknown (ANN-PC, ASSISTMAN-PC, megan, Laptop4, LAPTOP-8P7HDSEI, DESKTOP-F94M8UT/-LPOPV30/-MD6UQI3/-ROK7VNM). RECEPTIONIST-PC dup (2 RMM agents + 2 SC sessions) needs manual console removal + investigation. 2 no SC session: DESKTOP-KQSL232 (Lois Lane/CareTakers EOL), Health-Services-Director.

Workflow captured (memory feedback_screenconnect_cleanup_wiki_source): SC/RMM cleanup uses the client wiki as source of truth for machine->dept/location; where missing, enrich the wiki as we learn. Next sites ranked earlier: Valley Wide (cloud UDM), Grabb/Russo (small multi-site + UniFi); Safesite deferred (no UniFi, mobile fleet).

Update: 21:05 PT — ScreenConnect fleet easy-win pass (Company + Device Type)

Built projects/gps-rmm-audit/tools/sc-cleanup.py (DIRECT ScreenConnect API via urllib - CTRLAuthHeader + Origin to the RESTApi extension Service.ashx; ~10x faster than subprocess-per-call which timed out at 9min on ~500 sessions). Ran the safe no-guess pass across all ~45 worked RMM clients (skip: AZ Computer Guru, Unassigned, Dataforth, Cascades). Company (CP1) -> RMM client name (trimmed), keeping deliberate 'CODE - Name' conventions (GND -, LAB -); Device Type (CP4) from hostname/os; Department (CP3) only on high-confidence hostname tokens; Site (CP2) untouched.

Result: 112 Company sessions normalized (fixed person-surname values Osgood->Design and Brand Envoys, Parkinson->Leeann Maddux; trimmed 'Patriot Internal Medicine '), Device Type set fleet-wide, 1 Department (Valley Wide). 4 Company changes to eyeball: Shinn,Sharon 'Starr Pass Realty'->'Shinn, Sharon' (maybe rename the RMM client to Starr Pass Realty instead), Sombra->Sombra Residential LLC, VWP dropped '(VWP)', Wolkin->Wolkin, Robert. 11 duplicate SC sessions flagged for manual console removal (mostly SERVER-named reinstalls). Departments + Site stay for per-client wiki passes (feedback_screenconnect_cleanup_wiki_source).

Update: 21:40 PT — SC cleanup BUG + remediation (shared-name cross-contamination)

BUG in the fleet SC cleanup (sc-cleanup.py): GetSessionsByName matches by NAME across ALL clients (not client-scoped). Writing Company to shared/generic names cross-contaminated other clients' sessions with the last-processed client's value. Affected: SERVER (11 sessions across ~8 clients, all stamped 'Zeus Nestora'), Accounting (3, stamped Kittle), + within-client dups touched. Detected via non-idempotency (company changes never converged to 0).

Remediation:

  • Fixed tool: added if len(ss)>1: continue (never auto-write to a name returning >1 session). WAN map must be built ONLY from unique-named (len==1) machines.
  • Remapped contaminated sessions to correct client via GuestNetworkAddress (WAN IP) -> client, matched against a WAN map from unique machines + verified hardcoded values: SERVER 8 correct (T&C Sorensen 71.228.149.143, Cutting Edge 174.79.36.35, Four Paws 184.183.92.86, Kittle 174.67.176.116, Brett 162.191.251.126, Gonzvar 70.166.119.148, Zeus 68.177.35.66 via CindyW, ACE Portables via aceadmin user); Accounting 1 -> Kittle (174.67.176.116). Restored single-RMM-owner dups: 0226-Lenovo/MSI->Safesite, DESKTOP-NFK4F5P->GND-Grabb, eng-dev-server->DF-Dataforth, RECEPTIONIST-PC->Cascades of Tucson.
  • 5 sessions UNRECOVERABLE (originals overwritten, unidentifiable via RMM/UniFi/Syncro/WAN/reverse-DNS): 3 SERVER (67.1.219.218, 72.219.229.144 cox, 184.183.92.124 cox) + 2 Accounting (none, 184.191.16.88) -> left BLANK, need manual SC-console tagging. Logged to errorlog (correction + friction). Rest of fleet (unique-named machines) unaffected.

Update: Valley Wide ScreenConnect cleanup

Valley Wide Plastering (cloud UDM-Valley Wide Plastering). Confirmed SINGLE location via UniFi - devices are areas within one campus (North Bridge, East/West Portables, Server Room, staff-desk switches JR/Jesse/Rose/T's/Ty's), not separate buildings, so no site split. Company + Device Type already done by the fleet pass. This pass: fixed 5 mislabeled server device-types (VWP-FILES, VWP_ADSRVR, WINFileSvr, VWP-QBS, VWP-FIN -> Server, per wiki server fleet), set Department on ~16 (Estimator1->Estimating, FRONT64->Front Desk, WIN-ACCT->Accounting, VWP-FIN->Finance, ORDERSTY->Orders, all servers->IT). Skipped 12 person-named/generic (DESKTOP-*, JG64-2, JRW10-64, VWP-ChrisG-L/JESSEIII/JR-L/REANNA/ROSE) as unsure - come-back via wiki. Clean run, no dupes/contamination.

Update: Grabb & Durando ScreenConnect cleanup

Grabb & Durando Law Office (multi-site: Main Office/Bob's House/Jeff's House; office on UOS UGW3). Company + Device Type done by fleet pass. Set Department on 4 sure machines: GND-SERVER->IT, GND-BOB-PC (Robert Grabb, name partner)->Attorney, GND-JWILL (Jeff Williams)->Attorney, GND-JEFF-2->Attorney. Skipped as unsure (come-back via wiki/user): GND-Reyna + GND-TDIAZ + GND-L-1 (support staff, role unclear), LapTopNinja/DESKTOP-NFK4F5P (generic). SITE skipped: ambiguous - RMM has GND-BOB-PC/GND-JEFF-2 on 'Main Office' but hostnames suggest Bob's/Jeff's House; Grabb office UGW3 UniFi only tracks wireless (0), home offices not on it - can't verify which machines are at Main vs the home-office sites. DESKTOP-NFK4F5P has a duplicate SC session (manual console removal). Come-back: confirm Bob/Jeff machine locations + support-staff departments.

Update: Russo + IMC + fleet servers->IT pass

Russo Law Firm (3 machines, single site): Department set on all (RUSSO-SRV->IT, steve2020->Attorney [Steve Russo principal], STRIONFO->Administration [Shannon Trionfo, account/billing contact per wiki]). Empty 'Shannon' RMM site is vestigial.

Instrumental Music Center (music retail/repair, single site, ~12 in RMM): fixed IMC1 type->Server, Department set on 8 (IMC1->IT; IMC-STATION1/2 + IMC-L1-STATION9 + DESKTOP-44L80C0/MR3ALTK POS workstations->Sales; IMC-Lessons->Lessons; IMC-SvcStr->Repair). Skipped generics (C2B, DESKTOP-GHG12G3, IMC-Mini, LAPTOP-DCHQ3F92).

Fleet pass: added servers->IT fallback to sc-cleanup.py (device type Server + no role token -> IT). Re-ran fleet-wide (safe - the >1-session skip prevents re-contamination). Set +20 departments (servers->IT + role tokens) across ~16 clients (AMT, BirthBiologic, Cutting Edge, Glaztech, Horseshoe, Len's, Lonestar, Patriot, Peaceful Spirit, QWM, Safesite, Sif-oidak, Sombra, Prairie Schooner, Golden Corral, Universal Cryogenics), fixed 6 device types. Company converged to 0 (contamination fully resolved). Remaining departments = person-named/generic machines, need per-client wiki passes (the long tail).

Update: Staging auto-enroll system (generic installer + reassignment)

Problem: onboarded Bucket-C orgs show 0 agents - machines are offline/briefly-online, so per-client site-specific SC pushes don't land before they disconnect. Enrollment stalled ~90/189.

Solution (Howard's idea): a SINGLE generic installer pushed by a Syncro policy to ALL managed machines; they enroll into a catch-all "Staging" site whenever they come online; then auto-reassign to the real client by hostname->Syncro customer->GuruRMM client.

Built:

  • GuruRMM client "Staging - Auto Enroll" (04b24e18-5dee-4eb4-b7a4-3f967b997a27) / site "Staging" (7c980f78-075a-4c09-915c-ba961936bc95) / code DARK-STORM-3150. Key vaulted at infrastructure/gururmm-staging-site.sops.yaml.
  • Syncro one-liner to add to a policy (runs on all assets when online): irm 'https://rmm.azcomputerguru.com/install/DARK-STORM-3150/windows' | iex
  • projects/gps-rmm-audit/tools/reassign-staging.py: moves Staging agents to their real client via POST /api/agents/:id/move, matching hostname->Syncro customer business_name->GuruRMM client (main site). Idempotent; --dry supported. Verified runs clean (0 staging agents currently).

Next: Howard adds the one-liner to the Syncro all-machines policy; schedule reassign-staging.py (or fold into the daily GPS-RMM-Progress task). Unmatched agents stay in Staging + flagged.

Update: Corrected online detection + Staging auto-enroll (WORKS end-to-end)

Howard caught that machines marked "offline 45+ days" were actually online. Two bugs found + fixed:

  1. Online field: was reading GuestConnectedCount (always null -> everything looked offline). Correct = ActiveConnections with ProcessType==2 (guest agent connected); recency via GuestInfoUpdateTime.
  2. Age metric: needs-list used Syncro last_synced_at (Syncro agent check-in, goes stale when the agent breaks) instead of ScreenConnect reachability.

Built projects/gps-rmm-audit/tools/rebuild-and-push.py: rebuilds needs-screenconnect.md by SC reachability (ONLINE now / active<=14d / stale15-45 / dead) and pushes the generic Staging installer (DARK-STORM-3150, base64 -EncodedCommand via SendCommandToSession) to every ONLINE machine. Ran it: 17 online -> pushed all 17 -> 15 enrolled into Staging within ~2 min. (Counts: online 17, active<=14d 31, stale 4, dead 58.)

reassign-staging.py sorted all 15 to their real clients (Curtis, Mineralogical, IMC, Prairie Schooner, Zeus, Stamback + Ridgetop/Gary Hartman/Robyn Pittman). Surfaced 3 clients I'd wrongly written off as no-footprint (Ridgetop has 3 machines, Gary Hartman 1, Robyn Pittman 1) - onboarded them. Added name normalization to reassign (strip LLC/Inc/Corp/etc.) so Syncro 'Safesite LLC' matches GuruRMM 'Safesite'. Staging now empty (all reassigned).

Proven pipeline: correct-online -> push staging installer -> enroll -> auto-reassign. Ready to repeat as machines come online (or via the Syncro all-machines policy Howard will add).

Update: dedup fix deployed + full cleanup executed (2026-07-04 afternoon)

Deploy: cherry-picked the 2 dedup commits onto latest origin/main (hotfix-agent-dedup -> main, 1c1cac2); webhook auto-built and deployed (server 6.66 -> 6.77, CI version-bump c4f24de confirms). NOTE: fix authored on feature/av-removal-recipes which is 11 unrelated commits ahead - deployed via cherry-pick, NOT merge.

Live verification on Howard-Home (this machine, was unenrolled): (1) staging installer -> 1 record in Staging; (2) re-ran installer -> SAME record reused (id 7bf450ff), count stayed 1 = dedup works; (3) moved to AZ Computer Guru/Main Office + agent restart (re-presents staging code) -> STAYED in ACG, count 1 = placement persists. Howard-Home now properly enrolled under ACG/Main Office.

Step 2 (deletes): regenerated plan fresh (group by device_id ONLY, skip hostname-fallback ids, skip mixed-hostname groups, keep most-recent last_seen). 40 groups -> 40 stale orphans deleted, 0 failed (DELETE /api/agents/:id, throttled 2s + retry; deletes are async background cascade per BUG-018). Same-device dup groups after: 0.

Step 3 (moves): 39 one-time moves, 0 failed (throttled + 5x retry rode over the intermittent move-endpoint 500s): 24 Dataforth D1->D2 (AD2, D2-*, DANC0619, DESKTOP-02VT7TO/3PFA5I7/Q33I5H1/U8PHS72, DF-SVR-D2-Sync, DFORTH-Ship, DForth-Shipp, My9-PC, STATION43, Station20-PC, Station21, Station_41, Test01, goldstar19); 13 Staging -> real clients (Gary Hartman, IMC, Mineralogical x2, Ridgetop x2, Robyn Pittman, Safesite/Bell, Prairie Schooner x2, Zeus x2, Stamback); corrected my 2 diagnostic mis-moves (CNX-LAB-00 -> Ridgetop, CURTIS-002-W7 -> Curtis Plumbing). Staging now EMPTY.

Manual-review pile (13 same-hostname/different-device pairs, NEVER auto-touched): GURU-5070 (Mike's laptop - old record offline since 2026-06-06, new device_id since 2026-06-09 = likely reimage; Mike's call to delete stale), SERVER (ONE device with records in BOTH T&C Sorensen and Zeus Nestora - live one is Zeus; if physically T&C's it needs a move), BridgettePSHomeComputer, DESKTOP-BTR2AM3, MSI, Maras-HP-Laptop, PST-SURFACE, RECEPTIONIST-PC, RMM-TEST-MACHINE, Sif-Laptop554/555, bfcfbc739d23, gururmm.

Remaining known issue: move endpoint intermittently 500s under repeated calls (transient DB/load, handler code is a clean UPDATE) - throttle+retry works; worth a server-side look eventually.

Update: post-cleanup fresh sweep (2026-07-04 evening)

Holiday evening sweep: only 2 GPS machines online in SC (CP-QB Curtis Plumbing, IMC-PRINTSERVER IMC). Pushed staging installer to both, twice (second send wrapped in try/catch writing errors to C:\Windows\Temp\gururmm-install-err.txt on each box) - neither enrolled after ~5 min despite live SC connections and current SC client. Mechanism proven (15 enrolled this morning) so cause is local to these boxes - COME-BACK: read the err file via SC backstage next session. 25 more machines active<=14d will be caught by the daily sweep as offices reopen after the holiday. Morning-sweep machines confirmed correctly enrolled post-dedup (BookKeeper->Reliant, PLS-TWO/THREE->PUTT, CURTIS-HOME->Curtis).

Update: GPS-RMM-AutoEnroll loop registered (2026-07-04 close)

Built .claude/scripts/gps-rmm-autoenroll.sh: each run = rebuild-and-push.py (push staging installer DARK-STORM-3150 to GPS machines online-in-SC but missing from RMM) -> sleep 150 -> reassign-staging.py (sort to real client) -> #dev-alerts post when machines land. Log: projects/gps-rmm-audit/autoenroll.log. Registered as Windows task GPS-RMM-AutoEnroll on HOWARD-HOME: starts Monday 2026-07-06 06:00, repeats every 30 min (Howard: skip the holiday weekend). Test runs clean (fixed grep -c || echo 0 double-output bug). The ~25 active<=14d machines get harvested Monday as offices reopen; CP-QB + IMC-PRINTSERVER auto-retry each pass. Delete the task when targets.json is fully enrolled: schtasks /Delete /TN GPS-RMM-AutoEnroll.

Session close: enrollment 111/189 (true, post-dedup); server dedup fix deployed v6.77 + verified; RMM duplicate-free; Staging empty; auto-enroll loop armed for Monday.

Update: SC<->RMM consistency pass (2026-07-04 evening)

Verified all 333 RMM agents against SC sessions. Results: 66 "company mismatches" = the CODE-Name convention (DF - Dataforth, LAB - Len's Auto) - correct, ignored. Fixed: 6 blank device types (new enrollments), 8 site-label normalizations (cas/cascades->CascadesTucson, TUC->TUS - Tucson, Server - Tucson->Tucson, Jeff Laptop->Jeff's House, len's->Main, Sif-oidak District->Main Office). Flagged (info-bearing, do NOT overwrite): GTS-PEDRO-H SC-site=Willcox (Gonzvar may have a Willcox location), DESKTOP-JDQMLCE=Turner (Len's), TPM-PC=Moore (Mineralogical), DESKTOP-P5F7NQU=CC (Dataforth) - onsite verification.

NO-SC gap closed via RMM: 17 RMM agents had no SC session. 6 online Windows fixed by pushing the parameterized SC installer THROUGH the RMM agent (POST /api/agents/:id/command, powershell, elevated): DESKTOP-UQRN4K3 was a fresh install; the other 5 (Health-Services-Director, DF-Joel2, LeeannCyberPowerPC, FrontDeskReception, rednourcarrievirt) were ORPHANED SC CLIENTS - client software installed + connecting, but their console sessions had been deleted (service ImagePath still had the dead session GUID). Fix = registry-uninstall the orphan + reinstall parameterized (self-tags Company/Site/DeviceType via c= params). 6/6 online + correctly tagged. GOTCHA: PS via RMM command - use $f=$env:TEMP+'\scsetup.msi' (escaped quotes in the JSON payload break variable assignment; first attempt uninstalled without reinstalling - recovered). Command output is in the .stdout field (NOT .output) of GET /api/commands/:id.

Remaining no-SC (come-back): offline Windows (DESKTOP-KQSL232, DF-D2-Training-LT, 1122-Suzanne-Dell, StambackLaptopNew, Seth-PC, BridgettePSHomeComputer, DESKTOP-KQSL232), linux (bfcfbc739d23 x2 Lonestar, SL-SERVER Scileppi), macos (Mac-mini-2 Scileppi). Within-client SC dup sessions for manual console removal: RECEPTIONIST-PC (Cascades), eng-dev-server (Dataforth), DESKTOP-NFK4F5P (Grabb), 0226-Lenovo + MSI (Safesite).

Update: Phase 4 kickoff — AV coverage matrix (2026-07-04 evening)

Built the AV coverage matrix: GPS billed qty vs RMM vs Bitdefender vs Datto EDR per client. Join key discovery: BD company names end with _ = exact cid match against targets.json (fuzzy-name matching had false NO-AV for reversed names like Sheila Heieck). Results in tracker.md Phase 4 section: 9 clients / 22 paid devices with NO AV visible; 7 partial; EDR fleet = 14 orgs (Default RMM Org has 35 unassigned agents - cleanup); AV migration scope = 27 clients / 141 BD endpoints (excl Glaztech 242 + Dataforth). Glaz-Tech BD 242 vs 159 GPS billed feeds the anomaly item. Remaining Phase 4: backup verification (B2/MSP360 vs billed backup lines), email.

Update: EDR Default-org attribution (2026-07-04 night)

Dismantled Datto EDR "Default RMM Org": discovered its 23 Locations were already client-named (IMC - Speedway 10 agents, Putt 4, Reliant 4, Russo 2, etc.) - agents attributed at Location level, wrong org umbrella. Fix = re-parent Locations to per-client Organizations. Verified mechanics with [TEST] org+location first (POST /Organizations, POST /Locations, PATCH /Locations/{id} {organizationId}, DELETE both - all work; LoopBack undocumented endpoints). Executed: 18 orgs created, 21 locations re-parented (Glaztech-ALB and ACG locs to existing orgs), 0 failures. GOTCHAS: edr.py orgs agent/site counts are stale rollups after re-parenting (trust GET /Locations); one raw GET ignored the --filter (returned all 42) - verify with unfiltered fetch + group-by. Default org now = OnDemand(0)+Managed(0) only. Revised AV gap list in tracker (8 clients / 20 devices NO-AV; Andy's Mobile Fuel cleared).

Update: NO-AV remediation round 1 (2026-07-04 night)

Created 3 EDR orgs + target groups + reg keys (Ridgetop/ridgetop01, Gary A Hartman/hartman001, Robyn Pittman/pittman001) via the datto-edr skill (create-group -> mint-key). Pushed Install-EDR through RMM to the 5 online NO-AV machines. 4/5 registered ACTIVE in the correct orgs within minutes (RGI-DC needed a Start-Service kick after os error 1053). DESKTOP-PL2RCGL install still running at wrap - verify. avInstalled null right after registration - confirm Datto AV enables via org policy Monday. Remaining NO-AV clients (Little Hearts 8, R&R Eng 2, Altschuler 2, BSoT 2, Marty Ryan 1) unreachable tonight - autoenroll->RMM->EDR path as they surface.

Update: verification sweep + session close (2026-07-04 night)

Re-verified every workstream live. Findings (both handled): (1) guru-rmm submodule pointer referenced the feature-branch commit - realigned to deployed main c4f24de (a270f40); (2) DESKTOP-PL2RCGL EDR install FAILED (machine went offline mid-delivery, "command undeliverable") - Monday retry with key pittman001; protected count tonight = 4 of 5, not 5. All else green: RMM 0 same-device dups, Staging empty, moves stuck (Dataforth D1=25/D2=26 + spot-checks), both scheduled tasks Ready, dedup fix live in deployed main, EDR Default org = OnDemand+Managed only, SC fixes verified in place, staging key vaulted, memory updated.

Also logged correction: used emoji symbols in verification tables (rendered as ? in terminal) violating the CLAUDE.md NO EMOJIS rule - ASCII markers only.

Monday queue: autoenroll 6AM (~25 active machines), Dataforth EDR tail x4 (RegKey 27lzj6npdb), PL2RCGL EDR retry, CP-QB + IMC-PRINTSERVER err-file reads, avInstalled policy check, held billing findings, Phase 4 backup verification.

Update: Phase 4 backup verification + EDR pre-provisioning (2026-07-04 late night)

JANC (Syncro 2248945) confirmed SEPARATE from Janet Altschuler (457710) - Howard. Janet fully NO-AV.

Backup verification (billed Service-Data-Backup lines vs GuruRMM /api/mspbackups mappings vs B2 bucket freshness): 2 DEAD buckets (ACG-REDNOUR 194d, ACG-GLAZTECH 249d), 4 clients billed real money with no backup found (Tedards $45, Reliant $40, Mineralogical $10, Prairie Schooner $10), Jimmy 12-billed/1-real hard-verified, 11 clients running backups UNBILLED, 5 dead legacy MSP360 prefixes in the shared bucket (purge candidates). ACG-PST = Peaceful Spirit not Prairie Schooner. Full matrix in tracker. RMM mspbackups mappings synced tonight (28 mapped, only covers RMM-enrolled machines - caveat).

DATAFORTH-PC surfaced online (Howard) -> enrolled in RMM (landed Dataforth/D1 directly) -> EDR pushed via RMM, installed, ACTIVE. Dataforth tail now 3 (SURFACEOPS, D1-ENGI-006, MING-HP).

EDR pre-provisioning: 26 reg keys minted on existing targets + 18 new clients provisioned (org->group->key), 0 failures. All 48 client keys vaulted msp-tools/datto-edr-regkeys.sops.yaml. Monday deployment = pure Install-EDR pushes via RMM as machines come online.

Update: HELD billing findings VERIFIED + SENT (2026-07-05)

Re-verified all held findings live before sending. Changes since held: Dataforth now +9 (52 RMM vs 43 billed, was +8); Stamback improved 2->4 of 8; RECEPTIONIST-PC finding CORRECTED - not a stale duplicate but TWO LIVE machines (different device_ids, both online simultaneously) sharing one hostname - needs a rename, not a delete. All other findings unchanged (Glaz 159/5, Jimmy 12/1, LAB-SVR offline since 6/18, backup matrix from tonight).

Sent the consolidated findings via Discord DM to Mike, Winter, AND Howard (Howard requested inclusion) at ~2026-07-05 03:00 UTC (message ids 1523172715039559760 / ...725391233044 / ...736904593588). Findings-hold is now RELEASED. Tracker remains the detailed reference.

Reference in New Issue View Git Blame Copy Permalink