wiki: compile gps-rmm-audit (full) — Phase 4 backup verification, master-status list, 20 client wikis, onboardings, onsite installer kit
This commit is contained in:
@@ -72,7 +72,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks.
|
||||
|---|---|---|
|
||||
| [GuruRMM](projects/gururmm.md) | RMM platform, Rust/Axum server + React dashboard + cross-platform agent; **agent v0.6.78 fleet-wide stable / 0.6.79 beta / server v0.3.103 / dashboard = Fleet-Policy Console**; ~360 enrolled; migrations through 068_visible_agents_folder_id. Merge to main = build+deploy; channel default FAIL-CLOSED beta (BUG-023), explicit promote to stable. Recent (2026-07-06): **Fleet-Policy Management Console** shipped to PROD (policy-ui — folder tree table / place-devices / effective-policy viewer / version-history / backfill; migration 068, folder_id in API); **WS-looper comms-durability** hardening (bounded WS write-timeout + heartbeat-count blackhole-vs-stall diagnostic + AIMD adaptive keepalive; agent 0.6.79 beta / server 0.3.102). Prior (2026-07-04/05): four bug-fix batches v0.3.95->v0.3.99 + policy-core (migration 067). **BUG-024 OPEN/CRITICAL** — Linux glibc incident, Linux binaries QUARANTINED (re-quarantined every build) pending glibc-floor fix; 4 chronic one-way-WS agents now have a server-visible blackhole diagnostic. Active development | 2026-07-06 |
|
||||
| [GuruConnect](projects/guruconnect.md) | ACG's proprietary Rust remote-access/remote-support tool (ScreenConnect-class) — Windows agent + web dashboard; **v0.3.0 production** at connect.azcomputerguru.com; versioned integration contract with GuruRMM; co-located on the physical .30 box (shares the PG cluster); next: SPEC-018 session broker / capture worker as SYSTEM | 2026-06-12 |
|
||||
| [GPS -> GuruRMM Coverage Audit](projects/gps-rmm-audit.md) | Verify every GPS-billed client is fully enrolled in GuruRMM with services real (backup/AV/email); enrollment 111/189 (baseline 46); staging auto-enroll pipeline + 30-min harvest loop; found+fixed GuruRMM cross-site dup bug (v6.77); Phase 4 AV matrix (8 clients/20 devices NO-AV); billing findings HELD for Winter/Mike; AV migration BD->EDR (exception: Glaztech only) | 2026-07-04 |
|
||||
| [GPS -> GuruRMM Coverage Audit](projects/gps-rmm-audit.md) | Verify every GPS-billed client is fully enrolled in GuruRMM with billed services real (backup/AV/email). Single current list = `gps-master-status.md`. Phase 4 backup verification: authoritative source = MSP360 API (not B2 buckets); only 16/40 bill backup (7 OK / 2 review / 7 NOT backing up, endpoint-verified). Built msp360/seafile/owncloud/datto-workplace skills; 20 client wikis (56->76); onsite installer kit (Desktop/CLIENTS). Onboarded Janet Altschuler/Marty Ryan/BST + 3 shells. Datto Workplace key pending Mike; billing findings HELD for Winter/Mike; AutoEnroll task disabled | 2026-07-07 |
|
||||
| [Dataforth DOS — Test Datasheet Pipeline](projects/dataforth-dos.md) | DOS update system + TestDataDB pipeline (Node.js, PostgreSQL, Hoffman API); 469K records, 458.5K live on website; 2025 crypto attack recovery; security incident 2026-03-27; SCMVAS/SCMHVAS extension; email notifications via Graph API | 2026-05-24 |
|
||||
| [ClaudeTools Discord Bot](projects/discord-bot.md) | Claude Agent SDK bot in Discord; one persistent session per thread; Phase 1.5 complete (native tools, no hand-written tools); Phases 2-4 (API integration, remediation, UX) pending; runs as NSSM service on BEAST | 2026-05-24 |
|
||||
| [The Computer Guru Show](projects/radio-show.md) | Radio show archive processing pipeline (Whisper + pyannote + SQLite FTS5) + post-show content workflow; 572 episodes indexed; FastAPI UI redesigned; Jupiter audio-file gap open | 2026-05-24 |
|
||||
|
||||
@@ -2,17 +2,28 @@
|
||||
type: project
|
||||
name: gps-rmm-audit
|
||||
display_name: "GPS -> GuruRMM Coverage Audit"
|
||||
last_compiled: 2026-07-04
|
||||
last_compiled: 2026-07-07
|
||||
compiled_by: HOWARD-HOME/claude-main
|
||||
sources:
|
||||
- wiki/_templates/project.md
|
||||
- projects/gps-rmm-audit/gps-master-status.md
|
||||
- projects/gps-rmm-audit/backup-map.md
|
||||
- projects/gps-rmm-audit/tracker.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-03-howard-gps-rmm-coverage-audit.md
|
||||
- projects/gps-rmm-audit/dedup-plan.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-03-howard-gps-rmm-coverage-audit.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-backup-verification-skills.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-edr-mass-deploy-imc-printserver.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-cpqb-fixed-imcprintserver-tls-blocked.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-gps-backup-map-msp360.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-07-howard-gps-onboard-endpoint-backup-verify.md
|
||||
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-07-howard-gps-master-status-wikis-onsite-kit.md
|
||||
- .claude/memory/project_av_migration_bitdefender_to_edr.md
|
||||
- .claude/memory/reference_rmm_deploy_via_screenconnect.md
|
||||
- .claude/memory/reference_screenconnect_custom_property_slots.md
|
||||
- .claude/memory/feedback_screenconnect_cleanup_wiki_source.md
|
||||
- .claude/memory/reference_msp360_backup_monitoring.md
|
||||
- .claude/memory/feedback_backup_targets_never_guessed.md
|
||||
- .claude/memory/feedback_backup_check_only_billed.md
|
||||
backlinks: []
|
||||
---
|
||||
|
||||
@@ -24,39 +35,98 @@ Verifies every business paying ACG for GPS (Guru Protection Service) has GuruRMM
|
||||
correctly enrolled: org/site exists, all billed machines present and reporting, billed
|
||||
services (backup/AV/email/VoIP) actually configured. "Should have" = Syncro active
|
||||
recurring GPS schedules (device counts + service lines); "reality" = live GuruRMM
|
||||
`/api/agents`, cross-checked against Datto EDR, Bitdefender, and Syncro assets.
|
||||
`/api/agents`, cross-checked against Datto EDR, Bitdefender, MSP360, and Syncro assets.
|
||||
|
||||
Scope: 40 active GPS clients (4 paused excluded: Marcia Ashton, Tucson Mountain Motors,
|
||||
Richard Pittman, Brenda Lopez). GPS device count = workstation + server SKUs only (AV
|
||||
add-on/discount/setup lines excluded). Started 2026-07-03 (Howard).
|
||||
add-on/discount/setup lines excluded). Started 2026-07-03 (Howard); still open as of
|
||||
2026-07-07, now mostly in a backup-verification and per-client cleanup tail.
|
||||
|
||||
## Current State (2026-07-04 night close)
|
||||
**Current single source of truth is `projects/gps-rmm-audit/gps-master-status.md`**
|
||||
(regenerable from live GuruRMM `/api/agents` + Syncro backup seats + `backup-map.md`
|
||||
verdicts) — the at-a-glance per-client list: GPS device count, RMM enrollment, backup
|
||||
billing + status, wiki presence. `tracker.md` holds the detailed bucket-by-bucket history
|
||||
and remains the reference for how each finding was reached; `backup-map.md` holds the
|
||||
full Phase-4 backup verification detail.
|
||||
|
||||
- **Enrollment 111/189 devices** (true count, post-dedup). Baseline 2026-07-03: 46/189,
|
||||
32 clients short.
|
||||
- **Bucket A (7, count matched)** — verified end-to-end: machines/backup/AV/vault/wiki.
|
||||
1 wiki article created (Arizona Medical Transit). Findings held for Winter/Mike.
|
||||
- **Bucket B (8, present but short)** — RMM-vs-Datto-vs-Bitdefender matrix split it into
|
||||
real deploy gaps (IMC, Safesite, Horseshoe, Grabb & Durando, Quantum Wealth Mgmt — all
|
||||
pushed via SC) and billing flags (Jimmy Company, Stamback Septic, Glaz-Tech — held).
|
||||
- **Bucket C (25, no RMM org)** — 16 onboarded + ~44 agents deployed 2026-07-03; 3 of the
|
||||
"no footprint" clients were later found via the staging pipeline and onboarded
|
||||
(Ridgetop Group, Gary A Hartman LLC, Robyn Pittman). True no-footprint remaining:
|
||||
Little Hearts Little Hands, Residential and Renovation Engineering, Janet Altschuler,
|
||||
Business Services of Tucson LLC, Marty Ryan (~5 clients, ~16 devices) — need an online
|
||||
window, discovery, or onsite visit.
|
||||
- **Phase 4 AV matrix (2026-07-04)** — cid-matched Bitdefender + EDR vs GPS billing,
|
||||
fleet-wide. NO-AV: 8 clients / 20 paid devices (down from 9/22). 4 of 5 reachable NO-AV
|
||||
machines now protected (round 1).
|
||||
- Everything reachable via ScreenConnect is deployed; remaining gaps are offline
|
||||
machines tracked by the daily/30-min automation until they surface.
|
||||
## Current State (2026-07-07)
|
||||
|
||||
- **Enrollment**: closed steadily via ScreenConnect push + the Staging auto-enroll
|
||||
pipeline since the 2026-07-03 baseline (46/189) through 112/189 at the 2026-07-05
|
||||
close (that denominator excludes Glaz-Tech's 159-device billing anomaly). Per-client
|
||||
counts are no longer tracked as one fleet-wide fraction — `gps-master-status.md` is
|
||||
now the live source, listing RMM agent count against GPS billed count for each of the
|
||||
40 clients individually. **7 clients still have zero RMM footprint**: Andy's Mobile
|
||||
Fuel, Business Services of Tucson LLC, Heieck (Sheila), Little Hearts Little Hands,
|
||||
MVAN Enterprises Inc, Residential and Renovation Engineering, The Marc Group — all
|
||||
now have a GuruRMM client+site provisioned (site codes minted for the onsite kit,
|
||||
see below) but no machine has enrolled yet; onboarding proceeds onsite or as machines
|
||||
come online.
|
||||
- **Backup verification (Phase 4) — the major addition this week.** Authoritative
|
||||
backup-status source is now the **MSP360 Managed Backup Service API**
|
||||
(`/api/Monitoring`, via the `msp360` skill), not B2 bucket contents — a bucket with
|
||||
files does not prove a recent backup ran. Only **16 of the 40 GPS clients actually
|
||||
bill a backup line** (Syncro-verified 2026-07-06; the tracker's older `Svc=B` flags
|
||||
were wrong for Horseshoe, Quantum Wealth Management, T&C Sorensen, and Business
|
||||
Services of Tucson — none of those four bill backup). Verdict on the 16:
|
||||
**7 OK** (Dataforth, Valley Wide Plastering, Curtis Plumbing, Ridgetop Group, Brett
|
||||
Interiors, Gary A Hartman LLC, Mineralogical Record via ownCloud), **2 REVIEW**
|
||||
(Glaz-Tech — 2nd server 251d stale; Jimmy Company — 12 seats billed, 1 machine),
|
||||
**7 NOT BACKING UP** — 3 found in the MSP360/B2 pass (Grabb & Durando cloud plan
|
||||
failing/0 bytes, Bill Tedards ~359d stale, Arizona Medical Transit 0 plans) plus
|
||||
**4 confirmed by direct endpoint scan 2026-07-06/07** (Reliant Well Drilling, The
|
||||
Prairie Schooner, Janet Altschuler, Design and Brand Envoys — no backup software,
|
||||
service, scheduled task, Windows Server Backup, or Datto backup client on any
|
||||
machine; Prairie's `TPS-SERVER` has a dead Windows Server Backup to USB, last ran
|
||||
3/7/2018).
|
||||
- **Revenue leaks identified** (backing up in MSP360/B2/Seafile, but Syncro has no Data
|
||||
Backup line — billing question, held for Mike/Winter, nothing sent): Cascades of
|
||||
Tucson, Len's Auto Brokerage, Russo Law Firm, Horseshoe Management, Business Services
|
||||
of Tucson. **$0-priced backup lines** (billed a seat at $0 — bundled or under-billed,
|
||||
same hold): Ridgetop Group, Janet Altschuler, Design and Brand Envoys, Arizona
|
||||
Medical Transit.
|
||||
- **Orphan B2 buckets resolved via MSP360 CompanyName**: `ACG-PST` = Peaceful Spirit
|
||||
(not Prairie Schooner), `ACG-TCA` = Tucson Coin and Autograph (not T&C Sorensen —
|
||||
corrects an earlier tentative guess; T&C Sorensen does not bill for backup).
|
||||
- **Datto Workplace enumeration still BLOCKED** on a Mike-provisioned Super
|
||||
Administrator API key (current key = 0 projects visible; the public API is
|
||||
file-sharing-only with no admin/team/device endpoints — confirmed via Datto's
|
||||
published docs, not assumption). The 4 backup-map "held" clients were resolved by
|
||||
direct endpoint verification instead of waiting on that key.
|
||||
- **20 client wiki articles created** (wiki count 56 -> 76) closing the wiki gaps
|
||||
surfaced by `gps-master-status.md`.
|
||||
- **Onboarded to GuruRMM this week**: Janet Altschuler, Marty Ryan, Business Services
|
||||
of Tucson LLC (all real clients with a reachable machine); Andy's Mobile Fuel,
|
||||
Little Hearts Little Hands, Residential and Renovation Engineering (onboarded with
|
||||
no machine yet, specifically to mint site codes for the onsite installer kit).
|
||||
MVAN Enterprises and The Marc Group turned out to already exist from the 2026-07-03
|
||||
Bucket C onboarding (0 agents, not "no client") — the earlier "no footprint" label
|
||||
meant zero agents, not zero RMM presence.
|
||||
- **GuruRMM confirmed to support Windows, Linux, and macOS agents**
|
||||
(`/install/<code>/{windows,linux,macos}`), resolving the two Mac-only clients
|
||||
(Heieck, Residential and Renovation Engineering) as onsite-installable rather than
|
||||
blocked.
|
||||
- **Onsite installer kit built** at `C:\Users\Howard\Desktop\CLIENTS\` — one folder per
|
||||
site (20) with Windows/Linux/macOS installer scripts + README, per-location
|
||||
subfolders for the two multi-site clients (Grabb & Durando: Main/Bob's/Jeff's;
|
||||
Glaz-Tech: Involta/SLC/Tucson) so the correct site code reaches the correct location.
|
||||
Howard is taking this onsite to close the remaining no-footprint gaps in person.
|
||||
- **A wrong weekend findings DM to Winter was deleted** (2026-07-06) — superseded by
|
||||
the MSP360 + Syncro-seat data that arrived after it was sent. The corrected findings
|
||||
were PATCHed into the existing Mike/Winter/Howard message thread in place rather than
|
||||
sent as new messages, to keep one authoritative thread.
|
||||
- **GPS-RMM-AutoEnroll scheduled task is now DISABLED** per Howard — manual sweeps only
|
||||
(`bash .claude/scripts/gps-rmm-autoenroll.sh`) rather than automatic every 30 minutes.
|
||||
**GPS-RMM-Progress** (daily 8:07am gap-check DM) remains active.
|
||||
|
||||
## Tooling
|
||||
|
||||
`projects/gps-rmm-audit/tools/`:
|
||||
- **bucketc-onboard-deploy.sh** `<"Client"> <bd_company_id> <slug>` — onboards a GuruRMM
|
||||
client+site (idempotent), vaults the enrollment key, deploys via SC to every machine in
|
||||
the client's Bitdefender company with a live session. Used for Bucket C (16 clients).
|
||||
the client's Bitdefender company with a live session. Used for the original Bucket C
|
||||
(16 clients) and later ad hoc onboardings (Janet Altschuler, Marty Ryan, BST, Andy's,
|
||||
Little Hearts, Residential and Renovation Engineering).
|
||||
- **needs-sc.py** — reconciles Syncro assets (authoritative inventory) vs live RMM agents
|
||||
per client; writes `needs-screenconnect.md`; `MAXDAYS` env filters by Syncro
|
||||
`last_synced_at` recency. Online-detection superseded by `rebuild-and-push.py`.
|
||||
@@ -67,20 +137,27 @@ add-on/discount/setup lines excluded). Started 2026-07-03 (Howard).
|
||||
guard added after the contamination bug (see Findings).
|
||||
- **rebuild-and-push.py** — rebuilds the "not in RMM" list using the correct SC online
|
||||
signal (`ActiveConnections` `ProcessType==2`, not stale Syncro `last_synced_at`), pushes
|
||||
the generic **Staging** installer (`DARK-STORM-3150`) to every machine online now.
|
||||
the generic **Staging** installer (`DARK-STORM-3150`) to every machine online now. Now
|
||||
run manually via `gps-rmm-autoenroll.sh` (loop task disabled).
|
||||
- **reassign-staging.py** — moves agents out of `Staging - Auto Enroll` into their real
|
||||
client by hostname -> Syncro `business_name` -> GuruRMM client (LLC/Inc/Corp-normalized).
|
||||
Idempotent; `--dry` supported.
|
||||
- **edr-deploy.py** — throttled Bitdefender -> Datto EDR deployer: recomputes coverage
|
||||
from a live Datto EDR agent dump each run, targets ONLINE Windows RMM agents of GPS
|
||||
clients with a vaulted registration key not already in EDR, pushes the official
|
||||
Install-EDR one-liner through RMM one machine at a time (20s gap, retry-on-5xx,
|
||||
poll-to-completion). Glaz-Tech excluded by design. Re-run to sweep newly-online
|
||||
machines; idempotent.
|
||||
|
||||
`.claude/scripts/`:
|
||||
- **gps-rmm-autoenroll.sh** — closed loop: `rebuild-and-push.py` -> sleep 150s ->
|
||||
`reassign-staging.py` -> post to `#dev-alerts`. Logs to
|
||||
`projects/gps-rmm-audit/autoenroll.log`. Windows task **GPS-RMM-AutoEnroll**
|
||||
(HOWARD-HOME): starts Mon 2026-07-06 06:00, every 30 min. Safe to re-run (server
|
||||
v6.77+ dedups by device_id). Retire: `schtasks /Delete /TN GPS-RMM-AutoEnroll`.
|
||||
`projects/gps-rmm-audit/autoenroll.log`. **Windows task GPS-RMM-AutoEnroll is
|
||||
DISABLED** (Howard, 2026-07-07) — run this script manually instead of on the 30-min
|
||||
schedule. Safe to re-run (server v6.77+ dedups by device_id).
|
||||
- **gps-rmm-progress-check.sh** — daily read-only: `targets.json` vs live `/api/agents`,
|
||||
DMs Howard the per-client gaps, reports COMPLETE when met. Windows task
|
||||
**GPS-RMM-Progress**, daily 8:07am, Howard-Home.
|
||||
**GPS-RMM-Progress**, daily 8:07am, Howard-Home — still active.
|
||||
|
||||
**Staging site pattern (`DARK-STORM-3150`)**: Bucket-C machines are often offline or
|
||||
briefly online, so per-client SC pushes miss the window. A Syncro policy pushes one
|
||||
@@ -88,182 +165,351 @@ generic installer to all managed machines; anything that comes online enrolls in
|
||||
catch-all client `Staging - Auto Enroll` / site `Staging` (key vaulted at
|
||||
`infrastructure/gururmm-staging-site.sops.yaml`); `reassign-staging.py` auto-sorts it to
|
||||
the real client. Proven end-to-end (15 machines enrolled + correctly reassigned in one
|
||||
run, 2026-07-04); this is what `gps-rmm-autoenroll.sh` runs every 30 minutes.
|
||||
run, 2026-07-04). The 30-minute scheduled loop was disabled 2026-07-07; the mechanism is
|
||||
run on demand instead.
|
||||
|
||||
**Reassign / move flow**: `POST /api/agents/:id/move {site_id}` is a clean UPDATE
|
||||
(confirmed not the dup root cause). Throttled 2s + 5x retry to ride out intermittent
|
||||
500s. Used for both Staging reassignment and one-time site corrections (e.g. Dataforth
|
||||
D1/D2 split, 39 moves).
|
||||
500s. Used for Staging reassignment, one-time site corrections (Dataforth D1/D2 split,
|
||||
39 moves), and per-client reassignment (Janet Altschuler, Marty Ryan, Reliant).
|
||||
|
||||
### Backup verification tooling (new, Phase 4)
|
||||
|
||||
- **`msp360` skill** (`.claude/skills/msp360/`) — full CRUD against the MSP360 MBS API
|
||||
(`https://api.mspbackups.com`, Bearer login via `msp-tools/msp360-api.sops.yaml`).
|
||||
Reads: `status`, `monitoring` (`--failed`/`--company`/`--stale`/`--json`), `companies`,
|
||||
`users`, `licenses`, `billing`, raw GET. Writes (all `--confirm`-gated): create/delete
|
||||
company, create/delete user, grant/release/revoke license, raw passthrough. This is
|
||||
the **authoritative "is this client actually backing up" source** for the whole audit —
|
||||
`/api/Monitoring` returns per-computer LastStart/Status/DataCopied across 66 plans /
|
||||
37 companies. `/api/Computers` returns HTTP 400 (Remote Management API not enabled on
|
||||
the account) — use Monitoring/Companies/Users instead.
|
||||
- **`seafile` skill** — Seafile Pro Web API (admin creds vaulted `services/seafile-pro.sops.yaml`).
|
||||
Reference implementation for the pattern; `usage` / `audit [--client] [--rmm] [--json]`.
|
||||
- **`owncloud` skill** — SSH + `occ` (no OCS web-admin cred exists in the vault, only root
|
||||
SSH to the VM); per-user size read from ownCloud's own MySQL filecache, `du` fallback
|
||||
for dirty-cache users. Same `usage`/`audit` interface.
|
||||
- **`datto-workplace` skill** — HTTP Basic (api-key:api-secret,
|
||||
`msp-tools/datto-workplace.sops.yaml`); file-access API only, currently blind (0
|
||||
projects visible) until Mike enables Super Administrator scope on the integration.
|
||||
GuruRMM endpoint-detection (`Datto Workplace`/`Workplace2` patterns — never bare
|
||||
`"Datto"`, which false-matches Datto EDR/RMM/AV) is the practical fallback signal.
|
||||
- **`.claude/scripts/backup_common.py`** — shared plumbing for all four skills: repo-root/
|
||||
vault resolution, minimal JSON HTTP, and a GuruRMM `RmmClient` (login, list agents, run
|
||||
a read-only detection PowerShell to confirm a sync client is installed and signed in).
|
||||
- **`b2` skill** (pre-existing) — Backblaze B2 buckets; `bucket-size` per-bucket (the
|
||||
full `usage` scan across all buckets times out at 400s listing every object version).
|
||||
- Skill routing consolidated (`.claude/SKILL_ROUTING.md`): a backup request is a two-step
|
||||
route — (1) identify which backend the client actually uses via the client wiki +
|
||||
`backup-map.md` (never assume B2/MSP360), then (2) route to that backend's skill.
|
||||
Provisioning writes go through the target backend's own skill (all gated).
|
||||
|
||||
## Key Infrastructure & Integrations
|
||||
|
||||
- **GuruRMM API** — `http://172.16.3.30:3001`, vault `infrastructure/gururmm-server.sops.yaml`.
|
||||
Server at close: v6.77 (dedup fix deployed).
|
||||
Dashboard `rmm.azcomputerguru.com`; agent WS `wss://rmm-api.azcomputerguru.com/ws`
|
||||
(resolves direct IP, not Cloudflare). Server at last check: v6.77+ (dedup fix deployed).
|
||||
Install endpoints: `https://rmm.azcomputerguru.com/install/<site_code>/<platform>`,
|
||||
platform = `windows | windows-x86 | windows-legacy | windows-legacy-x86 | linux | macos`
|
||||
— confirmed Windows + Linux + macOS agents all exist. Offline-install path: raw binaries
|
||||
at `/install/<code>/download/<variant>`.
|
||||
- **ScreenConnect** — `https://computerguru.screenconnect.com`, RESTful API Manager ext
|
||||
(Service.ashx). Primary deploy + metadata channel for the audit. CP slots: CP1=Company,
|
||||
CP2=Site, CP3=Department, CP4=Device Type, CP5-7 unused, CP8=Tag. API hides property
|
||||
labels (values only); `UpdateSessionCustomProperties` replaces the whole 8-element
|
||||
array — always read current values before writing.
|
||||
array — always read current values before writing. Extension exposes only
|
||||
`GetSessionsByName` (no full-fleet listing) — need a hostname to target a machine.
|
||||
- **Syncro assets** — authoritative managed-device inventory (more complete than
|
||||
BD/Datto for discovery). Public API has no policy endpoint (`GET /policies` = 404); SC
|
||||
install policy is console-only. Assets carry `policy_folder_id` to target it.
|
||||
install policy is console-only. Assets carry `policy_folder_id` to target it. Recurring
|
||||
schedules are the authoritative "billed backup seat" source — must match on line NAME,
|
||||
not description (base GPS Workstation/Server SKUs carry "Backups" in boilerplate
|
||||
description text, producing false positives if matched loosely).
|
||||
- **MSP360 Managed Backup Service (MBS)** — `https://api.mspbackups.com`, vault
|
||||
`msp-tools/msp360-api.sops.yaml`. Authoritative "is it backing up" source for the whole
|
||||
fleet (37 companies / 59 users / 66 plans at discovery). `msp360` skill.
|
||||
- **Backblaze B2** — 16 buckets, destination-only (not a freshness signal). `b2` skill.
|
||||
- **Seafile Pro** — Docker on Jupiter, `http://172.16.3.20:8082` / `https://sync.azcomputerguru.com`.
|
||||
5 active accounts / ~5 TB at discovery (Russo Law 2.2 TB is the only GPS client on it).
|
||||
- **ownCloud** — VM `172.16.3.22` (cloud.acghosting.com), Rocky Linux 9.6, ownCloud 10.16
|
||||
Community. 10 users / 19.2 TB at discovery (`pavon` 14.2 TB is a video archive, not a
|
||||
backup; GPS clients on it: Mineralogical Record `minrec`, Business Services of Tucson
|
||||
`bst`; The Marc Group `themarcgroup` is non-GPS/email-only).
|
||||
- **Datto Workplace** — SaaS `acg.workplace.datto.com` (team 6) + on-prem Datto Workplace
|
||||
Server VM `ACG-DWP-X-BB` on Jupiter (confirmed on Birth Biologic via endpoint
|
||||
detection, non-GPS). File-access-only API; full enumeration blocked pending Mike's
|
||||
Super Administrator key.
|
||||
- **Datto EDR** — target AV end-state for nearly all clients. Managed via the
|
||||
`datto-edr` skill (`create-group` -> `mint-key` -> deploy). `avInstalled` reads null
|
||||
right after registration — confirm via org policy, not the raw field.
|
||||
right after registration — confirm via org policy, not the raw field. Authorized
|
||||
agent count rose 228 -> 302 during the 2026-07-05 mass-deploy pass.
|
||||
- **Bitdefender GravityZone** — retired as a coverage target, kept only as a discovery
|
||||
source. Company names carry the Syncro cid suffix (`_<cid>`) — exact join key vs
|
||||
`targets.json`, used for the Phase 4 matrix (fuzzy name matching gave false negatives
|
||||
on reversed names, e.g. Sheila Heieck).
|
||||
`targets.json` / `gps-clients.json`.
|
||||
|
||||
## Findings & Decisions
|
||||
|
||||
- **Billing findings HELD for Winter/Mike** (an early DM to Winter was retracted before
|
||||
this rule was set): Dataforth +8 machines over billed; Cascades duplicate agent
|
||||
(`RECEPTIONIST-PC`) + unbilled backup bucket (`ACG-Cascades` exists, no Data Backup
|
||||
line); Len's Auto `LAB-SVR` offline since 2026-06-18; AMT billed backup with no B2
|
||||
destination found; Jimmy Company (12 billed/1 real) and Stamback Septic (8 billed/~2
|
||||
real) — billing-vs-reality mismatch, nothing to enroll.
|
||||
### Machine / AV findings (2026-07-03/05, preserved)
|
||||
- **Billing findings VERIFIED + SENT 2026-07-05** (an early DM to Winter was retracted
|
||||
before the send-when-verified rule was set): Dataforth +9 machines over billed (52 RMM
|
||||
vs 43 billed); Cascades `RECEPTIONIST-PC` — re-verified as **two live machines sharing
|
||||
one hostname** (different device_ids, both online simultaneously), not a stale
|
||||
duplicate — needs a rename, not a delete; Len's Auto `LAB-SVR` offline since
|
||||
2026-06-18; Jimmy Company (12 billed/1 real) and Stamback Septic (8 billed/4 real, up
|
||||
from 2) — billing-vs-reality mismatch. Sent via Discord DM to Mike, Winter, and Howard
|
||||
2026-07-05; hold released.
|
||||
- **Glaz-Tech anomaly**: 159 GPS billed vs 5 real machines in RMM/Datto EDR, vs 242
|
||||
Bitdefender records (stale, years of ghosts). RMM+Datto agree on the real footprint;
|
||||
flagged for Mike's billing review, not treated as 154 missing agents.
|
||||
- **AV strategy**: Bitdefender -> Datto EDR for all clients. Exception narrowed during
|
||||
the audit (Dataforth originally excepted, then included 2026-07-04 — already 51 EDR
|
||||
agents, only 5 BD endpoints left to convert: D1-ENGI-006, DESKTOP-L2LE31M,
|
||||
DATAFORTH-PC, SURFACEOPS, MING-HP). **Final exception: Glaztech only.** Target state
|
||||
per machine (non-exempt) = GuruRMM + Datto EDR + Bitdefender removed. Detail: memory
|
||||
`project_av_migration_bitdefender_to_edr`.
|
||||
- **NO-AV list (revised 2026-07-04 night)**: 8 clients / 20 paid devices — Little Hearts
|
||||
Little Hands (8), Ridgetop Group (3), Residential and Renovation Engineering (2), Janet
|
||||
Altschuler (2 — confirmed SEPARATE from JANC Excavation, Syncro 457710 vs 2248945), Business
|
||||
Services of Tucson LLC (2), Gary A Hartman LLC (1), Robyn Pittman (1), Marty Ryan (1).
|
||||
- **NO-AV round 1**: Datto EDR deployed to the 5 reachable machines. 4/5 confirmed
|
||||
ACTIVE; Robyn Pittman's `DESKTOP-PL2RCGL` failed mid-delivery (went offline) — retry
|
||||
Monday, reg key in tracker.md. Rest ride autoenroll -> RMM -> EDR-push as they surface.
|
||||
flagged for Mike's billing review, not treated as 154 missing agents. Its 2nd backup
|
||||
seat (`GTI-INV-VMHOST`) is also 251 days stale — folded into the backup REVIEW verdict.
|
||||
- **AV strategy**: Bitdefender -> Datto EDR for all clients. **Final exception:
|
||||
Glaztech only** (Dataforth was originally excepted too, then fully migrated
|
||||
2026-07-04 — its final 5 legacy BD endpoints converted during the 2026-07-05 mass
|
||||
deploy, except two genuine legacy-OS blocks below). Target state per machine
|
||||
(non-exempt) = GuruRMM + Datto EDR + Bitdefender removed. Bitdefender removal is
|
||||
**not yet started** anywhere — deferred per-client, only after that client's EDR is
|
||||
confirmed ACTIVE. Detail: memory `project_av_migration_bitdefender_to_edr`.
|
||||
- **NO-AV mass deploy (2026-07-05)**: `edr-deploy.py` pushed EDR to ~78 machines across
|
||||
~25 clients in one throttled run (74/79 clean on the first pass). Two genuine
|
||||
legacy-OS blocks remain: **AMT-PC** (Windows 7 Home SP1, PowerShell 2.0 — modern
|
||||
Infocyte agent crashes on execution) and **SRV02** (Curtis Plumbing, Windows Server
|
||||
2008 R2 — agent launches but no-ops silently). Both need OS prerequisites or an
|
||||
onsite/legacy-agent path. Five apparent "failures" that session were mostly false
|
||||
negatives (Install-EDR is silent in non-interactive mode) — see Patterns below.
|
||||
- **GuruRMM cross-site duplication bug — found, fixed, deployed (v6.77)**: both agent
|
||||
enroll paths in `server/src/ws/mod.rs` deduped only by `(site_id, device_id)`. A
|
||||
machine already enrolled at one site that ran a different site's installer (new code,
|
||||
or Staging) wasn't found in the target site, so the server minted a duplicate row with
|
||||
the same device_id. Fix: `db::get_agent_by_device_id()` (global lookup) re-homes the
|
||||
existing record instead of creating a duplicate; the enrollment-key path only uses the
|
||||
global match when a real device_id exists, falling back to site-scoped hostname
|
||||
matching otherwise (so distinct same-named machines, e.g. 4x `SERVER`, are never
|
||||
merged). No blanket `UNIQUE(device_id)` constraint added (would collide on legacy
|
||||
hostname-fallback rows). Deployed via cherry-pick (`hotfix-agent-dedup` -> main, not a
|
||||
merge); webhook auto-built 6.66 -> 6.77. Verified live: re-running the staging
|
||||
installer reused the same record; move+restart kept placement. Cleanup: 40 stale
|
||||
orphans deleted (0 failed), 39 one-time moves (0 failed — 24 Dataforth D1->D2, 13
|
||||
Staging->real client, 2 diagnostic corrections). 13 same-hostname/different-device_id
|
||||
pairs deliberately left for manual review. Full plan: `dedup-plan.md`.
|
||||
machine already enrolled at one site that ran a different site's installer minted a
|
||||
duplicate row with the same device_id. Fix: `db::get_agent_by_device_id()` (global
|
||||
lookup) re-homes the existing record instead of creating a duplicate. Deployed via
|
||||
cherry-pick (`hotfix-agent-dedup` -> main). Cleanup: 40 stale orphans deleted, 39
|
||||
one-time moves, 0 failures. Full plan: `dedup-plan.md`.
|
||||
- **ScreenConnect contamination incident**: fleet `sc-cleanup.py` used
|
||||
`GetSessionsByName`, which matches by name across ALL clients, not client-scoped.
|
||||
Writing Company to shared names (`SERVER`, `Accounting`) cross-contaminated other
|
||||
clients' sessions with the last-processed value (11 `SERVER` sessions across ~8
|
||||
clients stamped "Zeus Nestora"; 3 `Accounting` sessions mis-stamped). Fixed (skip any
|
||||
lookup returning >1 session); remediated via `GuestNetworkAddress` (WAN IP) -> client,
|
||||
using a WAN map from unique-named machines only. **5 sessions unrecoverable** (original
|
||||
overwritten, unidentifiable) — 3 `SERVER` + 2 `Accounting`, left blank for manual
|
||||
SC-console tagging. Logged as both correction and friction.
|
||||
clients' sessions with the last-processed value. Fixed (`if len(ss) > 1: continue`);
|
||||
remediated via `GuestNetworkAddress` (WAN IP) -> client. **5 sessions unrecoverable**
|
||||
(3 `SERVER` + 2 `Accounting`) — left blank for manual SC-console tagging. Logged as
|
||||
both correction and friction.
|
||||
- **CP-QB (Curtis Plumbing) root cause found and fixed (2026-07-04/05)**: corrupt WMI
|
||||
class store (`Win32_Processor` = Invalid class, broken since ~2022) killed the
|
||||
universal installer's architecture check and froze its SC inventory since 2022;
|
||||
`winmgmt /verifyrepository` reported CONSISTENT (doesn't catch a missing class) —
|
||||
`mofcomp cimwin32.mof` restored it. Installer then ran clean; enrolled + reassigned.
|
||||
- **IMC-PRINTSERVER installed but TLS-blocked**: agent fully installed and running
|
||||
(manual no-PowerShell install over LAN, since PowerShell is dead on the box and SMB
|
||||
hangs), but the box silently drops ALL outbound TLS (even google.com — ClientHello
|
||||
then silence) while ScreenConnect works (non-TLS relay). Escalated to Mike (by-design
|
||||
lockdown vs. fix); **no reinstall needed** — the agent self-enrolls the instant TLS
|
||||
egress is unblocked.
|
||||
|
||||
### Backup findings (2026-07-05/07, new this week)
|
||||
- **Only 16 of 40 GPS clients actually bill a backup line** — a Syncro sub-agent pull
|
||||
of recurring-schedule Data Backup line-item NAME (not description, to avoid the
|
||||
boilerplate false-positive) corrected the tracker's old `Svc=B` flags for Horseshoe
|
||||
Management, Quantum Wealth Management, T&C Sorensen, and Business Services of
|
||||
Tucson — none of the four actually bill backup.
|
||||
- **7 of the 16 billed clients are NOT backing up**: Grabb & Durando (`GND-SERVER`
|
||||
cloud plan failing, 0 bytes — flapping), Bill Tedards (`DESKTOP-SUFJR0J` last good
|
||||
359 days ago), Arizona Medical Transit (MSP360 company exists, 0 plans), and 4
|
||||
**endpoint-verified 2026-07-06** with a full on-machine scan (no backup software,
|
||||
service, scheduled task, WSB, or Datto backup client on any machine, and nothing in
|
||||
MSP360/B2/Seafile/ownCloud/Datto Workplace): Reliant Well Drilling, The Prairie
|
||||
Schooner (a Windows Server Backup to USB exists on `TPS-SERVER` but last ran
|
||||
3/7/2018, ~8 years dead), Janet Altschuler (only a stopped Dropbox client), Design
|
||||
and Brand Envoys (`OSGOOD-PAUL` was offline/unscanned).
|
||||
- **2 REVIEW verdicts** — seats billed exceed machines actually backing up: Jimmy
|
||||
Company (12 seats / 1 machine, `Blaster2`, known 12/1 anomaly) and Glaz-Tech
|
||||
Industries (2 seats / 1 truly current — `GTI-INV-VMHOST` 251 days stale).
|
||||
- **Revenue leaks** (backing up, not billed — held for Mike/Winter, nothing sent):
|
||||
Cascades of Tucson (`CS-SERVER` daily, B2 `ACG-Cascades`), Len's Auto Brokerage
|
||||
(`LAB-SVR`, now stale, B2 `ACG-Lens`), Russo Law Firm (`RUSSO-SRV` 2.7 TB cloud +
|
||||
Seafile 2.2 TB), Horseshoe Management (`HSM-NewServer` daily, B2 `Horseshoe`),
|
||||
Business Services of Tucson (`KAT1` daily, B2 `ACG-BST` + ownCloud `bst`).
|
||||
- **$0-priced backup lines** (billed a seat at $0 — bundled or under-billed, flag for
|
||||
Mike/Winter): Ridgetop Group, Janet Altschuler, Design and Brand Envoys, Arizona
|
||||
Medical Transit.
|
||||
- **Orphan B2 buckets resolved via MSP360 CompanyName**: `ACG-PST` = Peaceful Spirit
|
||||
(22,054 files, non-GPS), `ACG-TCA` = Tucson Coin and Autograph (non-GPS) — corrects an
|
||||
earlier tentative guess that `ACG-TCA` was T&C Sorensen. Other identified orphans:
|
||||
`ACG-REDNOUR` = Rednour Law, `ACG-Rohrbach` = Rohrbach, `ACG-IX`/`ACG-Internal`/
|
||||
`ACG-Tech03L` = ACG internal, `MSPBackups20200311` = stale ex-client FSG (deletion
|
||||
candidate). ownCloud `mara`/`jburger` still unidentified (verify).
|
||||
- **Datto Workplace research**: confirmed there is no admin/team/device REST API at
|
||||
all — the public API is strictly file-sharing; user/storage administration is
|
||||
Workplace-Manager-portal-only. A Super Administrator toggle on the existing API
|
||||
integration (or team-sharing projects to it) is the only way to make `/file/projects`
|
||||
enumerate the whole team. That toggle is a portal action Claude cannot perform — Mike
|
||||
is provisioning a Super-Administrator-scoped integration; parked until it lands.
|
||||
- **Rule (Howard, standing)**: backup targets are never guessed. A billed seat count
|
||||
higher than machines-currently-backing-up is a Mike/Winter billing question, not a
|
||||
deploy instruction. Memory: `feedback_backup_targets_never_guessed`. Corollary added
|
||||
2026-07-07: only check backup status for clients that actually bill a backup line
|
||||
(e.g. stop checking Marty Ryan, which has none) — memory
|
||||
`feedback_backup_check_only_billed`.
|
||||
|
||||
## Patterns & Known Issues
|
||||
|
||||
- **`GetSessionsByName` matches across ALL SC clients** — never write properties to a
|
||||
name returning >1 session (`if len(ss) > 1: continue`); root cause of the contamination
|
||||
incident above.
|
||||
name returning >1 session (`if len(ss) > 1: continue`); root cause of the
|
||||
contamination incident above. The extension also exposes no full-fleet listing —
|
||||
finding a specific client's machines requires already knowing hostnames.
|
||||
- **`/api/agents/:id/move` intermittently 500s under rapid calls** (transient DB/load;
|
||||
handler is a clean UPDATE) — throttle 2s + retry 5x reliably rides over it.
|
||||
- **Syncro `last_synced_at` goes stale** when agent check-in breaks, making an online
|
||||
machine look offline 45+ days. Use SC `ActiveConnections` `ProcessType==2` +
|
||||
`GuestInfoUpdateTime` instead (originally read `GuestConnectedCount`, always null).
|
||||
- **Datto EDR org agent/site counts are stale rollups** after re-parenting Locations
|
||||
(`PATCH /Locations/{id} {organizationId}`) — trust a fresh `GET /Locations` grouped by
|
||||
org, not the org-list summary. One raw `GET` also silently ignored a `--filter` param —
|
||||
verify with an unfiltered fetch + manual group-by when in doubt.
|
||||
`GuestInfoUpdateTime` instead.
|
||||
- **Datto EDR org agent/site counts are stale rollups** after re-parenting Locations —
|
||||
trust a fresh `GET /Locations` grouped by org, not the org-list summary.
|
||||
- **`Install-EDR` is silent in non-interactive mode** — every status/error line is
|
||||
gated behind `if ($Interactive)`; a "completed / blank output" RMM result means
|
||||
nothing. The visible "Installed RTS agent" text comes from `agent.windows.exe`
|
||||
itself, so its absence just means the exe never ran (early-return on an existing
|
||||
service, or a crash). Verify with `-Interactive *> file` + read-back, or check
|
||||
`Get-Service huntAgent` + the binary path directly. `-Force` reinstalls over a stale
|
||||
huntAgent service left by a broken partial install.
|
||||
- **RMM command output is in `.stdout`, not `.output`**, on `GET /api/commands/:id`.
|
||||
- **Syncro recurring-schedule line matching must use the line NAME, not description** —
|
||||
base GPS Workstation/Server SKUs carry the word "Backups" in boilerplate description
|
||||
text, producing false-positive "bills for backup" matches.
|
||||
- **MSP360 bucket presence != a current backup** — B2/Seafile/ownCloud show a
|
||||
destination exists; MSP360 `/api/Monitoring` LastStart+Status is the only real
|
||||
freshness signal. `/api/Computers` is gated off (HTTP 400, feature not enabled on
|
||||
this account) — use Monitoring/Companies/Users instead.
|
||||
- **Datto backup-detection regex must be precise** (`"Datto Workplace"`/`"Workplace2"`)
|
||||
— a bare `"Datto"` pattern false-matches Datto EDR/RMM/AV agents that have nothing to
|
||||
do with backup.
|
||||
- AV coverage splits by client size (Datto for large, Bitdefender historically for
|
||||
small) — check **both** before declaring a gap; formalized by the Phase 4 cid join.
|
||||
- SC coverage varies per client — universal on some (IMC/Horseshoe/QWM/Safesite), sparse
|
||||
on others (Grabb & Durando) — verify a live session exists before assuming the channel
|
||||
works; where absent, discover via Syncro assets instead.
|
||||
small) — check **both** before declaring a gap.
|
||||
- SC coverage varies per client — verify a live session exists before assuming the
|
||||
channel works; where absent, discover via Syncro assets instead.
|
||||
- The client wiki is the primary source of truth for machine -> department/person/
|
||||
location during SC cleanup; where missing, learn it and write it back.
|
||||
location during SC cleanup, and now also for **which backup backend a client uses**
|
||||
(per the consolidated wiki-first skill routing) — where missing, learn it and write
|
||||
it back.
|
||||
- **GuruRMM install scripts are bootstrappers, not offline installers** — the
|
||||
`/install/<code>/<platform>` endpoints return a script that downloads the
|
||||
arch-matched binary at run time (needs onsite internet); raw binaries for true
|
||||
offline install live at `/install/<code>/download/<variant>`.
|
||||
|
||||
## Operations Playbook
|
||||
|
||||
- **Daily 8:07am** — `GPS-RMM-Progress` runs `gps-rmm-progress-check.sh`, DMs Howard the
|
||||
gaps vs `targets.json`; reports COMPLETE and retires when met.
|
||||
- **Every 30 min from Mon 2026-07-06 06:00** — `GPS-RMM-AutoEnroll` runs
|
||||
`gps-rmm-autoenroll.sh` (push Staging to online-but-missing machines, wait 150s,
|
||||
reassign, post to `#dev-alerts`). Retire once fully enrolled.
|
||||
- **Add a client (Bucket C pattern)**: `bucketc-onboard-deploy.sh "<Client>" <bd_company_id>
|
||||
<slug>` — creates client+site, vaults the key, deploys to reachable Bitdefender
|
||||
machines. Domain-joined pattern: DC already in RMM -> AD is the authoritative list ->
|
||||
push the site installer to domain members via SC (IMC template — DC remote-exec,
|
||||
WMI/DCOM, `schtasks /S`, WinRM all failed on default Win10/11 clients; SC
|
||||
`send-command` as SYSTEM was the only clean channel). Steps + encoding:
|
||||
`reference_rmm_deploy_via_screenconnect`.
|
||||
- **Auto-enroll (manual, on demand)** — `GPS-RMM-AutoEnroll`'s 30-minute schedule is
|
||||
**disabled** as of 2026-07-07 (Howard: manual sweeps only). Run
|
||||
`bash .claude/scripts/gps-rmm-autoenroll.sh` by hand to push Staging to online-but-
|
||||
missing machines, wait, reassign, and post to `#dev-alerts`.
|
||||
- **Add a client (no-footprint pattern)**: `bucketc-onboard-deploy.sh "<Client>"
|
||||
<bd_company_id> <slug>` — creates client+site, vaults the key, deploys to reachable
|
||||
Bitdefender machines. Domain-joined pattern: DC already in RMM -> AD is the
|
||||
authoritative list -> push the site installer to domain members via SC. If no
|
||||
BD/Datto footprint exists at all, onboard anyway (mints the site code) and deploy
|
||||
onsite or via the flash-drive kit.
|
||||
- **Deploy Datto EDR**: create-group -> mint-key (via `datto-edr` skill) -> push
|
||||
`Install-EDR` through GuruRMM (`POST /api/agents/:id/command`, visible stdout) rather
|
||||
than blind SC push (failed on offline-prone boxes, e.g. Dataforth `DATAFORTH-PC`, same
|
||||
as `CP-QB`). Reg keys tracked in `tracker.md`, not reproduced here.
|
||||
`Install-EDR` through GuruRMM (`POST /api/agents/:id/command`, visible stdout only
|
||||
with `-Interactive`) rather than blind SC push. Re-run `edr-deploy.py` to sweep newly
|
||||
online machines; it recomputes coverage from a live EDR dump each run.
|
||||
- **SC metadata cleanup, per client**: wiki first, then hostname role tokens, then UniFi
|
||||
switch/AP naming; run `sc-cleanup.py` for the safe no-guess pass; never touch Site
|
||||
without a topology source.
|
||||
- **Backup verification, per client**: check the client wiki + `backup-map.md` first
|
||||
for which backend applies (never assume B2/MSP360); confirm with the matching skill
|
||||
(`msp360`/`b2`/`seafile`/`owncloud`/`datto-workplace`); only check clients that
|
||||
actually bill a backup line (`gps-master-status.md` "Billed for backup" table).
|
||||
Endpoint-scan directly (RMM PowerShell) when the destination-side tools are
|
||||
inconclusive or blocked (as done for the 4 Datto-pending clients).
|
||||
- **Onsite visit**: use the flash-drive kit at `Desktop\CLIENTS\<slug>\` — per-site
|
||||
installer scripts (Windows/Linux/macOS) + README; multi-site clients have
|
||||
per-location subfolders with the correct site code for that location.
|
||||
|
||||
## History Highlights
|
||||
|
||||
- **2026-07-03** — Audit started; 40 active GPS clients identified; `tracker.md` built
|
||||
(Bucket A 7 / B 8 / C 25); `GPS-RMM-Progress` registered (baseline 46/189).
|
||||
- **2026-07-03** — Bucket A verified end-to-end; renamed mislabeled client "Russo,
|
||||
Steve" -> Russo Law Firm; set the AV strategy (Bitdefender -> Datto EDR).
|
||||
- **2026-07-03** — IMC deploy as the template: DC remote-exec failed 4 ways; pivoted to
|
||||
SC send-command (1 -> 12 agents); saved as `reference_rmm_deploy_via_screenconnect`.
|
||||
- **2026-07-03** — Bucket B push (IMC/Horseshoe/QWM/Safesite/Grabb) via SC; Bucket C
|
||||
onboarded (16 clients, ~44 agents) via `bucketc-onboard-deploy.sh`.
|
||||
- **2026-07-03** — Discovery pivoted to Syncro assets (authoritative); `needs-sc.py`
|
||||
built; found Syncro's stored SC session GUID on assets was stale.
|
||||
- **2026-07-03 evening** — SC session-hygiene cleanup began (Dataforth, then Cascades):
|
||||
reverse-engineered CP1-CP8 slots; normalized Company, set Site/Dept/Device Type via
|
||||
UniFi + client wiki.
|
||||
- **2026-07-04** — Fleet-wide SC easy-win pass (~45 clients); found + remediated the
|
||||
cross-contamination bug from shared hostnames.
|
||||
- **2026-07-04** — Corrected online-detection logic (SC `ActiveConnections` vs stale
|
||||
Syncro `last_synced_at`); Staging auto-enroll pipeline proven end-to-end (15 machines).
|
||||
- **2026-07-04** — GuruRMM dup-agent bug found/fixed/deployed (v6.77); 40 orphans deleted
|
||||
+ 39 moves executed, 0 failures.
|
||||
- **2026-07-04** — SC<->RMM consistency pass across all 333 agents; closed the no-SC gap
|
||||
for 6 machines by pushing the SC installer through RMM itself.
|
||||
- **2026-07-04** — Phase 4 AV matrix built (Bitdefender cid join vs GPS billing vs
|
||||
EDR): 9/22 NO-AV, 27-client/141-endpoint BD->EDR migration scope identified.
|
||||
- **2026-07-04 night** — EDR "Default RMM Org" dismantled: 18 orgs created, 21 locations
|
||||
re-parented; NO-AV revised to 8 clients/20 devices.
|
||||
- **2026-07-04 night** — NO-AV round 1: EDR deployed to 5 machines, 4/5 active;
|
||||
`GPS-RMM-AutoEnroll` registered; session closed at 111/189, verification sweep green.
|
||||
(Bucket A 7 / B 8 / C 25); `GPS-RMM-Progress` registered (baseline 46/189). IMC deploy
|
||||
established the ScreenConnect send-command channel as the working deploy pattern
|
||||
after 4 remote-exec methods failed. Bucket B/C pushed and onboarded (~60 agents).
|
||||
- **2026-07-04** — Fleet-wide SC cleanup pass (~45 clients); found + remediated the
|
||||
shared-hostname cross-contamination bug. Corrected online-detection logic (SC
|
||||
`ActiveConnections` vs stale Syncro `last_synced_at`); Staging auto-enroll pipeline
|
||||
proven end-to-end. GuruRMM dup-agent bug found/fixed/deployed (v6.77); 40 orphans
|
||||
deleted + 39 moves, 0 failures. Phase 4 AV matrix built (9/22 NO-AV); EDR "Default
|
||||
RMM Org" dismantled (18 orgs created, 21 locations re-parented), NO-AV revised to
|
||||
8 clients/20 devices; round-1 EDR remediation (4/5 active). Session closed at
|
||||
111/189 enrollment.
|
||||
- **2026-07-05** — CP-QB fixed (corrupt WMI class store) and enrolled. IMC-PRINTSERVER
|
||||
installed but found TLS-egress-blocked (escalated to Mike). Full Bitdefender ->
|
||||
Datto EDR mass deploy: `edr-deploy.py` protected ~78 machines across ~25 clients in
|
||||
one throttled run (authorized EDR agents 228 -> 302); 2 genuine legacy-OS blocks
|
||||
identified (AMT-PC Win7/PS2, SRV02 Server 2008 R2). Held billing findings verified
|
||||
and sent to Mike/Winter/Howard via Discord DM. Built the `seafile`, `owncloud`, and
|
||||
`datto-workplace` skills (+ shared `backup_common.py`) after Howard corrected the
|
||||
assumption that backup destinations were B2-only. Built the `msp360` skill (full
|
||||
CRUD) once its API was found to be the authoritative backup-status source; code
|
||||
review (18 agents) fixed 10 findings including a credential-leak in dry-run output.
|
||||
Consolidated backup skill routing behind a wiki-first rule. Ran the first full
|
||||
40-client backup verification (16 of 40 actually bill backup; 7 OK / 2 REVIEW /
|
||||
3 confirmed-not-backing-up + 4 held pending Datto).
|
||||
- **2026-07-06** — Deleted the wrong weekend findings DM to Winter (superseded by the
|
||||
MSP360 + Syncro-seat data). Endpoint-verified the 4 held clients (Reliant, Prairie
|
||||
Schooner, Janet Altschuler pending onboard, Design and Brand Envoys) — all confirmed
|
||||
NOT backing up. `backup-map.md` fully rewritten with MSP360 as source of truth.
|
||||
- **2026-07-07** — Onboarded Janet Altschuler and Marty Ryan to GuruRMM (machines
|
||||
located via SC hostnames, agents pushed and enrolled); confirmed Janet's machines
|
||||
already have Datto EDR (left alone). Updated the Mike/Winter Discord messages in
|
||||
place (PATCH) with the endpoint-verified results rather than sending new ones. Built
|
||||
`gps-master-status.md` as the single current per-client list (superseding repeated
|
||||
reconciliation of 3 stale docs), consolidating GPS device count, RMM enrollment,
|
||||
backup billing + verdict, and wiki presence for all 40 clients. Created the 20
|
||||
missing client wiki articles via two parallel sub-agents (wiki count 56 -> 76).
|
||||
Onboarded Business Services of Tucson (real machine) and, to generate site codes for
|
||||
the onsite kit, Andy's Mobile Fuel and Little Hearts Little Hands and Residential and
|
||||
Renovation Engineering (no machine yet). Confirmed GuruRMM has Windows, Linux, and
|
||||
macOS agents. Built the flash-drive onsite installer kit at `Desktop\CLIENTS\` (20
|
||||
site folders, per-location subfolders for Grabb & Durando and Glaz-Tech). Disabled
|
||||
the `GPS-RMM-AutoEnroll` 30-minute schedule (manual sweeps only, per Howard).
|
||||
|
||||
## Pending / Next Steps (Monday queue, 2026-07-06)
|
||||
## Pending / Next Steps
|
||||
|
||||
- Run `GPS-RMM-AutoEnroll` (armed 06:00) to harvest the ~25 machines active within 14
|
||||
days as offices reopen.
|
||||
- Dataforth EDR tail: convert the remaining 4 BD endpoints (D1-ENGI-006, DATAFORTH-PC,
|
||||
SURFACEOPS, MING-HP — DESKTOP-L2LE31M was reinstalled/retired, stale BD record deleted
|
||||
2026-07-04), then remove BD from Dataforth.
|
||||
- Retry the failed EDR install on `DESKTOP-PL2RCGL` (Robyn Pittman).
|
||||
- Read the error files left on `CP-QB` and `IMC-PRINTSERVER`
|
||||
(`C:\Windows\Temp\gururmm-install-err.txt`) from the failed holiday-evening pushes.
|
||||
- Confirm `avInstalled` actually enables via org policy for the round-1 EDR deploys.
|
||||
- Locate or confirm-unmanaged: Little Hearts Little Hands, Residential and Renovation
|
||||
Engineering, Janet Altschuler (separate from JANC — confirmed 2026-07-04), Business
|
||||
Services of Tucson LLC, Marty Ryan.
|
||||
- Send held billing findings to Winter/Mike once fully verified (Dataforth +8, Cascades
|
||||
dup+unbilled backup, Len's LAB-SVR offline, AMT backup destination unknown,
|
||||
Jimmy/Stamback/Glaz-Tech anomalies).
|
||||
- Continue Phase 4: backup verification (B2/MSP360 vs billed lines), then email host
|
||||
verification (several "Exchange Hosted Email" lines still undocumented).
|
||||
- Manual SC-console cleanup: remove flagged duplicate sessions (Cascades
|
||||
`RECEPTIONIST-PC`, Dataforth `eng-dev-server`, Grabb `DESKTOP-NFK4F5P`, Safesite
|
||||
`0226-Lenovo`/`MSI`); manually tag the 5 unrecoverable blank sessions.
|
||||
- **Howard onsite tour** with the `Desktop\CLIENTS\` flash-drive kit to close the 7
|
||||
zero-RMM-footprint clients in person (Andy's Mobile Fuel, Business Services of
|
||||
Tucson — machine `KAT1` etc. queued for reconnect, Heieck/Little Hearts/Residential
|
||||
and Renovation Engineering/The Marc Group/MVAN — verify which still need a physical
|
||||
visit vs. a reconnect). Report machine details back into each client wiki +
|
||||
`gps-master-status.md` as they land.
|
||||
- **Mike/Winter still owe replies** (as of 2026-07-07) on: the 4 endpoint-verified
|
||||
not-backing-up clients (Reliant, Prairie Schooner, Janet Altschuler, Design and Brand
|
||||
Envoys — decision needed: bill correctly, deploy real backup, or drop the line), the
|
||||
Datto Workplace Super Administrator API key (current key = 0 projects, blocking full
|
||||
Datto enumeration), and the general backup billing decisions below.
|
||||
- **7 FAIL + 2 REVIEW backup verdicts need a billing/remediation decision**: Grabb &
|
||||
Durando (failing cloud plan), Bill Tedards (359d stale), Arizona Medical Transit (0
|
||||
plans), Reliant/Prairie Schooner/Janet Altschuler/Design and Brand Envoys (endpoint-
|
||||
confirmed none), Jimmy Company (12/1 seat mismatch), Glaz-Tech (2/1, 2nd server 251d
|
||||
stale). None of these are deploy instructions per the standing rule — Mike/Winter
|
||||
decide bill-vs-deploy-vs-drop.
|
||||
- **Revenue-leak decisions** (bill vs. decommission, held, nothing sent): Cascades,
|
||||
Len's, Russo, Horseshoe, Business Services of Tucson; plus the 4 $0-priced lines
|
||||
(Ridgetop, Janet Altschuler, Design and Brand Envoys, AMT).
|
||||
- **Two legacy-OS EDR blocks** remain unresolved: AMT-PC (Win7 SP1/PS2) and SRV02
|
||||
(Server 2008 R2) — need OS prerequisites, the legacy-agent path, or an onsite visit.
|
||||
- **Bitdefender removal not started anywhere** — per-client, only after EDR is
|
||||
confirmed ACTIVE for that client; do after the current tail is closed.
|
||||
- **IMC-PRINTSERVER** still awaiting Mike's ruling on the TLS-egress lockdown
|
||||
(by-design vs. fix); no reinstall needed once unblocked.
|
||||
- **Manual SC-console cleanup**: remove flagged duplicate sessions (Cascades
|
||||
`RECEPTIONIST-PC` needs a RENAME not a delete — confirmed two live machines sharing
|
||||
one hostname; Dataforth `eng-dev-server`; Grabb `DESKTOP-NFK4F5P`; Safesite
|
||||
`0226-Lenovo`/`MSI`); manually tag the 5 unrecoverable blank sessions from the
|
||||
contamination incident.
|
||||
- **Identify remaining unknowns**: ownCloud `mara`/`jburger` accounts (verify);
|
||||
Mineralogical Record's ownCloud vs. `universal-minerals.md` wiki article — confirm
|
||||
which is current (verify).
|
||||
- Continue Phase 4 to email host verification (several "Exchange Hosted Email" lines
|
||||
across Bucket A clients still undocumented) once the backup tail closes.
|
||||
|
||||
## Backlinks
|
||||
|
||||
*(none yet — link from client wiki articles touched by this audit: Dataforth, Cascades
|
||||
of Tucson, Instrumental Music Center, Arizona Medical Transit, and others as their pages
|
||||
come to reference this project)*
|
||||
*(link from client wiki articles touched by this audit — Dataforth, Cascades of
|
||||
Tucson, Instrumental Music Center, Arizona Medical Transit, Janet Altschuler, Marty
|
||||
Ryan, Reliant Well Drilling, and the 20 newly created client articles as they come to
|
||||
reference this project)*
|
||||
|
||||
Reference in New Issue
Block a user