366 lines
34 KiB
Markdown
366 lines
34 KiB
Markdown
# GPS -> GuruRMM Coverage Audit
|
|
|
|
**Goal:** For every business/client paying for GPS (Guru Protection Service), verify that
|
|
GuruRMM is set up correctly — the org/account exists, the machines they pay for are all
|
|
enrolled and reporting, and the services they pay for (backups, AV, email) are actually
|
|
configured and working. Where the client wiki is missing host/login/provider info, fill
|
|
those gaps as we go (credentials -> SOPS vault via `/vault`).
|
|
|
|
**Source of truth for "should have":** Syncro active recurring schedules (device counts +
|
|
service line items). **Reality:** GuruRMM `/api/agents`, plus backup/AV/email tooling.
|
|
|
|
- Started: 2026-07-03 (Howard)
|
|
- **AV STRATEGY (Howard 2026-07-03):** migrate **Bitdefender -> Datto EDR for ALL clients except Glaztech and Dataforth** (those two keep Bitdefender). Target end-state per machine (non-exempt) = GuruRMM agent + Datto EDR + Bitdefender removed. Bitdefender inventory is now only a discovery source (which machines exist), not a coverage target. See memory `project_av_migration_bitdefender_to_edr`.
|
|
- Scope: 40 **active** GPS clients (4 paused clients excluded: Marcia Ashton, Tucson Mountain Motors, Richard Pittman, Brenda Lopez)
|
|
- GPS device count = sum of GPS workstation + server SKUs (excludes AntiVirus add-on, discounts, setup)
|
|
|
|
## Per-client verification checklist (each client)
|
|
|
|
- [ ] 1. RMM org/account exists and is named correctly
|
|
- [ ] 2. Machine count in RMM matches GPS devices billed (reconcile every host)
|
|
- [ ] 3. Services billed are actually configured + working: Backup / AV / Email / VoIP
|
|
- [ ] 4. Client wiki has: host/provider (email, DNS, web — and whether ACG-managed), admin logins (-> vault), key contacts
|
|
- [ ] 5. Discrepancies logged + remediation started
|
|
|
|
Legend: `MATCH` RMM >= billed · `SHORT (n)` RMM under billed by n · `MISSING` no RMM org ·
|
|
`?` needs investigation. Svc flags from billing: B=Backup A=AV E=Email V=VoIP.
|
|
|
|
---
|
|
|
|
## A. Present in RMM — counts match (verify services + wiki) — 7
|
|
|
|
| done | Client | Syncro CID | GPS billed | RMM machines | Status | Svc | Notes |
|
|
|------|--------|-----------|-----------:|-------------:|--------|-----|-------|
|
|
| [ ] | Dataforth Corp | 578095 | 43 | 51 | MATCH (RMM+8) | B A E | RMM has more than billed — reconcile extras |
|
|
| [ ] | Cascades of Tucson | 20149445 | 29 | 33 | MATCH (RMM+4) | A E V | |
|
|
| [ ] | Valley Wide Plastering | 31694734 | 29 | 28 | MATCH (~) | B | short 1, within reason |
|
|
| [ ] | Len's Auto Brokerage | 3289131 | 8 | 8 | MATCH | E | |
|
|
| [ ] | Arizona Medical Transit | 7088349 | 1 | 2 | MATCH (RMM+1) | B E V | |
|
|
| [ ] | AT Trebesch | 238740 | 1 | 1 | MATCH | - | |
|
|
| [ ] | Russo Law Firm | 23331699 | 3 | 3 | MATCH | A E V | Renamed 2026-07-03 from mislabeled "Russo, Steve" (Steve Russo owner, Shannon Trionfo contact) |
|
|
|
|
### Bucket A findings (discovery 2026-07-03)
|
|
|
|
- **Dataforth Corp** — 51 agents vs 43 billed GPS (**+8**). Possible under-billing / uncounted machines — several look like personal boxes (DESKTOP-*, LAPTOP-RD47E88A, Test01). Reconcile host-by-host with Mike; confirm which are billable. Wiki: `dataforth.md` exists.
|
|
- **Cascades of Tucson** — 33 agents vs 29 billed (**+4**). `RECEPTIONIST-PC` appears **twice** in RMM — likely a duplicate/stale agent record to clean up. Wiki: `cascades-tucson.md` exists.
|
|
- **Valley Wide Plastering** — 28 agents vs 29 billed (short 1). Effectively reconciled. Wiki: `valleywide.md` exists.
|
|
- **Len's Auto Brokerage** — 8 agents = 8 billed (MATCH). **FLAG: LAB-SVR (production Server 2019) agent offline since 2026-06-18** (~2 wks) — verify box/agent health. Email = 1x M365 Apps for Business; **email host/provider not documented** in wiki (gap). Wiki: `lens-auto-brokerage.md` thorough.
|
|
- **Arizona Medical Transit** — 2 agents (AMT-HYPERV + AMT-PC) vs 1 billed. **No wiki article exists** — create one (host/provider, logins -> vault).
|
|
- **AT Trebesch** — 1 agent = 1 billed (MATCH). Wiki: `attrebesch.md` exists.
|
|
- **Russo Law Firm** — 3 agents = 3 billed (MATCH). Org rename applied today. Sites: Main (has all 3) + empty "Shannon" site — consider moving STRIONFO to the Shannon site. Wiki: `russo-law.md` exists.
|
|
|
|
**Still to verify per client (services + wiki):** backups (none billed for most of A except Dataforth/VWP/AMT), AV coverage vs billed AV seats, email host documented, admin logins in vault.
|
|
|
|
#### Backup layer (B2/MSP360) findings
|
|
- **Dataforth** — `ACG-Dataforth` bucket present w/ data (billed B) [OK dest exists]
|
|
- **Valley Wide** — `VWP-Backup` bucket present w/ data (billed B) [OK dest exists]
|
|
- **Arizona Medical Transit** — **billed Data Backup but NO dedicated B2 bucket** — destination unknown (Datto? shared bucket?). VERIFY where AMT backup lands.
|
|
- **Cascades** — `ACG-Cascades` bucket present w/ data **but no Data Backup line item billed** — possible unbilled backup / revenue leak, or legacy. Confirm w/ Mike.
|
|
- **Len's Auto** — `ACG-Lens` bucket present w/ data **but backup not billed** (Svc=E only) — same question as Cascades.
|
|
- Caveat: bucket file lists are name-ordered, not time-ordered — "backup ran today" freshness must be confirmed in the MSP360 console; bucket presence only proves a destination is configured.
|
|
- Other buckets not tied to a bucket-A client: ACG-BST, ACG-Brett, ACG-GLAZTECH, ACG-IX, ACG-PST, ACG-REDNOUR, ACG-Rohrbach, ACG-TCA, Horseshoe, ACG-Internal, MSPBackups20200311 (stale — 2021, ex-client FSG).
|
|
|
|
#### AV layer findings (AV split across TWO tools — Datto AV is primary for big clients, Bitdefender for smaller)
|
|
- **Dataforth** — billed 43 AV. **Datto EDR: 51 agents** (org 4a2664bf) — covered [OK]. (Bitdefender also has 5 — legacy/partial; Datto is primary.)
|
|
- **Cascades** — billed 29 AV. **Datto EDR: 34 agents** (org 2d5ea96e) — covered [OK]. Bitdefender company exists but 0 endpoints — Cascades AV lives entirely in Datto.
|
|
- **Russo Law Firm** — billed ~5 AV. **Bitdefender: 6 endpoints** (company 60abfa4c) — covered [OK], but STRIONFO listed **twice** in Bitdefender (dedupe stale record). Not the primary in Datto.
|
|
- Lesson for the audit: AV coverage is NOT single-tool — must check BOTH Datto EDR and Bitdefender before declaring an AV gap. Bitdefender company names carry the Syncro CID suffix (`_NNNNN`) which makes mapping exact.
|
|
- Datto "Default RMM Org" (35 agents, 23 sites) is a catch-all — small clients' Datto agents may sit there unsegmented; relevant when we reach buckets B/C.
|
|
|
|
#### Email + vault findings
|
|
- **Vault:** all 7 A clients have entries. **Dupes to consolidate:** `russo` + `russo-law`, and `valleywide` + `vwp`. AMT had a vault entry (RMM keys) but no wiki (now created).
|
|
- **Email hosts (from billing — several need the actual mail host documented):**
|
|
- Dataforth — Pax8 M365 (Exchange Online P1 + M365 Business Std): ACG-managed M365 [OK]
|
|
- Cascades — 45 M365 Business Premium **+ 235 "Exchange Hosted Email"**: large hosted-Exchange footprint, **host not documented** [GAP]
|
|
- Len's Auto — only 1 M365 *Apps for Business* (no mailbox license): actual **email host unknown** [GAP]
|
|
- Arizona Medical Transit — 5 "Exchange Hosted Email": **host not documented** [GAP]
|
|
- Russo Law — 5 "Exchange Hosted Email": **host not documented** [GAP]
|
|
- AT Trebesch — no email billed
|
|
- "Exchange Hosted Email" is a recurring unknown across A (and likely B/C) — one host to identify (ACG-hosted Exchange vs a third party). Resolve once, apply everywhere.
|
|
|
|
#### Bucket A verification rollup (2026-07-03)
|
|
- **Machines:** reconciled 7/7 (findings above). **Backups:** mapped 7/7 (3 billing flags held for Winter). **AV:** verified 3/3 AV-billed clients covered (Datto + Bitdefender). **Vault:** present 7/7. **Wiki:** 6 existed + AMT created = 7/7.
|
|
- **Remaining open (documentation, not coverage gaps):** email host for Cascades/Len's/AMT/Russo; Dataforth +8 billing reconcile; Cascades dup agent + Bitdefender dup (STRIONFO); Len's LAB-SVR offline; vault dupe consolidation. All logged; nothing outbound to Winter until the full list is verified.
|
|
|
|
## B. Present in RMM — SHORT (missing agents to deploy) — 8
|
|
|
|
| done | Client | Syncro CID | GPS billed | RMM machines | Gap | Svc | Notes |
|
|
|------|--------|-----------|-----------:|-------------:|----:|-----|-------|
|
|
| [ ] | Glaz-Tech Industries | 143932 | 159 | 5 | 154 | B A E | ANOMALY — 149x GPS basic + 10x GPS Pro Server billed; verify billing is real vs legacy before treating as 154 missing |
|
|
| [ ] | Instrumental Music Center | 7088508 | 20 | 1 | 19 | A E V | |
|
|
| [ ] | Jimmy Company | 18560272 | 12 | 1 | 11 | B A | |
|
|
| [ ] | Horseshoe Management | 625269 | 9 | 1 | 8 | B E | |
|
|
| [ ] | Safesite LLC | 26563106 | 37 | 31 | 6 | A E | |
|
|
| [ ] | Stamback Septic | 11513046 | 8 | 3 | 5 | V | |
|
|
| [ ] | Grabb & Durando Law Office | 14232794 | 12 | 9 | 3 | B A E | |
|
|
| [ ] | Quantum Wealth Management | 7088747 | 3 | 2 | 1 | B E V | |
|
|
|
|
### Bucket B coverage matrix (RMM vs Datto AV vs Bitdefender, 2026-07-03)
|
|
|
|
| Client | GPS billed | RMM | Datto | Bitdef | Read |
|
|
|--------|----------:|----:|------:|-------:|------|
|
|
| Glaz-Tech Industries | 159 | 5 (all servers) | 5 | 242 | **ANOMALY** — RMM+Datto = 5 real infra boxes; Bitdefender 242 is years of stale enrollments; 149 GPS-basic billing not backed by real machines. HUMAN review (Mike). |
|
|
| Instrumental Music Center | 20 | 1 | 0 | 22 | **Real gap** — ~22 workstations exist (Bitdefender AV) but only IMC1 in RMM. Deploy ~19 RMM agents. |
|
|
| Horseshoe Management | 9 | 1 | 6 | 7 | **Real gap** — 6-7 machines exist (Datto+BD), only HSM-NewServer in RMM. Deploy ~5-8 agents. |
|
|
| Safesite LLC | 37 | 31 | 48 | 16 | **Real gap** — 48 in Datto, RMM 31. Machines exist; RMM short ~6+. Dedupe RMM `MSI` (listed twice). |
|
|
| Grabb & Durando | 12 | 9 | 0 | 15 | **Real gap** — 15 in Bitdefender, RMM 9. Deploy ~3-6 agents. |
|
|
| Quantum Wealth Mgmt | 3 | 2 | 0 | 4 | **Small gap** — BD 4, RMM 2. Add ~1-2 agents. |
|
|
| Jimmy Company | 12 | 1 | 0 | 1 | **BILLING FLAG** — only 1 machine managed anywhere (RMM Blaster2 / BD 1). Billed 12 -> either stale billing OR 11 unmanaged+unprotected machines. Investigate. |
|
|
| Stamback Septic | 8 | 3 (2 uniq) | 0 | 2 | **BILLING FLAG** — 2-3 machines managed anywhere, billed 8. Same question as Jimmy. RMM `DESKTOP-BTR2AM3` listed twice (dedupe). |
|
|
|
|
**Split:** Real RMM-deploy gaps -> IMC, Horseshoe, Safesite, Grabb, QWM (~34-52 agents to push where the box already runs Datto/BD AV). Billing/coverage review (for Winter/Mike, document only) -> Glaz-Tech, Jimmy, Stamback. RMM dedupes -> Safesite `MSI` x2, Stamback `DESKTOP-BTR2AM3` x2.
|
|
Bitdefender companies exist for ALL bucket-B (and nearly all bucket-C) clients with the Syncro CID in the name — AV is broadly deployed even where RMM is not.
|
|
|
|
#### IMC deep-dive (template client for the deploy pattern, 2026-07-03)
|
|
- **IMC1 = Primary DC** for domain `IMC.local` (192.168.0.2), already in RMM; Domain Admin cred `IMC\guru` vaulted (`clients/imc/imc1.sops.yaml`). RMM site: **IMCMain / INNER-BRIDGE-8354**.
|
|
- **True active fleet ~22** (AD objects with 2026 logons == Bitdefender's 22). Billed 20 GPS — legit.
|
|
- **RMM has only IMC1** -> **21 active domain machines need the agent.**
|
|
- Deploy vehicle: push GuruRMM site MSI (INNER-BRIDGE-8354) from the DC to domain members using the vaulted Domain Admin cred (Invoke-Command or a software-install GPO). This is the reusable pattern for any **domain** client (DC already in RMM -> AD is the authoritative list -> push from DC).
|
|
- **AD hygiene finding:** ~24 stale computer objects in IMC.local (Windows 7, last logon 2015-2019) never removed — separate cleanup task.
|
|
- Deploy targets (in Bitdefender, active, not IMC1): IMC-M-EDSERVICE, IMC-SVCSTR, IMC-L1-STATION9, IMC-MINI, IMC-LESSONS, IMC-STATION2, IMC-STATION1, PURCHASINGCOMP, IMC-L1-GRAPHICS, LAPTOP-DCHQ3F92, LAPTOP-PNVA9G51, PHIL2021LAPTOP, IMC-LUIS, DESKTOP-GHG12G3, DESKTOP-JQ0D38J, DESKTOP-URV3UGR, C2B, IMC-PRINTSERVER, DESKTOP-44L80C0, DESKTOP-MR3ALTK, REPAIRADMIN (21).
|
|
|
|
#### IMC DEPLOY EXECUTED 2026-07-03 — via ScreenConnect (channel finding: see memory `reference_rmm_deploy_via_screenconnect`)
|
|
- **DC remote-exec is a dead end** on IMC's Win10/11 clients: DCOM firewalled (WMI "RPC unavailable"), schtasks/S rejected by Win11 from the 2016 DC ("request not supported"), WinRM off. SYSTEM on the DC also can't create GPOs; SSH to IMC1 blocked (Tailscale route not accepting 192.168.0.0/24 + no local key).
|
|
- **Working channel = ScreenConnect send-command** (runs as SYSTEM on the guest, no creds, no firewall issue). Every IMC machine has an SC agent.
|
|
- Pushed `powershell -enc <base64 of: irm '<site>/windows'|iex>` to 20 of 21 targets (2 test + 18 rollout). **IMC-L1-GRAPHICS** has NO SC session (stale 2025 box — handle separately).
|
|
- Result: **RMM IMC agents 1 -> 12 and climbing** (online machines enrolled in ~1-3 min; offline ones queued in SC, install on reconnect). Daily check task tracks to completion.
|
|
- DA-password attempts via RMM were scrubbed (`DELETE /api/commands/:id`, HTTP 204) — no credential persisted. No partial installs from the failed methods.
|
|
|
|
### Bucket B enrollment progress (via ScreenConnect send-command)
|
|
- **IMC** — 1 -> 12 enrolled (site INNER-BRIDGE-8354); ~8 offline queued in SC; IMC-L1-GRAPHICS no SC session.
|
|
- **Horseshoe Management** — 1 -> 4-5 enrolled (site GOLD-OCEAN-4982); pushed to hsm-bill/cathy/frank02/server + desktop-jk4e68n; hsm-cathy + desktop-jk4e68n still installing.
|
|
- **Grabb & Durando** — multi-site (Main LIGHT-PEAK-6399, Bob's House LIGHT-GATE-7086, Jeff's House UPPER-FALCON-8240). **Most BD "gap" machines have NO SC session and are likely stale/duplicate BD records** (real gap ~3, not 6). Only GND-L-3 had an SC session (pushed). HOMEPC flagged — needs house-site assignment. Grabb needs closer per-machine review, not bulk push.
|
|
- **Channel finding:** ScreenConnect coverage VARIES per client — universal on IMC/Horseshoe, sparse on Grabb. Check SC session existence per machine before assuming the channel; where SC is absent, the machine may be stale in Bitdefender or need another channel.
|
|
- **Quantum Wealth** — 2 -> **3 (target met)**. Pushed QUANTUMSERVER + DESKTOP-K89A8CF (site GREEN-CLOUD-1199).
|
|
- **Safesite** — 31 -> **34 and climbing** (20 gap machines pushed, 3 had no SC). NOTE: Safesite has ~48 real machines in Datto vs 37 billed — likely under-billed AND under-deployed. Deployed to the **"Unknown" catch-all site (LIGHT-CLOUD-3585)** because the 3-site split (Bell/Glendale/Unknown) can't be mapped from the asset-tag hostnames — **needs re-siting in the come-back pass.**
|
|
- **Jimmy Company / Stamback Septic** — billing flags: only 1 / ~2 machines exist anywhere (BD/Datto), nothing to enroll. For Winter/Mike billing review.
|
|
|
|
### For the come-back pass (missing machines + issues to fix)
|
|
- Bucket B stragglers: offline machines queued in SC (install on reconnect) — daily check tracks.
|
|
- IMC-L1-GRAPHICS (no SC), Grabb's ~3 real-gap machines (no SC), Safesite's 3 without SC.
|
|
- Safesite: re-site the ~20 machines from "Unknown" to Bell/Glendale; reconcile 48-Datto-vs-37-billed (under-billing?).
|
|
- Grabb HOMEPC: assign Bob's vs Jeff's house site.
|
|
- Billing flags to Winter: Jimmy (12 billed, 1 real), Stamback (8 billed, ~2 real), Glaz-Tech (159 anomaly), + backup mismatches (AMT/Cascades/Len's).
|
|
- Bucket C (25 clients): no RMM org yet — must /rmm onboard (client+site) BEFORE deploying.
|
|
|
|
## C. MISSING from RMM entirely (no org found) — 25
|
|
|
|
| done | Client | Syncro CID | GPS billed | Svc | Notes / verify not under an alias |
|
|
|------|--------|-----------|-----------:|-----|-------|
|
|
| [ ] | Reliant Well Drilling and Pump | 10736261 | 9 | B V | |
|
|
| [ ] | Zeus Nestora | 1196974 | 8 | - | |
|
|
| [ ] | Little Hearts Little Hands | 1144233 | 8 | E | |
|
|
| [ ] | PUTT Land Surveying | 7180175 | 7 | A E | |
|
|
| [ ] | Curtis Plumbing | 416585 | 6 | B A E | |
|
|
| [ ] | The Prairie Schooner | 3664974 | 5 | B E V | |
|
|
| [ ] | Mineralogical Record | 207770 | 5 | B A V | |
|
|
| [ ] | T & C Sorensen | 344886 | 4 | B E | |
|
|
| [ ] | MVAN Enterprises Inc | 29462761 | 4 | A E | |
|
|
| [ ] | Ridgetop Group | 9413367 | 3 | B | |
|
|
| [ ] | Multicultural Counseling Center | 35483539 | 3 | A E | |
|
|
| [ ] | Brett Interiors | 15726057 | 3 | B | |
|
|
| [ ] | Heieck, Sheila | 12045942 | 3 | E | individual-named account |
|
|
| [ ] | The Marc Group | 869073 | 2 | E | |
|
|
| [ ] | Residential and Renovation Engineering | 7088403 | 2 | A V | |
|
|
| [ ] | Bill Tedards | 487887 | 2 | B E V | |
|
|
| [ ] | Janet Altschuler | 457710 | 2 | B | individual-named account |
|
|
| [ ] | Business Services of Tucson LLC | 29338800 | 2 | B | |
|
|
| [ ] | Andy's Mobile Fuel | 27364453 | 2 | E | |
|
|
| [ ] | Design and Brand Envoys | 26747288 | 2 | B A E | |
|
|
| [ ] | Pro-Tech Services | 23702122 | 2 | A | |
|
|
| [ ] | Inside Track Productions | 3021358 | 1 | - | |
|
|
| [ ] | Gary A Hartman LLC | 29038261 | 1 | B | |
|
|
| [ ] | Robyn Pittman | 17031534 | 1 | - | individual-named account |
|
|
| [ ] | Marty Ryan | 140717 | 1 | A E | individual-named account |
|
|
|
|
---
|
|
|
|
## Daily progress check (automated)
|
|
- Windows scheduled task **GPS-RMM-Progress** runs daily 8:07am (Howard-Home), script `.claude/scripts/gps-rmm-progress-check.sh`, targets `projects/gps-rmm-audit/targets.json`. Compares live RMM agent counts (unique hostnames) to GPS device targets and DMs Howard the remaining gaps; reports COMPLETE when all met (then retire via `schtasks /Delete /TN GPS-RMM-Progress`). Baseline 2026-07-03: **46/189 devices in RMM, 32 clients short.** Glaz-Tech excluded pending billing review.
|
|
|
|
### Bucket C — onboarded + deployed 2026-07-03 (via helper `tools/bucketc-onboard-deploy.sh`)
|
|
16 clients onboarded (RMM client+site created, enrollment key vaulted at `clients/<slug>/gururmm-site-main.sops.yaml`), agent pushed via ScreenConnect to SC-reachable machines:
|
|
|
|
| Client | Site code | Deployed via SC | No-SC (come-back) |
|
|
|--------|-----------|:--:|:--:|
|
|
| Reliant Well Drilling | CALM-HAWK-3954 | 4 | 8 (+ FW*/WILCOX* = other entities, skipped) |
|
|
| Curtis Plumbing | SILVER-WOLF-6785 | 4 | 2 |
|
|
| PUTT Land Surveying | EAST-CASTLE-3313 | 3 | 4 |
|
|
| The Prairie Schooner | UPPER-HARBOR-4168 | 3 | 2 |
|
|
| T & C Sorensen | IRON-FORGE-1700 | 4 | 0 |
|
|
| Zeus Nestora | GREEN-TIGER-6194 | 3 | 0 |
|
|
| Brett Interiors | IRON-EAGLE-4784 | 4 | 0 |
|
|
| Bill Tedards | CALM-PEAK-4628 | 2 (Datto src) | 3 (BD, no SC) |
|
|
| Design and Brand Envoys | SOUTH-STAR-8736 | 3 | 0 |
|
|
| Heieck, Sheila | WILD-MOON-9773 | 0 | 3 (BD, no SC) |
|
|
| Multicultural Counseling | EAST-OCEAN-2818 | 3 | 0 |
|
|
| MVAN Enterprises | LOWER-FORGE-6736 | 1 | 1 |
|
|
| The Marc Group | SILVER-OCEAN-6422 | 2 | 0 |
|
|
| Mineralogical Record | BLUE-MOON-8542 | 5 (BD+Datto) | 1 |
|
|
| Pro-Tech Services | INNER-GATE-4746 | 2 | 0 |
|
|
| Inside Track Productions | CALM-GATE-2273 | 1 | 0 |
|
|
|
|
~44 machines deployed. Discovery source = Bitdefender company (mostly), Datto EDR where BD empty (Bill Tedards, Mineralogical extras).
|
|
|
|
**Bucket C NOT onboarded — no machines found in Bitdefender OR Datto (come-back: locate machines or confirm unmanaged):** Little Hearts Little Hands, Janet Altschuler, Business Services of Tucson, Andy's Mobile Fuel, Gary A Hartman LLC, Marty Ryan, Residential and Renovation Engineering, Ridgetop Group, Robyn Pittman (9 clients, 1-8 GPS each). These have no BD/Datto footprint — machines may be SC-only, or genuinely unmanaged/decommissioned.
|
|
|
|
**Reliant caveat:** its Bitdefender company mixes Reliant + Farwest (FW*) + Wilcox (WILCOX*) machines — only clearly-Reliant ones (RWD-*, generics) were targeted; FW*/WILCOX* skipped (separate clients).
|
|
|
|
## Rollup
|
|
|
|
- **7** clients match on machine count (still need service + wiki verification).
|
|
- **8** clients present but short — ~50 agents to deploy (excl. Glaz-Tech anomaly).
|
|
- **25** clients with **no RMM org** — ~86 GPS devices billed, zero RMM presence (some may be under an alias / not yet deployed — verify per client).
|
|
- **Biggest single flag:** Glaz-Tech Industries billed 159 GPS but only 5 RMM agents — confirm the billing is current before acting.
|
|
|
|
## Method notes
|
|
- GPS SKUs matched: GPS basic/monthly, GPS pro/monthly, GPS Workstation, GPS Server, GPS Pro Server (+ variants). Excluded: GPS AntiVirus Add-on, GPS addon, GPS Discount, GPS Set-up, GPS trial.
|
|
- RMM counts from `GET /api/agents` grouped by `client_name`, 2026-07-03.
|
|
- "MISSING" = no `client_name` match in RMM; each must be double-checked for an alias (person name / DBA) before onboarding a duplicate.
|
|
|
|
## Phase 4 — AV coverage matrix (2026-07-04, cid-matched BD + EDR vs GPS billing)
|
|
|
|
Method: BD company names carry the Syncro customer id suffix (_<cid>) = hard join vs targets.json; EDR orgs matched by name. GPS qty = billed devices.
|
|
|
|
**NO AV AT ALL (9 clients, 22 paid devices)** — paying for GPS, zero AV visible:
|
|
Little Hearts Little Hands(8), Ridgetop Group(3), Residential and Renovation Engineering(2), Janet Altschuler(2), Business Services of Tucson LLC(2), Andy's Mobile Fuel(2), Gary A Hartman LLC(1), Robyn Pittman(1), Marty Ryan(1)
|
|
|
|
**PARTIAL (7)**: Valley Wide 27/29, IMC 12/20, Jimmy Company 1/12, Stamback 2/8, Zeus Nestora 4/8, Len's Auto 6/8, MVAN 2/4
|
|
|
|
**Already on EDR**: Dataforth 51, Safesite 48, Cascades 34, VWP 27, Horseshoe 6, Glaztech 5, BirthBiologic 5, Mineralogical 4, Tedards 2, Peaceful Spirit 1 (+Default RMM Org holds 35 UNASSIGNED agents - cleanup)
|
|
|
|
**AV migration scope (task #5, BD->EDR excl Glaztech+Dataforth): 27 clients, 141 BD endpoints.**
|
|
Note: Glaz-Tech BD = 242 endpoints vs 159 GPS billed (feeds the #4 anomaly discussion).
|
|
Next Phase-4 chunk: backup verification (B2/MSP360 per client vs billed backup lines), then email.
|
|
|
|
**Scope update 2026-07-04 (Howard):** AV migration exception narrowed — **ONLY Glaztech stays on Bitdefender**. Dataforth migrates fully to EDR (already 51 EDR agents; remaining 5 BD endpoints to convert: D1-ENGI-006, DESKTOP-L2LE31M, DATAFORTH-PC, SURFACEOPS, MING-HP).
|
|
|
|
**Dataforth EDR tail — Monday 2026-07-06 list:** DATAFORTH-PC, SURFACEOPS, D1-ENGI-006, MING-HP.
|
|
Path: autoenroll loop pulls them into GuruRMM as they come online -> push EDR THROUGH RMM (visible stdout; RegKey 27lzj6npdb, group "Dataforth - D1" 64144044). SC blind-push attempted on DATAFORTH-PC 7/4 did not land (no output channel - same pattern as CP-QB). DESKTOP-L2LE31M: reinstalled/gone (Howard) - stale BD record deleted. After the 4 land: remove Bitdefender from Dataforth entirely.
|
|
|
|
## Phase 4 update — EDR Default-org attribution DONE (2026-07-04 night)
|
|
|
|
Datto EDR "Default RMM Org" dismantled: its 21 client-named Locations re-parented to per-client Organizations (18 orgs created; Glaztech-ALB -> existing Glaztech org, Arizona Computer Guru loc -> existing ACG org). Mechanics: POST /Organizations + PATCH /Locations/{id} {organizationId} (LoopBack, undocumented but verified with [TEST] articles first; org-list agent/site counts are STALE rollups - trust GET /Locations organizationId). Default RMM Org now holds only OnDemand(0) + Managed(0).
|
|
|
|
Corrected EDR coverage this reveals: IMC 10, Reliant 4 (+Home), PUTT 4, Russo 2, MVAN 2, Andy's Mobile Fuel 2, Key Paul 2, Roharbach 2, BG Builders 2, Rednour 1, Len's 1, JANC 1, Bardach 1.
|
|
|
|
REVISED AV gaps: NO-AV now 8 clients / 20 paid devices (Andy's Mobile Fuel came off the list): Little Hearts Little Hands(8), Ridgetop(3), Residential and Renovation Engineering(2 - EDR org+location exist, 0 agents), Janet Altschuler(2 - own location empty; NOTE org "JANC Excavation and Construction"(1 agent, janc-qb) may be her business - verify), Business Services of Tucson(2), Gary A Hartman(1), Robyn Pittman(1), Marty Ryan(1). PARTIAL improved: MVAN now 4/4 OK, Len's 7/8, IMC 12BD+10EDR (overlap likely; effectively covered).
|
|
|
|
## Phase 4 — NO-AV remediation round 1 (2026-07-04 night)
|
|
|
|
Deployed Datto EDR to the 5 reachable NO-AV machines via RMM push (Install-EDR one-liner, visible stdout):
|
|
- Ridgetop Group (org 3db1059a, group 436d7e55, key ridgetop01): CNX-LAB-00 [OK] RGI-DC [OK after service kick, 1053 on first start] RTG-host01 [OK] - all 3 ACTIVE in EDR
|
|
- Gary A Hartman LLC (org 962e2986, key hartman001): DESKTOP-EVA4H1A [OK] ACTIVE
|
|
- Robyn Pittman (org dc47a7a5, key pittman001): DESKTOP-PL2RCGL install still running at wrap - VERIFY next session
|
|
Note: avInstalled field null right after registration - verify Datto AV component enables per org policy (check Monday).
|
|
|
|
Remaining NO-AV (machines unreachable, need online windows / discovery / onsite): Little Hearts Little Hands (8 - no RMM/SC at all, biggest gap), Residential and Renovation Engineering (2 - EDR org ready, 0 agents), Janet Altschuler (2 - verify if JANC Excavation org/janc-qb is hers), Business Services of Tucson (2), Marty Ryan (1). Path: autoenroll loop -> RMM -> EDR push (same as tonight).
|
|
|
|
**Verification sweep 2026-07-04 close:** all workstreams re-checked live. Two findings, both handled: (1) guru-rmm submodule pointer referenced the feature branch — realigned to deployed main c4f24de; (2) DESKTOP-PL2RCGL (Robyn Pittman) EDR install FAILED — machine went offline mid-delivery ("command undeliverable") — RETRY MONDAY (key pittman001). Everything else green: RMM 0 dups + staging empty + moves stuck (Dataforth D1=25/D2=26), both scheduled tasks Ready (AutoEnroll Mon 6:00, Progress daily 8:07), dedup fix in deployed main, EDR Default org = OnDemand+Managed only, 4/5 NO-AV deploys ACTIVE, SC fixes verified (sites/types/deploys), staging key vaulted, memory updated, 22 errorlog entries.
|
|
|
|
**Resolved 2026-07-04 (Howard):** JANC Excavation and Construction (Syncro cid 2248945) and Janet Altschuler (cid 457710) are SEPARATE clients — janc-qb is NOT Janet's machine. Janet Altschuler remains fully NO-AV (2 paid devices, no footprint). JANC's EDR org (1 agent) stands on its own.
|
|
|
|
## Phase 4 — Backup verification (2026-07-04 night)
|
|
|
|
Method: Syncro recurring-schedule "Service - Data Backup" lines (the SHOULD) vs GuruRMM /api/mspbackups mappings (MSP360 job data, synced tonight - only covers RMM-enrolled machines) vs B2 bucket freshness (max uploadTimestamp sampled; CBB_Configuration touched each run). ACG-PST bucket = Peaceful Spirit (PST-SERVER), NOT Prairie Schooner (TPS = The Prairie Schooner).
|
|
|
|
**[ERROR] BROKEN backups (bucket dead while backup expected):**
|
|
- ACG-REDNOUR: last upload 2025-12-22 (194d dead) - Rednour backups broken since December
|
|
- ACG-GLAZTECH: last upload 2025-10-28 (249d dead) - Glaztech backups stopped October
|
|
- ACG-Rohrbach: 13d stale - verify job schedule
|
|
|
|
**[ERROR] BILLED with real $ but NO backup found anywhere (no MSP360, no bucket, not in shared):**
|
|
- Bill Tedards $45/mo (qty 1) | Reliant Well Drilling $40/mo | Mineralogical Record $10/mo | The Prairie Schooner $10/mo (TPS-SVR enrolled RMM today, no MSP360 yet)
|
|
- Billed at $0 (free lines, cosmetic but should be real or removed): AMT, Design and Brand Envoys, Janet Altschuler (qty 2), T&C Sorensen, Ridgetop(covered anyway)
|
|
|
|
**[WARNING] UNDER-covered vs billed qty:** Jimmy Company 12 billed/$240 vs 1 machine (Blaster2) - hard-verified over-bill; Dataforth 7 billed vs 5 machines (AD1, AD2, DF-HYPERV-B, FILES-D1, SAGE-SQL); Curtis 2 vs 1 (SRV02); BSoT 2/$240 - bucket ACG-BST fresh(1d) but machines not identifiable via RMM (not enrolled) - verify machine count.
|
|
|
|
**[WARNING] Backup RUNS but NOT billed (revenue leakage, 11 clients):** Cascades (CS-SERVER, bucket fresh), Horseshoe (HSM-NewServer + bucket), IMC (IMC1), Kittle (SERVER), Len's Auto (LAB-SVR + LAB-Becky, ACG-Lens fresh), Peaceful Spirit (PST-SERVER, ACG-PST fresh), Gonzvar (Server), Russo (RUSSO-SRV), Universal Cryogenics (UC2-SERVER, shared bucket fresh), Tucson Coin (Seth-PC, ACG-TCA fresh), Rednour (mapped but bucket DEAD - broken AND unbilled). CAVEAT: some may be bundled in GPS pricing - Winter/Mike to rule.
|
|
|
|
**[OK] VERIFIED billed+working:** VWP 4/4 (VWP-Backup fresh), Grabb (GND-SERVER), Gary Hartman (DESKTOP-EVA4H1A), Ridgetop (RGI-DC), Brett (ACG-Brett fresh 0d - machine not RMM-mapped, identify), Curtis partial, Dataforth partial.
|
|
|
|
**Legacy storage burning money (shared bucket MSPBackups20200311, all dead):** FSG-SRV-02 (1678d), SALMON/TROUT (1435d), LAB-BECKY old path (867d), VWP-SERVER old path (47d), DROBO (40d). Candidates for delete-prefix purge after Mike confirms.
|
|
|
|
**Dataforth EDR tail update:** DATAFORTH-PC enrolled RMM (D1) + EDR installed ACTIVE 2026-07-04 night. Tail = SURFACEOPS, D1-ENGI-006, MING-HP.
|
|
|
|
## EDR pre-provisioning COMPLETE (2026-07-05 early)
|
|
|
|
Every migration/GPS client now has a Datto EDR org + target group + registration key - Monday is pure deployment. 26 keys minted on existing targets (attribution orgs + Safesite/VWP/Tedards/BirthBio/Gonzvar/Peaceful Spirit/ACG) + 18 new clients provisioned org->group->key (Grabb, Stamback, Zeus, Jimmy, Prairie Schooner, QWM, Brett, Sorensen, Marc Group, Pro-Tech, Inside Track, AMT, Trebesch, Multicultural, Little Hearts, BSoT, Marty Ryan, Kittle). 0 failures. All 48 keys (incl. earlier ridgetop01/hartman001/pittman001/Dataforth) vaulted at **msp-tools/datto-edr-regkeys.sops.yaml**. Glaztech deliberately NOT keyed (stays Bitdefender).
|
|
|
|
Deploy recipe per machine (Monday): RMM online -> POST /api/agents/:id/command powershell elevated: Install-EDR -URL https://azcomp4587.infocyte.com -RegKey <client key from vault>.
|
|
|
|
**Rule (Howard 2026-07-05): backup targets are never guessed.** The backup matrix above is a BILLING-reconciliation input only. A billed qty higher than machines-currently-backing-up is a question for Mike/Winter (bill wrong? machine offline? deliberate?), NOT an instruction to add backup to more machines. Which machine gets backed up is a deliberate per-client decision - do not infer targets, do not deploy backup jobs from this audit.
|
|
|
|
## 2026-07-05 overnight — CP-QB FIXED+ENROLLED; IMC-PRINTSERVER installed but TLS-blocked
|
|
|
|
- **CP-QB (Curtis Plumbing) DONE:** WMI class store was corrupt (Win32_Processor Invalid
|
|
class 0x80041010, broken since ~2022 — universal installer dies at its Get-CimInstance
|
|
arch check; also why SC GuestInfo was frozen at 2022). verifyrepository said CONSISTENT
|
|
(does not catch missing classes); `mofcomp cimwin32.mof` fixed it. Installer re-ran clean,
|
|
enrolled, reassigned to Curtis Plumbing (5/6). **Enrollment 112/189.**
|
|
- **IMC-PRINTSERVER (IMC) agent INSTALLED, service RUNNING, watchdog installed — but the
|
|
box drops ALL outbound TLS** (CF hosts, rmm-api direct IP, even google.com: ClientHello
|
|
then silence; SC works only because its relay protocol is not TLS; LAN + DNS fine).
|
|
Agent log ends at "Connecting to wss://rmm-api...". **It will self-enroll the moment TLS
|
|
egress is unblocked — do NOT reinstall.** Next: check IMC UniFi gateway (192.168.0.254)
|
|
for a per-device rule on 192.168.0.79 / A0-1E-0B-11-60-1D; else hands-on for a broken
|
|
local WFP/AV filter. Then reassign-staging.py + EDR push.
|
|
- **Box quirks (reusable):** powershell.exe never executes via SC on this box (all PS
|
|
one-liner pushes silently die + wedge the SC queue); only single-line cmd commands run;
|
|
SC extension eats `&`-chains and multi-line commands; SMB to \IMC1\NETLOGON hangs.
|
|
Working channels: curl.exe over LAN; HTTP file-serve from IMC1 via temp HttpListener
|
|
(RMM command); readback via listener query log or nginx access log on the RMM server
|
|
(`/install/<code>/windows?gpsdiag=...` — CF passes the install path).
|
|
Full detail: session-logs/2026-07/2026-07-05-howard-cpqb-fixed-imcprintserver-tls-blocked.md
|
|
|
|
**Onsite-check group created (2026-07-05, Howard's rule):** machines/clients with no SC,
|
|
no Syncro checkin, and no Bitdefender go to `onsite-check-list.md` — handled at the next
|
|
site visit, not chased remotely. IMC-PRINTSERVER pending Mike's ruling (TLS lockdown by
|
|
design vs fix — DM sent 2026-07-05).
|
|
|
|
## AV switch (Bitdefender -> Datto EDR) mass deploy — 2026-07-05
|
|
|
|
Throttled push via GuruRMM (one command at a time, 20s gaps) using per-client vaulted
|
|
reg keys. Tool: `tools/edr-deploy.py` (recomputes coverage from a live `edr.py --json
|
|
agents` dump each run; targets ONLINE Windows RMM agents of GPS clients with a vaulted
|
|
key that are not already in EDR; Glaztech excluded by design).
|
|
|
|
- **Result: ~78 machines newly protected across ~25 clients.** Datto EDR authorized-agent
|
|
count 228 -> 302 over the session. Run 1: 74/79 clean "Installed RTS agent". Follow-up
|
|
runs cleared the rest except the two legacy boxes below.
|
|
- **Horseshoe key gap closed:** org+group already existed, key `jthm976pwo` was just never
|
|
vaulted -> added to `msp-tools/datto-edr-regkeys.sops.yaml`.
|
|
- **AD1 (Dataforth, Server 2016):** had a broken partial EDR install (huntAgent service
|
|
present but Stopped, no binary) -> Install-EDR silently skipped it (the "already
|
|
installed" early-return is gagged in non-interactive mode). Fixed with **-Force** reinstall
|
|
-> Success, service Running.
|
|
- **DESKTOP-PL2RCGL (Robyn Pittman):** the 7/4 mid-delivery failure — online now, deployed
|
|
clean (key pittman001), service Running.
|
|
|
|
### Genuinely BLOCKED — legacy OS (need prerequisites or onsite)
|
|
- **AMT-PC** (Arizona Medical Transit) — Windows 7 Home SP1, **PowerShell 2.0**. The modern
|
|
Datto EDR `agent.windows.exe` (v3.17.1) **crashes on execution** (process tree dies, zero
|
|
output, even to a file). Downloads fine; won't run. Needs Win7 SHA-2 servicing + WMF/.NET
|
|
prereqs, or the legacy-agent path, or onsite. Same box that needed the manual legacy
|
|
GuruRMM install.
|
|
- **SRV02** (Curtis Plumbing) — Windows **Server 2008 R2**. Agent downloads + launches but
|
|
`--version` emits nothing (silent no-op, missing runtime prereq). Same legacy class.
|
|
|
|
### Loose ends to verify (protected, but console visibility unclear)
|
|
- **SS-SP1** (Safesite, Server 2016): huntAgent=Running + agent.exe present locally, but NOT
|
|
visible in the EDR console under "sp1" — likely registered under its real hostname or
|
|
awaiting approval. Verify it reports in.
|
|
- **DFORTH-Ship vs DForth-Shipp** (Dataforth): the Win11 box is in EDR as `dforth-shipp`;
|
|
the single-p "DFORTH-Ship" RMM record is a duplicate-hostname artifact of the same machine.
|
|
|
|
### Diagnostic insight (reusable)
|
|
Install-EDR is SILENT in non-interactive mode — every status/error line is gated behind
|
|
`if ($Interactive)`. A "completed / blank output" RMM result means nothing; the visible
|
|
"Installed RTS agent" text comes from `agent.windows.exe` itself, so its ABSENCE = the exe
|
|
never ran (early-return or crash). Use `-Interactive *> file` + read-back, or check
|
|
`Get-Service huntAgent` + `C:\Program Files\infocyte\agent\agent.exe`, to know true state.
|
|
Reg keys are per-target-group; `-Force` reinstalls over a stale huntAgent service.
|
|
|
|
**Bitdefender removal still deferred** — per-client, only after that client's EDR is
|
|
confirmed ACTIVE. Not started.
|