From b0ebd44c9d6afab0a212efa27ab1f51a1b0154ef Mon Sep 17 00:00:00 2001 From: Howard Enos Date: Sun, 5 Jul 2026 19:34:42 -0700 Subject: [PATCH] sync: auto-sync from HOWARD-HOME at 2026-07-05 19:34:16 Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-07-05 19:34:16 --- .claude/memory/MEMORY.md | 1 + .../reference_msp360_backup_monitoring.md | 27 +++ projects/gps-rmm-audit/backup-map.md | 130 +++++++++++++++ ...2026-07-05-howard-gps-backup-map-msp360.md | 157 ++++++++++++++++++ 4 files changed, 315 insertions(+) create mode 100644 .claude/memory/reference_msp360_backup_monitoring.md create mode 100644 projects/gps-rmm-audit/backup-map.md create mode 100644 projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-gps-backup-map-msp360.md diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index 6a70ddea..fd9dbd6e 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -2,6 +2,7 @@ ## Reference - [ACG resource map](reference_resource_map.md) — **READ THIS FIRST** when a task references a server/service/tenant/API. What we have access to, how to connect from this machine, per-machine exceptions, gotchas. Points at the detail files below. +- [MSP360 backup monitoring](reference_msp360_backup_monitoring.md) — Authoritative "is client X actually backing up / last run" source for the whole fleet (NOT B2 bucket contents). MSP360 MBS API `/api/Monitoring`, vault `msp-tools/msp360-api.sops.yaml`. CompanyName also IDs B2 buckets (ACG-PST=Peaceful Spirit, ACG-TCA=Tucson Coin). - [ALIS (Medtelligent)](reference_alis_medtelligent.md) — Cascades assisted-living EHR. API host api.alisonline.com, community 622; username must be tenant-qualified (howard.enos@cascadestucson). Staff are READ-ONLY via API — create/change staff via web-UI Staff Import .xls. Use the `alis` skill. - [GuruRMM User Manager](reference_gururmm_user_manager.md) — GuruRMM has a built-in per-agent User Manager tab (reset_password/enable/disable/groups for local+domain+AAD endpoint users; domain users only on a DC via `is_dc`). Use it, NOT raw Set-ADAccountPassword via /rmm. Endpoints: /api/agents/{id}/users + /users/action. - [RMM map network drive (err67 double-hop)](reference_rmm_map_network_drive.md) — Pushing a persistent mapped drive to a remote share via /rmm user_session fails with err67/1702 (impersonated token = no network cred/double-hop). Plant HKCU:\Network\ keys + cmdkey; mounts at next interactive logon. Immediate visibility needs the live session (ScreenConnect). diff --git a/.claude/memory/reference_msp360_backup_monitoring.md b/.claude/memory/reference_msp360_backup_monitoring.md new file mode 100644 index 00000000..461608ff --- /dev/null +++ b/.claude/memory/reference_msp360_backup_monitoring.md @@ -0,0 +1,27 @@ +--- +name: reference_msp360_backup_monitoring +description: MSP360 MBS /api/Monitoring is the authoritative "is this client backing up" source for the whole ACG fleet +metadata: + type: reference +--- + +For any "is client X actually backing up / when did it last run" question, the authoritative +source is the **MSP360 Managed Backup Service API**, not B2 bucket contents (buckets are just +where Cloud plans land; a bucket with files does not prove a recent backup ran). + +- Creds: vault `msp-tools/msp360-api.sops.yaml` (API login+password, generated in MSP360 + console Settings>General — not the console login). +- Auth: `POST https://api.mspbackups.com/api/Provider/Login` `{UserName, Password}` -> Bearer + `access_token`. +- `GET /api/Monitoring` -> every backup plan across all companies: `CompanyName`, + `ComputerName`, `PlanName`, `StorageType`, `LastStart`, `Status` (0/1=Success, 2=Warning, + 3=Failed, 4=Running, 6=Interrupted, 7=completed-with-warnings), `DataCopied`, `TotalData`, + `ErrorMessage`. Also `GET /api/Companies` (37, with Destinations) and `GET /api/Users` (59). + `/api/Computers` is 404. +- Used 2026-07-05 for the GPS backup map (`projects/gps-rmm-audit/backup-map.md`): resolved 11 + of 12 "no destination found" clients, surfaced 4 not-backing-up findings (AMT + T&C Sorensen + = 0 plans, Grabb GND-SERVER failing, Bill Tedards ~12mo stale). MSP360 CompanyName also IDs + B2 buckets: `ACG-PST`=Peaceful Spirit, `ACG-TCA`=Tucson Coin and Autograph. +- Backup targets are a per-client decision — report mismatches, don't deploy jobs. See + [[feedback_backup_targets_never_guessed]]. Related: [[reference_backblaze_storage_rate]]. +- Worth building a read-only `msp360` skill (login + Monitoring) so this is repeatable. diff --git a/projects/gps-rmm-audit/backup-map.md b/projects/gps-rmm-audit/backup-map.md new file mode 100644 index 00000000..fe488823 --- /dev/null +++ b/projects/gps-rmm-audit/backup-map.md @@ -0,0 +1,130 @@ +# GPS Backup Verification — Per-Customer Destination Map + +**Purpose:** Resolve every billed GPS "Data Backup" line to a real, *working* destination. +Feeds Phase 4 of the GPS -> GuruRMM coverage audit. See `wiki/projects/gps-rmm-audit.md`. + +- Built 2026-07-05 (Howard). Sources: **MSP360 MBS API (authoritative backup layer)** + + B2 (destination) + Seafile + ownCloud. Datto Workplace pending Mike's super-admin key. +- **MSP360 is the source of truth for "is it backing up":** `/api/Monitoring` reports every + backup plan (computer, last run, status, data) across 37 companies / 66 plans. B2 buckets + are just where the Cloud plans land. Creds: `msp-tools/msp360-api.sops.yaml`. +- Billed-backup flags from tracker.md `Svc` column (B = Data Backup billed). +- Reconciliation note: bucket presence != a backup ran. MSP360 `LastStart` + `Status` is the + real signal — used below. + +--- + +## 1. Billed-backup GPS clients (Svc = B) -> verified status + +20 of the 40 active GPS clients bill a Data Backup line. + +| Client | Bkt | Backing up? | Evidence (MSP360 unless noted) | Verdict | +|--------|:---:|:-----------:|-------------------------------|---------| +| Dataforth Corp | A | YES | 12 plans, AD1/AD2/etc daily Success (-> `ACG-Dataforth`) | RESOLVED | +| Valley Wide Plastering | A | YES | 4 plans, VWP-* daily Success (-> `VWP-Backup`) | RESOLVED | +| Arizona Medical Transit | A | **NO** | MSP360 company exists, **0 backup plans** | **FINDING: billed, not backing up** | +| Glaz-Tech Industries | B | PARTIAL | GTI-FINANCESVR daily Success; **GTI-INV-VMHOST last 2025-10-26 (stale)** (-> `ACG-GLAZTECH`) | RESOLVED (1 server stale) | +| Jimmy Company | B | YES | Blaster2 Cloud Success; local plans failing (disk full) | RESOLVED (billing flag: 12 billed/1 box) | +| Horseshoe Management | B | YES | HSM-NewServer daily Success (-> `Horseshoe`) | RESOLVED | +| Grabb & Durando Law | B | **NO** | GND-SERVER Cloud plan **status=Failed, 0 bytes** | **FINDING: backup failing** | +| Quantum Wealth Mgmt | B | ? | **No MSP360 company** | MISSING — Datto candidate | +| Reliant Well Drilling | C | ? | **No MSP360 company** | MISSING — Datto candidate | +| Curtis Plumbing | C | YES | CP-QB + SRV02 daily Success | RESOLVED | +| The Prairie Schooner | C | ? | **No MSP360 company** | MISSING — Datto candidate | +| Mineralogical Record | C | YES | ownCloud `minrec` (98 GB) | RESOLVED (ownCloud) | +| T & C Sorensen | C | **NO** | MSP360 company "Sorensen" exists, **0 plans** (ACG-TCA is NOT theirs — see §3) | **FINDING: billed, not backing up** | +| Ridgetop Group | C | YES | RGI-DC Cloud Success | RESOLVED | +| Brett Interiors | C | YES | Server image daily Success (-> `ACG-Brett`) | RESOLVED | +| Bill Tedards | C | **STALE** | Tedards DESKTOP-SUFJR0J last Success **2025-07-12 (~1 yr ago)** | **FINDING: backup stale ~12 mo** | +| Janet Altschuler | C | ? | **No MSP360 company** | MISSING — Datto candidate | +| Business Svcs of Tucson | C | YES | KAT1 Local Profile Success (-> `ACG-BST` + ownCloud `bst`) | RESOLVED | +| Design and Brand Envoys | C | ? | **No MSP360 company** | MISSING — Datto candidate | +| Gary A Hartman LLC | C | YES | Hartman DESKTOP-EVA4H1A backing up (path warnings) | RESOLVED | + +**Tally (20):** 11 RESOLVED (actively backing up) · 4 FINDINGS (destination exists but +broken) · 5 MISSING (no destination anywhere — Datto candidates). + +- **MISSING (5) — check once Datto enumerates:** Quantum Wealth, Reliant Well Drilling, + The Prairie Schooner, Janet Altschuler, Design and Brand Envoys. If not Datto users, these + are billed-but-unbacked. + +--- + +## 2. Backup-health findings (billed + configured, but NOT protected right now) + +These are the highest-priority operational findings — the client pays for backup and a +destination/company exists, but no good recent restore point is being produced. + +| Client | Machine | Problem | Last good backup | +|--------|---------|---------|------------------| +| Arizona Medical Transit | — | MSP360 company but **zero plans** configured | never | +| T & C Sorensen | — | MSP360 company "Sorensen" but **zero plans** | never | +| Grabb & Durando | GND-SERVER | Cloud plan **Status=Failed, 0 bytes** | check console for last success | +| Bill Tedards | DESKTOP-SUFJR0J | Backing up but **last success ~2025-07-12** (~12 mo stale) | 2025-07-12 | +| Glaz-Tech | GTI-INV-VMHOST | Second server image **last 2025-10-26** (~8 mo stale); GTI-FINANCESVR is current | 2025-10-26 | +| Len's Auto | LAB-SVR | IMAGE last 2026-06-07, FILES last 2026-06-17 — matches tracker "LAB-SVR offline since 2026-06-18" | 2026-06-17 | + +Also non-GPS but worth a note to Mike: **Peaceful Spirit** PST-SERVER Files plan +**Status=Failed** (0 bytes) despite 22K historical files in `ACG-PST`; **Desert RV** +local-storage plan Critical (E:\ unreachable). These are outside GPS scope. + +--- + +## 3. Revenue-leak / billing-mismatch (data landing, backup NOT billed) — held for Mike/Winter + +| Client | Destination | Billed Svc | Note | +|--------|-------------|:----------:|------| +| Cascades of Tucson | MSP360 CS-SERVER -> `ACG-Cascades` (active Success) | A E V | Backing up, **no B line billed** | +| Len's Auto Brokerage | MSP360 LAB-SVR -> `ACG-Lens` | E | Has backups, no B line (and LAB-SVR now stale) | +| Russo Law Firm | MSP360 RUSSO-SRV -> Cloud (2.7 TB) + Seafile `russo@rrs-law.com` (2.2 TB) | A E V | Backing up + 2.2 TB Seafile, **no B line** | +| The Marc Group | ownCloud `themarcgroup` (29 GB) | E | 29 GB stored, billed email only | + +--- + +## 4. Orphan destinations — identified (non-GPS clients / ACG internal / stale) + +All resolved via MSP360 company names + IX cPanel map + client wiki. None are GPS gaps. + +**Now identified (were unknown):** +- `ACG-PST` (22,054 files) = **Peaceful Spirit** (MSP360 PST-SERVER; current plan failing). +- `ACG-TCA` (404 files) = **Tucson Coin and Autograph** (MSP360 DESKTOP-P36LUUN + Seth-PC, + both Success). NOT T&C Sorensen. +- `ownCloud mara` / `jburger` — still unidentified (small; low priority). + +**Non-GPS clients with active MSP360 backups (not audit gaps):** Financial Safeguards +(FSG-SRV-02 **active** — the `MSPBackups20200311` bucket is stale but FSG uses a current +one), Rednour Law (`ACG-REDNOUR`), Rohrbach (`ACG-Rohrbach` + ownCloud), Martell & +Associates (ownCloud `Martell`), Universal Cryogenics, Tucson Coin & Autograph, Tucson +Safety and Medical (last 2026-02-25 — stale), Khalsa Montessori, LaHC, Desert RV, Anaise, +Peaceful Spirit. + +**ACG internal:** `ACG-Internal`, `ACG-Tech03L-Backup`, ownCloud `sysadmin`, Seafile +`mike`/`howard`, MSP360 ACG-Internal (NEPTUNE Exchange backup). + +**Not a backup:** ownCloud `pavon` (14.2 TB) — Pavon video-surveillance archive, ownCloud +is source of truth (`wiki/clients/pavon.md`). + +**Stale / cleanup (Mike's call):** `MSPBackups20200311` (ex-client FSG, 2021). + +--- + +## 5. Datto Workplace (enumeration pending Mike's super-admin key) + +- Confirmed via RMM detection: Birth Biologic (`ACG-DWP-X-BB`, `BB-*`/`KSTEEN*`/`EVO-X1`). +- When the key lands: `py .claude/skills/datto-workplace/scripts/datto_workplace.py projects --json`, + then check the 5 MISSING clients (§1) against it. + +--- + +## Next actions + +1. **Datto (blocked on Mike):** enumerate; resolve the 5 MISSING — Quantum Wealth, Reliant, + Prairie Schooner, Janet Altschuler, Design & Brand Envoys. +2. **Backup-health findings (§2) — operational, fix these:** AMT + T&C Sorensen have no + backup plans at all; Grabb GND-SERVER failing; Bill Tedards ~12 mo stale; Glaz-Tech + GTI-INV-VMHOST ~8 mo stale; Len's LAB-SVR stale (offline). Route per client. +3. **Held for Mike/Winter (§3):** Cascades/Len's unbilled backups; Russo 2.7 TB Cloud + 2.2 TB + Seafile no B line; Marc Group ownCloud. +4. **Cleanup:** confirm `MSPBackups20200311` (FSG) for deletion; identify ownCloud `mara`/`jburger`. +5. Optional: build a small `msp360` read-only skill (login + Monitoring) — this pass was done + via direct API; a skill would make backup-status checks repeatable for the whole fleet. diff --git a/projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-gps-backup-map-msp360.md b/projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-gps-backup-map-msp360.md new file mode 100644 index 00000000..5465ec1e --- /dev/null +++ b/projects/gps-rmm-audit/session-logs/2026-07/2026-07-05-howard-gps-backup-map-msp360.md @@ -0,0 +1,157 @@ +# GPS Backup Verification — Datto API Path + MSP360 Backup Map + +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Resumed GPS -> GuruRMM audit Phase 4 (backup verification), picking up from the +morning's `backup-verification-skills` session. Two threads: (1) determine whether the +Datto Workplace API can be pushed past its file-only limitation, and (2) produce the +corrected per-customer GPS backup map across all backends. + +Confirmed the Datto Workplace file API is still a dead end for backup enumeration (0 +projects visible to the current key). Researched Datto's actual API surface: there is +**no admin/team/user/device REST API** — the public API is strictly file-sharing, and +user/storage administration is Workplace-Manager-portal-only. However, the file API DOES +have a "broader access" mode: enabling **Super Administrator access** on the API +integration (or team-sharing projects to it) makes `/file/projects` enumerate the whole +team. That is a portal action on the SaaS tenant, which Claude cannot perform. Howard +chose that path; **Mike is provisioning a Super-Administrator-scoped API integration**. +Datto enumeration is parked on that. + +With Datto blocked, moved to building the backup map. Pulled the billed-backup client +list from `tracker.md` (20 of 40 active GPS clients bill a Data Backup line), the live +B2 bucket inventory (16 buckets), and the prior session's Seafile (5 accounts) and +ownCloud (10 users) data. First pass resolved 7 clients cleanly against B2/ownCloud, left +1 as a name-guess (T&C Sorensen -> ACG-TCA), and 12 with no destination found. Identified +the orphan destinations via the IX-server cPanel account map + client wiki: `pavon` +(14.2 TB, the largest consumer) is a video-surveillance archive, not a backup; most +others are non-GPS clients or ACG internal. Confirmed all 7 resolved billed buckets are +non-empty (real backups), and flagged `ACG-PST` as a substantial 22K-file bucket with no +known owner. + +The breakthrough was finding the **MSP360 Managed Backup Service API** credentials in the +vault (`msp-tools/msp360-api.sops.yaml`) and querying `/api/Monitoring` directly (no skill +exists). This is the authoritative backup layer: 66 backup plans across 37 companies, each +with computer, last-run, status, and data size. It collapsed the 12 unresolved clients to +5 truly missing (Datto candidates), confirmed 11 actively backing up, and surfaced 4 +not-protected findings (AMT + T&C Sorensen have zero backup plans; Grabb GND-SERVER cloud +plan failing; Bill Tedards ~12 months stale). MSP360 CompanyName also resolved the two +orphan-bucket mysteries — `ACG-PST` = Peaceful Spirit, `ACG-TCA` = Tucson Coin and +Autograph — which corrected an earlier tentative guess that ACG-TCA was T&C Sorensen (it +is not; Sorensen is not backing up). Rewrote `backup-map.md` with MSP360 as the source of +truth and saved a reference memory so future backup audits start there. + +## Key Decisions + +- Pursue "broader Datto API access" via the existing integration's Super Administrator + toggle rather than assuming a separate admin API exists — research confirmed no admin + API exists, so the toggle (or project team-shares) is the only enumeration path. +- Treat MSP360 `/api/Monitoring` as the authoritative "is X backing up" source, not B2 + bucket contents — a bucket with files does not prove a recent backup ran. B2/Seafile/ + ownCloud are destinations; MSP360 LastStart+Status is the real signal. +- Queried MSP360 via direct API (read-only) since no skill covers it; flagged building a + small read-only `msp360` skill as a follow-up rather than doing it inline. +- Reported AMT and T&C Sorensen "zero backup plans" as findings routed to Mike/Winter/ + client, not as a deploy instruction — backup targets are a per-client decision + (`feedback_backup_targets_never_guessed`). +- Corrected the ACG-TCA -> T&C Sorensen guess once MSP360 proved ACG-TCA = Tucson Coin; + did not ship the guess. + +## Problems Encountered + +- Datto Workplace OpenAPI spec served is stripped (`info: {}`, no securitySchemes) — used + Datto's published help docs to confirm the API is file-sharing-only and how the + Super-Admin scope works. +- `b2 usage` (full per-bucket byte-size across 16 buckets) timed out at 400s (exit 124) — + it lists every object version. Switched to per-bucket `bucket-size` with a 40s timeout; + empty buckets return instantly, data buckets are the slow ones, which itself answers the + empty-vs-real question. All 7 resolved billed buckets confirmed non-empty. +- MSP360 `/api/Computers` returns 404; used `/api/Companies` + `/api/Users` + + `/api/Monitoring` for the full picture instead. + +## Configuration Changes + +Created: +- `projects/gps-rmm-audit/backup-map.md` — the per-customer backup destination + health map + (MSP360-authoritative; B2/Seafile/ownCloud destinations; Datto pending). +- `.claude/memory/reference_msp360_backup_monitoring.md` + index line in + `.claude/memory/MEMORY.md`. + +Scratch (not committed): MSP360 probe scripts in the session scratchpad +(`msp360_probe.py`, `msp360_report.py`). + +## Credentials & Secrets + +- **MSP360 Managed Backup Service API** — vault `msp-tools/msp360-api.sops.yaml` (already + present, created by Mike 2026-05-18). Base `https://api.mspbackups.com`; auth `POST + /api/Provider/Login` `{UserName, Password}` -> Bearer `access_token`; login `kY9PvDdWki` + (password in vault). Endpoints used: `/api/Monitoring`, `/api/Companies`, `/api/Users`. +- **Datto Workplace API** — vault `msp-tools/datto-workplace.sops.yaml` (from morning + session); file-access only, `/file/projects` empty until Super-Admin scope enabled. + +## Infrastructure & Servers + +- **MSP360 MBS** — SaaS `api.mspbackups.com`. 37 companies / 59 users / 66 backup plans. + Status codes: 0/1=Success, 2=Warning, 3=Failed, 4=Running, 6=Interrupted, + 7=completed-with-warnings. +- **Backblaze B2** — 16 buckets. Billed-GPS destinations: ACG-Dataforth, VWP-Backup, + ACG-GLAZTECH, Horseshoe, ACG-Brett, ACG-BST. Orphans identified: ACG-PST=Peaceful Spirit + (22,054 files), ACG-TCA=Tucson Coin and Autograph (404 files), ACG-REDNOUR=Rednour Law, + ACG-Rohrbach=Rohrbach, ACG-IX/ACG-Internal/ACG-Tech03L=ACG internal, + MSPBackups20200311=stale ex-client FSG. +- **Datto Workplace** — SaaS `acg.workplace.datto.com` (team 6); on-prem Datto Workplace + Server VM `ACG-DWP-X-BB` on Jupiter (Birth Biologic). + +## Commands & Outputs + +```bash +# Datto Workplace status (still 0 projects for current key) +py .claude/skills/datto-workplace/scripts/datto_workplace.py status + +# B2 buckets + per-bucket size (usage full-scan times out; use bucket-size per bucket) +py .claude/skills/b2/scripts/b2.py buckets +py .claude/skills/b2/scripts/b2.py bucket-size ACG-PST # 22,054 files + +# MSP360 (direct API, read-only) — authoritative backup status +# POST /api/Provider/Login {UserName,Password} -> Bearer token +# GET /api/Monitoring -> 66 plans, per computer: LastStart/Status/DataCopied +# GET /api/Companies -> 37 ; GET /api/Users -> 59 +``` + +Key MSP360 findings: Arizona Medical Transit + T&C Sorensen = MSP360 company but 0 plans +(not backing up); Grabb & Durando GND-SERVER Cloud plan Status=Failed 0 bytes; Bill +Tedards DESKTOP-SUFJR0J last Success 2025-07-12 (~12 mo); Glaz-Tech GTI-INV-VMHOST last +2025-10-26; Len's LAB-SVR last 2026-06-17 (matches offline-since-2026-06-18 tracker note). + +## Pending / Incomplete Tasks + +- **Datto (blocked on Mike):** enumerate once Super-Admin API integration is live, then + check the 5 MISSING billed-backup clients against it: Quantum Wealth, Reliant Well + Drilling, The Prairie Schooner, Janet Altschuler, Design & Brand Envoys. +- **Backup-health findings to route (Mike/Winter/client):** AMT + T&C Sorensen have no + backup plans; Grabb GND-SERVER failing; Bill Tedards ~12 mo stale; Glaz-Tech + GTI-INV-VMHOST ~8 mo stale; Len's LAB-SVR stale. +- **Held revenue-leaks (Mike/Winter):** Cascades (ACG-Cascades active, no B line), Len's + (ACG-Lens, no B line), Russo (2.7 TB Cloud + 2.2 TB Seafile, no B line), Marc Group + (ownCloud, billed email only). +- **Cleanup:** confirm MSPBackups20200311 (FSG) for deletion; identify ownCloud + `mara`/`jburger`. +- **Follow-up build:** read-only `msp360` skill (login + Monitoring) for repeatable + fleet backup-status checks. +- Prior-session publish still applies: backup skills + `backup_common.py` uncommitted; + Datto vault entry stored but not pushed. + +## Reference Information + +- Backup map: `projects/gps-rmm-audit/backup-map.md` +- Memory: `.claude/memory/reference_msp360_backup_monitoring.md`, + `.claude/memory/feedback_backup_targets_never_guessed.md` +- MSP360 API vault: `msp-tools/msp360-api.sops.yaml`; base `https://api.mspbackups.com` +- Datto WP API vault: `msp-tools/datto-workplace.sops.yaml`; OpenAPI + `https://acg.workplace.datto.com/api/v1/openapi.json` +- Datto WP API scope docs: help.workplace.datto.com API Integrations (Super Administrator + access enables full-team project visibility on the file API).