# GPS Backup Verification — Datto API Path + MSP360 Backup Map

## User
- **User:** Howard Enos (howard)
- **Machine:** Howard-Home
- **Role:** tech

## Session Summary

Resumed GPS -> GuruRMM audit Phase 4 (backup verification), picking up from the
morning's `backup-verification-skills` session. Two threads: (1) determine whether the
Datto Workplace API can be pushed past its file-only limitation, and (2) produce the
corrected per-customer GPS backup map across all backends.

Confirmed the Datto Workplace file API is still a dead end for backup enumeration (0
projects visible to the current key). Researched Datto's actual API surface: there is
**no admin/team/user/device REST API** — the public API is strictly file-sharing, and
user/storage administration is Workplace-Manager-portal-only. However, the file API DOES
have a "broader access" mode: enabling **Super Administrator access** on the API
integration (or team-sharing projects to it) makes `/file/projects` enumerate the whole
team. That is a portal action on the SaaS tenant, which Claude cannot perform. Howard
chose that path; **Mike is provisioning a Super-Administrator-scoped API integration**.
Datto enumeration is parked on that.

With Datto blocked, moved to building the backup map. Pulled the billed-backup client
list from `tracker.md` (20 of 40 active GPS clients bill a Data Backup line), the live
B2 bucket inventory (16 buckets), and the prior session's Seafile (5 accounts) and
ownCloud (10 users) data. First pass resolved 7 clients cleanly against B2/ownCloud, left
1 as a name-guess (T&C Sorensen -> ACG-TCA), and 12 with no destination found. Identified
the orphan destinations via the IX-server cPanel account map + client wiki: `pavon`
(14.2 TB, the largest consumer) is a video-surveillance archive, not a backup; most
others are non-GPS clients or ACG internal. Confirmed all 7 resolved billed buckets are
non-empty (real backups), and flagged `ACG-PST` as a substantial 22K-file bucket with no
known owner.

The breakthrough was finding the **MSP360 Managed Backup Service API** credentials in the
vault (`msp-tools/msp360-api.sops.yaml`) and querying `/api/Monitoring` directly (no skill
exists). This is the authoritative backup layer: 66 backup plans across 37 companies, each
with computer, last-run, status, and data size. It collapsed the 12 unresolved clients to
5 truly missing (Datto candidates), confirmed 11 actively backing up, and surfaced 4
not-protected findings (AMT + T&C Sorensen have zero backup plans; Grabb GND-SERVER cloud
plan failing; Bill Tedards ~12 months stale). MSP360 CompanyName also resolved the two
orphan-bucket mysteries — `ACG-PST` = Peaceful Spirit, `ACG-TCA` = Tucson Coin and
Autograph — which corrected an earlier tentative guess that ACG-TCA was T&C Sorensen (it
is not; Sorensen is not backing up). Rewrote `backup-map.md` with MSP360 as the source of
truth and saved a reference memory so future backup audits start there.

## Key Decisions

- Pursue "broader Datto API access" via the existing integration's Super Administrator
  toggle rather than assuming a separate admin API exists — research confirmed no admin
  API exists, so the toggle (or project team-shares) is the only enumeration path.
- Treat MSP360 `/api/Monitoring` as the authoritative "is X backing up" source, not B2
  bucket contents — a bucket with files does not prove a recent backup ran. B2/Seafile/
  ownCloud are destinations; MSP360 LastStart+Status is the real signal.
- Queried MSP360 via direct API (read-only) since no skill covers it; flagged building a
  small read-only `msp360` skill as a follow-up rather than doing it inline.
- Reported AMT and T&C Sorensen "zero backup plans" as findings routed to Mike/Winter/
  client, not as a deploy instruction — backup targets are a per-client decision
  (`feedback_backup_targets_never_guessed`).
- Corrected the ACG-TCA -> T&C Sorensen guess once MSP360 proved ACG-TCA = Tucson Coin;
  did not ship the guess.

## Problems Encountered

- Datto Workplace OpenAPI spec served is stripped (`info: {}`, no securitySchemes) — used
  Datto's published help docs to confirm the API is file-sharing-only and how the
  Super-Admin scope works.
- `b2 usage` (full per-bucket byte-size across 16 buckets) timed out at 400s (exit 124) —
  it lists every object version. Switched to per-bucket `bucket-size` with a 40s timeout;
  empty buckets return instantly, data buckets are the slow ones, which itself answers the
  empty-vs-real question. All 7 resolved billed buckets confirmed non-empty.
- MSP360 `/api/Computers` returns 404; used `/api/Companies` + `/api/Users` +
  `/api/Monitoring` for the full picture instead.

## Configuration Changes

Created:
- `projects/gps-rmm-audit/backup-map.md` — the per-customer backup destination + health map
  (MSP360-authoritative; B2/Seafile/ownCloud destinations; Datto pending).
- `.claude/memory/reference_msp360_backup_monitoring.md` + index line in
  `.claude/memory/MEMORY.md`.

Scratch (not committed): MSP360 probe scripts in the session scratchpad
(`msp360_probe.py`, `msp360_report.py`).

## Credentials & Secrets

- **MSP360 Managed Backup Service API** — vault `msp-tools/msp360-api.sops.yaml` (already
  present, created by Mike 2026-05-18). Base `https://api.mspbackups.com`; auth `POST
  /api/Provider/Login` `{UserName, Password}` -> Bearer `access_token`; login `kY9PvDdWki`
  (password in vault). Endpoints used: `/api/Monitoring`, `/api/Companies`, `/api/Users`.
- **Datto Workplace API** — vault `msp-tools/datto-workplace.sops.yaml` (from morning
  session); file-access only, `/file/projects` empty until Super-Admin scope enabled.

## Infrastructure & Servers

- **MSP360 MBS** — SaaS `api.mspbackups.com`. 37 companies / 59 users / 66 backup plans.
  Status codes: 0/1=Success, 2=Warning, 3=Failed, 4=Running, 6=Interrupted,
  7=completed-with-warnings.
- **Backblaze B2** — 16 buckets. Billed-GPS destinations: ACG-Dataforth, VWP-Backup,
  ACG-GLAZTECH, Horseshoe, ACG-Brett, ACG-BST. Orphans identified: ACG-PST=Peaceful Spirit
  (22,054 files), ACG-TCA=Tucson Coin and Autograph (404 files), ACG-REDNOUR=Rednour Law,
  ACG-Rohrbach=Rohrbach, ACG-IX/ACG-Internal/ACG-Tech03L=ACG internal,
  MSPBackups20200311=stale ex-client FSG.
- **Datto Workplace** — SaaS `acg.workplace.datto.com` (team 6); on-prem Datto Workplace
  Server VM `ACG-DWP-X-BB` on Jupiter (Birth Biologic).

## Commands & Outputs

```bash
# Datto Workplace status (still 0 projects for current key)
py .claude/skills/datto-workplace/scripts/datto_workplace.py status

# B2 buckets + per-bucket size (usage full-scan times out; use bucket-size per bucket)
py .claude/skills/b2/scripts/b2.py buckets
py .claude/skills/b2/scripts/b2.py bucket-size ACG-PST   # 22,054 files

# MSP360 (direct API, read-only) — authoritative backup status
#   POST /api/Provider/Login {UserName,Password} -> Bearer token
#   GET  /api/Monitoring  -> 66 plans, per computer: LastStart/Status/DataCopied
#   GET  /api/Companies   -> 37 ; GET /api/Users -> 59
```

Key MSP360 findings: Arizona Medical Transit + T&C Sorensen = MSP360 company but 0 plans
(not backing up); Grabb & Durando GND-SERVER Cloud plan Status=Failed 0 bytes; Bill
Tedards DESKTOP-SUFJR0J last Success 2025-07-12 (~12 mo); Glaz-Tech GTI-INV-VMHOST last
2025-10-26; Len's LAB-SVR last 2026-06-17 (matches offline-since-2026-06-18 tracker note).

## Pending / Incomplete Tasks

- **Datto (blocked on Mike):** enumerate once Super-Admin API integration is live, then
  check the 5 MISSING billed-backup clients against it: Quantum Wealth, Reliant Well
  Drilling, The Prairie Schooner, Janet Altschuler, Design & Brand Envoys.
- **Backup-health findings to route (Mike/Winter/client):** AMT + T&C Sorensen have no
  backup plans; Grabb GND-SERVER failing; Bill Tedards ~12 mo stale; Glaz-Tech
  GTI-INV-VMHOST ~8 mo stale; Len's LAB-SVR stale.
- **Held revenue-leaks (Mike/Winter):** Cascades (ACG-Cascades active, no B line), Len's
  (ACG-Lens, no B line), Russo (2.7 TB Cloud + 2.2 TB Seafile, no B line), Marc Group
  (ownCloud, billed email only).
- **Cleanup:** confirm MSPBackups20200311 (FSG) for deletion; identify ownCloud
  `mara`/`jburger`.
- **Follow-up build:** read-only `msp360` skill (login + Monitoring) for repeatable
  fleet backup-status checks.
- Prior-session publish still applies: backup skills + `backup_common.py` uncommitted;
  Datto vault entry stored but not pushed.

## Reference Information

- Backup map: `projects/gps-rmm-audit/backup-map.md`
- Memory: `.claude/memory/reference_msp360_backup_monitoring.md`,
  `.claude/memory/feedback_backup_targets_never_guessed.md`
- MSP360 API vault: `msp-tools/msp360-api.sops.yaml`; base `https://api.mspbackups.com`
- Datto WP API vault: `msp-tools/datto-workplace.sops.yaml`; OpenAPI
  `https://acg.workplace.datto.com/api/v1/openapi.json`
- Datto WP API scope docs: help.workplace.datto.com API Integrations (Super Administrator
  access enables full-team project visibility on the file API).

## Update: 21:23 PT — built the msp360 skill (full CRUD) + consolidated backup routing

Built the `msp360` skill (Howard asked for full management, not read-only), then reviewed,
fixed, and published it; consolidated all backup-related skill routing behind a wiki-first
rule.

**MSP360 API surface** — confirmed authoritative endpoint list against
`help.mspbackups.com/mbs-api-specification/methods` and live `Allow`-header probing: full
CRUD on Users, Companies, Licenses (Grant/Release/Revoke), Destinations, Accounts,
Packages, Administrators; read-only Monitoring/Billing. `/api/Computers` returns HTTP 400
"Remote Management API methods are not enabled" (feature off on this account). Pulled live
User/Company/License schemas so write payloads use real field names, not guesses.

**Skill built** (`.claude/skills/msp360/`): `msp360_client.py` (Bearer login + typed
helpers, reuses `backup_common.py`), `msp360.py` (CLI), `SKILL.md`, `.gitignore`.
Reads: status, monitoring (`--failed`/`--company`/`--stale`/`--json`), companies, users,
licenses, billing, raw GET. Writes (all `--confirm`-gated, dry-run preview otherwise):
create/delete company, create/delete user (`--keep-data`), grant/release/revoke license,
raw passthrough. Errors -> `errorlog.md`; expected conditions (feature-gate, no-match) not.

**Code review** (workflow, high effort, 18 agents): 10 findings, all fixed + re-verified.
Notable: (1) CREDENTIAL LEAK — create-user dry-run echoed plaintext `--password`; now
redacted. (2) main() caught only `Msp360Error`, not parent `bc.BackupError`, so vault/HTTP
failures were mislabeled "unexpected"; now catches `bc.BackupError`. (3) non-list 2xx body
crashed reads; added `_as_list` guard. (4) licenses `Number` format crash on None; coalesced.
(5) `monitoring --failed` hid never-run/Unknown plans (the exact dead-plan case); now
included. Plus stale-date parse robustness, feature-gate substring broadening, dead
`--json` on billing, dead `update_*` client methods, and `_root()` now uses
`bc.resolve_root()`. Refuted 1 (SKILL.md does not document passwords on CLI).

**Dry-test sweep:** 14/14 reads OK; 8/8 writes correctly gated (nothing executed without
`--confirm`); password-redaction confirmed.

**Routing consolidation** (`.claude/SKILL_ROUTING.md`): seafile + owncloud had each written
their own provisioning rows. Collapsed all five backup rows (msp360/b2/seafile/owncloud/
datto) into ONE pointer row + a dedicated **"Backups (wiki-first routing)"** block: a backup
request is a two-step route — (1) identify which backend THIS customer uses via the client
wiki + `backup-map.md` (don't assume B2/MSP360), then (2) route to that backend's skill;
provisioning uses the target backend's skill (all writes gated). Encodes both behaviors
Howard asked for (check-the-wiki-for-status, and use-the-skills-to-set-up).

**Published:** commits `976bfe7` (skill) + `a8e76ba` (routing/memory) pushed to origin;
Phase 5c installed msp360 to global. Memory `reference_msp360_backup_monitoring` updated
(skill now exists, full not read-only).

## Update: 2026-07-06 — deleted wrong Winter DM + full GPS backup verification

**Deleted the wrong weekend Winter message.** The 2026-07-05 03:44 UTC GPS-audit findings
DM to Winter (`[GPS AUDIT] Billing + service findings`) was superseded/wrong now that we
have MSP360 + Syncro-seat data. Resolved Winter's DM channel via the bot (user id
624666486362996755 -> channel 1519394696873513060), confirmed the target message content,
`DELETE /channels/{ch}/messages/1523172725391233044` -> HTTP 204. The identical message
also went to Mike (`...715039559760`) and Howard (`...736904593588`) — left in place (Howard
only asked for Winter's). Bot can only delete its own messages (bounded risk).

**Full GPS backup verification (Howard: all 40 clients, flag seat-vs-machine gaps).**
- Built `projects/gps-rmm-audit/gps-clients.json` (40 clients + Syncro cids).
- Delegated a Syncro read-only sub-agent to pull recurring-schedule Data Backup line qty
  (billed backup seats) for all 40. It caught a false-positive: base GPS Workstation/Server
  products carry "Backups" in their boilerplate DESCRIPTION — matching restricted to line
  NAME. Result: **only 16 of 40 actually bill a backup line** (tracker `Svc=B` flags were
  wrong for Horseshoe, Quantum Wealth, T&C Sorensen, Business Svcs of Tucson).
- Reconciled billed seats vs live MSP360 current-machine backups. Verdict (16 billed):
  7 OK (Dataforth, Valley Wide, Curtis, Ridgetop, Brett, Gary Hartman, Mineralogical/ownCloud);
  2 REVIEW seat>machine (Jimmy 12/1, Glaz-Tech 2/1 w/ GTI-INV 251d stale);
  3 NOT BACKING UP (Grabb GND-SERVER failing 0-byte, Bill Tedards 359d stale, AMT 0 plans);
  4 HELD-unconfirmed pending Datto key (Reliant, Prairie Schooner, Janet Altschuler, D&B Envoys).
- Leaks (backing up, not billed): Cascades, Len's, Russo, Horseshoe, Business Svcs of Tucson.
  $0-priced backup lines: Ridgetop, Janet Altschuler, D&B Envoys, AMT.

**Howard's decisions:** hold the 4 Datto candidates as unconfirmed (not failing); HOLD all
outbound (no corrected Winter/Mike message yet) — just persist. `backup-map.md` fully
rewritten with the authoritative Syncro-seat + MSP360 reconciliation. Nothing sent.
