## User
- **User:** Howard Enos (howard)
- **Machine:** Howard-Home
- **Role:** tech

## Session Summary

Investigated MSP360 backup error 2525 (`RestorePointsTimeMismatch`, kb10096) reported on VWP-SERVER at Valley Wide Plastering. The error text ("current system time precedes the previous restore point creation time") pointed at a system-clock problem, so the investigation started on the endpoint rather than in the backup console.

Located VWP-SERVER via `rmm-search` (GuruRMM id `d2cc6200-07e5-4444-bb1c-1bcce273d615`, Main Office, online) and queried its clock over `/rmm`. The machine's clock read **2051-07-10** — roughly 25 years in the future — with the W32Time source stuck on `Local CMOS Clock` (Leap Indicator 3 = not synchronized) despite an `NT5DS` domain-sync configuration. Before changing anything, confirmed domain role and virtualization: VWP-SERVER is a **VMware VM** and a **domain member server** (DomainRole 3), not a DC. FSMO roles are held by VWP-DC1 (PDC/RID/Infra) and VWP_ADSRVR (schema/naming).

Because a 25-year skew on a domain member would normally break Kerberos, sampled the rest of the VWP.US fleet to rule out a domain-wide/ESXi-host time problem. VWP-DC1, VWP_ADSRVR, FRONT64, and VWP-HYPERV1 were all correct (2026-07-07) and syncing healthily (DCs from time.nist.gov, members from the DCs). So the 2051 clock was **isolated to VWP-SERVER**, not domain-wide.

Per user direction (Mike had already corrected the date/time out-of-band during the session), set VWP-SERVER's timezone from `Pacific Standard Time` to `US Mountain Standard Time` (Arizona, no DST). Re-verified afterward: VWP-SERVER now matches VWP-DC1 exactly (2026-07-07), timezone correct, and — critically — the W32Time source flipped from `Local CMOS Clock` to `VWP-DC1.VWP.US` with a fresh successful sync, so the root cause (broken domain time sync) is resolved and the clock should hold.

Then investigated the remaining backup breakage via `/msp360` and `cbb.exe` on the endpoint. The `Files - Remote Backup` plan (F:\Shares -> B2Storage, ~509 GB stored, block-level/new-format, unencrypted) had written its last restore point at 2051-07-10 6:00 PM while the clock was wrong; its next scheduled run is frozen at 2051. Now that time is back to 2026, new runs "precede" the 2051 point -> error 2525 persists, and the plan will not self-heal. Per-restore-point CLI enumeration (`cbb list -rp`) timed out against the 509 GB B2 bunch (240s server-side reaper), establishing that a surgical delete over RMM is impractical. DM'd Mike the root cause and two fix options and left the backup untouched pending his decision.

## Key Decisions

- Started diagnosis on the endpoint clock (not the MSP360 console) because error 2525's text is a time-ordering failure, not a backup-data failure.
- Checked domain role + virtualization and sampled the whole VWP fleet **before** touching the clock — a naive "set it back to 2026" on a domain member could have either been re-pushed forward by VMware Tools or dropped the member off the domain if the DCs were also wrong. Sampling proved the DCs were fine and the box was isolated.
- Chose `US Mountain Standard Time` (Arizona, no DST) over plain `Mountain Standard Time` (observes DST) — correct zone for Arizona; the old `Pacific Standard Time` only coincidentally matched offset in summer (PDT = UTC-7).
- Did not attempt the MSP360 restore-point delete over RMM after the listing call hit the reaper timeout; recommended GUI or a maintenance-window job with Mike to preserve the 509 GB history rather than a full wipe/re-upload.
- Kept the backup remediation gated behind Mike's decision (destructive to backup storage; his call on approach).

## Problems Encountered

- **jq parse error dispatching PowerShell with backslashes** (`root\\directory\\ldap` in a heredoc). Resolved by building the command payload with `jq -n --arg cmd "$SCRIPT"` instead of inlining into the JSON.
- **`/tmp` path blocked by `block-tmp-path.sh` pre-commit hook** when writing a scratch marker. Resolved by using repo-relative scratch files (`.vwp_cmd`, `.vwp_times`), later cleaned up.
- **`cbb help <cmd>` reprints the top-level index** rather than subcommand help; correct form is `<cmd> -?`. `account` needs `-l` to list.
- **`cbb list -rp` against the 509 GB B2 bunch timed out** at the 240s server-side reaper with no output (output was buffered to a file only read after all calls finished). Concluded remote CLI enumeration of individual restore points is not viable; deferred to GUI/maintenance-window.

## Configuration Changes

- **VWP-SERVER timezone:** `Pacific Standard Time` -> `US Mountain Standard Time` (via `Set-TimeZone -Id "US Mountain Standard Time"` over `/rmm`, cmd `61a2e177`). Only change made this session.
- No file changes in the repo other than this session log. Scratch files created during diagnostics (`.vwp_cmd`, `.vwp_times`) were removed.

## Credentials & Secrets

- None discovered, created, or rotated. RMM auth via existing vault path `infrastructure/gururmm-server.sops.yaml`; MSP360 via `msp-tools/msp360-api.sops.yaml`. No plaintext secrets handled.

## Infrastructure & Servers

- **VWP-SERVER** — Valley Wide Plastering, Main Office. GuruRMM id `d2cc6200-07e5-4444-bb1c-1bcce273d615`. VMware VM (VMware20,1). Domain member (DomainRole 3) of `VWP.US`. TZ now US Mountain Standard Time; W32Time source = VWP-DC1.VWP.US.
- **VWP.US domain FSMO:** PDC/RID/Infra = VWP-DC1 (`8eefbba6-28cf-4e3f-8fda-11900e0ac302`); Schema/Naming = VWP_ADSRVR (`bd2f2f86-ea33-4202-828f-b378e459e891`). VWP-DC1 syncs from time.nist.gov.
- **VWP-HYPERV1** — Hyper-V host (`bdc3e142-1eb3-4034-8ac2-300f46dbc1d9`), separate from the VMware host VWP-SERVER runs on. Clock healthy.
- **MSP360 company:** `Valleywide Plastering` (id `b271f525-2c00-4407-bcc6-a991973a3ecb`).
- **Backup destinations (cbb accounts, login `vwp`):** `B2Storage` (id `7faaaf4d-6bdb-4a4e-af52-38cebe33974b`, ~546,465,990,074 bytes / ~509 GB used); `VWP-Backup` (id `4b4f35b7-d83f-4df4-8270-5d23117e4873`, 0 bytes).
- **Backup agent (white-labeled):** `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`. Services: `Online Backup Service`, `Online Backup Service Remote Management` (both Running).

## Commands & Outputs

- `bash .claude/scripts/rmm-search.sh VWP-SERVER` -> single match, Valley Wide Plastering.
- Time check: VWP-SERVER Local `2051-07-10 18:45`, Source `Local CMOS Clock`, TZ `Pacific Standard Time`, Type `NT5DS`.
- Fleet sample (post): VWP-DC1/VWP_ADSRVR/FRONT64/VWP-HYPERV1 all `2026-07-07`, healthy sync.
- `Set-TimeZone -Id "US Mountain Standard Time"` -> Before `Pacific Standard Time`, After `US Mountain Standard Time`.
- Post-fix verify: VWP-SERVER `2026-07-07 08:36:58` == VWP-DC1; Source `VWP-DC1.VWP.US`; last successful sync 08:36:30.
- `py .claude/skills/msp360/scripts/msp360.py monitoring --company "Valleywide Plastering"` -> VWP-SERVER plan `Files - Remote Backup`, Status Warning, LastStart `2051-07-11T01:00:41`, 0.0B.
- `cbb getPlanDetails -id 6238c387-7bf6-47a2-a68b-86d590b5fff7` -> F:\Shares -> B2Storage, block-level, unencrypted, daily 6PM.
- `cbb list -a "B2Storage" -rp ...` -> timed out (server-side reaper, 240s).

## Pending / Incomplete Tasks

- **BLOCKED on Mike's decision** (DM'd 2026-07-07 15:56; no reply yet). Remaining work = clear MSP360 error 2525 on VWP-SERVER's `Files - Remote Backup` plan:
  - Option 1 (preferred, preserves ~509 GB history): in the agent's Backup Storage view, delete the single 2051-07-10 restore point, then run the plan once to create a fresh 2026 baseline; re-anchors the frozen 2051 schedule.
  - Option 2 (fallback): force a new full / new generation.
  - Do the delete via the agent GUI or a long-timeout maintenance-window job (CLI over RMM times out), with Mike.
- Verify VWP-SERVER's domain secure channel / Kerberos fully recovered after the 25-year skew (was flagged as a follow-up; not yet run).
- Other VWP plans showing Warning/0.0B (VWP-FILES, WINFileSvr) were noted but not investigated — separate from the 2525 issue.

## Reference Information

- MSP360 KB: kb10096 (RestorePointsTimeMismatch, error 2525) — https://mspbackups.com/backup-agent/2525
- Plan: `Files - Remote Backup`, PlanId `6238c387-7bf6-47a2-a68b-86d590b5fff7`.
- RMM cmd ids: timezone change `61a2e177`; various diagnostics.
- Discord DM to Mike: message_id `1524081568828882985`. Bot alert to #dev-alerts: message_id `1524076358605475982`.
- Client slug: `valleywide`.

## Update: 21:17 PT — waiting on Mike + repo scratch hygiene

**VWP-SERVER backup 2525 — still blocked on Mike.** DM'd him the root cause + two fix options
(15:56). No reply through the afternoon. Set up a self-paced poll of his Discord DM
(`discord-dm.sh read mike`) via ScheduleWakeup at ~20-25 min intervals; polled 3x, no reply;
user then said to end the checking, so the loop was stopped (ScheduleWakeup stop). **Nothing on
the backup has been changed** — the remediation (delete the 2051-07-10 restore point in the agent
Backup Storage view, then run the plan once for a fresh 2026 baseline, preserving ~509 GB history)
is still pending Mike's decision. Resume point: re-check his DM, act on his choice, then rebuild
the wiki (`/wiki-compile client:valleywide --full`).

**Repo scratch/token hygiene (from `/sync`).** A `git add -A` sweep committed stray root scratch
files. Cleaned up:
- Untracked + removed 9 `.rmm*` scratch files that prior sessions had committed to repo root
  (`.rmm_token`, `.rmm_b64_localadmin`, `.rmm-key-probe.ps1`, `.rmm-scan.ps1`, `.rmm_change_cmd`,
  `.rmm_cmds`, `.rmm_desktop_cmd`, `.rmm_server_cmd`, `.rmm_station2_cmd`). Committed `8b016ed`.
- Security check on the committed secrets: `.rmm_token` was a GuruRMM **admin** JWT but already
  **expired** (`exp 2026-07-06 06:28 UTC`, ~30h prior) — no rotation needed. `.rmm_b64_localadmin`
  was an `-EncodedCommand` PowerShell *script* (blanks a Localadmin password), not a stored secret.
  Files remain in git history (left as-is; nothing live exposed).
- Added `.gitignore` rules: `.rmm*` and `.vwp*` under the Session-scratch section.
- Later removed 3 more untracked scratch files (`scratch-audit-functions.ps1`,
  `scratch-audit-policies.ps1`, `.rmm_token.tmp`).
- Found 11 **tracked** `scratch-*.ps1` files (all Howard, dated today) that are a coherent
  **AD/DC-promotion + Windows Server Backup migration** working set (functional-level preflight,
  DC-promo/CA dry-runs, robocopy preseed, SMB share for machine-account migration, start-backup).
  Per user decision: **left tracked, untouched** — likely an active migration. Did NOT add a
  blanket `scratch-*` ignore (would misrepresent tracked files).

**Pending / next steps:**
- Get Mike's answer on the VWP-SERVER backup fix; execute; then `/wiki-compile client:valleywide --full`.
- The 11 `scratch-*.ps1` migration scripts are an open/active workstream (separate from the 2525 issue).

## Update: 2026-07-08 13:08 PT — Mike answered, backup re-seeding via cloned plan (RESOLVED, seeding)

**Mike's decision (via `/ask-forum`).** Re-checked his DM at day start — still no reply. Per user,
asked in #ct-forum (thread `1524444488251215945`, tagged Mike, `ask-forum.sh` background wait).
Mike replied **"OPTION 1"** (delete the 2051 restore point, re-run for a fresh 2026 baseline,
preserve history). He did not say who drives it; user chose to drive it via the ScreenConnect GUI.

**Option 1 turned out to be architecturally impossible.** In the agent GUI (VWP-SERVER, SC session
`4dac23c3-78f2-452e-b386-58dda770b2fb`), right-clicking the 2051 restore point offered no delete.
MSP360 KB confirmed: **you cannot delete an individual restore point — only an entire generation**
(kb.msp360.com). The documented 2525 fix ("wait out the period / correct the time") does not cover
a *future*-dated restore point, so it was useless here (25-yr gap, time already fixed).

**The "509 GB history" was not real history.** The Backup Storage view showed the ENTIRE backup —
both restore points, generation spanning **3/20/2051 -> 7/10/2051**, 561.6 GB original / **502.7 GB
on storage** — was written with the wrong clock. There is NO 2026 data. So the whole reason to avoid
a fresh re-seed (preserving history) was moot; every restore point is 2051-stamped. Plan config:
`Files - Remote Backup` -> `F:\Shares` -> B2Storage, Forever-Forward-Incremental, 90-day retention.
Agent build 8.6.0.338, branded "ACG-Backup Server".

**Resolution (user's approach): clone the plan.** Rather than delete data, user **cloned** the plan
to `Files - Remote Backup_2` (new PlanId `976585c5-927e-47c9-849e-717b4650ef66`) — a genuinely
separate plan/bunch with no 2051 restore point, so it seeds fresh at correct 2026 dates. Old plan
`Files - Remote Backup` (`6238c387...`) to be **disabled** (it was stopped; its schedule still
showed NextStart 2026-07-09 01:00, so it must be disabled, not just stopped, or it re-throws 2525).
First clone run at 19:52 showed Failed/no-error (a Consistency Check plan was Running on B2Storage
concurrently and blocked it). After the check, the clone re-ran and is now **uploading — 2.30 GB and
climbing** at 2026 dates. Core fix confirmed working; full ~500 GB seed will take hours.

**Verification note:** MSP360 MBS monitoring feed LAGS real-time — it still showed the 19:52 Failed
attempt while the agent locally showed Running/uploading. Trust the local agent for in-flight state;
the feed catches up on completion/heartbeat.

**Pending / next steps (user will check later and update):**
- Let `Files - Remote Backup_2` finish the full; verify via `/msp360 monitoring --company "Valleywide
  Plastering"` -> expect Status **Success** + a **2026** LastStart. That formally clears error 2525.
- **Disable** (not just stop) the old `Files - Remote Backup` plan so it stops erroring.
- **B2 cleanup:** once the new baseline is healthy, purge the old ~502 GB 2051 generation from B2 to
  avoid ~1 TB double-billing.
- Then `/wiki-compile client:valleywide --full`.

## Update: 2026-07-08 13:1x PT — session close

Backup (`Files - Remote Backup_2`) confirmed actively uploading when session closed; user will check
tomorrow (2026-07-09). Sent a reminder DM to Howard (Discord message_id `1524565994075324477`)
covering the 3 check-tomorrow items above (verify Success/2026 date, disable old plan, B2 cleanup).
Session closed here.
