azcomputerguru/claudetools
1
0
Fork 0
Code Issues 58 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
cda32b53bbe3604b2dee39b151e7f207db2345e5
claudetools/clients/valleywide/session-logs/2026-07/2026-07-07-howard-vwp-server-clock-backup-2525.md
Howard Enos 447b18d62b sync: auto-sync from HOWARD-HOME at 2026-07-08 17:00:33 
Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-08 17:00:33
2026-07-08 17:01:23 -07:00

14 KiB
Raw Blame History

User

  • User: Howard Enos (howard)
  • Machine: Howard-Home
  • Role: tech

Session Summary

Investigated MSP360 backup error 2525 (RestorePointsTimeMismatch, kb10096) reported on VWP-SERVER at Valley Wide Plastering. The error text ("current system time precedes the previous restore point creation time") pointed at a system-clock problem, so the investigation started on the endpoint rather than in the backup console.

Located VWP-SERVER via rmm-search (GuruRMM id d2cc6200-07e5-4444-bb1c-1bcce273d615, Main Office, online) and queried its clock over /rmm. The machine's clock read 2051-07-10 — roughly 25 years in the future — with the W32Time source stuck on Local CMOS Clock (Leap Indicator 3 = not synchronized) despite an NT5DS domain-sync configuration. Before changing anything, confirmed domain role and virtualization: VWP-SERVER is a VMware VM and a domain member server (DomainRole 3), not a DC. FSMO roles are held by VWP-DC1 (PDC/RID/Infra) and VWP_ADSRVR (schema/naming).

Because a 25-year skew on a domain member would normally break Kerberos, sampled the rest of the VWP.US fleet to rule out a domain-wide/ESXi-host time problem. VWP-DC1, VWP_ADSRVR, FRONT64, and VWP-HYPERV1 were all correct (2026-07-07) and syncing healthily (DCs from time.nist.gov, members from the DCs). So the 2051 clock was isolated to VWP-SERVER, not domain-wide.

Per user direction (Mike had already corrected the date/time out-of-band during the session), set VWP-SERVER's timezone from Pacific Standard Time to US Mountain Standard Time (Arizona, no DST). Re-verified afterward: VWP-SERVER now matches VWP-DC1 exactly (2026-07-07), timezone correct, and — critically — the W32Time source flipped from Local CMOS Clock to VWP-DC1.VWP.US with a fresh successful sync, so the root cause (broken domain time sync) is resolved and the clock should hold.

Then investigated the remaining backup breakage via /msp360 and cbb.exe on the endpoint. The Files - Remote Backup plan (F:\Shares -> B2Storage, ~509 GB stored, block-level/new-format, unencrypted) had written its last restore point at 2051-07-10 6:00 PM while the clock was wrong; its next scheduled run is frozen at 2051. Now that time is back to 2026, new runs "precede" the 2051 point -> error 2525 persists, and the plan will not self-heal. Per-restore-point CLI enumeration (cbb list -rp) timed out against the 509 GB B2 bunch (240s server-side reaper), establishing that a surgical delete over RMM is impractical. DM'd Mike the root cause and two fix options and left the backup untouched pending his decision.

Key Decisions

  • Started diagnosis on the endpoint clock (not the MSP360 console) because error 2525's text is a time-ordering failure, not a backup-data failure.
  • Checked domain role + virtualization and sampled the whole VWP fleet before touching the clock — a naive "set it back to 2026" on a domain member could have either been re-pushed forward by VMware Tools or dropped the member off the domain if the DCs were also wrong. Sampling proved the DCs were fine and the box was isolated.
  • Chose US Mountain Standard Time (Arizona, no DST) over plain Mountain Standard Time (observes DST) — correct zone for Arizona; the old Pacific Standard Time only coincidentally matched offset in summer (PDT = UTC-7).
  • Did not attempt the MSP360 restore-point delete over RMM after the listing call hit the reaper timeout; recommended GUI or a maintenance-window job with Mike to preserve the 509 GB history rather than a full wipe/re-upload.
  • Kept the backup remediation gated behind Mike's decision (destructive to backup storage; his call on approach).

Problems Encountered

  • jq parse error dispatching PowerShell with backslashes (root\\directory\\ldap in a heredoc). Resolved by building the command payload with jq -n --arg cmd "$SCRIPT" instead of inlining into the JSON.
  • /tmp path blocked by block-tmp-path.sh pre-commit hook when writing a scratch marker. Resolved by using repo-relative scratch files (.vwp_cmd, .vwp_times), later cleaned up.
  • cbb help <cmd> reprints the top-level index rather than subcommand help; correct form is <cmd> -?. account needs -l to list.
  • cbb list -rp against the 509 GB B2 bunch timed out at the 240s server-side reaper with no output (output was buffered to a file only read after all calls finished). Concluded remote CLI enumeration of individual restore points is not viable; deferred to GUI/maintenance-window.

Configuration Changes

  • VWP-SERVER timezone: Pacific Standard Time -> US Mountain Standard Time (via Set-TimeZone -Id "US Mountain Standard Time" over /rmm, cmd 61a2e177). Only change made this session.
  • No file changes in the repo other than this session log. Scratch files created during diagnostics (.vwp_cmd, .vwp_times) were removed.

Credentials & Secrets

  • None discovered, created, or rotated. RMM auth via existing vault path infrastructure/gururmm-server.sops.yaml; MSP360 via msp-tools/msp360-api.sops.yaml. No plaintext secrets handled.

Infrastructure & Servers

  • VWP-SERVER — Valley Wide Plastering, Main Office. GuruRMM id d2cc6200-07e5-4444-bb1c-1bcce273d615. VMware VM (VMware20,1). Domain member (DomainRole 3) of VWP.US. TZ now US Mountain Standard Time; W32Time source = VWP-DC1.VWP.US.
  • VWP.US domain FSMO: PDC/RID/Infra = VWP-DC1 (8eefbba6-28cf-4e3f-8fda-11900e0ac302); Schema/Naming = VWP_ADSRVR (bd2f2f86-ea33-4202-828f-b378e459e891). VWP-DC1 syncs from time.nist.gov.
  • VWP-HYPERV1 — Hyper-V host (bdc3e142-1eb3-4034-8ac2-300f46dbc1d9), separate from the VMware host VWP-SERVER runs on. Clock healthy.
  • MSP360 company: Valleywide Plastering (id b271f525-2c00-4407-bcc6-a991973a3ecb).
  • Backup destinations (cbb accounts, login vwp): B2Storage (id 7faaaf4d-6bdb-4a4e-af52-38cebe33974b, ~546,465,990,074 bytes / ~509 GB used); VWP-Backup (id 4b4f35b7-d83f-4df4-8270-5d23117e4873, 0 bytes).
  • Backup agent (white-labeled): C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe. Services: Online Backup Service, Online Backup Service Remote Management (both Running).

Commands & Outputs

  • bash .claude/scripts/rmm-search.sh VWP-SERVER -> single match, Valley Wide Plastering.
  • Time check: VWP-SERVER Local 2051-07-10 18:45, Source Local CMOS Clock, TZ Pacific Standard Time, Type NT5DS.
  • Fleet sample (post): VWP-DC1/VWP_ADSRVR/FRONT64/VWP-HYPERV1 all 2026-07-07, healthy sync.
  • Set-TimeZone -Id "US Mountain Standard Time" -> Before Pacific Standard Time, After US Mountain Standard Time.
  • Post-fix verify: VWP-SERVER 2026-07-07 08:36:58 == VWP-DC1; Source VWP-DC1.VWP.US; last successful sync 08:36:30.
  • py .claude/skills/msp360/scripts/msp360.py monitoring --company "Valleywide Plastering" -> VWP-SERVER plan Files - Remote Backup, Status Warning, LastStart 2051-07-11T01:00:41, 0.0B.
  • cbb getPlanDetails -id 6238c387-7bf6-47a2-a68b-86d590b5fff7 -> F:\Shares -> B2Storage, block-level, unencrypted, daily 6PM.
  • cbb list -a "B2Storage" -rp ... -> timed out (server-side reaper, 240s).

Pending / Incomplete Tasks

  • BLOCKED on Mike's decision (DM'd 2026-07-07 15:56; no reply yet). Remaining work = clear MSP360 error 2525 on VWP-SERVER's Files - Remote Backup plan:
    • Option 1 (preferred, preserves ~509 GB history): in the agent's Backup Storage view, delete the single 2051-07-10 restore point, then run the plan once to create a fresh 2026 baseline; re-anchors the frozen 2051 schedule.
    • Option 2 (fallback): force a new full / new generation.
    • Do the delete via the agent GUI or a long-timeout maintenance-window job (CLI over RMM times out), with Mike.
  • Verify VWP-SERVER's domain secure channel / Kerberos fully recovered after the 25-year skew (was flagged as a follow-up; not yet run).
  • Other VWP plans showing Warning/0.0B (VWP-FILES, WINFileSvr) were noted but not investigated — separate from the 2525 issue.

Reference Information

  • MSP360 KB: kb10096 (RestorePointsTimeMismatch, error 2525) — https://mspbackups.com/backup-agent/2525
  • Plan: Files - Remote Backup, PlanId 6238c387-7bf6-47a2-a68b-86d590b5fff7.
  • RMM cmd ids: timezone change 61a2e177; various diagnostics.
  • Discord DM to Mike: message_id 1524081568828882985. Bot alert to #dev-alerts: message_id 1524076358605475982.
  • Client slug: valleywide.

Update: 21:17 PT — waiting on Mike + repo scratch hygiene

VWP-SERVER backup 2525 — still blocked on Mike. DM'd him the root cause + two fix options (15:56). No reply through the afternoon. Set up a self-paced poll of his Discord DM (discord-dm.sh read mike) via ScheduleWakeup at ~20-25 min intervals; polled 3x, no reply; user then said to end the checking, so the loop was stopped (ScheduleWakeup stop). Nothing on the backup has been changed — the remediation (delete the 2051-07-10 restore point in the agent Backup Storage view, then run the plan once for a fresh 2026 baseline, preserving ~509 GB history) is still pending Mike's decision. Resume point: re-check his DM, act on his choice, then rebuild the wiki (/wiki-compile client:valleywide --full).

Repo scratch/token hygiene (from /sync). A git add -A sweep committed stray root scratch files. Cleaned up:

  • Untracked + removed 9 .rmm* scratch files that prior sessions had committed to repo root (.rmm_token, .rmm_b64_localadmin, .rmm-key-probe.ps1, .rmm-scan.ps1, .rmm_change_cmd, .rmm_cmds, .rmm_desktop_cmd, .rmm_server_cmd, .rmm_station2_cmd). Committed 8b016ed.
  • Security check on the committed secrets: .rmm_token was a GuruRMM admin JWT but already expired (exp 2026-07-06 06:28 UTC, ~30h prior) — no rotation needed. .rmm_b64_localadmin was an -EncodedCommand PowerShell script (blanks a Localadmin password), not a stored secret. Files remain in git history (left as-is; nothing live exposed).
  • Added .gitignore rules: .rmm* and .vwp* under the Session-scratch section.
  • Later removed 3 more untracked scratch files (scratch-audit-functions.ps1, scratch-audit-policies.ps1, .rmm_token.tmp).
  • Found 11 tracked scratch-*.ps1 files (all Howard, dated today) that are a coherent AD/DC-promotion + Windows Server Backup migration working set (functional-level preflight, DC-promo/CA dry-runs, robocopy preseed, SMB share for machine-account migration, start-backup). Per user decision: left tracked, untouched — likely an active migration. Did NOT add a blanket scratch-* ignore (would misrepresent tracked files).

Pending / next steps:

  • Get Mike's answer on the VWP-SERVER backup fix; execute; then /wiki-compile client:valleywide --full.
  • The 11 scratch-*.ps1 migration scripts are an open/active workstream (separate from the 2525 issue).

Update: 2026-07-08 13:08 PT — Mike answered, backup re-seeding via cloned plan (RESOLVED, seeding)

Mike's decision (via /ask-forum). Re-checked his DM at day start — still no reply. Per user, asked in #ct-forum (thread 1524444488251215945, tagged Mike, ask-forum.sh background wait). Mike replied "OPTION 1" (delete the 2051 restore point, re-run for a fresh 2026 baseline, preserve history). He did not say who drives it; user chose to drive it via the ScreenConnect GUI.

Option 1 turned out to be architecturally impossible. In the agent GUI (VWP-SERVER, SC session 4dac23c3-78f2-452e-b386-58dda770b2fb), right-clicking the 2051 restore point offered no delete. MSP360 KB confirmed: you cannot delete an individual restore point — only an entire generation (kb.msp360.com). The documented 2525 fix ("wait out the period / correct the time") does not cover a future-dated restore point, so it was useless here (25-yr gap, time already fixed).

The "509 GB history" was not real history. The Backup Storage view showed the ENTIRE backup — both restore points, generation spanning 3/20/2051 -> 7/10/2051, 561.6 GB original / 502.7 GB on storage — was written with the wrong clock. There is NO 2026 data. So the whole reason to avoid a fresh re-seed (preserving history) was moot; every restore point is 2051-stamped. Plan config: Files - Remote Backup -> F:\Shares -> B2Storage, Forever-Forward-Incremental, 90-day retention. Agent build 8.6.0.338, branded "ACG-Backup Server".

Resolution (user's approach): clone the plan. Rather than delete data, user cloned the plan to Files - Remote Backup_2 (new PlanId 976585c5-927e-47c9-849e-717b4650ef66) — a genuinely separate plan/bunch with no 2051 restore point, so it seeds fresh at correct 2026 dates. Old plan Files - Remote Backup (6238c387...) to be disabled (it was stopped; its schedule still showed NextStart 2026-07-09 01:00, so it must be disabled, not just stopped, or it re-throws 2525). First clone run at 19:52 showed Failed/no-error (a Consistency Check plan was Running on B2Storage concurrently and blocked it). After the check, the clone re-ran and is now uploading — 2.30 GB and climbing at 2026 dates. Core fix confirmed working; full ~500 GB seed will take hours.

Verification note: MSP360 MBS monitoring feed LAGS real-time — it still showed the 19:52 Failed attempt while the agent locally showed Running/uploading. Trust the local agent for in-flight state; the feed catches up on completion/heartbeat.

Pending / next steps (user will check later and update):

  • Let Files - Remote Backup_2 finish the full; verify via /msp360 monitoring --company "Valleywide Plastering" -> expect Status Success + a 2026 LastStart. That formally clears error 2525.
  • Disable (not just stop) the old Files - Remote Backup plan so it stops erroring.
  • B2 cleanup: once the new baseline is healthy, purge the old ~502 GB 2051 generation from B2 to avoid ~1 TB double-billing.
  • Then /wiki-compile client:valleywide --full.

Update: 2026-07-08 13:1x PT — session close

Backup (Files - Remote Backup_2) confirmed actively uploading when session closed; user will check tomorrow (2026-07-09). Sent a reminder DM to Howard (Discord message_id 1524565994075324477) covering the 3 check-tomorrow items above (verify Success/2026 date, disable old plan, B2 cleanup). Session closed here.

Reference in New Issue View Git Blame Copy Permalink