claudetools/clients/valleywide/session-logs/2026-07/2026-07-07-howard-vwp-server-clock-backup-2525.md

User

• User: Howard Enos (howard)
• Machine: Howard-Home
• Role: tech

Session Summary

Investigated MSP360 backup error 2525 (RestorePointsTimeMismatch, kb10096) reported on VWP-SERVER at Valley Wide Plastering. The error text ("current system time precedes the previous restore point creation time") pointed at a system-clock problem, so the investigation started on the endpoint rather than in the backup console.

Located VWP-SERVER via rmm-search (GuruRMM id d2cc6200-07e5-4444-bb1c-1bcce273d615, Main Office, online) and queried its clock over /rmm. The machine's clock read 2051-07-10 — roughly 25 years in the future — with the W32Time source stuck on Local CMOS Clock (Leap Indicator 3 = not synchronized) despite an NT5DS domain-sync configuration. Before changing anything, confirmed domain role and virtualization: VWP-SERVER is a VMware VM and a domain member server (DomainRole 3), not a DC. FSMO roles are held by VWP-DC1 (PDC/RID/Infra) and VWP_ADSRVR (schema/naming).

Because a 25-year skew on a domain member would normally break Kerberos, sampled the rest of the VWP.US fleet to rule out a domain-wide/ESXi-host time problem. VWP-DC1, VWP_ADSRVR, FRONT64, and VWP-HYPERV1 were all correct (2026-07-07) and syncing healthily (DCs from time.nist.gov, members from the DCs). So the 2051 clock was isolated to VWP-SERVER, not domain-wide.

Per user direction (Mike had already corrected the date/time out-of-band during the session), set VWP-SERVER's timezone from Pacific Standard Time to US Mountain Standard Time (Arizona, no DST). Re-verified afterward: VWP-SERVER now matches VWP-DC1 exactly (2026-07-07), timezone correct, and — critically — the W32Time source flipped from Local CMOS Clock to VWP-DC1.VWP.US with a fresh successful sync, so the root cause (broken domain time sync) is resolved and the clock should hold.

Then investigated the remaining backup breakage via /msp360 and cbb.exe on the endpoint. The Files - Remote Backup plan (F:\Shares -> B2Storage, ~509 GB stored, block-level/new-format, unencrypted) had written its last restore point at 2051-07-10 6:00 PM while the clock was wrong; its next scheduled run is frozen at 2051. Now that time is back to 2026, new runs "precede" the 2051 point -> error 2525 persists, and the plan will not self-heal. Per-restore-point CLI enumeration (cbb list -rp) timed out against the 509 GB B2 bunch (240s server-side reaper), establishing that a surgical delete over RMM is impractical. DM'd Mike the root cause and two fix options and left the backup untouched pending his decision.

Key Decisions

• Started diagnosis on the endpoint clock (not the MSP360 console) because error 2525's text is a time-ordering failure, not a backup-data failure.
• Checked domain role + virtualization and sampled the whole VWP fleet before touching the clock — a naive "set it back to 2026" on a domain member could have either been re-pushed forward by VMware Tools or dropped the member off the domain if the DCs were also wrong. Sampling proved the DCs were fine and the box was isolated.
• Chose US Mountain Standard Time (Arizona, no DST) over plain Mountain Standard Time (observes DST) — correct zone for Arizona; the old Pacific Standard Time only coincidentally matched offset in summer (PDT = UTC-7).
• Did not attempt the MSP360 restore-point delete over RMM after the listing call hit the reaper timeout; recommended GUI or a maintenance-window job with Mike to preserve the 509 GB history rather than a full wipe/re-upload.
• Kept the backup remediation gated behind Mike's decision (destructive to backup storage; his call on approach).

Problems Encountered

• jq parse error dispatching PowerShell with backslashes (root\\directory\\ldap in a heredoc). Resolved by building the command payload with jq -n --arg cmd "$SCRIPT" instead of inlining into the JSON.
• /tmp path blocked by block-tmp-path.sh pre-commit hook when writing a scratch marker. Resolved by using repo-relative scratch files (.vwp_cmd, .vwp_times), later cleaned up.
• cbb help <cmd> reprints the top-level index rather than subcommand help; correct form is <cmd> -?. account needs -l to list.
• cbb list -rp against the 509 GB B2 bunch timed out at the 240s server-side reaper with no output (output was buffered to a file only read after all calls finished). Concluded remote CLI enumeration of individual restore points is not viable; deferred to GUI/maintenance-window.

Configuration Changes

• VWP-SERVER timezone: Pacific Standard Time -> US Mountain Standard Time (via Set-TimeZone -Id "US Mountain Standard Time" over /rmm, cmd 61a2e177). Only change made this session.
• No file changes in the repo other than this session log. Scratch files created during diagnostics (.vwp_cmd, .vwp_times) were removed.

Credentials & Secrets

• None discovered, created, or rotated. RMM auth via existing vault path infrastructure/gururmm-server.sops.yaml; MSP360 via msp-tools/msp360-api.sops.yaml. No plaintext secrets handled.

Infrastructure & Servers

• VWP-SERVER — Valley Wide Plastering, Main Office. GuruRMM id d2cc6200-07e5-4444-bb1c-1bcce273d615. VMware VM (VMware20,1). Domain member (DomainRole 3) of VWP.US. TZ now US Mountain Standard Time; W32Time source = VWP-DC1.VWP.US.
• VWP.US domain FSMO: PDC/RID/Infra = VWP-DC1 (8eefbba6-28cf-4e3f-8fda-11900e0ac302); Schema/Naming = VWP_ADSRVR (bd2f2f86-ea33-4202-828f-b378e459e891). VWP-DC1 syncs from time.nist.gov.
• VWP-HYPERV1 — Hyper-V host (bdc3e142-1eb3-4034-8ac2-300f46dbc1d9), separate from the VMware host VWP-SERVER runs on. Clock healthy.
• MSP360 company: Valleywide Plastering (id b271f525-2c00-4407-bcc6-a991973a3ecb).
• Backup destinations (cbb accounts, login vwp): B2Storage (id 7faaaf4d-6bdb-4a4e-af52-38cebe33974b, ~546,465,990,074 bytes / ~509 GB used); VWP-Backup (id 4b4f35b7-d83f-4df4-8270-5d23117e4873, 0 bytes).
• Backup agent (white-labeled): C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe. Services: Online Backup Service, Online Backup Service Remote Management (both Running).

Commands & Outputs

• bash .claude/scripts/rmm-search.sh VWP-SERVER -> single match, Valley Wide Plastering.
• Time check: VWP-SERVER Local 2051-07-10 18:45, Source Local CMOS Clock, TZ Pacific Standard Time, Type NT5DS.
• Fleet sample (post): VWP-DC1/VWP_ADSRVR/FRONT64/VWP-HYPERV1 all 2026-07-07, healthy sync.
• Set-TimeZone -Id "US Mountain Standard Time" -> Before Pacific Standard Time, After US Mountain Standard Time.
• Post-fix verify: VWP-SERVER 2026-07-07 08:36:58 == VWP-DC1; Source VWP-DC1.VWP.US; last successful sync 08:36:30.
• py .claude/skills/msp360/scripts/msp360.py monitoring --company "Valleywide Plastering" -> VWP-SERVER plan Files - Remote Backup, Status Warning, LastStart 2051-07-11T01:00:41, 0.0B.
• cbb getPlanDetails -id 6238c387-7bf6-47a2-a68b-86d590b5fff7 -> F:\Shares -> B2Storage, block-level, unencrypted, daily 6PM.
• cbb list -a "B2Storage" -rp ... -> timed out (server-side reaper, 240s).

Pending / Incomplete Tasks

• BLOCKED on Mike's decision (DM'd 2026-07-07 15:56; no reply yet). Remaining work = clear MSP360 error 2525 on VWP-SERVER's Files - Remote Backup plan:
  • Option 1 (preferred, preserves ~509 GB history): in the agent's Backup Storage view, delete the single 2051-07-10 restore point, then run the plan once to create a fresh 2026 baseline; re-anchors the frozen 2051 schedule.
  • Option 2 (fallback): force a new full / new generation.
  • Do the delete via the agent GUI or a long-timeout maintenance-window job (CLI over RMM times out), with Mike.
• Verify VWP-SERVER's domain secure channel / Kerberos fully recovered after the 25-year skew (was flagged as a follow-up; not yet run).
• Other VWP plans showing Warning/0.0B (VWP-FILES, WINFileSvr) were noted but not investigated — separate from the 2525 issue.

Reference Information

• MSP360 KB: kb10096 (RestorePointsTimeMismatch, error 2525) — https://mspbackups.com/backup-agent/2525
• Plan: Files - Remote Backup, PlanId 6238c387-7bf6-47a2-a68b-86d590b5fff7.
• RMM cmd ids: timezone change 61a2e177; various diagnostics.
• Discord DM to Mike: message_id 1524081568828882985. Bot alert to #dev-alerts: message_id 1524076358605475982.
• Client slug: valleywide.