From f30413cffc48b27326295ac39da49304a018bf5d Mon Sep 17 00:00:00 2001 From: Howard Enos Date: Tue, 7 Jul 2026 09:03:16 -0700 Subject: [PATCH] sync: auto-sync from HOWARD-HOME at 2026-07-07 09:02:48 Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-07-07 09:02:48 --- ...-07-howard-vwp-server-clock-backup-2525.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 clients/valleywide/session-logs/2026-07/2026-07-07-howard-vwp-server-clock-backup-2525.md diff --git a/clients/valleywide/session-logs/2026-07/2026-07-07-howard-vwp-server-clock-backup-2525.md b/clients/valleywide/session-logs/2026-07/2026-07-07-howard-vwp-server-clock-backup-2525.md new file mode 100644 index 00000000..ae7e1ef3 --- /dev/null +++ b/clients/valleywide/session-logs/2026-07/2026-07-07-howard-vwp-server-clock-backup-2525.md @@ -0,0 +1,77 @@ +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Investigated MSP360 backup error 2525 (`RestorePointsTimeMismatch`, kb10096) reported on VWP-SERVER at Valley Wide Plastering. The error text ("current system time precedes the previous restore point creation time") pointed at a system-clock problem, so the investigation started on the endpoint rather than in the backup console. + +Located VWP-SERVER via `rmm-search` (GuruRMM id `d2cc6200-07e5-4444-bb1c-1bcce273d615`, Main Office, online) and queried its clock over `/rmm`. The machine's clock read **2051-07-10** — roughly 25 years in the future — with the W32Time source stuck on `Local CMOS Clock` (Leap Indicator 3 = not synchronized) despite an `NT5DS` domain-sync configuration. Before changing anything, confirmed domain role and virtualization: VWP-SERVER is a **VMware VM** and a **domain member server** (DomainRole 3), not a DC. FSMO roles are held by VWP-DC1 (PDC/RID/Infra) and VWP_ADSRVR (schema/naming). + +Because a 25-year skew on a domain member would normally break Kerberos, sampled the rest of the VWP.US fleet to rule out a domain-wide/ESXi-host time problem. VWP-DC1, VWP_ADSRVR, FRONT64, and VWP-HYPERV1 were all correct (2026-07-07) and syncing healthily (DCs from time.nist.gov, members from the DCs). So the 2051 clock was **isolated to VWP-SERVER**, not domain-wide. + +Per user direction (Mike had already corrected the date/time out-of-band during the session), set VWP-SERVER's timezone from `Pacific Standard Time` to `US Mountain Standard Time` (Arizona, no DST). Re-verified afterward: VWP-SERVER now matches VWP-DC1 exactly (2026-07-07), timezone correct, and — critically — the W32Time source flipped from `Local CMOS Clock` to `VWP-DC1.VWP.US` with a fresh successful sync, so the root cause (broken domain time sync) is resolved and the clock should hold. + +Then investigated the remaining backup breakage via `/msp360` and `cbb.exe` on the endpoint. The `Files - Remote Backup` plan (F:\Shares -> B2Storage, ~509 GB stored, block-level/new-format, unencrypted) had written its last restore point at 2051-07-10 6:00 PM while the clock was wrong; its next scheduled run is frozen at 2051. Now that time is back to 2026, new runs "precede" the 2051 point -> error 2525 persists, and the plan will not self-heal. Per-restore-point CLI enumeration (`cbb list -rp`) timed out against the 509 GB B2 bunch (240s server-side reaper), establishing that a surgical delete over RMM is impractical. DM'd Mike the root cause and two fix options and left the backup untouched pending his decision. + +## Key Decisions + +- Started diagnosis on the endpoint clock (not the MSP360 console) because error 2525's text is a time-ordering failure, not a backup-data failure. +- Checked domain role + virtualization and sampled the whole VWP fleet **before** touching the clock — a naive "set it back to 2026" on a domain member could have either been re-pushed forward by VMware Tools or dropped the member off the domain if the DCs were also wrong. Sampling proved the DCs were fine and the box was isolated. +- Chose `US Mountain Standard Time` (Arizona, no DST) over plain `Mountain Standard Time` (observes DST) — correct zone for Arizona; the old `Pacific Standard Time` only coincidentally matched offset in summer (PDT = UTC-7). +- Did not attempt the MSP360 restore-point delete over RMM after the listing call hit the reaper timeout; recommended GUI or a maintenance-window job with Mike to preserve the 509 GB history rather than a full wipe/re-upload. +- Kept the backup remediation gated behind Mike's decision (destructive to backup storage; his call on approach). + +## Problems Encountered + +- **jq parse error dispatching PowerShell with backslashes** (`root\\directory\\ldap` in a heredoc). Resolved by building the command payload with `jq -n --arg cmd "$SCRIPT"` instead of inlining into the JSON. +- **`/tmp` path blocked by `block-tmp-path.sh` pre-commit hook** when writing a scratch marker. Resolved by using repo-relative scratch files (`.vwp_cmd`, `.vwp_times`), later cleaned up. +- **`cbb help ` reprints the top-level index** rather than subcommand help; correct form is ` -?`. `account` needs `-l` to list. +- **`cbb list -rp` against the 509 GB B2 bunch timed out** at the 240s server-side reaper with no output (output was buffered to a file only read after all calls finished). Concluded remote CLI enumeration of individual restore points is not viable; deferred to GUI/maintenance-window. + +## Configuration Changes + +- **VWP-SERVER timezone:** `Pacific Standard Time` -> `US Mountain Standard Time` (via `Set-TimeZone -Id "US Mountain Standard Time"` over `/rmm`, cmd `61a2e177`). Only change made this session. +- No file changes in the repo other than this session log. Scratch files created during diagnostics (`.vwp_cmd`, `.vwp_times`) were removed. + +## Credentials & Secrets + +- None discovered, created, or rotated. RMM auth via existing vault path `infrastructure/gururmm-server.sops.yaml`; MSP360 via `msp-tools/msp360-api.sops.yaml`. No plaintext secrets handled. + +## Infrastructure & Servers + +- **VWP-SERVER** — Valley Wide Plastering, Main Office. GuruRMM id `d2cc6200-07e5-4444-bb1c-1bcce273d615`. VMware VM (VMware20,1). Domain member (DomainRole 3) of `VWP.US`. TZ now US Mountain Standard Time; W32Time source = VWP-DC1.VWP.US. +- **VWP.US domain FSMO:** PDC/RID/Infra = VWP-DC1 (`8eefbba6-28cf-4e3f-8fda-11900e0ac302`); Schema/Naming = VWP_ADSRVR (`bd2f2f86-ea33-4202-828f-b378e459e891`). VWP-DC1 syncs from time.nist.gov. +- **VWP-HYPERV1** — Hyper-V host (`bdc3e142-1eb3-4034-8ac2-300f46dbc1d9`), separate from the VMware host VWP-SERVER runs on. Clock healthy. +- **MSP360 company:** `Valleywide Plastering` (id `b271f525-2c00-4407-bcc6-a991973a3ecb`). +- **Backup destinations (cbb accounts, login `vwp`):** `B2Storage` (id `7faaaf4d-6bdb-4a4e-af52-38cebe33974b`, ~546,465,990,074 bytes / ~509 GB used); `VWP-Backup` (id `4b4f35b7-d83f-4df4-8270-5d23117e4873`, 0 bytes). +- **Backup agent (white-labeled):** `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`. Services: `Online Backup Service`, `Online Backup Service Remote Management` (both Running). + +## Commands & Outputs + +- `bash .claude/scripts/rmm-search.sh VWP-SERVER` -> single match, Valley Wide Plastering. +- Time check: VWP-SERVER Local `2051-07-10 18:45`, Source `Local CMOS Clock`, TZ `Pacific Standard Time`, Type `NT5DS`. +- Fleet sample (post): VWP-DC1/VWP_ADSRVR/FRONT64/VWP-HYPERV1 all `2026-07-07`, healthy sync. +- `Set-TimeZone -Id "US Mountain Standard Time"` -> Before `Pacific Standard Time`, After `US Mountain Standard Time`. +- Post-fix verify: VWP-SERVER `2026-07-07 08:36:58` == VWP-DC1; Source `VWP-DC1.VWP.US`; last successful sync 08:36:30. +- `py .claude/skills/msp360/scripts/msp360.py monitoring --company "Valleywide Plastering"` -> VWP-SERVER plan `Files - Remote Backup`, Status Warning, LastStart `2051-07-11T01:00:41`, 0.0B. +- `cbb getPlanDetails -id 6238c387-7bf6-47a2-a68b-86d590b5fff7` -> F:\Shares -> B2Storage, block-level, unencrypted, daily 6PM. +- `cbb list -a "B2Storage" -rp ...` -> timed out (server-side reaper, 240s). + +## Pending / Incomplete Tasks + +- **BLOCKED on Mike's decision** (DM'd 2026-07-07 15:56; no reply yet). Remaining work = clear MSP360 error 2525 on VWP-SERVER's `Files - Remote Backup` plan: + - Option 1 (preferred, preserves ~509 GB history): in the agent's Backup Storage view, delete the single 2051-07-10 restore point, then run the plan once to create a fresh 2026 baseline; re-anchors the frozen 2051 schedule. + - Option 2 (fallback): force a new full / new generation. + - Do the delete via the agent GUI or a long-timeout maintenance-window job (CLI over RMM times out), with Mike. +- Verify VWP-SERVER's domain secure channel / Kerberos fully recovered after the 25-year skew (was flagged as a follow-up; not yet run). +- Other VWP plans showing Warning/0.0B (VWP-FILES, WINFileSvr) were noted but not investigated — separate from the 2525 issue. + +## Reference Information + +- MSP360 KB: kb10096 (RestorePointsTimeMismatch, error 2525) — https://mspbackups.com/backup-agent/2525 +- Plan: `Files - Remote Backup`, PlanId `6238c387-7bf6-47a2-a68b-86d590b5fff7`. +- RMM cmd ids: timezone change `61a2e177`; various diagnostics. +- Discord DM to Mike: message_id `1524081568828882985`. Bot alert to #dev-alerts: message_id `1524076358605475982`. +- Client slug: `valleywide`.