sync: auto-sync from HOWARD-HOME at 2026-07-07 09:02:48
Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-07-07 09:02:48
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
## User
|
||||
- **User:** Howard Enos (howard)
|
||||
- **Machine:** Howard-Home
|
||||
- **Role:** tech
|
||||
|
||||
## Session Summary
|
||||
|
||||
Investigated MSP360 backup error 2525 (`RestorePointsTimeMismatch`, kb10096) reported on VWP-SERVER at Valley Wide Plastering. The error text ("current system time precedes the previous restore point creation time") pointed at a system-clock problem, so the investigation started on the endpoint rather than in the backup console.
|
||||
|
||||
Located VWP-SERVER via `rmm-search` (GuruRMM id `d2cc6200-07e5-4444-bb1c-1bcce273d615`, Main Office, online) and queried its clock over `/rmm`. The machine's clock read **2051-07-10** — roughly 25 years in the future — with the W32Time source stuck on `Local CMOS Clock` (Leap Indicator 3 = not synchronized) despite an `NT5DS` domain-sync configuration. Before changing anything, confirmed domain role and virtualization: VWP-SERVER is a **VMware VM** and a **domain member server** (DomainRole 3), not a DC. FSMO roles are held by VWP-DC1 (PDC/RID/Infra) and VWP_ADSRVR (schema/naming).
|
||||
|
||||
Because a 25-year skew on a domain member would normally break Kerberos, sampled the rest of the VWP.US fleet to rule out a domain-wide/ESXi-host time problem. VWP-DC1, VWP_ADSRVR, FRONT64, and VWP-HYPERV1 were all correct (2026-07-07) and syncing healthily (DCs from time.nist.gov, members from the DCs). So the 2051 clock was **isolated to VWP-SERVER**, not domain-wide.
|
||||
|
||||
Per user direction (Mike had already corrected the date/time out-of-band during the session), set VWP-SERVER's timezone from `Pacific Standard Time` to `US Mountain Standard Time` (Arizona, no DST). Re-verified afterward: VWP-SERVER now matches VWP-DC1 exactly (2026-07-07), timezone correct, and — critically — the W32Time source flipped from `Local CMOS Clock` to `VWP-DC1.VWP.US` with a fresh successful sync, so the root cause (broken domain time sync) is resolved and the clock should hold.
|
||||
|
||||
Then investigated the remaining backup breakage via `/msp360` and `cbb.exe` on the endpoint. The `Files - Remote Backup` plan (F:\Shares -> B2Storage, ~509 GB stored, block-level/new-format, unencrypted) had written its last restore point at 2051-07-10 6:00 PM while the clock was wrong; its next scheduled run is frozen at 2051. Now that time is back to 2026, new runs "precede" the 2051 point -> error 2525 persists, and the plan will not self-heal. Per-restore-point CLI enumeration (`cbb list -rp`) timed out against the 509 GB B2 bunch (240s server-side reaper), establishing that a surgical delete over RMM is impractical. DM'd Mike the root cause and two fix options and left the backup untouched pending his decision.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- Started diagnosis on the endpoint clock (not the MSP360 console) because error 2525's text is a time-ordering failure, not a backup-data failure.
|
||||
- Checked domain role + virtualization and sampled the whole VWP fleet **before** touching the clock — a naive "set it back to 2026" on a domain member could have either been re-pushed forward by VMware Tools or dropped the member off the domain if the DCs were also wrong. Sampling proved the DCs were fine and the box was isolated.
|
||||
- Chose `US Mountain Standard Time` (Arizona, no DST) over plain `Mountain Standard Time` (observes DST) — correct zone for Arizona; the old `Pacific Standard Time` only coincidentally matched offset in summer (PDT = UTC-7).
|
||||
- Did not attempt the MSP360 restore-point delete over RMM after the listing call hit the reaper timeout; recommended GUI or a maintenance-window job with Mike to preserve the 509 GB history rather than a full wipe/re-upload.
|
||||
- Kept the backup remediation gated behind Mike's decision (destructive to backup storage; his call on approach).
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **jq parse error dispatching PowerShell with backslashes** (`root\\directory\\ldap` in a heredoc). Resolved by building the command payload with `jq -n --arg cmd "$SCRIPT"` instead of inlining into the JSON.
|
||||
- **`/tmp` path blocked by `block-tmp-path.sh` pre-commit hook** when writing a scratch marker. Resolved by using repo-relative scratch files (`.vwp_cmd`, `.vwp_times`), later cleaned up.
|
||||
- **`cbb help <cmd>` reprints the top-level index** rather than subcommand help; correct form is `<cmd> -?`. `account` needs `-l` to list.
|
||||
- **`cbb list -rp` against the 509 GB B2 bunch timed out** at the 240s server-side reaper with no output (output was buffered to a file only read after all calls finished). Concluded remote CLI enumeration of individual restore points is not viable; deferred to GUI/maintenance-window.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
- **VWP-SERVER timezone:** `Pacific Standard Time` -> `US Mountain Standard Time` (via `Set-TimeZone -Id "US Mountain Standard Time"` over `/rmm`, cmd `61a2e177`). Only change made this session.
|
||||
- No file changes in the repo other than this session log. Scratch files created during diagnostics (`.vwp_cmd`, `.vwp_times`) were removed.
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- None discovered, created, or rotated. RMM auth via existing vault path `infrastructure/gururmm-server.sops.yaml`; MSP360 via `msp-tools/msp360-api.sops.yaml`. No plaintext secrets handled.
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **VWP-SERVER** — Valley Wide Plastering, Main Office. GuruRMM id `d2cc6200-07e5-4444-bb1c-1bcce273d615`. VMware VM (VMware20,1). Domain member (DomainRole 3) of `VWP.US`. TZ now US Mountain Standard Time; W32Time source = VWP-DC1.VWP.US.
|
||||
- **VWP.US domain FSMO:** PDC/RID/Infra = VWP-DC1 (`8eefbba6-28cf-4e3f-8fda-11900e0ac302`); Schema/Naming = VWP_ADSRVR (`bd2f2f86-ea33-4202-828f-b378e459e891`). VWP-DC1 syncs from time.nist.gov.
|
||||
- **VWP-HYPERV1** — Hyper-V host (`bdc3e142-1eb3-4034-8ac2-300f46dbc1d9`), separate from the VMware host VWP-SERVER runs on. Clock healthy.
|
||||
- **MSP360 company:** `Valleywide Plastering` (id `b271f525-2c00-4407-bcc6-a991973a3ecb`).
|
||||
- **Backup destinations (cbb accounts, login `vwp`):** `B2Storage` (id `7faaaf4d-6bdb-4a4e-af52-38cebe33974b`, ~546,465,990,074 bytes / ~509 GB used); `VWP-Backup` (id `4b4f35b7-d83f-4df4-8270-5d23117e4873`, 0 bytes).
|
||||
- **Backup agent (white-labeled):** `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`. Services: `Online Backup Service`, `Online Backup Service Remote Management` (both Running).
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
- `bash .claude/scripts/rmm-search.sh VWP-SERVER` -> single match, Valley Wide Plastering.
|
||||
- Time check: VWP-SERVER Local `2051-07-10 18:45`, Source `Local CMOS Clock`, TZ `Pacific Standard Time`, Type `NT5DS`.
|
||||
- Fleet sample (post): VWP-DC1/VWP_ADSRVR/FRONT64/VWP-HYPERV1 all `2026-07-07`, healthy sync.
|
||||
- `Set-TimeZone -Id "US Mountain Standard Time"` -> Before `Pacific Standard Time`, After `US Mountain Standard Time`.
|
||||
- Post-fix verify: VWP-SERVER `2026-07-07 08:36:58` == VWP-DC1; Source `VWP-DC1.VWP.US`; last successful sync 08:36:30.
|
||||
- `py .claude/skills/msp360/scripts/msp360.py monitoring --company "Valleywide Plastering"` -> VWP-SERVER plan `Files - Remote Backup`, Status Warning, LastStart `2051-07-11T01:00:41`, 0.0B.
|
||||
- `cbb getPlanDetails -id 6238c387-7bf6-47a2-a68b-86d590b5fff7` -> F:\Shares -> B2Storage, block-level, unencrypted, daily 6PM.
|
||||
- `cbb list -a "B2Storage" -rp ...` -> timed out (server-side reaper, 240s).
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **BLOCKED on Mike's decision** (DM'd 2026-07-07 15:56; no reply yet). Remaining work = clear MSP360 error 2525 on VWP-SERVER's `Files - Remote Backup` plan:
|
||||
- Option 1 (preferred, preserves ~509 GB history): in the agent's Backup Storage view, delete the single 2051-07-10 restore point, then run the plan once to create a fresh 2026 baseline; re-anchors the frozen 2051 schedule.
|
||||
- Option 2 (fallback): force a new full / new generation.
|
||||
- Do the delete via the agent GUI or a long-timeout maintenance-window job (CLI over RMM times out), with Mike.
|
||||
- Verify VWP-SERVER's domain secure channel / Kerberos fully recovered after the 25-year skew (was flagged as a follow-up; not yet run).
|
||||
- Other VWP plans showing Warning/0.0B (VWP-FILES, WINFileSvr) were noted but not investigated — separate from the 2525 issue.
|
||||
|
||||
## Reference Information
|
||||
|
||||
- MSP360 KB: kb10096 (RestorePointsTimeMismatch, error 2525) — https://mspbackups.com/backup-agent/2525
|
||||
- Plan: `Files - Remote Backup`, PlanId `6238c387-7bf6-47a2-a68b-86d590b5fff7`.
|
||||
- RMM cmd ids: timezone change `61a2e177`; various diagnostics.
|
||||
- Discord DM to Mike: message_id `1524081568828882985`. Bot alert to #dev-alerts: message_id `1524076358605475982`.
|
||||
- Client slug: `valleywide`.
|
||||
Reference in New Issue
Block a user