From 4b30e36a0e12835c8f2077e218caeb2042d2ab92 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Wed, 17 Jun 2026 15:12:59 -0700 Subject: [PATCH] dataforth(datasheet): correct Cause 2 of missing units - cryptolocker incident, not import timing Per Mike: import runs every 15 min, so routine timing isn't the cause. The 379 absent units are confined to 2025-10..2026-01 (stop after Jan 2026) on TS-4L/4R/1R - fingerprint of a one-time overwrite during the incident/recovery (fresh DOS logs overwrote accumulated appended server-side logs for ~2 weeks). One-time, not recurring; backfill from the surviving staged .TXT. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...ISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md | 27 ++++++++++++------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/projects/dataforth-dos/MISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md b/projects/dataforth-dos/MISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md index d5cf157c..585dc209 100644 --- a/projects/dataforth-dos/MISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md +++ b/projects/dataforth-dos/MISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md @@ -14,7 +14,7 @@ I cross-checked **all 11,921 staged datasheet files** (the `.TXT` the test stati | Cause | Units | Recoverable? | |---|---:|---| | **1. Encoded / non-standard serial numbers the importer skips** | **229** | **Yes** — the data exists, the importer just doesn't read it | -| **2. Source log (`.DAT`) was overwritten before import** | **379** | Only from a backup, or from the staged `.TXT` itself | +| **2. One-time log loss during the cryptolocker incident/recovery** | **379** | Yes — from the staged `.TXT`, which still exists on disk | The first cause is the important one: it is a **software limitation we can fix**, and its true reach is far larger than these 608 — see "Full scope" below. @@ -62,25 +62,32 @@ So this single serial-format limitation is keeping on the order of **~9,500 test --- -## Cause 2 — Source log overwritten before import (379 units) +## Cause 2 — One-time log loss during the cryptolocker incident/recovery (379 units) -These have ordinary numeric serials (no encoding issue), but their test data is **no longer in any log file** we import from. The per-model `.DAT` log is reused for later production runs, and the older records get overwritten. +These have ordinary numeric serials (no encoding issue), but their raw test data is **no longer in any log file** we import from. **This is not an ongoing gap** — the import runs every 15 minutes and everything since has come through cleanly. -**Confirmed example:** units `177097-1 ... 177097-16` (model DSCA33-05, tested 10-17-2025, TS-1R) appear in **no** log file anywhere under the test share. Their model's log (`DSCLOG\33-05.DAT`) now contains a **different** work order (`178644-*`, tested 02-26-2026). The 10-17-2025 results were overwritten; only the staged `.TXT` datasheet survived. +**What happened:** the DOS stations append all of a given model's results into a single shared per-model `.DAT` file. During the crypto incident, stations were failing to sync, and in the recovery the appended-to-one-filename behavior wasn't yet understood — so for ~2 weeks, freshly-created logs coming off the DOS machines overwrote the accumulated server-side logs, wiping the earlier history those files held. -**Recovery options for these 379:** -1. **From the staged `.TXT` itself** — the rendered datasheet still exists on disk for these units; they can be imported directly from the `.TXT` (store the existing sheet as-is) rather than re-derived from the `.DAT`. This is the most practical recovery and would cover most of the 379. -2. **From a pre-overwrite backup** of the `.DAT` logs, if one exists. (The `Recovery-TEST` backup area referenced by the importer does not exist on this server; if Engineering keeps log backups elsewhere, those could be imported.) +**Evidence (date/station fingerprint of a one-time overwrite):** the 379 affected units are confined to **2025-10 → 2026-01 and stop cold after January 2026**, concentrated on three stations: -If neither is pursued, these units' results remain only as the staged `.TXT` files and will not appear in the database or on the website. +| Test month | Units | | Station | Units | +|---|---:|---|---|---:| +| 2025-10 | 95 | | TS-4L | 185 | +| 2025-11 | 66 | | TS-4R | 171 | +| 2025-12 | 210 | | TS-1R | 23 | +| 2026-01 | 8 | | | | + +**Confirmed example:** units `177097-1 ... 177097-16` (model DSCA33-05, tested 10-17-2025, TS-1R) appear in **no** log file anywhere under the test share. Their model's log (`DSCLOG\33-05.DAT`) now holds a later work order (`178644-*`, tested 02-26-2026) — the older appended history was overwritten; only the staged `.TXT` datasheet survived. + +**Recovery:** the rendered datasheet `.TXT` still exists on disk for these units, so they can be backfilled directly from the `.TXT` (store the existing sheet as-is). The raw `.DAT` history is gone (the `Recovery-TEST` backup area the importer references does not exist on this server). Backfilling from the staged `.TXT` is the practical path and recovers all 379. --- ## Recommended actions 1. **Fix the importer's serial handling** (Cause 1) — recovers 229 staged units and ~9,500 total dropped records across 141 models. Highest value, single code change. Proposal to follow for review. -2. **Backfill the 379 overwritten units (Cause 2) from their staged `.TXT` files** — recovers the datasheets that still exist on disk. Lower-risk than chasing log backups. -3. **Prevent recurrence:** the per-model log overwrite is the underlying reason Cause 2 data is lost. If retaining every run matters, the logs should be archived before reuse (or the import scheduled frequently enough that records are captured before a log is overwritten). +2. **Backfill the 379 incident-era units (Cause 2) from their staged `.TXT` files** — recovers the datasheets that still exist on disk. +3. **Recurrence:** Cause 2 was a one-time incident artifact (the 15-minute import has captured everything since January 2026), so no ongoing process change is required. Worth confirming the current sync no longer overwrites accumulated server-side logs with fresh DOS-side copies. ---