claudetools/projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md

# Dataforth Datasheet Fixes — Handoff to the AD2 Local Session (2026-06-18)

**For:** the Claude session running locally on AD2 (`C:\Shares\testdatadb`).
**Why you:** the remote operator's SSH/VPN to AD2 keeps flapping; you run on the box, so you have stable access. Pick up the remaining work below. **Ref Syncro ticket #32441.**

You are working on the **deployed** pipeline at `C:\Shares\testdatadb` (Node + PostgreSQL 18). The repo copies under `projects/dataforth-dos/datasheet-pipeline/implementation/` are **STALE** — edit the DEPLOYED files, reconcile the repo after.

---

## What's already DONE (live on the website)
- **Fix 1 (RTD label):** `datasheet-exact.js` folds RTD (sensorNum 7) into the temperature path (`Temp. (C)`, signed). Deployed; **all ~24k RTD certs re-pushed** (8B35/DSCA34/SCM5B34/SCM5B35). Audit finding resolved.
- **Fix 4 (encoded serials):** `parsers/multiline.js` decode rule `^[A-Z]\d{3,}-` + `raw_serial_number`; `import.js` cross-model guard. Recovered 603 units, 3 quarantined (`test_records_quarantine`). Schema added: `raw_serial_number`, `test_records_quarantine`.
- **Fix 3 (retest latest-wins):** `import.js` conflict rule = same-model AND (FAIL OR newer-by-date OR (same-date AND `raw_data IS DISTINCT` AND higher `ingest_seq`)). Parser stamps `ingest_seq = source-mtime*1e6 + line`. Schema: `ingest_seq`. Settled + validated (old-vs-new render diff vs the 16:30 dump: no corruption).
- Save-states on AD2: `datasheet-exact.js.bak-2026-06-17-1646`, `multiline.js`/`import.js` `.bak-2026-06-17-1713` (Fix4) and `.bak-2026-06-17-1726` (Fix3).

## DISCIPLINE (follow exactly — these are customer calibration certs)
1. **Backup first:** a `pg_dump` exists at `C:\Shares\testdatadb\_backups\testdatadb-2026-06-17-1630.dump`. **Take a FRESH `pg_dump` + a VSS shadow before any schema change or re-import.** (superuser `postgres` / `Paper123!@#`; app `testdatadb_app` / `DfTestDB2026!` — both vaulted at `clients/dataforth/testdatadb-postgres`.)
2. **Per-file save-state:** copy any file to `<file>.bak-YYYY-MM-DD-HHMM` before editing.
3. **Atomic, asserted patches:** prepare all string-replacements in memory, assert each matches exactly once, then write; `require()` load-check after.
4. **Validate before WRITE; validate before PUBLISH.** Never re-push to Hoffman until the per-subtype byte-validation passes.
5. **Re-publication:** the `testdatadb` service caches the template — **restart it** (`Restart-Service testdatadb`) after changing `datasheet-exact.js` so the live service uses the new template; do re-pushes from a **fresh node process** via `uploadBySerialNumbers(serials)`. Hoffman is idempotent (returns Unchanged/Updated/Created). Re-push in batches; the DSCA fleet is ~78k certs.

---

## FIX 2 — DSCA Final-Test rebuild (Defect B). STAGE 1 done; do STAGE 2–3.

**Root cause:** `datasheet-exact.js` has ONE hardcoded `DATA_LINES['DSCA']` + a single DSCA branch in `buildTSpecs()`. Real DSCA modules have **26 distinct Final-Test layouts** (per subtype), so names/specs/row-alignment are wrong and lines drop (e.g. Output Noise on DSCA38-05). The ACCURACY block also uses 5B/8B titles (`Vout (V)` + `====`) instead of DSCA's (`Output (V|mA)` + `----`).

**STAGE 1 (DONE):** `C:\Shares\testdatadb\dsca-templates.json` already exists — **126 DSCA models**, each `{ "accOut": "Output (V)"|"Output (mA)", "rows": [ {"name": "...", "spec": "..."}, ... ] }`, extracted byte-accurately from the staged originals (the extractor used the `===` separator under the Final-Test header for exact column spans). Verified DSCA38-05 now has `Output Noise | <= 2000 uVrms`. **This JSON is the authoritative template source — use it, don't reverse DSCFIN.DAT.**

**STAGE 2 (do this):** wire it into `templates/datasheet-exact.js`.
1. Read the current DSCA render path FIRST: `parseRawData()` (how the Final-Test STATUS groups are parsed from `raw_data` — they're 5-per-line groups like `"PASS 28.42","PASS","PASS 252.2",...`), `buildTSpecs()` DSCA branch, the Final-Test render loop in `generateExactDatasheet()`, and the accuracy header line (~line 567, currently `inputHeader + '        Vout (V)  Vout (V)*  Error (%)  Status'` with `==========` separators).
2. Load `dsca-templates.json` at module top. For `family === 'DSCA'`:
   - Replace the param **names** and **specs** source: instead of `DATA_LINES['DSCA']` + `buildTSpecs` DSCA specs, use `DSCA_TEMPLATES[record.model_number].rows` (each row gives the name + spec text directly — no spec-file lookup needed for DSCA).
   - Map the parsed `raw_data` STATUS groups **positionally** onto the template rows for the **measured value + PASS/FAIL status**. The staged template row order == the raw_data group order (same DOS source). **Reconcile the existing skip rule** (`if (status.length <= 4) continue`): rows like `240VAC Withstand`/`Hi-Pot` have NO measured value and an empty spec — they must still render (blank measured + blank spec + PASS), so don't drop them; align by position, show the value when the group carries one.
   - Fix the ACCURACY block for DSCA: use `DSCA_TEMPLATES[model].accOut` (`Output (V)` or `Output (mA)`) in place of `Vout (V)`, and `----------` dash separators in place of `==========`. (The input column is already correct post-Fix-1.)
   - If a model is missing from `dsca-templates.json` (no staged original): **do not guess** — skip/flag it.
3. Save-state + atomic patch + load-check + restart the service.

**STAGE 3 — validate per subtype (the gate):** render ALL DSCA certs and **byte-compare vs the staged originals, grouped by the 26 layouts**; require zero content-delta per layout before any re-push. Use a content-normalized compare (strip the leading `===` letterhead-separator line and trailing whitespace — those are the known, deferred cosmetic gaps; focus on the Final-Test + ACCURACY content matching). Report match/mismatch per layout; investigate any mismatch before publishing. Then re-push DSCA in batches via `uploadBySerialNumbers`, diff-gated, watching Updated/Unchanged.

**Scope:** ~78k DSCA certs across DSCLOG. **Do NOT re-push until STAGE 3 is clean per subtype.**

---

## FIX 5 — Backfill 379 cryptolocker-era units (operational)
379 units (Oct 2025–Jan 2026, 3 stations) have no surviving `.DAT` but their staged `.TXT` still exist — and those were rendered by the ORIGINAL (correct) software, so **publish them directly, don't round-trip through raw_data**.
1. Add column `legacy_cert_text TEXT` to `test_records` (backup first).
2. Identify the 379 (staged `.TXT` present, serial absent from DB — the `MISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md` on the `ad2` branch has the method/list; Cause 2 set).
3. Insert them: `raw_data = NULL`, `legacy_cert_text` = the staged `.TXT` content, plus model/serial/test_date/station/overall_result parsed from the `.TXT`.
4. Modify `render-datasheet.renderContent()` to return `legacy_cert_text` when `raw_data IS NULL`.
5. **Validate:** cross-reference the 379 serials against the ERP/work-order system to confirm valid shipped units before publishing; spot-check rendered vs staged. Then publish.

---

## Minor cleanup
- **1 Hoffman push error** during the Fix-3 publish — find which serial (check `notify` logs / re-run that batch with per-record fallback) and resolve.
- **~420 "skipped" units** (no spec entry / model not registered in Hoffman — `upload-to-api.js` `UNREGISTERED_MODELS` + null-render skips). They're safely in the DB; decide per model whether to register in Hoffman or add spec coverage.

## Report back
Commit your work to the `ad2` branch and update ticket #32441 (hidden internal notes for engineering detail; the customer-facing thread is John Lehman). The remote operator will see your commits on sync. **Byte-for-byte DOS fidelity** (a leading `===` line + ~1-space input-column spacing, on ALL families) is intentionally deferred — there's a ticket note to disclose it to John when the whole effort is done.