claudetools/projects/dataforth-dos/DATASHEET-FIX-SPEC-2026-06-17.md

Dataforth Test-Datasheet Pipeline — Fix Spec (hardened)

Date: 2026-06-17 · Host: AD2 (C:\Shares\testdatadb, Node + PostgreSQL 18) · Status: SPEC for review — implementation driven on AD2 Inputs: AD2 diagnosis (DATASHEET-RTD-BUG-DIAGNOSIS, PARSING-FIDELITY-VERDICT, MISSING-UNITS-REPORT, CONFLICT-RULE-FIX-PROPOSAL) + independent multi-AI review (Grok adversarial + Gemini). Owner direction on retest handling (Mike, 2026-06-17).

All defects are in the regeneration/ingestion pipeline that replaced the cryptolocker-destroyed original parser/publisher. Source test data is intact (DB matches staged originals across 11,239 records, 0 parse faults).

---

0. Ground truth (verified live, 2026-06-17)

• test_records: 473,780 rows = 473,780 distinct serial_number → exactly one row per serial.
• Unique constraints: uq_test_records_sn UNIQUE(serial_number) [operative]; redundant UNIQUE(log_type,model_number,serial_number,test_date,test_station).
• Columns incl. raw_data (verbatim .DAT), overall_result, api_uploaded_at, forweb_exported_at, datasheet_exported_at, work_order.
• Deployed database/import.js uses ON CONFLICT (serial_number) with WHERE overall_result='FAIL' OR (EXCLUDED PASS AND EXCLUDED.test_date > test_records.test_date).
• WARNING — repo drift: the repo copy …/implementation/database/import.js is STALE (shows a 5-tuple ON CONFLICT). Edit the DEPLOYED file; reconcile the repo copy after. Verify every file's deployed content before changing it.

0a. CROSS-CUTTING — re-publication discipline (MANDATORY for any fix that changes cert text)

Every fix below that alters rendered output must be published deliberately, not by blanket cache-clear:

1. Diff before re-push. For each candidate serial, render OLD vs NEW and only act where output actually changes. Do not clear api_uploaded_at/forweb_exported_at for unchanged renders.
2. Re-POST semantics. Hoffman bulk API is idempotent — returns Unchanged when content matches, overwrites when it differs (per diagnosis §6). Confirm this holds before bulk re-push; watch for dedup/version behavior.
3. Targeted, staged rollout. Re-publish in bounded batches (start with the Phytec 102), confirm counts, then widen. Log every batch.
4. Rollback. Keep the prior rendered text (or the prior template commit) for every re-published serial so a bad batch can be reverted.
5. Audit framing. A cert's text changing after initial publication is a bug correction — record it (ticket #32441 + an internal change log of affected serial ranges) so it's defensible in an audit.

---

Fix 1 — Defect A: RTD input labeled resistance, not temperature (the audit finding)

File: templates/datasheet-exact.js · Scope: ~24,000 certs (8B35, DSCA34, SCM5B34/35) · Status: fix written, needs scope/ground-truth proof.

Root cause: getSensorNum() returns 7 for RTD sentypes (s.includes('RTD')); two branches on sensorNum===7 emit ' Rin (ohms)' header + unsigned value. Dataforth RTD certs report the input as Temperature (deg C); raw_data stimulus is already deg C.

Approach (AD2 diff): fold sensorNum===7 into the temperature branch (3–6) for header (' Temp. (C)') and value (formatSigned). Leave the i===13 ohm/ohm Lead-R override intact.

Hardening (multi-AI):

• "15 renders changed / 0 non-RTD" proves a branch moved, not correctness. Before deploy: (a) byte-compare the fixed render against the staged original .TXT for a real RTD sample (8B35 incl. SN 179553-13, DSCA34, SCM5B34/35) — require exact match; (b) confirm RTD-detection coverage: count RTD-family rows in the DB (model_number LIKE '%34%'/'%35%' etc.) and confirm the regenerator's s.includes('RTD') actually classifies all of them as 7 (only 15 changing across 184 renders may mean the sample was RTD-thin, or some RTD sentypes aren't matched).

Risk: LOW-MODERATE (localized; main risk is under-detecting which modules are RTD). Deploy: after byte-match + coverage check; then targeted re-push (Phytec 102 first).

---

Fix 2 — Defect B: DSCA Final-Test table wrong / dropped lines

File: templates/datasheet-exact.js (DATA_LINES['DSCA'], buildTSpecs() DSCA branch, accuracy-block titles) · Scope: up to ~78,000 DSCA certs · Status: NEEDS DESIGN — highest structural risk.

Root cause: a single hardcoded DATA_LINES['DSCA'] + single DSCA buildTSpecs branch; real DSCA modules have per-subtype Final-Test layouts → wrong names, garbage specs (< 0 mA, +/- 0 %), rows misaligned, lines dropped (e.g. Output Noise on DSCA38-05). Accuracy block also uses 5B/8B titles (Vout (V) / ====) instead of DSCA's (Output (V|mA) / ----).

Approach (multi-AI consensus — REVISED from AD2's DSCFIN.DAT idea):

• Do NOT reverse-engineer DSCFIN.DAT (legacy DOS config; QB writer has hardcoded overrides outside the config → guaranteed edge-case drift).
• Derive per-subtype templates from the staged original .TXT (the actual correct customer certs are ground truth): group staged DSCA .TXT by subtype, extract each subtype's Final-Test parameter name/unit/spec list and accuracy titles directly.
• Key subtype selection on model_number (prefix) + SENTYPE / output-signal type. Build an explicit subtype→layout map.
• Fix the DSCA accuracy block titles/separators (Output (V|mA), ----).

Validation (hard gate): generate ALL DSCA certs and byte-for-byte diff vs the staged originals across every subtype; zero-delta required. Any subtype with no staged original → flag, do not guess.

Risk: HIGH (largest population + wrong numeric labels/limits). The longer pole; do after 1/3/4. Deploy: only after zero-delta validation per subtype; re-publish in batches with diff-gating.

---

Fix 3 — Retest handling: latest test supersedes (OWNER DIRECTION + hardening)

File: deployed database/import.js (ON CONFLICT (serial_number) WHERE clause) · Scope: ~311 stuck units now + all future retests · Status: design owner-set, implementation hardened.

Owner rule (Mike): one row per serial; a new test on the SAME unit (same model / "everything else checks out") supersedes anything prior — latest test wins. A reused serial on a DIFFERENT product is NOT the same unit — recognize the collision, don't blindly overwrite.

Why the current rule fails: strictly-greater date + date-only granularity → same-day reruns can't replace (~311 stuck on a non-final run).

Approach (hardened — both AIs refute pure scan-order as the recency signal):

• Conflict on serial_number. Update when the incoming row is the SAME unit and is genuinely newer:
  • EXCLUDED.model_number = test_records.model_number (same unit) AND EXCLUDED.raw_data IS DISTINCT FROM test_records.raw_data (real change; avoids re-push churn) AND incoming is at least as new.
  • Recency must not rely on import scan order alone. Use EXCLUDED.test_date >= test_records.test_date, and break same-date ties with a monotonic signal captured at parse time — source .DAT mtime or an ingest sequence number (add a column, e.g. ingest_seq / source_mtime). Last-by-(date, tiebreaker) wins.
• Collision handling (different model_number, same serial): do NOT overwrite. Route to a test_records_quarantine table (or a flagged status) + alert. These are the reused generic serials (1-1, 1-2) — genuinely different units.
• Keep the FAIL → PASS override.

Validation: ingest the owner's 4-retest sample IN REVERSE chronological order; final DB state MUST be the mathematically newest run. Re-run tools/validate-parsing.js; same-day violations → ~0. Confirm the 311 settle on the latest run.

Risk: HIGH if scan-order is trusted (a future bulk re-import could overwrite newer with older across the whole DB). MITIGATED by the date+tiebreaker rule. Deploy: after the reverse-order sample passes; then re-import + diff-gated re-push of the 311.

---

Fix 4 — Importer drops letter-prefixed encoded serials

File: parsers/multiline.js (serial/date regex) · Scope: ~9,510 records / 840 serials / 141 models · Status: needs design (PK-boundary mutation).

Root cause: line.match(/^"(\d+-\d+[A-Za-z]?)","(\d{2}-\d{2}-\d{4})"$/) — \d+-\d+ requires leading digits, so DOS 8.3-encoded serials (10243-1 → A243-1; first two digits → letter, prefix = charCodeAt(0)-55) never match → whole record silently dropped.

Approach (hardened — keep the literal, both AIs):

• Widen regex to allow an optional leading letter: /^"([A-Za-z]?\d+-\d+[A-Za-z]?)","(\d{2}-\d{2}-\d{4})"$/.
• Store BOTH: add raw_serial_number (the literal file bytes, e.g. A243-1) and keep serial_number = decoded numeric (10243-1). UNIQUE stays on serial_number. Preserves a perfect audit trail.
• Decode only when the captured serial matches ^[A-Za-z]\d.

Pre-flight (MANDATORY before any import): run the parser over the ~9,510 dropped records read-only → emit CSV raw_serial, decoded_serial, model_number. Search decoded serials for collisions against the existing 473,780 rows. For each collision decide policy (a decoded A243-1 colliding with a genuine 10243-1 of a different model = a real conflict — quarantine, don't merge two physical units). Only proceed once collisions are enumerated and a policy set.

Risk: MODERATE (transforms the uniqueness key + customer lookup id). Deploy: after the collision CSV is clean/resolved; the re-import re-exercises the upsert path → run under the Fix-3 rule, diff-gated re-push.

---

Fix 5 — Backfill 379 cryptolocker-era units from staged originals

Scope: 379 units (Oct 2025–Jan 2026, 3 stations), no surviving .DAT; staged .TXT exist · Status: operational.

Key fact: the staged .TXT were produced by the ORIGINAL (pre-crypto) renderer → already correct (no Defect A/B).

Approach (both AIs — publish directly, don't round-trip):

• Add legacy_cert_text column. Insert the 379 with raw_data = NULL, legacy_cert_text = the staged .TXT content.
• Publisher serves legacy_cert_text when raw_data IS NULL (bypasses regeneration).
• Do NOT reverse .TXT→raw_data→re-render (two translation-loss points; guarantees drift).

Validation: cross-reference the 379 serials against the ERP / work-order system to confirm they are valid shipped units before exposing via the API. Spot-check rendered vs staged text.

Consistency note: these rows are permanently "original-renderer" output, divergent from what the fixed template would emit. Acceptable (and safer) given no raw source; document the class.

Risk: LOW (bounded 379, immutable text) — but zero tolerance for wrong text (no raw fallback). Deploy: after ERP cross-check.

---

Risk ranking (combined) & recommended order

Rank	Fix	Why
1 (tie)	Fix 3 retest	scan-order recency could corrupt DB-wide on any future re-import (Gemini #1)
1 (tie)	Fix 2 DSCA	largest scope + wrong numeric labels/limits; design-heavy (Grok #1)
3	Fix 4 serials	mutates the uniqueness/lookup key; merge risk
4	Fix 1 RTD	localized; risk is under-scoping RTD detection
5	Fix 5 backfill	small, immutable, but no raw fallback

Suggested execution order (lowest-risk customer win first, hardest last):

1. Fix 1 (RTD label) — after byte-match + scope check → clears the audit + corrects the Phytec 102 (deploy tonight candidate).
2. Re-publish Phytec 102 (diff-gated).
3. Fix 4 (serial decode) — after clean collision CSV.
4. Fix 3 (retest rule) — after reverse-order sample passes.
5. Fix 2 (DSCA rebuild) — after per-subtype zero-delta validation.
6. Fix 5 (backfill 379) — after ERP cross-check.

Open items needing an owner decision

• Fix 3: add a real tie-breaker column (ingest_seq/source_mtime) vs accept test_date >= + scan-order? (recommend the column.)
• Fix 3/4 collisions: quarantine table + alert vs reject-and-log? (recommend quarantine table.)
• Fix 4: add raw_serial_number column (recommended) — schema change.
• Fix 5: add legacy_cert_text column + publisher branch (recommended) — schema + publisher change.
• Tonight: scope = Fix 1 + Phytec re-push only (smallest safe win); the rest staged after.