17 KiB
Test-Datasheet Bug — End-to-End Trace & Diagnosis
Date: 2026-06-17 Host: AD2 (192.168.0.6) — testdatadb generator + PostgreSQL 18 Author: Mike Swanson / AZ Computer Guru Status: DIAGNOSIS ONLY — no code or DB changes made Reported by: John Lehman / Peter Iliya (Dataforth) — customer Wellbore Integrity (Joseph Swinehart) cal-cert audit on 8B35 4-wire RTD certs
0. TL;DR
There are two independent defects, both in the datasheet renderer (templates/datasheet-exact.js). Ingestion (.DAT parsing), the database contents, and the spec files are all correct — the raw test data in the DB holds the right values; the renderer mislabels and mis-maps them.
| # | Defect | Symptom on cert | Scope | Severity |
|---|---|---|---|---|
| A | RTD input column rendered as resistance instead of temperature | Header reads Rin (ohms), should read Temp. (C); positive input values lost their leading + |
~24,000 RTD certs (8B35, DSCA34, SCM5B34/35, and any RTD variant) | HIGH — this is the audit finding |
| B | Entire DSCA Final-Test parameter list is wrong | Wrong parameter names, garbage specs (< 0 mA, +/- 0 %), values aligned to the wrong rows, output column mislabeled (Vout (V) vs Output (mA)), lines missing/added |
up to 78,343 DSCA certs (all DSCLOG) | HIGH |
Defect A is a small, surgical fix. Defect B requires rebuilding the DSCA template against the legacy spec (DSCFIN.DAT) and is a larger effort.
Root cause is confirmed against the original ground-truth files (the DOS-station-generated staged .TXT), not assumptions.
1. The Original File IS the Ground Truth — and We Found It
Mike's key point was correct: the rendered datasheet exists as a file BEFORE it ever reaches the database. That original file is produced by the DOS test station itself and is the source of truth. Our DB-based regeneration is what introduces the error.
Where the original files physically live
The DOS station's QuickBASIC ATE program writes a fully-rendered .TXT datasheet to C:\STAGE\ on the station, then CTONWTXT.BAT uploads it to the NAS STAGE share. It is mirrored to AD2:
ORIGINAL (ground truth) staged datasheets:
AD2: C:\Shares\test\STAGE\<TS-station>\<encoded-SN>.TXT
NAS: /data/test/STAGE/<TS-station>/<encoded-SN>.TXT (\\192.168.0.9\test\STAGE)
Filenames use the 8.3 hex-prefix serial encoding (first two digits → letter, 55 + n):
179553-13 → 17→H → H9553-13.TXT; 180224-7 → 18→I → I0224-7.TXT.
Confirmed files for this investigation:
| Module | SN | Original staged file (ground truth) | Source .DAT |
|---|---|---|---|
| 8B35-04 (4-wire RTD) | 179553-13 | C:\Shares\test\STAGE\TS-4L\H9553-13.TXT |
C:\Shares\test\TS-4L\LOGS\8BLOG\35-04.DAT |
| DSCA38-05 (full bridge) | 180224-7 | C:\Shares\test\STAGE\TS-11R\I0224-7.TXT |
C:\Shares\test\TS-11R\LOGS\DSCLOG\38-05.DAT |
| DSCA34-05C (3-wire RTD) | 180007-8 | C:\Shares\test\STAGE\TS-4R\I0007-8.TXT |
C:\Shares\test\TS-4R\LOGS\DSCLOG\34-05C.DAT |
Note re Peter:
179553-13cannot be found on X: / DFWDS / For_Web because the website copy is regenerated from the DB at upload time — it is never written to disk. The on-disk ground truth is the STAGE copy above, not For_Web. (For_Web on AD2 holds only ~7,500 legacy files and does not contain this SN.)
2. End-to-End Pipeline (upstream emphasis)
[1] DOS Test Station (TS-xx, DOS 6.22, QuickBASIC ATE)
- Runs the unit test, measures everything.
- Writes TWO artifacts:
(a) C:\ATE\...\<model>.DAT raw CSV-ish multi-line test log (-> network LOGS)
(b) C:\STAGE\<encSN>.TXT FULLY RENDERED datasheet <-- GROUND TRUTH
- The station ALREADY knows the sensor type, so it prints the correct
input column ("Temp. (C)" for RTD) and the correct Final-Test parameter
list for that exact model. This is the format we must reproduce.
[2] Boot upload (CTONW.BAT / CTONWTXT.BAT)
- .DAT -> \\NAS\test\<TS>\LOGS\<logtype>\<model>.DAT
- .TXT -> \\NAS\test\STAGE\<TS>\<encSN>.TXT
[3] NAS <-> AD2 sync (Sync-FromNAS.ps1, 15 min)
- Mirrors to C:\Shares\test\... on AD2.
[4] testdatadb ingest (THIS host) ----- the original .TXT is IGNORED here -----
- import.js scans .DAT files (NOT the staged .TXT).
- parsers/multiline.js parses the .DAT into a record:
{ log_type, model_number, serial_number, test_date, test_station,
overall_result, raw_data (the verbatim .DAT block), source_file }
- INSERT ... ON CONFLICT(serial_number) into PostgreSQL test_records.
raw_data = the exact .DAT text block (this is faithful & correct).
[5] Render + upload (the bug lives here)
- render-datasheet.js -> templates/datasheet-exact.js regenerates the
datasheet text FROM raw_data + spec files, in memory.
- upload-to-api.js POSTs {SerialNumber, Content} to Hoffman bulk API.
- Hoffman serves it on the public product page.
The pivotal architectural fact: step [4] throws away the already-correct rendered .TXT and keeps only the raw .DAT block (raw_data). Step [5] then re-renders from scratch. Every rendering defect is introduced in step [5]; the upstream data is fine.
2.1 What the .DAT / raw_data actually contains (8B35-04, SN 179553-13)
"8B35-04 " <- model
-1.461694,-1.218078E-02,-.014174,-3.986431E-02,"PASS" <- accuracy pt 1: stim,calc,meas,err,status
151.5394,1.262828,1.26273,-1.966953E-03,"PASS" <- pt 2
303.6477,2.530397,2.531,1.204967E-02,"PASS" <- pt 3
448.7633,3.739694,3.7414,.0341177,"PASS" <- pt 4
598.0475,4.983729,4.9824,-2.658844E-02,"PASS" <- pt 5
"0","0",0 <- step-response placeholder
"PASS 28.424741","PASS","PASS 252.21681","PASS","PASS" <- final-test STATUS groups (5 per line)
"PASS","","PASS","PASS","PASS"
"PASS","PASS 3.478871E-023","PASS 3.986431E-023","PASS","PASS 26.328911"
"PASS","PASS","PASS 40.429370","PASS","PASS 140.50"
The first column of each accuracy point is the stimulus. For SN 179553-13 it is -1.46, 151.5, 303.6, 448.8, 598.0 — clearly temperatures in °C (model MAXIN = 600 °C), not ohms. The spec record confirms SENTYPE = "P1RTD4W", MAXIN = 600. So the DB has the right numbers and the right sensor type. Nothing upstream is wrong.
3. Diffs — Original (correct) vs DB-Generated (wrong)
3.1 8B35-04 RTD (SN 179553-13) — Defect A only
ACCURACY block header + first/last rows:
ORIGINAL (H9553-13.TXT, ground truth) GENERATED (current testdatadb)
----------------------------------- ------------------------------
Temp. (C) Vout (V) ... Rin (ohms) Vout (V) ... <-- WRONG LABEL
-1.46 ... -1.46 ... (same)
+151.54 ... 151.54 ... <-- lost '+'
+598.05 ... 598.05 ... <-- lost '+'
- Header:
Temp. (C)→ rendered asRin (ohms). This is the audit discrepancy. - Values: numerically identical (correct temperatures). Only difference: positive values lose the leading
+because the resistance formatter omits the sign. - Final Test Results: byte-for-byte IDENTICAL to the original (7 lines: Supply Current Nom, Exc. Current #1, Linearity, Accuracy, Supply Sensitivity, Frequency Response, Output Noise). No missing lines for 8B35. The 8B/5B Final-Test rendering is correct.
So for the Wellbore Integrity audit, the only defect on the 8B35 cert is the input column header label (and the cosmetic + sign). The measured data is right.
3.2 DSCA38-05 full-bridge (SN 180224-7) — Defect B
This is not a label tweak — the whole Final Test table is wrong:
ORIGINAL (I0224-7.TXT) GENERATED (current)
Supply Current 23.8 mA < 30 mA Supply Current, Nom 23.8 mA < 0 mA <- spec garbage
Supply Curr. w/ EXC Load 53.8 mA < 80 mA Supply Current @ Max Load 53.8 mA < 0 mA
Excitation Voltage 10.000 V 10+/-.003V Linearity, 50mA Load 10.000 % +/- 0 % <- wrong name+unit
Exc. Load Regulation -6 ppm/mA ... Accuracy, 50mA Load 6 % +/- 0 % <- wrong row
Output Reg. w/ EXC Load 0.00 % +/- .05 % Positive Current Limit 0.0 mA < 0 mA
Excitation Current Limit 54 mA < 65 mA Negative Current Limit 54 mA > 0 mA
Linearity 0.002 % +/- .02 % Overrange 0.002 % > 0 %
Accuracy -0.008 % +/- .05 % Power Supply Sensitivity 0.008 %/% +/-.0006
Power Supply Sens. 0.0000 %/% +/-.0006 Frequency Response 0.0000 dB 25+/-5 dB <- value=0
Frequency Response 25.0 dB 25+/-5 dB Compliance 25.0 % +/- 0 % <- 25.0 is freq!
Output Noise 1205 uVrms <=2000 (Output Noise line dropped entirely)
Also in the ACCURACY block: original column titles are Output (V) and separator dashes ----------; the generator emits Vout (V) and ==========. The input header happens to read Vin (mV) in both (bridge module), so the bridge input label is OK, but everything below it is misaligned.
3.3 DSCA34-05C 3-wire RTD (SN 180007-8) — Defect A and B together
ORIGINAL (I0007-8.TXT) GENERATED (current)
Temp. (C) Output (mA) ... Rin (ohms) Vout (V) ... <-- A: label + wrong out-unit
Supply Current 50.7 mA < 65 mA Supply Current, Nom 50.7 mA < 0 mA
Exc. Current @ -f.s. 264.0 uA 261 uA Linearity, 0mA Load 264.0 % +/- 0 % <-- 264.0 is uA, not %
Exc. Current @ +f.s. 281.0 uA 278 uA Accuracy, 0mA Load 281.0 % +/- 0 %
Linearity 0.017 % +/-.03% Overrange 0.017 % > 0 %
Accuracy -0.034 % +/-.05% Power Supply Sens. 0.034 %/% +/-.0005
... (9 real params) ... (wrong names, garbage specs, lines dropped)
The excitation currents (264.0 uA, 281.0 uA) get printed under the labels "Linearity, 0mA Load" / "Accuracy, 0mA Load" as percentages — visibly nonsensical.
4. Root-Cause Localization
Tested against the original files, the defect is (c) the renderer — templates/datasheet-exact.js. Ingestion/parsing (a) and the DB data (b) are correct.
Defect A — RTD treated as resistance
// getSensorNum(): RTD maps to 7
if (s.includes('RTD')) return 7; // line ~150
// Accuracy input-column header (generateExactDatasheet):
} else if (sensorNum === 7) {
inputHeader = ' Rin (ohms)'; // line ~564 <-- WRONG for Dataforth RTD
}
// Accuracy value formatting (formatAccuracyLine):
} else if (sensorNum === 7) {
stimStr = point.stim.toFixed(2).padStart(8); // line ~447 <-- resistance format, no sign
}
Dataforth RTD modules always express the input as temperature on the datasheet (the RTD curve converts resistance→°C; the .DAT already stores °C). sensorNum === 7 is reached only by RTD sentypes (P1RTD3W, P1RTD4W, NIRTD3W, …). There is currently no module for which Rin (ohms) is correct — true resistance/potentiometer inputs are not routed to 7 (they fall through to the voltage default). So fixing the 7 branch is safe and will not over-correct.
Defect B — DSCA parameter list mismatch
DATA_LINES['DSCA'] is a single hardcoded list (Supply Current Nom / @ Max Load / Linearity 0mA / Accuracy 0mA / Linearity 5mA / … / Compliance / Accuracy @ 5 ohm). Real DSCA modules use different Final-Test layouts per subtype (bridge/excitation modules list Excitation Voltage / Exc. Load Reg. / Output Reg.; RTD/TC list Exc. Current @ ±f.s.; etc.). The hardcoded list does not match, so:
raw_dataSTATUS groups are mapped positionally onto the wrong parameter names;buildTSpecs()for DSCA reads spec fields (ILIMIT,PERCOVER,COMPLIANCE,ACCURACY1/2/3,LINEAR1/2/3) that are zero/absent for these modules → specs print as< 0,+/- 0;- the skip rule
if (status.length <= 4) continuedrops every bare-PASSslot, but because the list is misaligned the wrong lines drop and the survivors land on wrong rows; - the ACCURACY block also uses the 5B/8B titles (
Vout (V)/==========) instead of DSCA's (Output (V|mA)/----------).
The "missing Final Test lines" complaint is a symptom of Defect B (DSCA misalignment + skip rule), not a separate bug. The 8B/5B skip rule is correct — the legacy station also prints only tested parameters (verified: 8B35 matches exactly).
Clarification for the thread: DSCA38 is a bridge/strain-gauge module (FBRIDGE/HBRIDGE), not an RTD. The DSCA RTD analog is DSCA34. So DSCA38's problem is Defect B; DSCA34's problem is A+B. If the audit specifically concerns RTD resistance-vs-temperature, the DSCA part to verify with the customer is DSCA34, while DSCA38 demonstrates the broader DSCA table breakage.
5. Correct Output & Proposed Fix
5.1 Correct RTD output (Defect A)
Per the originals, RTD modules must render:
- Input column header:
Temp. (C)(same column/format as thermocouples,sensorNum3–6). - Input values: the stimulus value straight from
raw_data(already °C — no conversion needed), signed format (+598.05,-1.46).
Surgical change in templates/datasheet-exact.js (fold RTD into the temperature path):
// (1) Header — replace the sensorNum===7 branch:
if ((sensorNum >= 3 && sensorNum <= 6) || sensorNum === 7) {
inputHeader = ' Temp. (C)';
} else if (sensorNum === 2 || sensorNum === 9) {
inputHeader = ' Iin (mA)';
} else {
inputHeader = (maxIn != null && maxIn < 1) ? ' Vin (mV)' : ' Vin (V)';
}
// (2) Value format — in formatAccuracyLine, treat 7 like 3–6:
if ((sensorNum >= 3 && sensorNum <= 6) || sensorNum === 7) {
stimStr = formatSigned(point.stim, 2, 8); // temperature, signed
} else { ... }
Leave the existing i===13 && sensorNum===7 → 'ohm/ohm' unit override (Lead-R-Effect) as-is; it is a separate, correct detail. Verify exact leading-space alignment against a known-good thermocouple original before pushing (the header column should byte-match; this is the same polish already in progress in generated-v2-*.TXT).
This fix corrects header + values for all RTD modules (8B35, DSCA34, SCM5B34/35, etc.) at once. Do not add any resistance→temperature math — the data is already temperature.
5.2 DSCA template (Defect B)
Not a one-liner. The fix is to drive the DSCA Final-Test parameter list (and ACCURACY column titles/units) per module subtype, matching the legacy QuickBASIC DSC writer. The legacy parameter selection lives in specdata\DSCFIN.DAT (DSC final-test definitions) alongside DSCMAIN4.DAT/DSCOUT.DAT. Recommended approach:
- Reverse the DSCFIN.DAT layout (or read the QB DSC datasheet source) to get the per-subtype parameter name/unit/spec list.
- Replace the single
DATA_LINES['DSCA']+ DSCA branch ofbuildTSpecs()with subtype-aware selection keyed on SENTYPE / output-signal type. - Fix the DSCA ACCURACY block to use
Output (V|mA)(perOUTSIGTYPE) and dash separators. - Validate byte-for-byte against staged originals across DSCA subtypes (bridge, RTD, TC, current-out, voltage-out).
Until B is fixed, all DSCA (DSCLOG) website datasheets should be treated as unreliable for Final-Test content.
6. Impact (records currently on the website)
| Group | On-web count | Defect |
|---|---|---|
| 8B35* | 5,476 | A |
| DSCA34* (RTD) | 3,573 | A + B |
| SCM5B34/35* (RTD) | 14,887 | A |
| All DSCA (DSCLOG) | 78,343 | B (RTD subset also A) |
| Total on website | 464,671 | — |
RTD-label exposure (Defect A) ≈ 24,000 certs; DSCA table exposure (Defect B) ≈ 78,000 certs.
After fixes are reviewed and deployed, affected records can be re-pushed by clearing api_uploaded_at for the affected models and letting the upload path re-render (RE-PUSH is idempotent; Hoffman returns Unchanged when content matches).
7. How to Reproduce / Verify (read-only)
# Render current generator output for a SN and compare to the staged original:
cd C:\Shares\testdatadb
node -e "const db=require('./database/db');const {renderContent}=require('./database/render-datasheet');(async()=>{const r=await db.queryOne('SELECT * FROM test_records WHERE serial_number=$1',['179553-13']);if(r.test_date&&r.test_date.toISOString)r.test_date=r.test_date.toISOString().slice(0,10);console.log(renderContent(r));await db.close();})()"
# Ground truth:
type C:\Shares\test\STAGE\TS-4L\H9553-13.TXT
8. Files Referenced
- Generator:
C:\Shares\testdatadb\templates\datasheet-exact.js(repo:projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js) - Render glue:
C:\Shares\testdatadb\database\render-datasheet.js - Specs:
C:\Shares\testdatadb\parsers\spec-reader.js,C:\Shares\testdatadb\specdata\*.DAT(incl.DSCFIN.DAT) - Ingest:
C:\Shares\testdatadb\database\import.js,C:\Shares\testdatadb\parsers\multiline.js - Ground-truth originals:
C:\Shares\test\STAGE\<TS>\<encSN>.TXT