azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

dataforth(datasheet): Fix 2 — per-model slot maps resolve ambiguous DSCA layouts

Browse Source 
Some DSCA subtypes' raw_data STATUS groups carry more (or fewer) value-bearing
entries than the template's spec-bearing rows (the test program measures slots the
printed sheet omits, e.g. DSCA49's 5mA load pair), so the in-order zip misaligned
values and those models were skipped by the count-guard.

New tool derive-dsca-slotmaps.js derives a per-model slotMap (absolute statusEntries
index per spec-bearing row) by greedily matching a staged original's printed values
to the DB raw_data STATUS entries (same fround formatting), then picking the
candidate map that validates against the most units. Models are grouped by identical
row-name signature and one map is derived per group from all sibling units — this
disambiguates duplicate values (e.g. a unit where 5mA != 50mA linearity forces the
correct slot; DSCA49-04 alone has only 2 staged units that can't, but its siblings'
25 units do). Stored as `slotMap` in dsca-templates.json.

Renderer: consults slotMap only when the sequential zip fails (value count !=
spec-row count), so the 88 already-clean models keep their path (no regression) and
ambiguous ones pull the right value via the map.

STAGE 3 re-validation: FINAL-TEST CLEAN 88 -> 92; 134 more certs now render
(null 450 -> 316); matches 2278 -> 2412. Same 6 retest-vintage dirty models, no
new mismatches. DSCA49 family + DSCA40-03 group now clean and validated.

Still blocked (separate gap, NOT layout ambiguity): DSCA45-* and most DSCA33-*
render null because they have NO spec-reader entries (render-datasheet bails before
rendering). Their slotMaps are derived and ready; they need spec coverage. One DSCA33
group (DSCA33-02/03/03A/04/05/1948) did not reach the slotMap validation threshold
(best 19/35 units) and stays skipped pending more/cleaner staged samples.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mike Swanson
2026-06-18 07:50:06 -07:00
parent 5a259ec641
commit 0579d05b66
5 changed files with 194 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt
View File

@@ -7,14 +7,14 @@ Corpus: 2806 staged DSCA originals across 126 models.
SUMMARY
models with staged originals: 126
models FINAL-TEST CLEAN (>=1 compared, 0 mismatch): 88
models FINAL-TEST CLEAN (>=1 compared, 0 mismatch): 92
models with FINAL-TEST mismatches: 6
certs compared: 2316
Final-Test match: 2278
certs compared: 2450
Final-Test match: 2412
Final-Test mismatch: 38
(certs with accuracy-section diffs: 1337 — informational)
(certs with accuracy-section diffs: 1369 — informational)
staged serials not in DB: 40
in DB but not rendered (skipped/null): 450
in DB but not rendered (skipped/null): 316
MODELS WITH FINAL-TEST CONTENT MISMATCHES (investigate before re-push):
DSCA38-05 compared=129 ftMatch=97 ftMismatch=32
@@ -51,8 +51,8 @@ MODELS WITH FINAL-TEST CONTENT MISMATCHES (investigate before re-push):
render: "Linearity 0.017 % +/- .05 % PASS"
golden: "Linearity 0.012 % +/- .05 % PASS"
FINAL-TEST CLEAN MODELS (88):
DSCA30-01, DSCA30-02, DSCA30-03, DSCA30-06, DSCA30-07, DSCA30-08, DSCA30-08C, DSCA30-09, DSCA30-09C, DSCA30-1944, DSCA30-1945, DSCA30-1946, DSCA31-02, DSCA31-03, DSCA31-06, DSCA31-07, DSCA31-11, DSCA31-12, DSCA31-1273, DSCA31-12C, DSCA31-13, DSCA31-13C, DSCA31-15, DSCA31-1918, DSCA32-01, DSCA32-01C, DSCA32-01E, DSCA34-01, DSCA34-02C, DSCA34-04, DSCA34-04C, DSCA34-05, DSCA34-05C, DSCA34-1858, DSCA36-01, DSCA36-02, DSCA36-03, DSCA36-04, DSCA36-04C, DSCA36-1949, DSCA38-02, DSCA38-03, DSCA38-07, DSCA38-08C, DSCA38-09, DSCA38-09E, DSCA38-12C, DSCA38-12E, DSCA38-1468, DSCA38-1544, DSCA38-15C, DSCA38-16, DSCA38-16C, DSCA38-18C, DSCA38-19, DSCA39-01, DSCA39-02, DSCA39-07, DSCA40-03, DSCA40-05, DSCA40-05C, DSCA40-06, DSCA40-1951, DSCA40-1952, DSCA41-01, DSCA41-02, DSCA41-03, DSCA41-05C, DSCA41-06, DSCA41-09, DSCA41-13, DSCA41-14, DSCA41-15, DSCA41-15E, DSCA42-01, DSCA42-01C, DSCA42-02, DSCA43-10, DSCA43-20E, DSCA47E-08C, DSCA47J-01C, DSCA47J-03, DSCA47K-05, DSCA47K-13, DSCA47K-14, DSCA47N-15, DSCA47T-06, DSCA47T-1928
FINAL-TEST CLEAN MODELS (92):
DSCA30-01, DSCA30-02, DSCA30-03, DSCA30-06, DSCA30-07, DSCA30-08, DSCA30-08C, DSCA30-09, DSCA30-09C, DSCA30-1944, DSCA30-1945, DSCA30-1946, DSCA31-02, DSCA31-03, DSCA31-06, DSCA31-07, DSCA31-11, DSCA31-12, DSCA31-1273, DSCA31-12C, DSCA31-13, DSCA31-13C, DSCA31-15, DSCA31-1918, DSCA32-01, DSCA32-01C, DSCA32-01E, DSCA34-01, DSCA34-02C, DSCA34-04, DSCA34-04C, DSCA34-05, DSCA34-05C, DSCA34-1858, DSCA36-01, DSCA36-02, DSCA36-03, DSCA36-04, DSCA36-04C, DSCA36-1949, DSCA38-02, DSCA38-03, DSCA38-07, DSCA38-08C, DSCA38-09, DSCA38-09E, DSCA38-12C, DSCA38-12E, DSCA38-1468, DSCA38-1544, DSCA38-15C, DSCA38-16, DSCA38-16C, DSCA38-18C, DSCA38-19, DSCA39-01, DSCA39-02, DSCA39-07, DSCA40-03, DSCA40-05, DSCA40-05C, DSCA40-06, DSCA40-1951, DSCA40-1952, DSCA41-01, DSCA41-02, DSCA41-03, DSCA41-05C, DSCA41-06, DSCA41-09, DSCA41-13, DSCA41-14, DSCA41-15, DSCA41-15E, DSCA42-01, DSCA42-01C, DSCA42-02, DSCA43-10, DSCA43-20E, DSCA47E-08C, DSCA47J-01C, DSCA47J-03, DSCA47K-05, DSCA47K-13, DSCA47K-14, DSCA47N-15, DSCA47T-06, DSCA47T-1928, DSCA49-04, DSCA49-05, DSCA49-1601, DSCA49-1895
MODELS WITH NO COMPARABLE CERT (no staged serial in DB, or all skipped/null):
DSCA31-1947(null), DSCA33-01(null), DSCA33-01A(null), DSCA33-02(null), DSCA33-02C(null), DSCA33-03(null), DSCA33-03A(null), DSCA33-03C(null), DSCA33-04(null), DSCA33-04C(null), DSCA33-05(null), DSCA33-05C(null), DSCA33-07C(null), DSCA33-1917(null), DSCA33-1919(null), DSCA33-1948(null), DSCA45-01(null), DSCA45-01C(null), DSCA45-02(null), DSCA45-03(null), DSCA45-03C(null), DSCA45-04(null), DSCA45-04C(null), DSCA45-05C(null), DSCA45-06(null), DSCA45-07(null), DSCA45-08(null), DSCA47N-15C(null), DSCA49-04(null), DSCA49-05(null), DSCA49-1601(null), DSCA49-1895(null)
DSCA31-1947(null), DSCA33-01(null), DSCA33-01A(null), DSCA33-02(null), DSCA33-02C(null), DSCA33-03(null), DSCA33-03A(null), DSCA33-03C(null), DSCA33-04(null), DSCA33-04C(null), DSCA33-05(null), DSCA33-05C(null), DSCA33-07C(null), DSCA33-1917(null), DSCA33-1919(null), DSCA33-1948(null), DSCA45-01(null), DSCA45-01C(null), DSCA45-02(null), DSCA45-03(null), DSCA45-03C(null), DSCA45-04(null), DSCA45-04C(null), DSCA45-05C(null), DSCA45-06(null), DSCA45-07(null), DSCA45-08(null), DSCA47N-15C(null)

2
projects/dataforth-dos/datasheet-pipeline/implementation/dsca-templates.json
View File

File diff suppressed because one or more lines are too long

21
projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js
View File

@@ -656,12 +656,15 @@ function generateExactDatasheet(record, specs) {
if (m) measurements.push(m);
}
const specRowCount = dscaTpl.rows.filter(r => (r.spec || '').trim()).length;
// The simple positional zip is only sound when there is exactly one measured
// value per spec-bearing row. When counts differ, this subtype uses a slot
// layout the zip can't resolve (e.g. raw_data records load points the template
// omits), so emitting would misalign values onto wrong rows. Skip and flag for
// STAGE 3 per-subtype mapping rather than publish wrong data ("do not guess").
if (measurements.length !== specRowCount) return null;
// The simple positional zip is sound only when there is exactly one measured
// value per spec-bearing row. When counts differ, this subtype measures slots
// the template omits (e.g. an extra load pair); use the per-model slotMap
// (absolute statusEntries index per spec-bearing row, derived from the staged
// originals) to pull the right value. With no usable slotMap, skip rather than
// misalign ("do not guess").
const useSlot = (measurements.length !== specRowCount)
&& Array.isArray(dscaTpl.slotMap) && dscaTpl.slotMap.length === specRowCount;
if (measurements.length !== specRowCount && !useSlot) return null;
let h1 = setCol('', 12, 'Parameter');
h1 = setCol(h1, 31, 'Measured Value*');
@@ -674,13 +677,15 @@ function generateExactDatasheet(record, specs) {
h2 = setCol(h2, 69, '='.repeat(6));
lines.push(h2);
let mi = 0;
let mi = 0, si = 0;
for (const row of dscaTpl.rows) {
const spec = (row.spec || '').trim();
let line = setCol('', 4, row.name);
if (spec) {
const su = splitSpecUnit(spec);
const m = measurements[mi++];
const m = useSlot
? formatMeasuredExact(parsed.statusEntries[dscaTpl.slotMap[si++]])
: measurements[mi++];
if (m) {
// measured value right-justified ending at col 38, unit at col 40
const v = String(m.valStr);

2
projects/dataforth-dos/dsca-clean-models.json
View File

@@ -1 +1 @@
["DSCA30-01","DSCA30-02","DSCA30-03","DSCA30-06","DSCA30-07","DSCA30-08","DSCA30-08C","DSCA30-09","DSCA30-09C","DSCA30-1944","DSCA30-1945","DSCA30-1946","DSCA31-02","DSCA31-03","DSCA31-06","DSCA31-07","DSCA31-11","DSCA31-12","DSCA31-1273","DSCA31-12C","DSCA31-13","DSCA31-13C","DSCA31-15","DSCA31-1918","DSCA32-01","DSCA32-01C","DSCA32-01E","DSCA34-01","DSCA34-02C","DSCA34-04","DSCA34-04C","DSCA34-05","DSCA34-05C","DSCA34-1858","DSCA36-01","DSCA36-02","DSCA36-03","DSCA36-04","DSCA36-04C","DSCA36-1949","DSCA38-02","DSCA38-03","DSCA38-07","DSCA38-08C","DSCA38-09","DSCA38-09E","DSCA38-12C","DSCA38-12E","DSCA38-1468","DSCA38-1544","DSCA38-15C","DSCA38-16","DSCA38-16C","DSCA38-18C","DSCA38-19","DSCA39-01","DSCA39-02","DSCA39-07","DSCA40-03","DSCA40-05","DSCA40-05C","DSCA40-06","DSCA40-1951","DSCA40-1952","DSCA41-01","DSCA41-02","DSCA41-03","DSCA41-05C","DSCA41-06","DSCA41-09","DSCA41-13","DSCA41-14","DSCA41-15","DSCA41-15E","DSCA42-01","DSCA42-01C","DSCA42-02","DSCA43-10","DSCA43-20E","DSCA47E-08C","DSCA47J-01C","DSCA47J-03","DSCA47K-05","DSCA47K-13","DSCA47K-14","DSCA47N-15","DSCA47T-06","DSCA47T-1928"]
["DSCA30-01","DSCA30-02","DSCA30-03","DSCA30-06","DSCA30-07","DSCA30-08","DSCA30-08C","DSCA30-09","DSCA30-09C","DSCA30-1944","DSCA30-1945","DSCA30-1946","DSCA31-02","DSCA31-03","DSCA31-06","DSCA31-07","DSCA31-11","DSCA31-12","DSCA31-1273","DSCA31-12C","DSCA31-13","DSCA31-13C","DSCA31-15","DSCA31-1918","DSCA32-01","DSCA32-01C","DSCA32-01E","DSCA34-01","DSCA34-02C","DSCA34-04","DSCA34-04C","DSCA34-05","DSCA34-05C","DSCA34-1858","DSCA36-01","DSCA36-02","DSCA36-03","DSCA36-04","DSCA36-04C","DSCA36-1949","DSCA38-02","DSCA38-03","DSCA38-07","DSCA38-08C","DSCA38-09","DSCA38-09E","DSCA38-12C","DSCA38-12E","DSCA38-1468","DSCA38-1544","DSCA38-15C","DSCA38-16","DSCA38-16C","DSCA38-18C","DSCA38-19","DSCA39-01","DSCA39-02","DSCA39-07","DSCA40-03","DSCA40-05","DSCA40-05C","DSCA40-06","DSCA40-1951","DSCA40-1952","DSCA41-01","DSCA41-02","DSCA41-03","DSCA41-05C","DSCA41-06","DSCA41-09","DSCA41-13","DSCA41-14","DSCA41-15","DSCA41-15E","DSCA42-01","DSCA42-01C","DSCA42-02","DSCA43-10","DSCA43-20E","DSCA47E-08C","DSCA47J-01C","DSCA47J-03","DSCA47K-05","DSCA47K-13","DSCA47K-14","DSCA47N-15","DSCA47T-06","DSCA47T-1928","DSCA49-04","DSCA49-05","DSCA49-1601","DSCA49-1895"]

171
projects/dataforth-dos/tools/derive-dsca-slotmaps.js Normal file
View File

@@ -0,0 +1,171 @@
// Fix 2 — derive per-model DSCA slot maps for the ambiguous layouts.
//
// Problem: for some DSCA subtypes the raw_data STATUS groups carry MORE (or fewer)
// value-bearing entries than the template's spec-bearing rows — the test program
// measures slots (e.g. an extra 5mA load pair) that the printed sheet omits. A
// simple in-order zip then misaligns values onto rows, so those models are skipped
// by the renderer's count-guard.
//
// Fix: for each such model, pair a staged original with its DB raw_data and greedily
// match each printed measured value to a STATUS entry (same fround formatting the
// renderer uses), recording the ABSOLUTE statusEntries index per spec-bearing row.
// That ordered subsequence is the slotMap; the renderer reads statusEntries[slotMap[s]]
// for the s-th spec-bearing row. Stored in dsca-templates.json as `slotMap`.
//
// Data-vintage safe: a staged unit that is a retest (printed values != DB record)
// won't match cleanly; we try multiple staged units per model and accept the first
// that matches ALL spec rows. Read-only except the templates JSON it rewrites.
const fs = require('fs'), path = require('path');
// This tool is specific to the deployed pipeline (it reads the staged originals and
// rewrites the deployed templates JSON), so it requires the deployed modules by
// absolute path and is runnable from anywhere.
const DEPLOY = 'C:/Shares/testdatadb';
const db = require(DEPLOY + '/database/db');
const dse = require(DEPLOY + '/templates/datasheet-exact');
const STAGE = 'C:/Shares/test/STAGE';
const OUT = DEPLOY + '/dsca-templates.json';
function walk(d, out) { let it = []; try { it = fs.readdirSync(d, { withFileTypes: true }); } catch { return out; } for (const e of it) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p, out); else if (/\.txt$/i.test(e.name)) out.push(p); } return out; }
function colSpans(sep) { const cols = []; let m; const re = /=+/g; while ((m = re.exec(sep))) cols.push([m.index, m.index + m[0].length]); return cols; }
// fround + toFixed(decimal-code), value start at index 4 — must match formatMeasuredExact.
function fmt(statusStr) {
if (!statusStr || statusStr.length <= 4) return null;
const decimalDigit = statusStr[statusStr.length - 1];
const valueStr = statusStr.substring(4, statusStr.length - 1).trim();
const parsed = parseFloat(valueStr);
if (isNaN(parsed)) return valueStr;
const v = Math.fround(parsed);
const d = parseInt(decimalDigit, 10);
return isNaN(d) ? v.toFixed(1) : v.toFixed(d);
}
// Parse the staged Final-Test section -> ordered list of spec-bearing printed values.
function stagedPrintedValues(t) {
const L = t.replace(/\r\n/g, '\n').split('\n');
const fi = L.findIndex(l => /FINAL TEST RESULTS/.test(l)); if (fi < 0) return null;
let hi = -1; for (let i = fi + 1; i < L.length; i++) { if (/Parameter\s+Measured/.test(L[i])) { hi = i; break; } } if (hi < 0) return null;
const sep = L[hi + 1] || ''; const cols = colSpans(sep); if (cols.length < 4) return null;
const [pc, mc, sc, stc] = cols;
const vals = [];
for (let i = hi + 2; i < L.length; i++) {
const l = L[i];
if (/Check List|^\s*_{5,}/.test(l)) break;
if (!l.trim()) continue;
if (/^\s*Standard output load/i.test(l)) continue;
const measured = (l.slice(mc[0], sc[0]) || '').trim();
// spec column ONLY (cols 48..69) — not the trailing Status column (PASS),
// so empty-spec rows (240VAC Withstand / Hi-Pot) are correctly skipped.
const spec = (l.slice(sc[0], stc[0]) || '').trim();
if (!spec) continue;
const v = measured.split(/\s+/)[0]; // strip trailing unit
if (v === '') return null; // a spec row with no printed value -> can't use this unit
vals.push(v);
}
return vals;
}
// Greedy in-order match printed values -> absolute statusEntries indices.
function greedyMap(printed, statusEntries) {
const map = []; let j = 0;
for (const pv of printed) {
let found = -1;
for (let k = j; k < statusEntries.length; k++) {
if (fmt(statusEntries[k]) === pv) { found = k; break; }
}
if (found < 0) return null;
map.push(found); j = found + 1;
}
return map;
}
// Does this slotMap reproduce a unit's printed values exactly?
function mapMatches(map, printed, statusEntries) {
if (map.length !== printed.length) return false;
for (let s = 0; s < map.length; s++) {
if (fmt(statusEntries[map[s]]) !== printed[s]) return false;
}
return true;
}
(async () => {
const onlyModels = process.argv.slice(2).filter(a => !a.startsWith('--'));
const tpl = JSON.parse(fs.readFileSync(OUT, 'utf8'));
// index staged DSCA files by model
const files = walk(STAGE, []);
const byModel = {};
for (const f of files) {
let t; try { t = fs.readFileSync(f, 'utf8'); } catch { continue; }
const model = (t.match(/^\s*Model:\s*(\S+)/m) || [])[1] || '';
if (!/^DSCA/i.test(model)) continue;
const sn = (t.match(/^\s*SN:\s*(\S+)/m) || [])[1] || '';
if (!sn) continue;
(byModel[model.trim()] = byModel[model.trim()] || []).push({ f, sn: sn.trim(), text: t });
}
let derived = 0, failed = [];
const UNITS_PER_MODEL = 8, MAX_SAMPLES = 48; // bounds DB lookups per signature group
// Seeds = the ambiguous models to solve (args), or all models if none given.
const seeds = new Set(onlyModels.length ? onlyModels.filter(m => tpl[m]) : Object.keys(tpl));
// Group ALL models by identical row-name signature. Same printed layout => same
// canonical-slot mapping, so one slotMap serves the whole group; pooling units
// across the group lets siblings disambiguate duplicate values (e.g. a unit where
// 5mA != 50mA linearity forces the correct slot). Only process groups that contain
// a seed, so a targeted run touches only the relevant families.
const sigOf = (m) => tpl[m].rows.map(r => r.name).join('|');
const groups = {};
for (const model of Object.keys(tpl)) { (groups[sigOf(model)] = groups[sigOf(model)] || []).push(model); }
for (const models of Object.values(groups)) {
if (![...models].some(m => seeds.has(m))) continue;
const specRowCount = tpl[models[0]].rows.filter(r => (r.spec || '').trim()).length;
const samples = [];
for (const model of models) {
if (samples.length >= MAX_SAMPLES) break;
const units = (byModel[model] || []).slice(0, UNITS_PER_MODEL);
for (const u of units) {
if (samples.length >= MAX_SAMPLES) break;
let row = await db.queryOne('SELECT raw_data FROM test_records WHERE serial_number=$1 AND model_number=$2 LIMIT 1', [u.sn, model]);
if (!row) row = await db.queryOne('SELECT raw_data FROM test_records WHERE raw_serial_number=$1 AND model_number=$2 LIMIT 1', [u.sn, model]);
if (!row || !row.raw_data) continue;
const printed = stagedPrintedValues(u.text);
if (!printed || printed.length !== specRowCount) continue;
const p = dse.parseRawData(row.raw_data, 'DSCA');
if (!p) continue;
samples.push({ printed, status: p.statusEntries });
}
}
if (!samples.length) continue;
// Candidate slotMaps = greedy map from each sample; the TRUE map reproduces the
// most units (a duplicate-confused map fails where the duplicated slots differ;
// retest-vintage units fail every map and are ignored).
const cands = new Map();
for (const s of samples) { const m = greedyMap(s.printed, s.status); if (m) cands.set(m.join(','), m); }
let best = null, bestScore = -1;
for (const m of cands.values()) {
let score = 0;
for (const s of samples) if (mapMatches(m, s.printed, s.status)) score++;
if (score > bestScore) { bestScore = score; best = m; }
}
const ratio = bestScore / samples.length;
const accept = best && (samples.length === 1 ? bestScore === 1 : (ratio >= 0.6 && bestScore >= 2));
if (accept) {
// Apply to every model in the group. The renderer only consults slotMap when
// the sequential value-zip fails (value count != spec-row count), so clean
// models keep their current path and only ambiguous ones use the map.
for (const model of models) tpl[model].slotMap = best;
derived += models.length;
console.log(' [' + models.length + '] ' + models[0].padEnd(13) + ' slotMap=[' + best.join(',') + '] matched ' + bestScore + '/' + samples.length + ' units' + (models.length > 1 ? ' (+' + (models.length - 1) + ' siblings)' : ''));
} else if (best) {
failed.push(models.join('/') + '(best ' + bestScore + '/' + samples.length + ')');
}
}
fs.writeFileSync(OUT, JSON.stringify(tpl));
console.log('\nderived slotMaps: ' + derived);
if (failed.length) console.log('no clean match (left as-is): ' + failed.join(', '));
await db.close();
})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });