dataforth(datasheet): wire DSCA33/45 Hoffman-mined templates (gated; accuracy-data WIP)

Per the 5070 handoff (DSCA33-45-HOFFMAN-RECOVERY): the lost DSCA33/45 specs are
recoverable from Hoffman, not John. Wired the mined dsca33-45-templates.json (56
models) into the renderer:

- datasheet-exact.js: load DSCA3345_TEMPLATES; for family DSCA, the Hoffman-mined
  template takes PRECEDENCE over the stale staged-extraction entry (which shadowed 25
  models with accOut "?"/no accHeader). Emit the verbatim 2-line accHeader for these
  families (Vin (mVAC)/Iin (AAC)/Frequency (Hz), Output (VDC)/(mADC)). Per-model
  `validated` GATE: a DSCA33/45 model renders only after byte-matching its Hoffman
  original; until then it returns null (skipped) so an unverified render can never
  overwrite a pristine live original. DSCA_VALIDATE_MODE env opens the gate for the
  validation harness only. Exposed rendersWithoutSpecs().
- render-datasheet.js: allow a null-specs render for DSCA33/45 (their spec files were
  lost; template-driven) instead of bailing on missing specs.
- derive-dsca-slotmaps.js: DSCA_TPL env to target the 3345 templates; derived 43 slot
  maps into them (22 models need none, 8 DSCA33 still below threshold).
- validate-dsca3345.js (new): renders each model's _srcSerial, fetches the live
  Hoffman original (GET TestReportDataFiles/{serial}, deployed uploader token — no
  vault needed), content-normalized compare; --apply marks validated.

STATUS: gate is CLOSED — 0 models validated, all DSCA33/45 still render null, nothing
published, no risk. Final-Test block + accuracy headers now byte-match the Hoffman
originals for all 56 models; the remaining blocker is accuracy-DATA numeric quirks that
must match to pass the gate:
  - DSCA33 calc column stored in A but displayed in mADC (x1000); measured stored in
    mA (not scaled) — an original-software unit quirk.
  - sign conventions differ per layout (DSCA33 stim/calc/meas unsigned, error signed;
    DSCA45 stim unsigned, calc/meas/error signed).
  - DSCA45 frequency-input stim formatting.
These need per-layout reverse-engineering against the originals (the validation harness
is the oracle). 8 DSCA33 models (DSCA33-02/03/03A/04/04A/05/05A/1642) also lack a slot
map (below threshold). DSCA33-1948 + DSCA45-1746 (24 units) have no Hoffman original.

Cleanups: deleted superseded memory project_dsca33_45_spec_gap; struck the obsolete
"ask John" TODO 2 from the handoff note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.claude/memory/MEMORY.md

@@ -112,7 +112,6 @@
 - [GURU-BEAST-ROG Setup Status](machine_windows_guru_setup_status.md) — Windows workstation fully configured except SSH key deployment to servers.
 
 ## Project
-- [DSCA33/DSCA45 spec gap](project_dsca33_45_spec_gap.md) — Datasheet pipeline: DSCA33/DSCA45 render null because their MAIN specs are missing from all recovered DAT files (DSCA33 nowhere; DSCA45 only in DSCFIN.DAT). Data gap, not a bug; ~8,763 certs blocked. Final-Test renders via slotMaps+stub; need real specs from Dataforth or accuracy-block templating.
 - [CyndyOffice physical HP lockups](cyndyoffice-physical-hp-lockups.md) — RMM "Howard-VM" site agent CyndyOffice is a PHYSICAL HP Pavilion TP01 (not a VM); ~20 hard freezes/6wk = Kernel-Power 41 bugcheck-0, no dump/WHEA = hardware (RAM/PSU/BIOS), SSD healthy. UUID re-enrolls.
 - [Automate memory consolidation/lint (phased)](project_memory_consolidation_automation.md) — Eventually auto-run /memory-dream; lint+additive fixes can automate early, merges/deletes stay human-approved. Engine: .claude/skills/memory-dream/ + .claude/scripts/sync-memory.sh.
 - [Trebesch PST consolidation (staged)](project_trebesch_pst_consolidation.md) — Address-book CSV from 24 PSTs on DESKTOP-QNP3ON5; scripts staged at .claude/tmp/treb-*.ps1, WAITING for Howard's 6pm-MST 2026-06-01 go signal (attended run). See [[reference_trebesch_qnp3on5]].

.claude/memory/project_dsca33_45_spec_gap.md

@@ -1,34 +0,0 @@
----
-name: project_dsca33_45_spec_gap
-description: Why DSCA33/DSCA45 datasheets won't render — missing main specs + special layouts, not a bug
-metadata:
-  type: project
----
-
-In the Dataforth test-datasheet pipeline (deployed `C:\Shares\testdatadb`, AD2 fork `ad2`
-branch), the **DSCA33** and **DSCA45** model families render null (no datasheet) for a
-**data-gap** reason, not a code bug:
-
-- The main spec DAT files we have (`specdata/DSCOUT.DAT`, `DSCMAIN4.DAT`) contain DSCA prefixes
-  30,31,32,34,36,37,38,39,40,41,42,43,47,49 — but **NOT 33 or 45**. DSCA33 appears in **no** DAT
-  file at all; DSCA45 appears only in `DSCFIN.DAT` (the Final-Test layout file, which lacks the
-  `SENTYPE`/`MAXIN`/input-type fields the accuracy block needs). Likely lost in the cryptolocker wipe.
-- `database/render-datasheet.js` bails (`if (!modelSpecs) return null`) before rendering, so these
-  models produce nothing. ~8,763 PASS certs blocked: DSCA33 = 3,350 (35 models), DSCA45 = 5,413 (23 models).
-
-The Fix 2 Final-Test rebuild is NOT the blocker — the per-model templates + slotMaps render the
-Final-Test block correctly for these (proven: DSCA45-01 Final-Test vs golden = 2 trivial diffs with a
-stubbed spec). The real blockers to publishing them correctly:
-1. **Missing main specs** → `render-datasheet.js` won't even call the renderer. For DSCA, specs are
-   only needed for the accuracy-block header + model name now (Final-Test is template/slotMap-driven).
-2. **Special accuracy headers** the sensor-type logic can't produce: DSCA45 = `Frequency (Hz)` /
-   `Output (V)` (frequency-response table); DSCA33 = `Vin (mVAC)` / `Output (VDC)` (AC-RMS module).
-3. **Non-status template rows**: DSCA45 has informational rows like `Zero-Crossing Input` / `TTL Input`
-   with NO PASS in the golden, but the renderer appends PASS to every empty-spec row (like the old
-   loadNote artifact — see [[project_test_datasheet_pipeline]]).
-
-Two paths to fix: (a) obtain the authoritative DSCA33/DSCA45 main spec files from Dataforth/engineering
-(clean, fixes accuracy headers properly — flag to John Lehman), or (b) extend the template approach to
-capture the accuracy header + special-row handling from the staged originals (self-contained, but a
-STAGE-4-sized effort across the special layouts). Final-Test data for the matchable models is already
-ready (slotMaps derived). Related: [[project_pipeline_rebuilt]].

projects/dataforth-dos/datasheet-pipeline/implementation/database/render-datasheet.js

@@ -0,0 +1,31 @@
+/**
+ * In-memory equivalent of what export-datasheets.js writes to
+ * X:\For_Web\<SN>.TXT. Lets upload-to-api.js POST directly to Hoffman's API
+ * from DB state without a filesystem intermediate.
+ *
+ * Returns a string (datasheet text) or null if the record cannot be rendered
+ * (no specs for the model, no raw_data for VASLOG_ENG, etc.).
+ */
+const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
+const { generateExactDatasheet, rendersWithoutSpecs } = require('../templates/datasheet-exact');
+
+let _specMap = null;
+function specs() {
+    if (_specMap === null) _specMap = loadAllSpecs();
+    return _specMap;
+}
+
+function renderContent(record) {
+    if (record.log_type === 'VASLOG_ENG') {
+        return record.raw_data || null;
+    }
+    const modelSpecs = getSpecs(specs(), record.model_number);
+    // DSCA33/45 lost their spec files in the wipe but render from the Hoffman-mined
+    // templates (template-driven names/specs + verbatim accuracy header), so allow a
+    // null-specs render for them. generateExactDatasheet still gates on per-model
+    // validation and returns null until the model is verified.
+    if (!modelSpecs && !rendersWithoutSpecs(record.model_number)) return null;
+    return generateExactDatasheet(record, modelSpecs) || null;
+}
+
+module.exports = { renderContent };

projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js

@@ -18,6 +18,18 @@ try {
     DSCA_TEMPLATES = {};
 }
 
+// DSCA33/DSCA45 templates recovered from the Hoffman API (their main spec files were
+// lost in the wipe). Superset schema: also carries a verbatim 2-line `accHeader`, a
+// `_srcSerial` validation oracle, and a `validated` gate flag. A model renders ONLY
+// after its render byte-matches the Hoffman original (validated:true) — until then it
+// stays null/skipped so the pipeline never overwrites a pristine original.
+let DSCA3345_TEMPLATES = {};
+try {
+    DSCA3345_TEMPLATES = require('../dsca33-45-templates.json');
+} catch (e) {
+    DSCA3345_TEMPLATES = {};
+}
+
 // -------------------------------------------------------------------------
 // DATA LINES: parameter names and units per family
 // -------------------------------------------------------------------------
@@ -552,13 +564,24 @@ function generateExactDatasheet(record, specs) {
         return generateSCMVASDatasheet(record);
     }
 
-    // DSCA STAGE 2: the per-model staged template is the authoritative Final-Test
-    // layout (names + specs + accuracy output label). No template means no staged
-    // original was available -> do not guess the layout; skip this cert.
+    // DSCA: the per-model template is the authoritative Final-Test layout (names +
+    // specs + accuracy label). Source is the staged-original set (dsca-templates) or
+    // the Hoffman-mined set (dsca33-45-templates) for the families whose specs were
+    // lost. No template -> do not guess; skip this cert.
+    const dscaKey = (record.model_number || '').trim();
+    // Hoffman-mined templates take PRECEDENCE: DSCA33/45 were also captured by the
+    // STAGE 1 staged extractor (sometimes with accOut "?" and no accHeader), and that
+    // stale entry must not shadow the authoritative Hoffman-mined one.
     const dscaTpl = (family === 'DSCA')
-        ? (DSCA_TEMPLATES[(record.model_number || '').trim()] || null)
+        ? (DSCA3345_TEMPLATES[dscaKey] || DSCA_TEMPLATES[dscaKey] || null)
         : null;
     if (family === 'DSCA' && !dscaTpl) return null;
+    // Hoffman-mined DSCA33/45 render only once the model is byte-validated against its
+    // Hoffman original — otherwise stay null so an unverified render can't overwrite a
+    // live original. The validation harness sets DSCA_VALIDATE_MODE to render
+    // unvalidated models for the byte-compare; the live service never sets it.
+    if (family === 'DSCA' && DSCA3345_TEMPLATES[dscaKey] && !DSCA3345_TEMPLATES[dscaKey].validated
+        && !process.env.DSCA_VALIDATE_MODE) return null;
 
     const parsed = (family === 'SCM7B')
         ? parse7BRawData(record.raw_data)
@@ -614,6 +637,14 @@ function generateExactDatasheet(record, specs) {
     } else {
     lines.push('                                 ACCURACY TEST');
     lines.push('');
+    if (dscaTpl && Array.isArray(dscaTpl.accHeader) && dscaTpl.accHeader.length >= 2) {
+        // DSCA33/45 (Hoffman-mined): the accuracy header carries model-specific tokens
+        // the sensor-type logic can't synthesize (Vin (mVAC), Iin (AAC), Frequency (Hz),
+        // Output (VDC)/(mADC)). Emit the verbatim 2-line header from the original.
+        lines.push(dscaTpl.accHeader[0]);
+        lines.push(dscaTpl.accHeader[1]);
+        lines.push(TAB5 + '-'.repeat(10) + '    ' + '-'.repeat(11) + '    ' + '-'.repeat(11) + '    ' + '-'.repeat(10) + '    ' + '-'.repeat(8));
+    } else {
     lines.push('                    Calculated       Measured');
 
     // Input column header based on sensor type
@@ -631,6 +662,7 @@ function generateExactDatasheet(record, specs) {
     const accSep = (family === 'DSCA') ? '-' : '=';
     lines.push('    ' + inputHeader + '        ' + accOut + '        ' + accOut + '*      Error (%)       Status');
     lines.push(TAB5 + accSep.repeat(10) + '      ' + accSep.repeat(10) + '      ' + accSep.repeat(10) + '      ' + accSep.repeat(9) + '      ' + accSep.repeat(8));
+    }
 
     for (const point of parsed.accuracy) {
         lines.push(formatAccuracyLine(point, sensorNum, maxIn));
@@ -1024,11 +1056,21 @@ function generateSCMVASDatasheet(record) {
     return lines.join('\r\n');
 }
 
+/**
+ * True if the model renders from a template that does NOT require spec-reader specs
+ * (the Hoffman-mined DSCA33/45 set). Lets render-datasheet.js skip the missing-specs
+ * bail for these. (Still gated on per-model `validated` inside generateExactDatasheet.)
+ */
+function rendersWithoutSpecs(modelNumber) {
+    return !!DSCA3345_TEMPLATES[(modelNumber || '').trim()];
+}
+
 module.exports = {
     generateExactDatasheet,
     generateSCMVASDatasheet,
     extractSCMVASAccuracy,
     parseRawData,
     parse7BRawData,
+    rendersWithoutSpecs,
     DATA_LINES,
 };

projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-dsca-fix2-stage2-3-publish.md

@@ -272,13 +272,12 @@ text ready to paste:
 > STAGE 1 extension. Byte-for-byte DOS cosmetic fidelity (leading rule line + ~1-space input-column
 > spacing) still deferred. Details: `ad2` branch commits + `DSCA-STAGE3-REPORT-2026-06-18.txt`.
 
-**TODO 2 — request from Dataforth/engineering (John):** the authoritative MAIN spec records for the
-**DSCA33-*** and **DSCA45-*** families (DSCMAIN/DSCOUT-style entries with SENTYPE/MAXIN/input-type).
-Missing from every recovered `specdata/*.DAT` (DSCA33 nowhere; DSCA45 only in DSCFIN.DAT) — lost in
-the wipe — blocking ~8,763 cert renders (DSCA33 3,350 / DSCA45 5,413). Drop into `specdata/`, reload,
-and those models render via the slot maps already derived this session. Detail: memory
-`project_dsca33_45_spec_gap`. (Also note the deferred byte-fidelity disclosure to John, per the
-original FIX2-5 handoff, once the whole effort is done.)
+**TODO 2 — SUPERSEDED (do NOT ask John for DSCA33/45 specs).** The 5070 session found the DSCA33/45
+originals survived on the Hoffman API and mined 56/58 models into `dsca33-45-templates.json`. The AD2
+session has since wired them in and validated against Hoffman (see the later update below). Only
+DSCA33-1948 + DSCA45-1746 (24 units) lack an original. Refs: memory
+`project_dsca33_45_resolved_via_hoffman`, doc `DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md`. (Still note
+the deferred byte-fidelity disclosure to John per the original FIX2-5 handoff once the effort is done.)
 
 Nothing else on AD2 is pending — all code is committed to `ad2` and the live Hoffman site is current
 (92 clean models published, 0 errors).

projects/dataforth-dos/tools/derive-dsca-slotmaps.js

@@ -24,7 +24,9 @@ const db = require(DEPLOY + '/database/db');
 const dse = require(DEPLOY + '/templates/datasheet-exact');
 
 const STAGE = 'C:/Shares/test/STAGE';
-const OUT = DEPLOY + '/dsca-templates.json';
+// Default to the staged-original templates; point at the Hoffman-mined DSCA33/45 set
+// via DSCA_TPL when deriving slotMaps for those families.
+const OUT = process.env.DSCA_TPL || (DEPLOY + '/dsca-templates.json');
 
 function walk(d, out) { let it = []; try { it = fs.readdirSync(d, { withFileTypes: true }); } catch { return out; } for (const e of it) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p, out); else if (/\.txt$/i.test(e.name)) out.push(p); } return out; }
 function colSpans(sep) { const cols = []; let m; const re = /=+/g; while ((m = re.exec(sep))) cols.push([m.index, m.index + m[0].length]); return cols; }

projects/dataforth-dos/tools/validate-dsca3345.js

@@ -0,0 +1,97 @@
+// Fix 2 — validate DSCA33/45 Hoffman-mined renders against the live Hoffman originals.
+// For each model: render its _srcSerial (an already-uploaded unit) via the new render
+// path and content-normalized-compare it to GET /api/v1/TestReportDataFiles/{_srcSerial}.
+// --apply marks passing models `validated:true` in dsca33-45-templates.json (the render
+// gate). Read-only otherwise (no DB writes, no Hoffman writes).
+process.env.DSCA_VALIDATE_MODE = '1'; // open the render gate for the compare
+const fs = require('fs');
+const https = require('https');
+const db = require('./database/db');
+const { renderContent } = require('./database/render-datasheet');
+
+const TPL_PATH = './dsca33-45-templates.json';
+const CREDS_PATH = 'C:\\ProgramData\\dataforth-uploader\\credentials.json';
+const APPLY = process.argv.includes('--apply');
+const only = process.argv.slice(2).filter(a => !a.startsWith('--'));
+
+function creds() { return JSON.parse(fs.readFileSync(CREDS_PATH, 'utf8')); }
+function httpReq(method, uri, headers, body) {
+    return new Promise((resolve, reject) => {
+        const u = new URL(uri);
+        const req = https.request({ hostname: u.hostname, port: u.port || 443, path: u.pathname + u.search, method, headers, timeout: 30000 }, res => {
+            let d = ''; res.on('data', c => d += c); res.on('end', () => { try { resolve({ status: res.statusCode, body: JSON.parse(d) }); } catch { resolve({ status: res.statusCode, body: { _raw: d } }); } });
+        });
+        req.on('error', reject); req.on('timeout', () => req.destroy(new Error('timeout')));
+        if (body) req.write(body); req.end();
+    });
+}
+async function getToken() {
+    const c = creds();
+    const form = Object.entries({ grant_type: 'client_credentials', client_id: c.CF_CLIENT_ID, client_secret: c.CF_CLIENT_SECRET, scope: c.CF_SCOPE })
+        .map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`).join('&');
+    const r = await httpReq('POST', c.CF_TOKEN_URL, { 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': Buffer.byteLength(form) }, form);
+    if (r.status !== 200 || !r.body.access_token) throw new Error('token fail ' + r.status);
+    return r.body.access_token;
+}
+async function fetchOriginal(token, serial) {
+    const c = creds();
+    const r = await httpReq('GET', `${c.CF_API_BASE}/api/v1/TestReportDataFiles/${encodeURIComponent(serial)}`, { Authorization: 'Bearer ' + token });
+    if (r.status !== 200) return null;
+    return r.body && r.body.Content ? r.body.Content : null;
+}
+// content-normalize: collapse whitespace per line; DROP rule lines (pure separators —
+// runs of = ~ _ -) and blank lines. Rule lines carry no content and their
+// presence/position is the deferred cosmetic gap (e.g. the leading === letterhead line
+// the originals have and our renders omit), so removing them isolates real content.
+function norm(s) {
+    return s.replace(/\r/g, '').split('\n')
+        .map(l => l.trim())
+        .filter(t => t.length > 0 && !(/^[=~_\- ]+$/.test(t) && /[=~_\-]/.test(t)))
+        .map(t => t.replace(/\s+/g, ' '));
+}
+
+(async () => {
+    const tpl = JSON.parse(fs.readFileSync(TPL_PATH, 'utf8'));
+    const models = (only.length ? only : Object.keys(tpl)).filter(m => tpl[m]);
+    const token = await getToken();
+    const pass = [], fail = [], noOracle = [], noRec = [];
+    for (const m of models) {
+        const sn = tpl[m]._srcSerial;
+        if (!sn) { noOracle.push(m); continue; }
+        const original = await fetchOriginal(token, sn);
+        if (!original) { noOracle.push(m + '(no Hoffman ' + sn + ')'); continue; }
+        const rec = await db.queryOne('SELECT * FROM test_records WHERE serial_number=$1 AND model_number=$2 LIMIT 1', [sn, m]);
+        if (!rec) { noRec.push(m + '(' + sn + ')'); continue; }
+        let rendered; try { rendered = renderContent(rec); } catch (e) { rendered = null; }
+        if (!rendered) { fail.push({ m, sn, reason: 'render null' }); continue; }
+        const a = norm(rendered), b = norm(original);
+        let diff = -1; const mx = Math.max(a.length, b.length);
+        for (let i = 0; i < mx; i++) { if (a[i] !== b[i]) { diff = i; break; } }
+        if (diff === -1) pass.push(m);
+        else fail.push({ m, sn, line: diff, render: a[diff], golden: b[diff] });
+    }
+    console.log('\n=== DSCA33/45 Hoffman validation ===');
+    console.log('PASS (' + pass.length + '): ' + pass.join(', '));
+    console.log('\nFAIL (' + fail.length + '):');
+    for (const f of fail) {
+        if (f.reason) { console.log('  ' + f.m + ' (' + f.sn + '): ' + f.reason); continue; }
+        console.log('  ' + f.m + ' (' + f.sn + ') first diff L' + f.line);
+        console.log('     render: ' + JSON.stringify(f.render));
+        console.log('     golden: ' + JSON.stringify(f.golden));
+    }
+    if (noOracle.length) console.log('\nNO ORACLE: ' + noOracle.join(', '));
+    if (noRec.length) console.log('NO DB REC: ' + noRec.join(', '));
+
+    if (APPLY) {
+        const passSet = new Set(pass);
+        for (const m of Object.keys(tpl)) {
+            if (passSet.has(m)) tpl[m].validated = true;
+            else if (only.length === 0) delete tpl[m].validated; // full run: clear stale
+        }
+        fs.writeFileSync(TPL_PATH, JSON.stringify(tpl));
+        console.log('\n[APPLY] marked validated on ' + pass.length + ' models in ' + TPL_PATH);
+    } else {
+        console.log('\n(dry run — pass --apply to mark validated)');
+    }
+    await db.close();
+})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });