claudetools/projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/IMPLEMENTATION_PLAN.md

SCMVAS/SCMHVAS Datasheet Pipeline Integration — Implementation Plan

Created: 2026-04-12 Basis: Discovery + sample analysis completed 2026-04-11 Target environment: AD2 server, C:\Shares\testdatadb\ Decision: Option C — simple Accuracy-only datasheet, generated directly from DB record, no hvin.dat lookup needed

---

Scope

Two product families need first-class support in the automated datasheet pipeline:

• SCMVAS-Mxxx — obsolete, datasheets end ~2024 plus occasional retests
• SCMHVAS-Mxxxx — replacement line (two test paths):
  • Production half → TESTHV3 software → logs at TS-3R\LOGS\VASLOG\*.DAT (multiline CSV format, same as 5BLOG/8BLOG)
  • Engineering half → plain .txt output pre-rendered at TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\*.txt

Sample datasheet format (the exact output we must produce)

        Dataforth Corporation                        Phone number: (520) 741-1404
        3331 E. Hemisphere Loop                      Fax: (520) 741-0762
        Tucson, AZ 85706 USA                         Email: info@dataforth.com




                                    TEST DATA SHEET
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Date: 10/04/2023
        Model: SCMHVAS-M0200
        SN: 166590-1
                                  FINAL TEST RESULTS
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Parameter           Measured Value      Specification       Status
        ================    ==============      =============       ======
        Accuracy            0.007%              +/- 0.03%           PASS

        _______________________________________________________________________
                                  Check List

        Module Appearance:      __X__           Mounting Screw:           __X__

        Pins Straight:          __X__           Module Header:            __X__

        It is hereby certified that the above product is in conformance with
        all requirements to the extent specified.  This product is not
        authorized or warranted for use in life support devices and/or systems.

        * NIST traceable calibration certificates support Measured Value data.
        Calibration services are available through ANSI/NCSL Z540-1 and
        ISO Guide 25 Certified Metrology Labs.

Each line prefixed with 8 spaces. Tilde separator line is 71 tildes. Specification string is constant +/- 0.03%. Check List uses __X__ markers (pre-filled, not blank like SCM7B).

---

Accuracy extraction rule (production VASLOG .DAT)

The raw_data PASS/FAIL line looks like "PASS-7.005501E-033" or "PASS 5.999184E-033". Format is:

<"PASS"|"FAIL"> <optional space> <signed-float-7digits> E - <2-digit-exponent> <trailing-status-digit>

The trailing single digit (observed: 2 or 3) is a test-status code, NOT part of the float. The captured float is already in percent units (not a fraction).

Extraction regex: /^(PASS|FAIL)\s*(-?\d+\.?\d*E[+-]?\d{2})\d?$/i applied to the stripped contents of the quoted status string.

Formatting: abs value, 3 decimals, trim trailing zeros → display like 0.007%, 0.01%, 0.005%.

Verified against samples:

• "PASS-7.005501E-033" → 7.005501E-03 = 0.007005501% → display 0.007% ✓
• "PASS 4.988443E-033" → 4.988443E-03 = 0.004988443% → display 0.005% ✓
• "PASS 1.524978E-023" → 1.524978E-02 = 0.01524978% → display 0.015%

---

Changes by file

1. parsers/spec-reader.js

Goal: allow SCMVAS/SCMHVAS model numbers to pass the MODNAME validation filter so they land in the spec map (with a synthetic no-specs stub), OR bypass spec lookup entirely for this family.

Approach: bypass entirely. Cleaner — no stub records.

Change getSpecs() to special-case SCMVAS/SCMHVAS and return a well-known "no-specs" sentinel (e.g. { _family: 'SCMVAS', _noSpecs: true }) instead of null. This lets exportNewRecords() proceed to formatter without silently skipping.

Also update the MODNAME prefix regex at line 287 (^(SCM5B|5B|SCM7B|7B|8B|DSCA|DSCT|SCT|BOGUS)) — this line rejects records in the binary DAT parser only, which doesn't affect VASLOG (VASLOG isn't read through that code path). No change needed here — leaving SCMVAS/SCMHVAS out of the binary parser filter is correct since we don't parse hvin.dat.

Diff scope: ~20 lines in getSpecs().

2. templates/datasheet-exact.js

Goal: new family branch emitting the simple Accuracy-only template.

Approach: Add SCMVAS to DATA_LINES (single-entry array: [['Accuracy', '%']]). At the top of generateExactDatasheet(), if the spec stub flags _family === 'SCMVAS', route to a dedicated generateSCMVASDatasheet(record) helper that builds the 35-line template above. This helper does NOT use specs — only record.model_number, record.serial_number, record.test_date, record.overall_result, and record.raw_data.

The helper must:

• Render 8-space left indent on every line
• Date formatted MM/DD/YYYY (matching newer samples) — note: "Corrected HVAS" uses MM-DD-YYYY; use MM/DD/YYYY per the most recent Engineering-Tested samples
• Extract accuracy value via the regex above
• Constants: specification = +/- 0.03%, withstand/Hi-Pot block omitted (SCMVAS has none), checklist uses __X__ markers

Also delete the vestigial startsWith('SCMHVAS') check at existing line 652 (it was inside the DSCT branch and is no longer reachable once SCMVAS gets its own branch).

Diff scope: ~80 new lines (new helper + DATA_LINES entry + router change + one deletion).

3. database/export-datasheets.js

Goal: do not skip SCMVAS/SCMHVAS records due to missing specs.

Approach: after changing getSpecs() to return a stub for this family, the existing if (!specs) continue; logic in both run() and exportNewRecords() just works. No explicit change needed — verify only.

4. database/import.js

Goal: ingest the Engineering-Tested plain .txt files.

Approach: Add a new dedicated import branch for the VASLOG - Engineering Tested subfolder:

• Add a new parser parsers/vaslog-engtxt.js that:
  • Takes a .txt filepath, parses SN from filename (pattern ^(\d+-\d+[A-Za-z]?)(?:\d{14})?\.txt$, capturing the SN segment like 166590-1 or 167601-4 — the optional 14-digit timestamp suffix MMDDYYYYhhmmss is dropped)
  • Reads the file, extracts Model:, Date:, SN:, Accuracy, and Status from the plain-text header rows
  • Returns one record with log_type='VASLOG_ENG', overall_result='PASS' (or derive from the Status field), raw_data=<full file contents>, source_file=<path>
• Register VASLOG_ENG in the LOG_TYPES map with a new vaslog-engtxt parser alias
• Make the importStationLogs() walk recurse into VASLOG/ one level to pick up the VASLOG - Engineering Tested/*.txt subfolder. Cleanest: parameterize the LOG_TYPES entry with subfolder and recursive flags.

For the pass-through copy to X:\For_Web\<SN>.TXT, the Engineering-Tested files already have the correct final format. Two sub-options:

• (4a) Pass-through: exportNewRecords() detects log_type === 'VASLOG_ENG', copies raw_data (the original file contents) verbatim into X:\For_Web\<SN>.TXT, sets forweb_exported_at. Zero risk of format drift.
• (4b) Re-render: treat the VASLOG_ENG record the same as a VASLOG record — run it through the same generateSCMVASDatasheet() helper. Consistent with production path.

Recommendation: (4a) pass-through. Reasons:

• The files already match the target format exactly (verified by comparing samples to Corrected HVAS Files/*.txt)
• Preserves any Engineering-hand-tweaked formatting
• If drift is ever needed, switching to (4b) is a one-line change later

Diff scope: new parsers/vaslog-engtxt.js (~60 lines) + ~30 lines across import.js + ~15 lines in export-datasheets.js for the pass-through branch.

5. C:\Shares\test\scripts\Sync-FromNAS-rsync.ps1

Goal: ensure the Engineering-Tested subfolder is included in the sync.

Approach: Verify that the existing TS-3R\LOGS\ rsync already pulls the full subtree (--recursive). Based on prior session logs, the rsync syncs TS-3R/*, so the subfolder likely rides along. Verify only — no change expected. If rsync uses explicit includes, add VASLOG - Engineering Tested/*** to the include list.

6. Database schema

Goal: no schema changes required.

The test_records table already has log_type, model_number, serial_number, test_date, overall_result, raw_data, source_file, forweb_exported_at. The new VASLOG_ENG log_type is just a new string value in an existing column.

7. Backfill strategy

Production VASLOG .DAT: these are already imported into test_records via the existing multiline parser. After the spec-reader/formatter changes deploy, run:

node database/export-datasheets.js --limit 0

to regenerate datasheets for all PASS SCMVAS/SCMHVAS records where forweb_exported_at IS NULL. This backfills historical SCMVAS/SCMHVAS records that were previously skipped due to "no specs".

Engineering-Tested .txt: run the full import once after the new parser is added. Should pick up all 434 existing files and copy them to X:\For_Web\.

---

Risks / edge cases

1. The VAS-MPT.DAT / HVAS-MPT.DAT "pass-through" models — might need a slightly different treatment (skip Check List? different wording?). Treat same as regular SCMVAS for now; revisit if user reports a mismatch.
2. FAIL records — the PASS regex above also matches FAIL. Verify the Status column in the output shows FAIL and that the existing exportNewRecords logic (which filters overall_result = 'PASS') skips FAIL datasheets by default. No action needed.
3. Filename SN extraction for Engineering-Tested — observed patterns: 166590-1.txt, 166590-110042023104524.txt (trailing timestamp). Regex must correctly split the timestamp. A small number of edge cases exist (e.g. 166594-1010042023090444.txt = SN 166594-10, timestamp 10042023090444) — the SN has variable-length second segment. Safe rule: SN ends at the last -<digits> segment before the optional 14-digit timestamp.
4. Duplicate files — 166593-4.txt (1519 bytes) and 166593-410042023114928.txt (1600 bytes) coexist. Treat the timestamped filename as canonical; untimestamped is a later re-render. Import both but dedupe on (log_type, model_number, serial_number, test_date, test_station) (existing unique constraint already handles this).
5. Date format variance — production VASLOG stores MM-DD-YYYY in raw_data; Engineering-Tested .txt uses MM/DD/YYYY or MM-DD-YYYY depending on vintage. Normalize all date displays to MM/DD/YYYY per the newest Engineering-Tested output.

---

Test plan (Coding Agent must verify)

Before declaring complete:

1. node database/export-datasheets.js --dry-run --serial 179379-1 → should preview a well-formed SCMHVAS datasheet (no "missing specs" skip).
2. node database/export-datasheets.js --serial 166590-1 → compare generated X:\For_Web\166590-1.TXT byte-for-byte against the existing samples/vaslog-engtxt/166590-110042023104524.txt. Expect visual match; char-level drift acceptable only in whitespace.
3. Full incremental import of the VASLOG - Engineering Tested subfolder → verify all 434 .txt files copy to X:\For_Web\.
4. Historical backfill of production VASLOG records → spot-check 5 SCMHVAS and 5 SCMVAS datasheets against any known-good reference in Corrected HVAS Files/.
5. Regression: pick 10 existing SCM5B + 10 DSCA datasheets, regenerate, confirm no format drift vs. their current X:\For_Web\*.TXT.

---

Delegation

Once this plan is approved, hand off to the Coding Agent with:

• This plan as the spec
• All research artifacts under projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/
• Sample output at samples/corrected-hvas/171087-1.txt and samples/vaslog-engtxt/166590-110042023104524.txt as golden references
• Access to AD2 via paramiko (creds from vault path clients/dataforth/ad2.sops.yaml)

After implementation, mandatory Code Review Agent pass before deploying to C:\Shares\testdatadb\.