claudetools/projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/IMPLEMENTATION_PLAN.md

# SCMVAS/SCMHVAS Datasheet Pipeline Integration — Implementation Plan

**Created:** 2026-04-12
**Basis:** Discovery + sample analysis completed 2026-04-11
**Target environment:** AD2 server, `C:\Shares\testdatadb\`
**Decision:** Option C — simple Accuracy-only datasheet, generated directly from DB record, no `hvin.dat` lookup needed

---

## Scope

Two product families need first-class support in the automated datasheet pipeline:

- **SCMVAS-Mxxx** — obsolete, datasheets end ~2024 plus occasional retests
- **SCMHVAS-Mxxxx** — replacement line (two test paths):
  - Production half → TESTHV3 software → logs at `TS-3R\LOGS\VASLOG\*.DAT` (multiline CSV format, same as 5BLOG/8BLOG)
  - Engineering half → plain `.txt` output pre-rendered at `TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\*.txt`

### Sample datasheet format (the exact output we must produce)

```
        Dataforth Corporation                        Phone number: (520) 741-1404
        3331 E. Hemisphere Loop                      Fax: (520) 741-0762
        Tucson, AZ 85706 USA                         Email: info@dataforth.com




                                    TEST DATA SHEET
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Date: 10/04/2023
        Model: SCMHVAS-M0200
        SN: 166590-1
                                  FINAL TEST RESULTS
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Parameter           Measured Value      Specification       Status
        ================    ==============      =============       ======
        Accuracy            0.007%              +/- 0.03%           PASS

        _______________________________________________________________________
                                  Check List

        Module Appearance:      __X__           Mounting Screw:           __X__

        Pins Straight:          __X__           Module Header:            __X__

        It is hereby certified that the above product is in conformance with
        all requirements to the extent specified.  This product is not
        authorized or warranted for use in life support devices and/or systems.

        * NIST traceable calibration certificates support Measured Value data.
        Calibration services are available through ANSI/NCSL Z540-1 and
        ISO Guide 25 Certified Metrology Labs.
```

Each line prefixed with 8 spaces. Tilde separator line is 71 tildes. Specification string is constant `+/- 0.03%`. Check List uses `__X__` markers (pre-filled, not blank like SCM7B).

---

## Accuracy extraction rule (production VASLOG .DAT)

The raw_data PASS/FAIL line looks like `"PASS-7.005501E-033"` or `"PASS 5.999184E-033"`. Format is:

```
<"PASS"|"FAIL"> <optional space> <signed-float-7digits> E - <2-digit-exponent> <trailing-status-digit>
```

The trailing single digit (observed: `2` or `3`) is a test-status code, NOT part of the float. The captured float is **already in percent units** (not a fraction).

**Extraction regex:** `/^(PASS|FAIL)\s*(-?\d+\.?\d*E[+-]?\d{2})\d?$/i` applied to the stripped contents of the quoted status string.

**Formatting:** abs value, 3 decimals, trim trailing zeros → display like `0.007%`, `0.01%`, `0.005%`.

Verified against samples:
- `"PASS-7.005501E-033"` → `7.005501E-03` = `0.007005501%` → display `0.007%` ✓
- `"PASS 4.988443E-033"` → `4.988443E-03` = `0.004988443%` → display `0.005%` ✓
- `"PASS 1.524978E-023"` → `1.524978E-02` = `0.01524978%` → display `0.015%`

---

## Changes by file

### 1. `parsers/spec-reader.js`

**Goal:** allow SCMVAS/SCMHVAS model numbers to pass the MODNAME validation filter so they land in the spec map (with a synthetic no-specs stub), OR bypass spec lookup entirely for this family.

**Approach:** bypass entirely. Cleaner — no stub records.

Change `getSpecs()` to special-case SCMVAS/SCMHVAS and return a well-known "no-specs" sentinel (e.g. `{ _family: 'SCMVAS', _noSpecs: true }`) instead of `null`. This lets `exportNewRecords()` proceed to formatter without silently skipping.

Also update the MODNAME prefix regex at line 287 (`^(SCM5B|5B|SCM7B|7B|8B|DSCA|DSCT|SCT|BOGUS)`) — this line rejects records in the binary DAT parser only, which doesn't affect VASLOG (VASLOG isn't read through that code path). No change needed here — leaving SCMVAS/SCMHVAS out of the binary parser filter is correct since we don't parse `hvin.dat`.

**Diff scope:** ~20 lines in `getSpecs()`.

### 2. `templates/datasheet-exact.js`

**Goal:** new family branch emitting the simple Accuracy-only template.

**Approach:** Add `SCMVAS` to DATA_LINES (single-entry array: `[['Accuracy', '%']]`). At the top of `generateExactDatasheet()`, if the spec stub flags `_family === 'SCMVAS'`, route to a dedicated `generateSCMVASDatasheet(record)` helper that builds the 35-line template above. This helper does NOT use `specs` — only `record.model_number`, `record.serial_number`, `record.test_date`, `record.overall_result`, and `record.raw_data`.

The helper must:
- Render 8-space left indent on every line
- Date formatted `MM/DD/YYYY` (matching newer samples) — note: "Corrected HVAS" uses `MM-DD-YYYY`; use `MM/DD/YYYY` per the most recent Engineering-Tested samples
- Extract accuracy value via the regex above
- Constants: specification = `+/- 0.03%`, withstand/Hi-Pot block omitted (SCMVAS has none), checklist uses `__X__` markers

Also delete the vestigial `startsWith('SCMHVAS')` check at existing line 652 (it was inside the DSCT branch and is no longer reachable once SCMVAS gets its own branch).

**Diff scope:** ~80 new lines (new helper + DATA_LINES entry + router change + one deletion).

### 3. `database/export-datasheets.js`

**Goal:** do not skip SCMVAS/SCMHVAS records due to missing specs.

**Approach:** after changing `getSpecs()` to return a stub for this family, the existing `if (!specs) continue;` logic in both `run()` and `exportNewRecords()` just works. No explicit change needed — verify only.

### 4. `database/import.js`

**Goal:** ingest the Engineering-Tested plain `.txt` files.

**Approach:** Add a new dedicated import branch for the `VASLOG - Engineering Tested` subfolder:

- Add a new parser `parsers/vaslog-engtxt.js` that:
  - Takes a `.txt` filepath, parses SN from filename (pattern `^(\d+-\d+[A-Za-z]?)(?:\d{14})?\.txt$`, capturing the SN segment like `166590-1` or `167601-4` — the optional 14-digit timestamp suffix `MMDDYYYYhhmmss` is dropped)
  - Reads the file, extracts `Model:`, `Date:`, `SN:`, `Accuracy`, and `Status` from the plain-text header rows
  - Returns one record with `log_type='VASLOG_ENG'`, `overall_result='PASS'` (or derive from the Status field), `raw_data=<full file contents>`, `source_file=<path>`
- Register `VASLOG_ENG` in the LOG_TYPES map with a new `vaslog-engtxt` parser alias
- Make the `importStationLogs()` walk recurse into `VASLOG/` one level to pick up the `VASLOG - Engineering Tested/*.txt` subfolder. Cleanest: parameterize the LOG_TYPES entry with `subfolder` and `recursive` flags.

For the **pass-through copy to `X:\For_Web\<SN>.TXT`**, the Engineering-Tested files already have the correct final format. Two sub-options:

- **(4a) Pass-through**: `exportNewRecords()` detects `log_type === 'VASLOG_ENG'`, copies `raw_data` (the original file contents) verbatim into `X:\For_Web\<SN>.TXT`, sets `forweb_exported_at`. Zero risk of format drift.
- **(4b) Re-render**: treat the `VASLOG_ENG` record the same as a VASLOG record — run it through the same `generateSCMVASDatasheet()` helper. Consistent with production path.

**Recommendation: (4a) pass-through.** Reasons:
- The files already match the target format exactly (verified by comparing samples to `Corrected HVAS Files/*.txt`)
- Preserves any Engineering-hand-tweaked formatting
- If drift is ever needed, switching to (4b) is a one-line change later

**Diff scope:** new `parsers/vaslog-engtxt.js` (~60 lines) + ~30 lines across `import.js` + ~15 lines in `export-datasheets.js` for the pass-through branch.

### 5. `C:\Shares\test\scripts\Sync-FromNAS-rsync.ps1`

**Goal:** ensure the Engineering-Tested subfolder is included in the sync.

**Approach:** Verify that the existing `TS-3R\LOGS\` rsync already pulls the full subtree (`--recursive`). Based on prior session logs, the rsync syncs `TS-3R/*`, so the subfolder likely rides along. **Verify only — no change expected.** If rsync uses explicit includes, add `VASLOG - Engineering Tested/***` to the include list.

### 6. Database schema

**Goal:** no schema changes required.

The `test_records` table already has `log_type`, `model_number`, `serial_number`, `test_date`, `overall_result`, `raw_data`, `source_file`, `forweb_exported_at`. The new `VASLOG_ENG` log_type is just a new string value in an existing column.

### 7. Backfill strategy

**Production VASLOG .DAT**: these are already imported into `test_records` via the existing multiline parser. After the spec-reader/formatter changes deploy, run:

```
node database/export-datasheets.js --limit 0
```

to regenerate datasheets for all PASS SCMVAS/SCMHVAS records where `forweb_exported_at IS NULL`. This backfills historical SCMVAS/SCMHVAS records that were previously skipped due to "no specs".

**Engineering-Tested .txt**: run the full import once after the new parser is added. Should pick up all 434 existing files and copy them to `X:\For_Web\`.

---

## Risks / edge cases

1. **The `VAS-MPT.DAT` / `HVAS-MPT.DAT` "pass-through" models** — might need a slightly different treatment (skip Check List? different wording?). Treat same as regular SCMVAS for now; revisit if user reports a mismatch.
2. **FAIL records** — the PASS regex above also matches `FAIL`. Verify the Status column in the output shows `FAIL` and that the existing `exportNewRecords` logic (which filters `overall_result = 'PASS'`) skips FAIL datasheets by default. No action needed.
3. **Filename SN extraction for Engineering-Tested** — observed patterns: `166590-1.txt`, `166590-110042023104524.txt` (trailing timestamp). Regex must correctly split the timestamp. A small number of edge cases exist (e.g. `166594-1010042023090444.txt` = SN `166594-10`, timestamp `10042023090444`) — the SN has variable-length second segment. Safe rule: SN ends at the last `-<digits>` segment before the optional 14-digit timestamp.
4. **Duplicate files** — `166593-4.txt` (1519 bytes) and `166593-410042023114928.txt` (1600 bytes) coexist. Treat the timestamped filename as canonical; untimestamped is a later re-render. Import both but dedupe on `(log_type, model_number, serial_number, test_date, test_station)` (existing unique constraint already handles this).
5. **Date format variance** — production VASLOG stores `MM-DD-YYYY` in raw_data; Engineering-Tested `.txt` uses `MM/DD/YYYY` or `MM-DD-YYYY` depending on vintage. Normalize all date displays to `MM/DD/YYYY` per the newest Engineering-Tested output.

---

## Test plan (Coding Agent must verify)

Before declaring complete:

1. `node database/export-datasheets.js --dry-run --serial 179379-1` → should preview a well-formed SCMHVAS datasheet (no "missing specs" skip).
2. `node database/export-datasheets.js --serial 166590-1` → compare generated `X:\For_Web\166590-1.TXT` byte-for-byte against the existing `samples/vaslog-engtxt/166590-110042023104524.txt`. Expect visual match; char-level drift acceptable only in whitespace.
3. Full incremental import of the `VASLOG - Engineering Tested` subfolder → verify all 434 `.txt` files copy to `X:\For_Web\`.
4. Historical backfill of production VASLOG records → spot-check 5 SCMHVAS and 5 SCMVAS datasheets against any known-good reference in `Corrected HVAS Files/`.
5. Regression: pick 10 existing SCM5B + 10 DSCA datasheets, regenerate, confirm no format drift vs. their current `X:\For_Web\*.TXT`.

---

## Delegation

Once this plan is approved, hand off to the Coding Agent with:
- This plan as the spec
- All research artifacts under `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/`
- Sample output at `samples/corrected-hvas/171087-1.txt` and `samples/vaslog-engtxt/166590-110042023104524.txt` as golden references
- Access to AD2 via `paramiko` (creds from vault path `clients/dataforth/ad2.sops.yaml`)

After implementation, mandatory Code Review Agent pass before deploying to `C:\Shares\testdatadb\`.