# Session Log: 2026-04-11

## Session Summary

Extending the Dataforth Test Datasheet Pipeline (rebuilt 2026-03-27/29) to add SCMVAS-Mxxx (obsolete) and SCMHVAS-Mxxxx (replacement) product families. Discovery phase substantially complete — uncovered that the ingestion half is already mostly wired up but spec-lookup and formatter for this family are missing.

### Request Summary

Add two product lines:
1. **SCMVAS-Mxxx** (obsolete ~2024) — spec DB at `\\AD1\Engineering\ENGR\ATE\High Voltage Input Module Test\HVDATA\hvin.dat`; test program at `...\Released\` (TESTHV3.BAS + NLIBATE3.BAS + TESTHV4.BAS)
2. **SCMHVAS-Mxxxx** (replacement) — half tested with same software above; other half tested in Engineering with plain .txt output at `TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\`

### Work Completed

1. **SSH access to AD2 via paramiko** — password auth, single-session batch scripts under `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/`.

2. **Source files pulled** to `source/`:
   - `TESTHV3.BAS` (116461B, 2020) — main test program (tests SCM5B41/8B/DSCA high-voltage variants)
   - `TESTHV4.BAS` (110498B, 2017) — alternate test program
   - `NLIBATE3.BAS` (59671B, 2020), `LIBATE3.BAS` (26496B parent) — ATE library (new + old)
   - `DBHV.BAS` (26192B) — database editor (holds TYPE DBASE definition for hvin.dat)
   - `hvin.dat` (6567B, 33 records × 199 bytes each), `hvsort.dat` (1065B) — spec binary files
   - `Readme.txt` — directory structure doc

3. **hvin.dat binary structure decoded** — TYPE DBASE from DBHV.BAS:
   - 4 strings (MODNAME*13, INTYPE*3, OUTSIGTYPE*7, WAVESHPCAL*8) + 42 SINGLEs = **199 bytes/record, 33 records**
   - Python parser: `parse_hvin.py` (verified: reads MODNAMEs correctly; IEEE 754 floats like the other existing parsers assume)
   - **Models actually in hvin.dat are engineering IDs**: SCM5B41-03, SCM5B41-09D/-10D, SCM5B41-1181/-1182/-1728/-1882, SCM5B31-1882, DSCA41-03/-1568, DSCA31-1273/-1947, 8B51-12/-13/-1831/-1832/-1837, 8B31-12/-13/-1841/-1842/-1843/-1876, 8B41-12/-13 — **NOT "SCMVAS-Mxxx"/"SCMHVAS-Mxxxx" marketing names**

4. **VASLOG location + format discovered**:
   - Path on AD2-mirrored NAS: `C:\Shares\test\TS-3R\LOGS\VASLOG\`
   - Contains CSV-like multiline `.DAT` files (same format as 5BLOG, 8BLOG, etc.): `HVAS-M01/M02/M03/M04/MPT.DAT` (5 models) + `VAS-M100/M200/M300/M400/M500/M600/M650/M700/MPT.DAT` (9 models)
   - Header per record: `"SCMHVAS-M0100 "` or similar marketing name
   - Data block: 5 CSV rows of measurements, status line `"PASS-7.005501E-033",...`, footer `"179379-1","04-09-2026"` (SN + test date)
   - Subfolder `VASLOG - Engineering Tested\` contains 434 `.txt` files (Engineering half of SCMHVAS — simple accuracy-only datasheets, 1600 bytes each)
   - Another folder `C:\Shares\test\Corrected HVAS Files\` has 200 pre-generated SCMHVAS datasheets (WO-unit.txt pattern, 171087-x, 171088-x)

5. **Existing pipeline state** (pulled from `C:\Shares\testdatadb\`):
   - `parsers/multiline.js` — already handles `.DAT` CSV format; doc comment explicitly mentions "DSCLOG, 5BLOG, 8BLOG, PWRLOG, SCTLOG, VASLOG"
   - `database/import.js` — **VASLOG is already registered in LOG_TYPES** (line 26) using multiline parser, so `*.DAT` files in TS-xx/LOGS/VASLOG are being ingested into the `test_records` table already
   - `parsers/spec-reader.js` — handles 5BMAIN, 8BMAIN, DSCOUT, SCTMAIN, DSCMAIN4, 5B45DATA, DB5B48, 5B49_2, 7BMAIN. **Does NOT load hvin.dat; MODNAME regex filter does NOT allow SCMVAS/SCMHVAS/VAS/HVAS/HVIN prefix** (line 287)
   - `templates/datasheet-exact.js` — has families SCM5B, 8B, DSCA, SCM7B, DSCT. Has one vestigial check at line 652 that skips 240VAC/Hi-Pot lines if modelName starts with SCMHVAS (inside DSCT branch) — suggests prior attempt but no full SCMVAS/SCMHVAS branch
   - **Engineering-Tested .txt subfolder is NOT recursed** by import.js (line 178 uses `recursive=false`)
   - **No mechanism** copies these .txt files to `X:\For_Web\` for web publishing

### Critical Open Question (Blocker)

**Model name mismatch**: hvin.dat contains engineering IDs like `SCM5B41-1181`, `8B51-1831`, `DSCA41-1568`. VASLOG .DAT logs use marketing names like `SCMHVAS-M0100`, `VAS-M100`. These can't be looked up against hvin.dat directly.

Three possibilities:
- (a) Each marketing SCMVAS/HVAS-Mxxxx maps to a specific engineering SCM5B41-xxxx part — need a mapping table from the user.
- (b) A separate spec DB (not hvin.dat) holds the marketing-named records.
- (c) SCMVAS/HVAS datasheets don't need parameter-level specs at all — they only print "Accuracy" pass/fail (as seen in Engineering-Tested .txt samples), which can be computed directly from the pass/fail line in the .DAT log.

**Recommendation**: (c) is most consistent with the sample datasheets. The "Corrected HVAS" and Engineering-Tested .txt files each have exactly **one** parameter (Accuracy). That format can be generated without consulting hvin.dat — we'd just need a simple SCMVAS/SCMHVAS branch in datasheet-exact.js that pulls model+SN+date+accuracy% from the DB record and formats the 35-line template. HVIN.DAT might only be referenced inside TESTHV3.EXE at station-level during testing (to drive the test equipment), not needed at datasheet generation time.

### Pending Tasks

- [x] User confirmed **Option C** (simple Accuracy-only, no hvin.dat lookup) on 2026-04-12
- [x] Decoded PASS log value format: `"PASS<signed><mantissa>E-<2digit-exp><status-digit>"` — captured float is already in percent units, trailing digit is extraneous test status
- [x] Implementation plan drafted: `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/IMPLEMENTATION_PLAN.md`
- [x] User approved plan (2026-04-12), delegated to Coding Agent
- [x] Coding Agent staged 6 files under `projects/dataforth-dos/datasheet-pipeline/implementation/`
- [x] Code Review Agent found 5 must-fix issues (recursive default regression, importFiles dispatch order, filename regex greedy match, hardcoded creds, binary passthrough integrity)
- [x] Coding Agent applied all 5 fixes + nice-to-haves; test harness still byte-matches golden sample
- [x] Code Review Agent re-approved — **Production Ready**
- [x] Dry-run deploy to AD2 (Task #7) — 4 updates + 1 create, paths valid
- [x] Deploy to AD2 `C:\Shares\testdatadb\` (Task #8) — 5 files + .bak-20260412 backups
- [x] Verify testdatadb service health (Task #9) — Restarted, :3000 200 OK
- [x] Dry-run export for known SCMHVAS (Task #10) — 179379-1 generated 1600B matching golden
- [x] Import Engineering-Tested .txt files (Task #11) — 434/434 VASLOG_ENG records imported
- [x] Full backfill on 27,571 records (Task #12) — 27,065 done (26,663 rendered + 402 passthrough), 438 skipped due to plain-decimal PASS format
- [x] Investigate 438 stragglers (Task #14) — root cause: QB STR$() plain-decimal output for values above threshold; 1.6% of records; no semantic difference
- [x] Patch regex to handle plain-decimal (Task #15) — added `SCMVAS_ACCURACY_RE_PLAIN` fallback
- [x] Code Review of patch (Task #16) — APPROVED
- [x] Redeploy template + backfill stragglers (Task #17) — 438/438 rendered, 0 remaining backlog, plain-decimal sample (SN 66260-12 from 2011-02-16) renders as 0.012% PASS correctly

### Research Artifacts

- `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/` — all research material
  - `source/` — pulled .BAS + hvin.dat
  - `existing-parsers/`, `existing-templates/`, `existing-database/` — snapshot of current prod code
  - `samples/vaslog-dat/` — 14 production .DAT logs (HVAS-M01..MPT, VAS-M100..MPT)
  - `samples/vaslog-engtxt/` — 10 sample Engineering-Tested .txt
  - `samples/corrected-hvas/` — 5 sample "Corrected HVAS" .txt
  - `parse_hvin.py`, `fetch_all.py`, `fetch_vaslog*.py`, `ssh_ad2.py` — helper scripts

### Related

- Prior rebuild: `session-logs/2026-03-28-session-ad2.md`
- Project: `projects/dataforth-dos/`
- AD2 ssh creds: `sysadmin / Paper123!@#` (via paramiko, banner_timeout=30)