Mike Swanson
45083f4735
Extends the Test Datasheet Pipeline on AD2:C:\Shares\testdatadb to
generate web-published datasheets for the SCMVAS-Mxxx (obsolete) and
SCMHVAS-Mxxxx (replacement) High Voltage Input Module product lines.
Both are tested either with the existing TESTHV3 software (production
VASLOG .DAT logs) or in Engineering with plain .txt output.
Key changes on AD2 (all deployed 2026-04-12 with dated backups):
- parsers/spec-reader.js: getSpecs() returns a `{_family:'SCMVAS',
_noSpecs:true}` sentinel for SCMVAS/SCMHVAS/VAS-M/HVAS-M model prefixes
so the export pipeline does not silently skip them for missing specs.
- templates/datasheet-exact.js: new Accuracy-only template branch
(generateSCMVASDatasheet + helpers) that mirrors the existing shipped
format byte-for-byte. Extraction regex covers both QuickBASIC STR$()
output formats: scientific-with-trailing-status-digit (98.4% of
records) and plain-decimal (1.6% of records above QB's threshold).
- parsers/vaslog-engtxt.js (new): parses the Engineering-Tested .txt
files in TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\. Filename SN
regex strips optional trailing 14-digit timestamp; in-file "SN:"
header is the authoritative source when the filename is malformed.
- database/import.js: LOG_TYPES grows a VASLOG_ENG entry with
subfolder + recursive flags. Pre-existing 7 log types keep their
implicit recursive=true behaviour (config.recursive !== false).
importFiles() routes VASLOG_ENG paths before the generic loop so a
VASLOG - Engineering Tested/*.txt path does not mis-dispatch to the
multiline parser.
- database/export-datasheets.js: VASLOG_ENG records are written
verbatim via fs.copyFileSync(source_file, For_Web/<SN>.TXT) for true
byte-level pass-through, with a graceful raw_data fallback when the
source file is no longer on disk.
Deploy outcome:
- 27,503 SCMVAS/SCMHVAS datasheets rendered (27,065 from scientific +
438 from plain-decimal PASS lines, post-patch rerun)
- 434 Engineering-Tested .txt files pass-through-copied to For_Web
- 0 errors across both batches
Repo layout added here:
- scmvas-hvas-research/: discovery artifacts (source .BAS, hvin.dat,
sample .DAT + .txt, binary-format notes, IMPLEMENTATION_PLAN.md)
- implementation/: staged final code + deploy helpers + local test
harness + per-step verification scripts
- backups/pre-deploy-20260412/: independent local snapshot of the 4
AD2 files replaced, pulled byte-for-byte before deploy
All helper scripts fetch the AD2 password at runtime from the SOPS
vault (clients/dataforth/ad2.sops.yaml). None of the committed files
contain the plaintext credential. Known vault-entry hygiene issue
(stale shell-escape backslash before the `!`) is documented in the
fetcher comments and stripped at read-time; flagged separately for
cleanup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
93 lines
7.9 KiB
Markdown
93 lines
7.9 KiB
Markdown
# Session Log: 2026-04-11
|
||
|
||
## Session Summary
|
||
|
||
Extending the Dataforth Test Datasheet Pipeline (rebuilt 2026-03-27/29) to add SCMVAS-Mxxx (obsolete) and SCMHVAS-Mxxxx (replacement) product families. Discovery phase substantially complete — uncovered that the ingestion half is already mostly wired up but spec-lookup and formatter for this family are missing.
|
||
|
||
### Request Summary
|
||
|
||
Add two product lines:
|
||
1. **SCMVAS-Mxxx** (obsolete ~2024) — spec DB at `\\AD1\Engineering\ENGR\ATE\High Voltage Input Module Test\HVDATA\hvin.dat`; test program at `...\Released\` (TESTHV3.BAS + NLIBATE3.BAS + TESTHV4.BAS)
|
||
2. **SCMHVAS-Mxxxx** (replacement) — half tested with same software above; other half tested in Engineering with plain .txt output at `TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\`
|
||
|
||
### Work Completed
|
||
|
||
1. **SSH access to AD2 via paramiko** — password auth, single-session batch scripts under `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/`.
|
||
|
||
2. **Source files pulled** to `source/`:
|
||
- `TESTHV3.BAS` (116461B, 2020) — main test program (tests SCM5B41/8B/DSCA high-voltage variants)
|
||
- `TESTHV4.BAS` (110498B, 2017) — alternate test program
|
||
- `NLIBATE3.BAS` (59671B, 2020), `LIBATE3.BAS` (26496B parent) — ATE library (new + old)
|
||
- `DBHV.BAS` (26192B) — database editor (holds TYPE DBASE definition for hvin.dat)
|
||
- `hvin.dat` (6567B, 33 records × 199 bytes each), `hvsort.dat` (1065B) — spec binary files
|
||
- `Readme.txt` — directory structure doc
|
||
|
||
3. **hvin.dat binary structure decoded** — TYPE DBASE from DBHV.BAS:
|
||
- 4 strings (MODNAME*13, INTYPE*3, OUTSIGTYPE*7, WAVESHPCAL*8) + 42 SINGLEs = **199 bytes/record, 33 records**
|
||
- Python parser: `parse_hvin.py` (verified: reads MODNAMEs correctly; IEEE 754 floats like the other existing parsers assume)
|
||
- **Models actually in hvin.dat are engineering IDs**: SCM5B41-03, SCM5B41-09D/-10D, SCM5B41-1181/-1182/-1728/-1882, SCM5B31-1882, DSCA41-03/-1568, DSCA31-1273/-1947, 8B51-12/-13/-1831/-1832/-1837, 8B31-12/-13/-1841/-1842/-1843/-1876, 8B41-12/-13 — **NOT "SCMVAS-Mxxx"/"SCMHVAS-Mxxxx" marketing names**
|
||
|
||
4. **VASLOG location + format discovered**:
|
||
- Path on AD2-mirrored NAS: `C:\Shares\test\TS-3R\LOGS\VASLOG\`
|
||
- Contains CSV-like multiline `.DAT` files (same format as 5BLOG, 8BLOG, etc.): `HVAS-M01/M02/M03/M04/MPT.DAT` (5 models) + `VAS-M100/M200/M300/M400/M500/M600/M650/M700/MPT.DAT` (9 models)
|
||
- Header per record: `"SCMHVAS-M0100 "` or similar marketing name
|
||
- Data block: 5 CSV rows of measurements, status line `"PASS-7.005501E-033",...`, footer `"179379-1","04-09-2026"` (SN + test date)
|
||
- Subfolder `VASLOG - Engineering Tested\` contains 434 `.txt` files (Engineering half of SCMHVAS — simple accuracy-only datasheets, 1600 bytes each)
|
||
- Another folder `C:\Shares\test\Corrected HVAS Files\` has 200 pre-generated SCMHVAS datasheets (WO-unit.txt pattern, 171087-x, 171088-x)
|
||
|
||
5. **Existing pipeline state** (pulled from `C:\Shares\testdatadb\`):
|
||
- `parsers/multiline.js` — already handles `.DAT` CSV format; doc comment explicitly mentions "DSCLOG, 5BLOG, 8BLOG, PWRLOG, SCTLOG, VASLOG"
|
||
- `database/import.js` — **VASLOG is already registered in LOG_TYPES** (line 26) using multiline parser, so `*.DAT` files in TS-xx/LOGS/VASLOG are being ingested into the `test_records` table already
|
||
- `parsers/spec-reader.js` — handles 5BMAIN, 8BMAIN, DSCOUT, SCTMAIN, DSCMAIN4, 5B45DATA, DB5B48, 5B49_2, 7BMAIN. **Does NOT load hvin.dat; MODNAME regex filter does NOT allow SCMVAS/SCMHVAS/VAS/HVAS/HVIN prefix** (line 287)
|
||
- `templates/datasheet-exact.js` — has families SCM5B, 8B, DSCA, SCM7B, DSCT. Has one vestigial check at line 652 that skips 240VAC/Hi-Pot lines if modelName starts with SCMHVAS (inside DSCT branch) — suggests prior attempt but no full SCMVAS/SCMHVAS branch
|
||
- **Engineering-Tested .txt subfolder is NOT recursed** by import.js (line 178 uses `recursive=false`)
|
||
- **No mechanism** copies these .txt files to `X:\For_Web\` for web publishing
|
||
|
||
### Critical Open Question (Blocker)
|
||
|
||
**Model name mismatch**: hvin.dat contains engineering IDs like `SCM5B41-1181`, `8B51-1831`, `DSCA41-1568`. VASLOG .DAT logs use marketing names like `SCMHVAS-M0100`, `VAS-M100`. These can't be looked up against hvin.dat directly.
|
||
|
||
Three possibilities:
|
||
- (a) Each marketing SCMVAS/HVAS-Mxxxx maps to a specific engineering SCM5B41-xxxx part — need a mapping table from the user.
|
||
- (b) A separate spec DB (not hvin.dat) holds the marketing-named records.
|
||
- (c) SCMVAS/HVAS datasheets don't need parameter-level specs at all — they only print "Accuracy" pass/fail (as seen in Engineering-Tested .txt samples), which can be computed directly from the pass/fail line in the .DAT log.
|
||
|
||
**Recommendation**: (c) is most consistent with the sample datasheets. The "Corrected HVAS" and Engineering-Tested .txt files each have exactly **one** parameter (Accuracy). That format can be generated without consulting hvin.dat — we'd just need a simple SCMVAS/SCMHVAS branch in datasheet-exact.js that pulls model+SN+date+accuracy% from the DB record and formats the 35-line template. HVIN.DAT might only be referenced inside TESTHV3.EXE at station-level during testing (to drive the test equipment), not needed at datasheet generation time.
|
||
|
||
### Pending Tasks
|
||
|
||
- [x] User confirmed **Option C** (simple Accuracy-only, no hvin.dat lookup) on 2026-04-12
|
||
- [x] Decoded PASS log value format: `"PASS<signed><mantissa>E-<2digit-exp><status-digit>"` — captured float is already in percent units, trailing digit is extraneous test status
|
||
- [x] Implementation plan drafted: `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/IMPLEMENTATION_PLAN.md`
|
||
- [x] User approved plan (2026-04-12), delegated to Coding Agent
|
||
- [x] Coding Agent staged 6 files under `projects/dataforth-dos/datasheet-pipeline/implementation/`
|
||
- [x] Code Review Agent found 5 must-fix issues (recursive default regression, importFiles dispatch order, filename regex greedy match, hardcoded creds, binary passthrough integrity)
|
||
- [x] Coding Agent applied all 5 fixes + nice-to-haves; test harness still byte-matches golden sample
|
||
- [x] Code Review Agent re-approved — **Production Ready**
|
||
- [x] Dry-run deploy to AD2 (Task #7) — 4 updates + 1 create, paths valid
|
||
- [x] Deploy to AD2 `C:\Shares\testdatadb\` (Task #8) — 5 files + .bak-20260412 backups
|
||
- [x] Verify testdatadb service health (Task #9) — Restarted, :3000 200 OK
|
||
- [x] Dry-run export for known SCMHVAS (Task #10) — 179379-1 generated 1600B matching golden
|
||
- [x] Import Engineering-Tested .txt files (Task #11) — 434/434 VASLOG_ENG records imported
|
||
- [x] Full backfill on 27,571 records (Task #12) — 27,065 done (26,663 rendered + 402 passthrough), 438 skipped due to plain-decimal PASS format
|
||
- [x] Investigate 438 stragglers (Task #14) — root cause: QB STR$() plain-decimal output for values above threshold; 1.6% of records; no semantic difference
|
||
- [x] Patch regex to handle plain-decimal (Task #15) — added `SCMVAS_ACCURACY_RE_PLAIN` fallback
|
||
- [x] Code Review of patch (Task #16) — APPROVED
|
||
- [x] Redeploy template + backfill stragglers (Task #17) — 438/438 rendered, 0 remaining backlog, plain-decimal sample (SN 66260-12 from 2011-02-16) renders as 0.012% PASS correctly
|
||
|
||
### Research Artifacts
|
||
|
||
- `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/` — all research material
|
||
- `source/` — pulled .BAS + hvin.dat
|
||
- `existing-parsers/`, `existing-templates/`, `existing-database/` — snapshot of current prod code
|
||
- `samples/vaslog-dat/` — 14 production .DAT logs (HVAS-M01..MPT, VAS-M100..MPT)
|
||
- `samples/vaslog-engtxt/` — 10 sample Engineering-Tested .txt
|
||
- `samples/corrected-hvas/` — 5 sample "Corrected HVAS" .txt
|
||
- `parse_hvin.py`, `fetch_all.py`, `fetch_vaslog*.py`, `ssh_ad2.py` — helper scripts
|
||
|
||
### Related
|
||
|
||
- Prior rebuild: `session-logs/2026-03-28-session-ad2.md`
|
||
- Project: `projects/dataforth-dos/`
|
||
- AD2 ssh creds: `sysadmin / Paper123!@#` (via paramiko, banner_timeout=30)
|