formatMeasuredExact now applies Math.fround to the parsed value before toFixed.
The DOS QuickBASIC computed/stored these as single-precision floats, so the
last-digit rounding at the .5 boundary follows single, not double, precision.
Without it, double-precision toFixed flipped boundaries (9.9995 -> "9.999" vs
golden "10.000"; 46.85 -> "46.9" vs "46.8"; .45 -> "0.5" vs "0.4"; 3.3325 ->
"3.333" vs "3.332"). Verified each against the staged golden.
STAGE 3 re-validation: FINAL-TEST CLEAN models 68 -> 85 (+17), mismatches
26 -> 9, cert matches 2123 -> 2206. Zero regression — every remaining dirty
model was already dirty pre-fix; no previously-clean model flipped.
The 9 remaining are NOT rounding:
- 4 models (DSCA38-05/-1793/-19C/-19E): Supply Current retest data-vintage — the
staged .TXT is an older test run than the DB latest-wins record; not a render
bug, can't reconcile against an older sheet.
- 5 models (DSCA39-01/02/05/07/1950): STAGE 1 template artifact — the footer note
"Standard output load for test is 250 ohms." was mis-captured as a truncated
parameter row. A renderer-side fix was attempted but emitted the note for all
current-output models (regressed 24 clean -C models), so it was reverted; needs
a targeted STAGE 1 extractor fix instead.
Renamed dsca-clean68-models.json -> dsca-clean-models.json (now 85 models).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restarted testdatadb service (new template live), canary-pushed 1 cert (updated,
0 errors), then re-pushed all PASS certs for the 68 Final-Test-content-clean
models via uploadBySerialNumbers from a fresh node process.
Result over 30,423 PASS certs: updated=26022 unchanged=2738 created=0 errors=0
skipped=1663. The 26,022 updates replace the old defective DSCA renders on the
live site with the rebuilt, byte-content-validated ones. The 1,663 skips are the
count-guard correctly refusing any individual cert whose value count != its
template row count (ambiguous) — never publishing misaligned data.
Artifacts: push-clean68.js (driver), dsca-clean68-models.json (the published
set). NOT yet published: the 26 last-digit-diff models, the ~32 ambiguous/null
layout families, and the 231 untemplated models.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
validate-dsca-stage3.js: read-only harness that, for every staged DSCA original
we have ground truth for (2806 across 126 models), looks up the DB record,
renders it through the live path, and content-compares. GATE = the FINAL TEST
RESULTS section (rule lines canonicalized, whitespace collapsed — so the deferred
column-spacing cosmetic doesn't register); accuracy-section diffs reported
separately as informational.
First run verdict (report attached):
- 68 models FINAL-TEST CONTENT-CLEAN (0 mismatch over compared certs).
- 2123/2316 certs match the Final-Test content exactly (91.7%).
- 26 models show measured-value last-digit diffs only — structure (names, specs,
row alignment, statuses) is correct. Two root causes, neither structural:
* rounding-mode: JS double toFixed vs QB single-precision half-up
(e.g. raw 9.9995 code3 -> "9.999" here, "10.000" in golden). Fixable but
float-precision-sensitive; risks regressing currently-clean values.
* data vintage: staged .TXT is a different test run than the DB latest-wins
record (Fix 3) — e.g. Supply Current 19.6 vs 20.3, 0.7 apart. Not a render
bug; can't be reconciled against an older staged sheet.
- ~32 models render null (count-guard): DSCA33-*, DSCA45-*, DSCA49-* families
whose raw_data carries load points the template omits -> need per-subtype slot
mapping (the canonical-slot approach) before they can render.
Still NOT published: service not restarted, nothing re-pushed to Hoffman.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deployed file is C:\Shares\testdatadb\templates\datasheet-exact.js; this
reconciles the repo copy + adds dsca-templates.json (STAGE 1 output).
What changed in generateExactDatasheet (DSCA family only; 5B/8B/7B/DSCT/SCMVAS
paths byte-unchanged):
- Load dsca-templates.json once at module top (126 per-model layouts).
- DSCA Final-Test now renders names + specs from the staged template rows, not
the single hardcoded DATA_LINES['DSCA'] + buildTSpecs DSCA branch.
- Value-bearing raw_data STATUS groups map positionally onto the spec-bearing
template rows; empty-spec rows (240VAC Withstand / Hi-Pot) render blank+PASS.
Removed the duplicate hardcoded 240VAC/Hi-Pot footer for DSCA (now rows).
- ACCURACY header uses the template accOut ("Output (V)"/"Output (mA)") with '-'
rule separators instead of "Vout (V)" + '='.
- Header/columns match the staged originals (Measured Value*, 25/15/19/6 rule).
Two real bugs fixed (both are the handoff's "lines drop" / wrong-value defect):
- formatMeasuredExact reads the value from index 4 so negative signs survive
("PASS-4.24..." -> "-4", not "4"); also decimal-code N -> toFixed(N) exactly
(DSCA differs from 5B/8B where code 2 means 1 decimal).
- parseRawData no longer consumes the first DSCA STATUS group as a bare
step-response line when that line is absent (dropped 3 rows on e.g. DSCA39-01).
Safety: when value count != spec-row count the positional zip is ambiguous
(subtype measures load points the template omits, e.g. DSCA49 5mA pair), so the
cert is SKIPPED (null) and left for STAGE 3 per-subtype mapping rather than
emitting misaligned data.
Validation: DSCA38-05 (SN 180224-1) Final-Test block byte-identical to its
staged original. 92/126 templated models render cleanly; 7 ambiguous + 27
no-spec skip. Remaining ACCURACY-block spacing diffs are the deferred cosmetic
gap. NOT YET LIVE: testdatadb service not restarted, nothing re-pushed to
Hoffman (STAGE 3 gate).
Coverage gap to resolve before publish: only 126/357 DSCA models in the DB have
a staged template (56,074 certs, 70.1%); 231 models / 23,866 certs have none and
now render null — needs a STAGE 1 extension (more staged originals).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plain-language summary for John (EE, non-developer) covering all 4 problems
(RTD label, DSCA table, same-day run, missing units), framed as ACG-owned fixes.
Cross-links the four supporting technical docs. Draft - not yet sent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per Mike: import runs every 15 min, so routine timing isn't the cause. The 379
absent units are confined to 2025-10..2026-01 (stop after Jan 2026) on TS-4L/4R/1R
- fingerprint of a one-time overwrite during the incident/recovery (fresh DOS logs
overwrote accumulated appended server-side logs for ~2 weeks). One-time, not
recurring; backfill from the surviving staged .TXT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
608 staged datasheets absent from DB. Two causes: (1) 229 units with encoded/
non-standard serials the importer's leading-digit regex silently skips - data is
in the .DAT, recoverable; full blind spot is 840 serials / 9,510 records / 141
models dropped fleet-wide. (2) 379 units whose per-model .DAT was overwritten by a
later work order - recoverable only from the staged .TXT or a log backup. Adds
John-facing report, raw data, and the chase-missing-units.js tool.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Whole-source sweep (981,716 records / 406,549 serials): 6,515 same-day multi-run
events; DB holds a NON-latest run for 311 (the strictly-greater-date conflict rule
freezes on an arbitrary same-day run). Corrects the verdict doc to flag same-day
retests as a latest-wins faithfulness violation (not benign). Adds the proposed
>= -with-data-differs conflict-rule fix (diagnose-only) and the sweep tool.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated all 11,922 staged original .TXT datasheets against test_records.
0 genuine parse faults across 11,239 comparable records; mismatches all explained
(retests, reused serials, VAS format, legacy out-of-scope units). Adds the
validate-parsing.js tool, raw report, and verdict. Two follow-ups (NOT parse bugs):
608 staged units absent from DB (ingestion completeness), and same-day retests keep
the first run (ON CONFLICT strictly-greater-date).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo copy only (review before deploying to C:\Shares\testdatadb). Folds RTD
(sensorNum 7) into the temperature path so the ACCURACY input column shows
'Temp. (C)' with signed temperature values, matching the original DOS-generated
datasheets and thermocouples (3-6). raw_data stimulus is already in deg C; no
conversion. getSensorNum and the i==13 ohm/ohm unit override are untouched.
Verified read-only against deployed env: 8B35 SN 179553-13 now shows Temp (C);
regression over 184 5B/8B renders -> 15 RTD changed (intended), 0 non-RTD changed.
Does NOT address Defect B (DSCA template). See DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the ~5,148 unpublished 8B/5B/SCM PASS records (driving the fix):
(1) parseRawData wrongly consumes a PASS/FAIL line as the step-response line for
non-DSCA families that omit the "0","0",v line (8B45/8B49/5B39/SCM5B33...) ->
drops the first Final-Test group -> measurement-count mismatch -> null render.
(2) Even parsed, the renderer has ONE hardcoded DATA_LINES['8B'] (RTD-shaped), so
models like 8B45 (frequency input, == DSCA45 structurally) get wrong param
names/specs. Same class as DSCA -> needs per-model templates.
Mined per-model templates from the Hoffman originals (published siblings) for all
136 mineable models via tools/mine-hoffman-dsca.py (family-agnostic extractor):
8b5bscm-templates.json = {accOut, accHeader, rows[name,spec], _srcSerial}.
Input-type split: 72 voltage / 18 temp / 12 current (accuracy already handled) +
10 frequency (8B45/5B45 — same unsolved freq-accuracy as DSCA45) + 24 7B/other.
Only 3 niche models (17 units) have no Hoffman original.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend (deployed live on AD2, service restarted, + repo copy resynced — it was
far behind the deployed server):
- /api/search: add whitelisted sort/dir (NULLS LAST) so sortable headers and the
"Latest uploads" preset work. web_status filter and POST /api/upload already
existed on the server; the stale repo copy now matches live.
Frontend (redesign prototype):
- "Latest uploads" preset (web_status=on + sort=api_uploaded_at desc) and
"Not yet published" (web_status=off) are now active presets.
- Push to Web (inspector) + Re-push (multi-select) wired to POST /api/upload
behind a confirm() gate; refresh WEB status after. Validated idempotently on a
published record (unchanged:1, errors:0).
- "Retested units" stays disabled — needs a retest flag in the pipeline (next).
tools/preview-proxy.py: forward POST so the publish buttons work in same-origin preview.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- fitCert: replace the flaky CSS `zoom` (Firefox support is recent/inconsistent)
with transform:scale() measured against the widest line (+ right margin and
font-load retries) so the cert always scales to fit the inspector with no
horizontal clip. Validated live on a narrow 5B cert (0.74x) and a wide DSCA45
cert (0.55x) against the real AD2 dataset.
- inspector Web field -> Published (green) / Not published (amber) chips.
- widen default inspector 480 -> 500px.
- tools/preview-proxy.py: serve the prototype AND reverse-proxy /api to the live
AD2 server so the cert iframe is same-origin during preview — styleCert/fitCert
read iframe.contentDocument, which silently no-ops when the iframe is loaded
cross-origin straight from AD2 (why the fit looked broken in earlier previews).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DSCA33/DSCA45 main spec files lost in the cryptolocker wipe are recoverable:
the original software published correct certs to the Hoffman product API before
the wipe and our null-skipping renderer never overwrote them. Mine per-model
Final-Test templates (names + specs + verbatim accuracy headers) straight from
those originals instead of requesting spec files from Dataforth/John.
- dsca33-45-templates.json: 56 models (DSCA33 34/35, DSCA45 22/23); only
DSCA33-1948 + DSCA45-1746 (24 units) lack an original.
- mine-hoffman-dsca.py: the re-runnable miner.
- DSCA33-45-HOFFMAN-RECOVERY handoff for the AD2 session (incl. the gate:
validate each render vs its Hoffman original before enabling live rendering).
- memories: Hoffman recovery (supersedes the spec-gap "need John" note) and the
AD2 SSH MTU-blackhole root cause/fix; errorlog entries (syncro jq, ssh correction).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vanilla single-file (no build/CDN) command-center redesign of the testdatadb search
UI: omni-search with serial/model/text routing + auto-select fast path, dense
monospace results table with PASS/FAIL pills + web-published indicator, persistent
split-pane datasheet inspector (iframe to /api/datasheet), left filter rail
(result/date/model/station/log), server pagination, CSV export, URL state, keyboard
nav (/ ↑↓ ↵ Esc), clinical light theme. Hits the existing API; deployed to AD2 as
public/index.redesign.html (preview at :3000/index.redesign.html). Synthesized from
Grok + Gemini concepts (both converged on this command-center design).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remote SSH/VPN to AD2 keeps flapping; hand the remaining datasheet fixes to the
local AD2 session. Includes the per-subtype approach (DSCA_TEMPLATES from staged
originals — STAGE 1 done, dsca-templates.json on AD2 = 126 models), the render-wiring
+ per-subtype byte-validation gate, Fix 5 (379 backfill via legacy_cert_text), the
discipline (backup/save-state/validate-before-publish), and the derive-dsca-templates
tool. Ref ticket #32441.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- UNREGISTERED_MODELS set: 9 model numbers not in Hoffman API catalog; skipped
silently instead of generating errors
- batch-500 fallback: when a bulk batch returns HTTP 500, retry each record
individually so good records get stamped and only truly-bad records count
as errors
- FAIL-parameter filter: records with any FAIL on a parameter line are excluded
from the push before the batch is assembled
- notify.js integration: wired in existing notification module
Files added:
- projects/dataforth-dos/database/upload-to-api.js
- projects/dataforth-dos/database/notify.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Establishes inter-session coordination for 29 projects/clients:
- Full lock/component format for active projects (dataforth-dos,
radio-show, cascades-tucson, valleywide, instrumental-music-center,
lens-auto-brokerage, msp-audit-scripts)
- Light format for complete/stalled/planning (msp-pricing, pavon,
wrightstown-*, gururmm-agent, community-forum, glaztech, etc.)
- Onboarding stubs for recently added clients
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- TEST-DATASHEET-PROCESS.md: comprehensive pipeline documentation for
Dataforth engineering (10 sections, data flow, state diagram, FAQ)
- signing-attestation/: domain ownership attestation letter with
in-place signature for Azure Trusted Signing identity validation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Installed C:\ProgramData\dataforth-uploader\ on AD2 with:
- credentials.json (SYSTEM+Administrators ACL only)
- run-pipeline.ps1 (DFWDS-process -> enumerate For_Web -> upload-delta)
- dfwds-process.js + upload-delta.js (copied from prior install dir)
- logs/ with 60-day retention
Scheduled Task 'DataforthTestDatasheetUploader' registered as SYSTEM,
hourly trigger, 30-min execution limit. First SYSTEM-context run verified:
received=7061 unchanged=7061 errors=0 in 8.7s.
Initial registration via inline base64 mangled the backslashes in the -File
argument (resulted in ERROR_DIRECTORY 0x8007010B). Fixed by running the
registration PowerShell from a file rather than an encoded command string.
Also deleted throwaway tmp/list_amtransit.py + tmp/reset_cansley.py which
had hardcoded ACG admin password.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Built the missing piece between the test datasheet pipeline and Dataforth's
new product API. End-to-end:
- Pulled DFWDS (Dataforth Web Datasheet System) VB6 source from
AD1\Engineering\ENGR\ATE\Test Datasheets\DFWDS to local for analysis
- Decoded its filename validation: A-J prefix decodes (A=10..J=19), all-
numeric WO# valid (no leading 0), anything else bad
- Ported the validation + move logic to Node (dfwds-process.js)
- Built bulk uploader (upload-delta.js) for Hoffman's Swagger API
(POST /api/v1/TestReportDataFiles/bulk with OAuth client_credentials)
Sanitized 3 prior reference scripts (fetch-server-inventory, test-scenarios,
test-upload-two) to read CF_* env vars instead of hardcoded creds.
Live drain results:
- 897 files moved Test_Datasheets -> For_Web (all valid, no renames, no
bad), DFWDS port summary in 1.1s
- Pushed entire For_Web (7,061 files) to Hoffman API in 49.7s @ 142/s:
Created=803 Updated=114 Unchanged=6,144 Errors=0
- Server count: 489,579 -> 490,382 (+803 net new)
Also:
- Added clients/dataforth/.gitignore to exclude plaintext Oauth.txt note
- Added clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md
(ticket write-up of 2026-04-11/12/13 IMC1 RDS removal/SQL migration work)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive record of 2026-04-11/12 work extending the Dataforth Test
Datasheet Pipeline: discovery, implementation, deploy to AD2, full
backfill of 27,937 datasheets, post-deploy regex patch for QB plain-
decimal PASS lines, and repo commit 0dd3d82.
Includes credentials, infrastructure paths, commit reference, open
items (vault hygiene, rsync coverage), and accuracy-extraction
reference logic for future sessions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extends the Test Datasheet Pipeline on AD2:C:\Shares\testdatadb to
generate web-published datasheets for the SCMVAS-Mxxx (obsolete) and
SCMHVAS-Mxxxx (replacement) High Voltage Input Module product lines.
Both are tested either with the existing TESTHV3 software (production
VASLOG .DAT logs) or in Engineering with plain .txt output.
Key changes on AD2 (all deployed 2026-04-12 with dated backups):
- parsers/spec-reader.js: getSpecs() returns a `{_family:'SCMVAS',
_noSpecs:true}` sentinel for SCMVAS/SCMHVAS/VAS-M/HVAS-M model prefixes
so the export pipeline does not silently skip them for missing specs.
- templates/datasheet-exact.js: new Accuracy-only template branch
(generateSCMVASDatasheet + helpers) that mirrors the existing shipped
format byte-for-byte. Extraction regex covers both QuickBASIC STR$()
output formats: scientific-with-trailing-status-digit (98.4% of
records) and plain-decimal (1.6% of records above QB's threshold).
- parsers/vaslog-engtxt.js (new): parses the Engineering-Tested .txt
files in TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\. Filename SN
regex strips optional trailing 14-digit timestamp; in-file "SN:"
header is the authoritative source when the filename is malformed.
- database/import.js: LOG_TYPES grows a VASLOG_ENG entry with
subfolder + recursive flags. Pre-existing 7 log types keep their
implicit recursive=true behaviour (config.recursive !== false).
importFiles() routes VASLOG_ENG paths before the generic loop so a
VASLOG - Engineering Tested/*.txt path does not mis-dispatch to the
multiline parser.
- database/export-datasheets.js: VASLOG_ENG records are written
verbatim via fs.copyFileSync(source_file, For_Web/<SN>.TXT) for true
byte-level pass-through, with a graceful raw_data fallback when the
source file is no longer on disk.
Deploy outcome:
- 27,503 SCMVAS/SCMHVAS datasheets rendered (27,065 from scientific +
438 from plain-decimal PASS lines, post-patch rerun)
- 434 Engineering-Tested .txt files pass-through-copied to For_Web
- 0 errors across both batches
Repo layout added here:
- scmvas-hvas-research/: discovery artifacts (source .BAS, hvin.dat,
sample .DAT + .txt, binary-format notes, IMPLEMENTATION_PLAN.md)
- implementation/: staged final code + deploy helpers + local test
harness + per-step verification scripts
- backups/pre-deploy-20260412/: independent local snapshot of the 4
AD2 files replaced, pulled byte-for-byte before deploy
All helper scripts fetch the AD2 password at runtime from the SOPS
vault (clients/dataforth/ad2.sops.yaml). None of the committed files
contain the plaintext credential. Known vault-entry hygiene issue
(stale shell-escape backslash before the `!`) is documented in the
fetcher comments and stripped at read-time; flagged separately for
cleanup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DOS 6.22 MD fails on existing dirs, then COPY creates a file instead of
copying into a directory. Dirs now pre-created on NAS server side.
Also removed trailing backslash from COPY target path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mixed-case paths caused "invalid directory" on TS-27. All paths now uppercase
to match DOS convention and Samba default case = upper setting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
No conditionals, no DEL. Just MD dirs and COPY. Deployed to AD2 and synced to D2TESTNAS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CTONWTXT.BAT was never being called. Now invoked from CTONW.BAT on every boot.
Destination changed from X: (broken Novell serve.sys check) to T:\Stage\%MACHINE%\.
Removed IF NOT clauses for DOS 6.22 compatibility. Deployed to AD2 and synced to D2TESTNAS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added EXE copy from Ate\ProdSW to C:\ATE in NWTOC.BAT
- Added /Y overwrite flag to all COPY commands
- Removed cyclic DATA folder copies from NWTOC
- Session log for 2026-03-16 DF DOS troubleshooting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Problem: Data syncs to AD2 but import.js not importing to database
- Newest DB record stuck at 2026-01-19 (2 months stale)
- rsync transfers working (9.8MB at 06:30 & 07:15)
- import.js not being called or failing silently
Instructions for PC to diagnose and fix.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
DEPLOY v4.1 fixes critical bug where ECHO >> redirects included
trailing space in MACHINE variable, causing "Too many parameters"
on all COPY commands with subdirectory paths. TS-4L data upload
confirmed working - 84 test files + 90 reports on NAS.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DOS 6.22 IF EXIST with wildcards on SMB1 network paths causes
"Bad command" and "Too many parameters" errors. Rewrote CTONW,
NWTOC, and AUTOEXEC to v4.0 with direct COPY/MD commands.
Pre-created all station LOGS directories on new D2TESTNAS.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
