claudetools/clients/dataforth/session-logs/email-draft-updated.md

Subject: Test Datasheets to Website - Pipeline Rebuilt, Status Update & Open Questions

John, Ken,

Following up on John's email from this morning — I've completed a full analysis and rebuild of the test datasheet pipeline. Here's where things stand.

What We Found

The Complete Original Pipeline (5 stages)

1. QuickBASIC test programs write .TXT datasheets to C:\STAGE on each DOS machine
2. CTONWTXT.BAT copies C:\STAGE*.txt to X:\Test_Datasheets on boot
3. DFWDS.exe validates/renames files, moves valid ones to X:\For_Web
4. TestDataSheetUploader syncs X:\For_Web to dataforth.com via HTTP web services
5. Website serves datasheets at dataforth.com/TestDataReport

What Was Broken

• CTONWTXT.BAT is never called in the current AUTOEXEC.BAT v4.1 (deployed 2026-03-12). ARCHBAT.BAT isn't called either. TXT datasheets have been accumulating in C:\STAGE on test stations since ~Sept 2025.
• DFWDS.exe config (C:\DFWDS\DFWDS_NAMES.TXT) was lost in the crypto wipe. No scheduled task. We found the production config and VB6 source on the ENGR share at \AD1\Engineering\ENGR\ATE\Test Datasheets\DFWDS\Program.
• TestDataSheetUploader was not deployed/scheduled on AD2. The app.config still had dev paths.
• Website upload endpoints (dataforth.com/Services/Uploader.aspx, DirectoryManifest.aspx, DeleteFile.aspx) all return 404. These old ASP.NET services no longer exist on the current website.

What We Rebuilt (New Approach)

Rather than restoring the old DFWDS + TestDataSheetUploader chain, we built a new pipeline that generates datasheets directly from the DAT file data that's already being imported into the test database. This eliminates the need for CTONWTXT.BAT, DFWDS.exe, and the old VB.NET uploader entirely.

New Pipeline

1. Sync-FromNAS pulls .DAT files from the NAS every 15 minutes (already working)
2. import.js parses DAT files into the database (already working)
3. NEW: After import, the system automatically generates exact-match TXT datasheets and writes them to X:\For_Web

What Was Built (all integrated into the TestDataDB app on AD2)

Spec Parser (parsers/spec-reader.js)

• Reads the QuickBASIC binary spec files (5BMAIN.DAT, 8BMAIN.DAT, DSCOUT.DAT, SCTMAIN.DAT)
• Decodes all 4 product family TYPE structures (SCM5B, 8B, DSCA, DSCT)
• 751 model specifications loaded and available for lookup

Exact-Match Datasheet Formatter (templates/datasheet-exact.js)

• Reverse-engineered from the QuickBASIC DATASHEETWRITE subroutine source code
• Produces TXT output identical to the original format: header, accuracy test table, final test results with spec limits, footer with checklist, NIST certification
• Handles all 4 product families with correct column headers per sensor type (Vin, Temp, Iin, Rin), conditional parameters, and family-specific footers
• Spec limits displayed using the same TSPECS logic as the original QuickBASIC code

Export System (database/export-datasheets.js)

• Generates {serial_number}.TXT files in X:\For_Web
• Tracks export status via datasheet_exported_at field in the database (prevents duplicates)
• Can run standalone for batch export or automatically after each import
• Integrated into import.js as a post-import hook

On-Demand PDF Generation

• Added /api/datasheet/:id/pdf endpoint to the web app
• Users can download a PDF of any test record from the database UI
• Uses the exact-match formatter so PDFs match the original TXT layout

Database/UI Improvements

• Added datasheet_exported_at column for tracking export status
• Search now defaults to showing all records (was previously empty on load)
• Added "ALL" quick filter button
• Added "DOWNLOAD PDF" button in the record detail view

Data Reconciliation

• Cross-referenced 2.27M database records against the 501K existing files in X:\For_Web
• 1,436,006 records matched existing files and were marked as already exported
• ~830K records remain unexported (many are 7B series which we don't have spec files for yet)

For_Web Archive

• X:\For_Web had 501K files, making directory scans very slow
• Archiving pre-2026 files into year subfolders (X:\For_Web\2013, X:\For_Web\2014, etc.)
• This reduces the active folder from 501K to ~1,058 current-year files
• Archived files remain accessible but won't interfere with the upload process

What Still Needs To Be Done

1. Website Upload Replacement (NEEDS INPUT)

The old TestDataSheetUploader's web service endpoints (Uploader.aspx, DirectoryManifest.aspx, DeleteFile.aspx) are all dead — they return 404 on the current dataforth.com website.

Questions for Ken:

• When did the website change? Was the uploader working before the crypto attack?
• What does the current website use for test data lookup? The legacy site (legacy.dataforth.com/TestDataReport_Print.aspx) still works without authentication. The new site (dataforth.com/TestDataReport) requires OIDC login.
• Do we need to build a new upload mechanism, or does the website team need to provide new endpoints?
• Are the old credentials (DataforthWebShare / Data6277) still valid for anything?

2. CTONWTXT.BAT Fix on DOS Machines (NEEDS INPUT from John)

The current AUTOEXEC.BAT calls CTONW.BAT (DAT files only) but not CTONWTXT.BAT (TXT datasheets). Peter confirmed TXT files piling up in C:\STAGE since Sept 2025.

However, with the new pipeline, we may not need CTONWTXT.BAT at all — the datasheets are now generated from DAT data on the server side. The TXT files in C:\STAGE are redundant.

Question for John: Should we still fix CTONWTXT.BAT for redundancy/local backup purposes, or is the server-side generation sufficient? If we do fix it, the change is simple: add CALL C:\BAT\CTONWTXT.BAT %MACHINE% X: to CTONW.BAT and deploy via the NAS push.

3. SCM7B Spec File (OPTIONAL)

We have spec files for SCM5B, 8B, DSCA, and DSCT. The 7B series (~830K records) can't generate datasheets without a 7BMAIN.DAT or equivalent.

Question for John: Is there a 7B spec DAT file on the ENGR share? Where are the 7B test programs located?

4. TestDataDB Service Permissions

The Windows service (testdatadb) runs as SYSTEM, which creates file permission conflicts with maintenance operations. We should either:

• Change the service to run as INTRANET\sysadmin
• Or set proper inherited permissions on the testdatadb directory

Let me know your thoughts on the open questions, especially the website upload situation.

Mike