Subject: RE: Test Datasheets to Website - Full Pipeline Analysis & Recovery Plan

John, Ken,

Following up on John's email - I've done a thorough trace of the entire test datasheet pipeline from the DOS test stations through to dataforth.com. Here's what I've found and what needs to be fixed.

## Current Pipeline (5 Stages)

### Stage 1 - QuickBASIC Test Programs (WORKING)
Test programs (e.g., TEST5B2E.BAS) write results two ways:
- LOGIT: binary .DAT files → C:\ATE\{5BLOG,8BLOG,...}\
- DATASHEETWRITE: .TXT datasheets → C:\STAGE\

Filename encoding via GETDSFNAME:
- 5-digit WO#: {WO#}-{dash#}.TXT (e.g., 12345-1.TXT)
- 6-digit WO# (10xxxx-19xxxx): alpha-encoded (e.g., 123456 → C3456-1.TXT)
- Invalid WO#: BAD-{dash#}.TXT

### Stage 2 - Network Transfer on Boot (BROKEN)
AUTOEXEC.BAT v4.1 (deployed 2026-03-12) calls:
1. STARTNET.BAT → maps T: = \\D2TESTNAS\test, X: = \\D2TESTNAS\datasheets
2. NWTOC.BAT → software updates (working)
3. CTONW.BAT → copies .DAT files to T:\TS-XX\LOGS\ (working)
4. menux → test menu (working)

**PROBLEM: CTONWTXT.BAT is never called.** This is the batch file that copies C:\STAGE\*.txt to X: (the datasheets share). It exists on the machines (1,504 bytes, dated 03-12-26) but nothing invokes it. The old ARCHBAT.BAT called it, but ARCHBAT is not called in the current AUTOEXEC either.

**Result: .TXT datasheets have been accumulating in C:\STAGE on the test stations since approximately September 2025.** Peter confirmed finding old files there.

**Fix:** Add the following line to CTONW.BAT (or AUTOEXEC.BAT), after the existing .DAT transfer:
    CALL C:\BAT\CTONWTXT.BAT %MACHINE% X:

This can be deployed to all 64 stations via the existing NAS software push (Sync-FromNAS pushes updated ProdSW and BAT files to the NAS, and NWTOC pulls them to each station on boot).

### Stage 3 - DFWDS.exe Processing (BROKEN)
DFWDS.exe (Dataforth Website Datasheet program, version 2015-06-08) processes files that arrive in X:\Test_Datasheets:
- Validates filenames and content
- Renames DOS-encoded filenames to serial number format
- Moves valid files → X:\For_Web (currently ~501,000 files)
- Moves invalid files → X:\Bad_Datasheets (~18,800 files)
- Logs to X:\Datasheets_Log

**PROBLEM: The config file C:\DFWDS\DFWDS_NAMES.TXT was lost in the crypto wipe.** The exe is hardcoded to read from that path. The ENGR share has a copy of the exe and the config template at \\AD1\Engineering\ENGR\ATE\Test Datasheets\DFWDS\. We also don't have a scheduled task for it on AD2.

DFWDS last ran successfully on 2026-03-11 (processed 978 files, 0 bad). Before that, there's a gap back to September 2025. The X:\Test_Datasheets folder currently has 338 files waiting to be processed.

**Fix:**
1. Create C:\DFWDS\ on AD2 and restore DFWDS_NAMES.TXT with the correct paths
2. Copy DFWDS.exe from the ENGR share
3. Create a scheduled task to run it (daily? on a schedule that matches production?)

**Question for John/Ken:** How often was DFWDS.exe previously run, and was it on a schedule or triggered manually?

### Stage 4 - TestDataSheetUploader to Website (BROKEN)
Ken's VB.NET app (TestDataSheetUploader) syncs X:\For_Web to dataforth.com via three web services:
- Uploader.aspx - uploads new/changed files
- DirectoryManifest.aspx - gets server file list for comparison
- DeleteFile.aspx - removes obsolete files from server

The app compares local files against the server manifest by filename, size, and last-write-time, and only uploads files modified in the current year.

Source is on the ENGR share at: \\AD1\Engineering\ENGR\ATE\Test Datasheets\TestDataSheetUploader\
Last built: November 2, 2022

**PROBLEM: Not deployed or scheduled on AD2. The app.config still has dev paths (C:\Users\hoffm\...) instead of production paths.**

**Fix:**
1. Update app.config TestDataSheetPath to point to X:\For_Web
2. Deploy the built exe to AD2
3. Create a scheduled task

**Question for Ken:** What was the production config for TestDataSheetPath? Was it X:\For_Web or something else? Was this run on a schedule or manually? Are the web service credentials (DataforthWebShare) still valid?

### Stage 5 - PDF Generation (UNKNOWN)
X:\For_Web_PDF contains ~4,773 PDF versions of datasheets. I haven't identified what generates these.

**Question:** What creates the PDFs? Is this a separate process we need to restore?

## What's Currently Working
- Test programs are generating data (files from today in X:\Test_Datasheets)
- NAS sync (Sync-FromNAS-rsync.ps1) runs every 15 min, pulling .DAT files and pushing software updates
- TestDataDB server is running on AD2 (port 3000), importing .DAT files
- X: drive (\\AD2\webshare) is accessible with all folders intact
- 501K+ historical datasheets in X:\For_Web are preserved

## What's Broken (in pipeline order)
1. CTONWTXT.BAT not called → TXT files stuck on DOS machines
2. DFWDS.exe config missing → files in Test_Datasheets not processed
3. TestDataSheetUploader not deployed → files in For_Web not synced to website

## Proposed Fix Order
1. Fix CTONW.BAT to call CTONWTXT.BAT → deploy via NAS push
2. Restore DFWDS on AD2 (config + exe + scheduled task)
3. Deploy TestDataSheetUploader with production config + scheduled task

Let me know your thoughts and answers to the questions above so we can start restoring this.

Mike