ece3222d3a
- Imported AD1 Claude session files to clients/dataforth/session-logs/ - Created memory: project_datasheet_pipeline.md (full pipeline architecture) - Created memory: project_dataforth_incident_2026-03-27.md (security incident + MFA) - Updated MEMORY.md index - Updated session log with AD1 pipeline rebuild findings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
99 lines
5.2 KiB
Markdown
99 lines
5.2 KiB
Markdown
Subject: RE: Test Datasheets to Website - Full Pipeline Analysis & Recovery Plan
|
|
|
|
John, Ken,
|
|
|
|
Following up on John's email - I've done a thorough trace of the entire test datasheet pipeline from the DOS test stations through to dataforth.com. Here's what I've found and what needs to be fixed.
|
|
|
|
## Current Pipeline (5 Stages)
|
|
|
|
### Stage 1 - QuickBASIC Test Programs (WORKING)
|
|
Test programs (e.g., TEST5B2E.BAS) write results two ways:
|
|
- LOGIT: binary .DAT files → C:\ATE\{5BLOG,8BLOG,...}\
|
|
- DATASHEETWRITE: .TXT datasheets → C:\STAGE\
|
|
|
|
Filename encoding via GETDSFNAME:
|
|
- 5-digit WO#: {WO#}-{dash#}.TXT (e.g., 12345-1.TXT)
|
|
- 6-digit WO# (10xxxx-19xxxx): alpha-encoded (e.g., 123456 → C3456-1.TXT)
|
|
- Invalid WO#: BAD-{dash#}.TXT
|
|
|
|
### Stage 2 - Network Transfer on Boot (BROKEN)
|
|
AUTOEXEC.BAT v4.1 (deployed 2026-03-12) calls:
|
|
1. STARTNET.BAT → maps T: = \\D2TESTNAS\test, X: = \\D2TESTNAS\datasheets
|
|
2. NWTOC.BAT → software updates (working)
|
|
3. CTONW.BAT → copies .DAT files to T:\TS-XX\LOGS\ (working)
|
|
4. menux → test menu (working)
|
|
|
|
**PROBLEM: CTONWTXT.BAT is never called.** This is the batch file that copies C:\STAGE\*.txt to X: (the datasheets share). It exists on the machines (1,504 bytes, dated 03-12-26) but nothing invokes it. The old ARCHBAT.BAT called it, but ARCHBAT is not called in the current AUTOEXEC either.
|
|
|
|
**Result: .TXT datasheets have been accumulating in C:\STAGE on the test stations since approximately September 2025.** Peter confirmed finding old files there.
|
|
|
|
**Fix:** Add the following line to CTONW.BAT (or AUTOEXEC.BAT), after the existing .DAT transfer:
|
|
CALL C:\BAT\CTONWTXT.BAT %MACHINE% X:
|
|
|
|
This can be deployed to all 64 stations via the existing NAS software push (Sync-FromNAS pushes updated ProdSW and BAT files to the NAS, and NWTOC pulls them to each station on boot).
|
|
|
|
### Stage 3 - DFWDS.exe Processing (BROKEN)
|
|
DFWDS.exe (Dataforth Website Datasheet program, version 2015-06-08) processes files that arrive in X:\Test_Datasheets:
|
|
- Validates filenames and content
|
|
- Renames DOS-encoded filenames to serial number format
|
|
- Moves valid files → X:\For_Web (currently ~501,000 files)
|
|
- Moves invalid files → X:\Bad_Datasheets (~18,800 files)
|
|
- Logs to X:\Datasheets_Log
|
|
|
|
**PROBLEM: The config file C:\DFWDS\DFWDS_NAMES.TXT was lost in the crypto wipe.** The exe is hardcoded to read from that path. The ENGR share has a copy of the exe and the config template at \\AD1\Engineering\ENGR\ATE\Test Datasheets\DFWDS\. We also don't have a scheduled task for it on AD2.
|
|
|
|
DFWDS last ran successfully on 2026-03-11 (processed 978 files, 0 bad). Before that, there's a gap back to September 2025. The X:\Test_Datasheets folder currently has 338 files waiting to be processed.
|
|
|
|
**Fix:**
|
|
1. Create C:\DFWDS\ on AD2 and restore DFWDS_NAMES.TXT with the correct paths
|
|
2. Copy DFWDS.exe from the ENGR share
|
|
3. Create a scheduled task to run it (daily? on a schedule that matches production?)
|
|
|
|
**Question for John/Ken:** How often was DFWDS.exe previously run, and was it on a schedule or triggered manually?
|
|
|
|
### Stage 4 - TestDataSheetUploader to Website (BROKEN)
|
|
Ken's VB.NET app (TestDataSheetUploader) syncs X:\For_Web to dataforth.com via three web services:
|
|
- Uploader.aspx - uploads new/changed files
|
|
- DirectoryManifest.aspx - gets server file list for comparison
|
|
- DeleteFile.aspx - removes obsolete files from server
|
|
|
|
The app compares local files against the server manifest by filename, size, and last-write-time, and only uploads files modified in the current year.
|
|
|
|
Source is on the ENGR share at: \\AD1\Engineering\ENGR\ATE\Test Datasheets\TestDataSheetUploader\
|
|
Last built: November 2, 2022
|
|
|
|
**PROBLEM: Not deployed or scheduled on AD2. The app.config still has dev paths (C:\Users\hoffm\...) instead of production paths.**
|
|
|
|
**Fix:**
|
|
1. Update app.config TestDataSheetPath to point to X:\For_Web
|
|
2. Deploy the built exe to AD2
|
|
3. Create a scheduled task
|
|
|
|
**Question for Ken:** What was the production config for TestDataSheetPath? Was it X:\For_Web or something else? Was this run on a schedule or manually? Are the web service credentials (DataforthWebShare) still valid?
|
|
|
|
### Stage 5 - PDF Generation (UNKNOWN)
|
|
X:\For_Web_PDF contains ~4,773 PDF versions of datasheets. I haven't identified what generates these.
|
|
|
|
**Question:** What creates the PDFs? Is this a separate process we need to restore?
|
|
|
|
## What's Currently Working
|
|
- Test programs are generating data (files from today in X:\Test_Datasheets)
|
|
- NAS sync (Sync-FromNAS-rsync.ps1) runs every 15 min, pulling .DAT files and pushing software updates
|
|
- TestDataDB server is running on AD2 (port 3000), importing .DAT files
|
|
- X: drive (\\AD2\webshare) is accessible with all folders intact
|
|
- 501K+ historical datasheets in X:\For_Web are preserved
|
|
|
|
## What's Broken (in pipeline order)
|
|
1. CTONWTXT.BAT not called → TXT files stuck on DOS machines
|
|
2. DFWDS.exe config missing → files in Test_Datasheets not processed
|
|
3. TestDataSheetUploader not deployed → files in For_Web not synced to website
|
|
|
|
## Proposed Fix Order
|
|
1. Fix CTONW.BAT to call CTONWTXT.BAT → deploy via NAS push
|
|
2. Restore DFWDS on AD2 (config + exe + scheduled task)
|
|
3. Deploy TestDataSheetUploader with production config + scheduled task
|
|
|
|
Let me know your thoughts and answers to the questions above so we can start restoring this.
|
|
|
|
Mike
|