ece3222d3a
- Imported AD1 Claude session files to clients/dataforth/session-logs/ - Created memory: project_datasheet_pipeline.md (full pipeline architecture) - Created memory: project_dataforth_incident_2026-03-27.md (security incident + MFA) - Updated MEMORY.md index - Updated session log with AD1 pipeline rebuild findings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
74 lines
3.8 KiB
Markdown
74 lines
3.8 KiB
Markdown
---
|
|
name: Dataforth Test Datasheet Pipeline - Rebuilt 2026-03-27
|
|
description: Full pipeline from DOS test stations to website. New server-side generation replaces DFWDS/Uploader. 72/73 Quatronix datasheets generated. AD2 crypto wipe recovery.
|
|
type: project
|
|
---
|
|
|
|
## Background
|
|
AD2 (192.168.0.6) was wiped in a crypto/ransomware attack months ago. The test datasheet pipeline was broken. Customer Quatronix (China) blocking shipment of 328 modules (whittled to 54) without datasheets.
|
|
|
|
## Pipeline (5 stages, rebuilt 2026-03-27)
|
|
|
|
### Stage 1: DOS Test Stations (64 stations)
|
|
- QuickBASIC programs generate test data -> C:\STAGE on each DOS PC
|
|
- DAT files (raw test data) + TXT files (formatted datasheets)
|
|
- CTONW.BAT copies DAT files to NAS (working)
|
|
- CTONWTXT.BAT copies TXT files (NOT called in current AUTOEXEC v4.1 since 2026-03-12)
|
|
- TXT files piling up in C:\STAGE since Sept 2025
|
|
|
|
### Stage 2: NAS <-> AD2 Sync
|
|
- Script: C:\Shares\test\scripts\Sync-FromNAS-rsync.ps1 (every 15 min, WORKING)
|
|
- Rsync daemon on NAS: port 873, module "test", user rsync / IQ203s32119
|
|
- PULL: DAT files from NAS -> AD2, triggers database import
|
|
- PUSH: Software updates from AD2 -> NAS for DOS machines
|
|
|
|
### Stage 3: TestDataDB (Node.js/SQLite, WORKING)
|
|
- App: C:\Shares\testdatadb\ (Windows service "testdatadb", auto-start)
|
|
- API: http://192.168.0.6:3000
|
|
- Database: C:\Shares\testdatadb\database\testdata.db (2.27M records)
|
|
- Import: database/import.js (post-import hook calls export)
|
|
- **NEW: Spec parser** (parsers/spec-reader.js) - reads binary spec DATs, 1470 models
|
|
- **NEW: Exact-match formatter** (templates/datasheet-exact.js) - reverse-engineered from QB
|
|
- **NEW: Auto-export** (database/export-datasheets.js) - generates TXT to X:\For_Web
|
|
|
|
### Stage 4: WebShare (X: = \\ad2\webshare = C:\Shares\webshare)
|
|
- X:\Test_Datasheets - incoming (staging for old DFWDS)
|
|
- X:\For_Web - validated datasheets (501K+ files, pre-2026 archived to year subfolders)
|
|
- X:\For_Web_PDF - PDF versions (4.7K files)
|
|
- X:\Bad_Datasheets - invalid files (18K)
|
|
- X:\Datasheets_Log - DFWDS logs
|
|
|
|
### Stage 5: Website Upload (BROKEN)
|
|
- Old endpoints: dataforth.com/Services/{Uploader,DirectoryManifest,DeleteFile}.aspx - ALL 404
|
|
- Credentials: DataforthWebShare / Data6277
|
|
- TestDataSheetUploader (VB.NET, Hoffman) - not running, config pointed to dev paths
|
|
- Legacy site: legacy.dataforth.com/TestDataReport_Print.aspx (still works, no auth)
|
|
- New site: dataforth.com/TestDataReport (requires OIDC login)
|
|
|
|
## What Was Eliminated by Rebuild
|
|
- CTONWTXT.BAT (DOS TXT transfer) - no longer needed, server generates from DAT data
|
|
- DFWDS.exe (VB6 filename decoder) - no longer needed
|
|
- TestDataSheetUploader (VB.NET web uploader) - endpoints dead anyway
|
|
|
|
## Key File Encoding
|
|
H-prefix decode: A=10, B=11, C=12, D=13, E=14, F=15, G=16, H=17, I=18, J=19
|
|
Example: H8601-6.TXT -> serial 178601-6
|
|
New pipeline extracts SN from DAT record data directly, not filenames.
|
|
|
|
## Open Items
|
|
1. Website upload replacement (old ASP.NET endpoints dead)
|
|
2. 7B datasheet formatting (specs loaded, needs 7B-specific layout, ~830K records)
|
|
3. SCM5B49 spec file empty - need from John Lehman
|
|
4. Service permissions (runs as SYSTEM, causes SHM/WAL conflicts)
|
|
5. New product lines: MAQ20/PWRM (XLS), 10D (JSON, ~May 2026), DSCMHV
|
|
|
|
## Key Contacts
|
|
- John Lehman (jlehman@dataforth.com) - Engineering, QB code, specs
|
|
- Peter Iliya (pIliya@dataforth.com) - Applications Engineer, manual datasheet retrieval
|
|
- Ken Hoffman - TestDataSheetUploader author (VB.NET), DFWDS author, unresponsive
|
|
- Georg Haubner (ghaubner@dataforth.com) - D: drive has pre-crypto backup of network shares
|
|
- Ginger (gy@quatronix-cn.com) - Quatronix China, customer requesting datasheets
|
|
|
|
**Why:** Critical business issue - customer refusing shipments without datasheets.
|
|
**How to apply:** Pipeline is mostly rebuilt. Priority: website upload replacement, then 7B support.
|