ece3222d3a
- Imported AD1 Claude session files to clients/dataforth/session-logs/ - Created memory: project_datasheet_pipeline.md (full pipeline architecture) - Created memory: project_dataforth_incident_2026-03-27.md (security incident + MFA) - Updated MEMORY.md index - Updated session log with AD1 pipeline rebuild findings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
48 lines
2.6 KiB
Markdown
48 lines
2.6 KiB
Markdown
---
|
|
name: Test datasheet pipeline rebuilt 2026-03-27
|
|
description: New server-side pipeline replaces CTONWTXT/DFWDS/TestDataSheetUploader. Generates exact-match TXT from DAT imports. 1470 model specs loaded.
|
|
type: project
|
|
---
|
|
|
|
## What Was Built (2026-03-27)
|
|
|
|
All files in C:\Shares\testdatadb\:
|
|
|
|
### New Files
|
|
- `parsers/spec-reader.js` — Reads 8 binary spec DAT files, 1470 models across 6 families (SCM5B, 8B, DSCA, DSCT, SCM7B, plus sub-families 5B45, 5B48)
|
|
- `templates/datasheet-exact.js` — Exact-match TXT formatter reverse-engineered from QuickBASIC DATASHEETWRITE
|
|
- `database/export-datasheets.js` — Standalone/library export script, writes {SN}.TXT to X:\For_Web
|
|
- `database/archive-for-web.js` — Archives old For_Web files into year subfolders
|
|
- `database/generate-customer-pdfs.js` — One-off script for Quatronix customer order (72/73 generated)
|
|
- `specdata/` — Binary spec DAT files (5BMAIN, 8BMAIN, DSCOUT, SCTMAIN, DSCMAIN4, 7BMAIN, 5B45DATA, DB5B48)
|
|
|
|
### Modified Files
|
|
- `database/import.js` — Post-import hook calls export-datasheets after incremental imports
|
|
- `database/schema.sql` — Added datasheet_exported_at and forweb_exported_at columns
|
|
- `routes/api.js` — Added /api/datasheet/:id/pdf endpoint (pdfkit)
|
|
- `public/index.html` — Added ALL filter, auto-search on load, PDF download button, export status in detail view
|
|
- `package.json` — Added pdfkit dependency
|
|
|
|
### Database Changes
|
|
- `datasheet_exported_at` — When TXT was generated
|
|
- `forweb_exported_at` — When file was confirmed in X:\For_Web
|
|
- 1,436,007 records pre-marked as exported (matched existing For_Web files)
|
|
|
|
### Pipeline Flow
|
|
DAT file arrives via NAS sync → import.js parses to DB → export-datasheets generates {SN}.TXT → X:\For_Web
|
|
|
|
### What's Eliminated
|
|
- CTONWTXT.BAT (DOS-side TXT transfer) — no longer needed
|
|
- DFWDS.exe (filename decoder/validator) — no longer needed
|
|
- TestDataSheetUploader (VB.NET web uploader) — endpoints are 404, needs replacement
|
|
|
|
### Remaining Gaps
|
|
- Website upload: old ASP.NET endpoints dead (404), need new upload mechanism
|
|
- SCM5B49 spec file empty (0 bytes) — need John's help
|
|
- 7B datasheet formatting not yet added to datasheet-exact.js (7B has 30+ unique parameters)
|
|
- Service runs as SYSTEM causing permission issues with SHM/WAL files
|
|
- 500K files archived into year subfolders in X:\For_Web
|
|
|
|
**Why:** Rebuilt to recover from crypto wipe and unblock customer shipments (Quatronix/China blocking on missing datasheets).
|
|
**How to apply:** For new test data, the pipeline is automatic via Sync-FromNAS. For the 830K backlog of unexported records, run `node database/export-datasheets.js --limit N`.
|