Add AD1 session data, memory entries for datasheet pipeline and security incident
- Imported AD1 Claude session files to clients/dataforth/session-logs/ - Created memory: project_datasheet_pipeline.md (full pipeline architecture) - Created memory: project_dataforth_incident_2026-03-27.md (security incident + MFA) - Updated MEMORY.md index - Updated session log with AD1 pipeline rebuild findings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
47
clients/dataforth/session-logs/project_pipeline_rebuilt.md
Normal file
47
clients/dataforth/session-logs/project_pipeline_rebuilt.md
Normal file
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: Test datasheet pipeline rebuilt 2026-03-27
|
||||
description: New server-side pipeline replaces CTONWTXT/DFWDS/TestDataSheetUploader. Generates exact-match TXT from DAT imports. 1470 model specs loaded.
|
||||
type: project
|
||||
---
|
||||
|
||||
## What Was Built (2026-03-27)
|
||||
|
||||
All files in C:\Shares\testdatadb\:
|
||||
|
||||
### New Files
|
||||
- `parsers/spec-reader.js` — Reads 8 binary spec DAT files, 1470 models across 6 families (SCM5B, 8B, DSCA, DSCT, SCM7B, plus sub-families 5B45, 5B48)
|
||||
- `templates/datasheet-exact.js` — Exact-match TXT formatter reverse-engineered from QuickBASIC DATASHEETWRITE
|
||||
- `database/export-datasheets.js` — Standalone/library export script, writes {SN}.TXT to X:\For_Web
|
||||
- `database/archive-for-web.js` — Archives old For_Web files into year subfolders
|
||||
- `database/generate-customer-pdfs.js` — One-off script for Quatronix customer order (72/73 generated)
|
||||
- `specdata/` — Binary spec DAT files (5BMAIN, 8BMAIN, DSCOUT, SCTMAIN, DSCMAIN4, 7BMAIN, 5B45DATA, DB5B48)
|
||||
|
||||
### Modified Files
|
||||
- `database/import.js` — Post-import hook calls export-datasheets after incremental imports
|
||||
- `database/schema.sql` — Added datasheet_exported_at and forweb_exported_at columns
|
||||
- `routes/api.js` — Added /api/datasheet/:id/pdf endpoint (pdfkit)
|
||||
- `public/index.html` — Added ALL filter, auto-search on load, PDF download button, export status in detail view
|
||||
- `package.json` — Added pdfkit dependency
|
||||
|
||||
### Database Changes
|
||||
- `datasheet_exported_at` — When TXT was generated
|
||||
- `forweb_exported_at` — When file was confirmed in X:\For_Web
|
||||
- 1,436,007 records pre-marked as exported (matched existing For_Web files)
|
||||
|
||||
### Pipeline Flow
|
||||
DAT file arrives via NAS sync → import.js parses to DB → export-datasheets generates {SN}.TXT → X:\For_Web
|
||||
|
||||
### What's Eliminated
|
||||
- CTONWTXT.BAT (DOS-side TXT transfer) — no longer needed
|
||||
- DFWDS.exe (filename decoder/validator) — no longer needed
|
||||
- TestDataSheetUploader (VB.NET web uploader) — endpoints are 404, needs replacement
|
||||
|
||||
### Remaining Gaps
|
||||
- Website upload: old ASP.NET endpoints dead (404), need new upload mechanism
|
||||
- SCM5B49 spec file empty (0 bytes) — need John's help
|
||||
- 7B datasheet formatting not yet added to datasheet-exact.js (7B has 30+ unique parameters)
|
||||
- Service runs as SYSTEM causing permission issues with SHM/WAL files
|
||||
- 500K files archived into year subfolders in X:\For_Web
|
||||
|
||||
**Why:** Rebuilt to recover from crypto wipe and unblock customer shipments (Quatronix/China blocking on missing datasheets).
|
||||
**How to apply:** For new test data, the pipeline is automatic via Sync-FromNAS. For the 830K backlog of unexported records, run `node database/export-datasheets.js --limit N`.
|
||||
Reference in New Issue
Block a user