claudetools/clients/dataforth/session-logs/project_pipeline_rebuilt.md at b626f0dfca634a8fb226cae67da863a4c9727f2f

Files

OC-5070 ece3222d3a Add AD1 session data, memory entries for datasheet pipeline and security incident

- Imported AD1 Claude session files to clients/dataforth/session-logs/
- Created memory: project_datasheet_pipeline.md (full pipeline architecture)
- Created memory: project_dataforth_incident_2026-03-27.md (security incident + MFA)
- Updated MEMORY.md index
- Updated session log with AD1 pipeline rebuild findings

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-27 20:07:20 -07:00

2.6 KiB

Raw Blame History

name, description, type

name	description	type
Test datasheet pipeline rebuilt 2026-03-27	New server-side pipeline replaces CTONWTXT/DFWDS/TestDataSheetUploader. Generates exact-match TXT from DAT imports. 1470 model specs loaded.	project

What Was Built (2026-03-27)

All files in C:\Shares\testdatadb:

New Files

parsers/spec-reader.js — Reads 8 binary spec DAT files, 1470 models across 6 families (SCM5B, 8B, DSCA, DSCT, SCM7B, plus sub-families 5B45, 5B48)
templates/datasheet-exact.js — Exact-match TXT formatter reverse-engineered from QuickBASIC DATASHEETWRITE
database/export-datasheets.js — Standalone/library export script, writes {SN}.TXT to X:\For_Web
database/archive-for-web.js — Archives old For_Web files into year subfolders
database/generate-customer-pdfs.js — One-off script for Quatronix customer order (72/73 generated)
specdata/ — Binary spec DAT files (5BMAIN, 8BMAIN, DSCOUT, SCTMAIN, DSCMAIN4, 7BMAIN, 5B45DATA, DB5B48)

Modified Files

database/import.js — Post-import hook calls export-datasheets after incremental imports
database/schema.sql — Added datasheet_exported_at and forweb_exported_at columns
routes/api.js — Added /api/datasheet/:id/pdf endpoint (pdfkit)
public/index.html — Added ALL filter, auto-search on load, PDF download button, export status in detail view
package.json — Added pdfkit dependency

Database Changes

datasheet_exported_at — When TXT was generated
forweb_exported_at — When file was confirmed in X:\For_Web
1,436,007 records pre-marked as exported (matched existing For_Web files)

Pipeline Flow

DAT file arrives via NAS sync → import.js parses to DB → export-datasheets generates {SN}.TXT → X:\For_Web

What's Eliminated

CTONWTXT.BAT (DOS-side TXT transfer) — no longer needed
DFWDS.exe (filename decoder/validator) — no longer needed
TestDataSheetUploader (VB.NET web uploader) — endpoints are 404, needs replacement

Remaining Gaps

Website upload: old ASP.NET endpoints dead (404), need new upload mechanism
SCM5B49 spec file empty (0 bytes) — need John's help
7B datasheet formatting not yet added to datasheet-exact.js (7B has 30+ unique parameters)
Service runs as SYSTEM causing permission issues with SHM/WAL files
500K files archived into year subfolders in X:\For_Web

Why: Rebuilt to recover from crypto wipe and unblock customer shipments (Quatronix/China blocking on missing datasheets). How to apply: For new test data, the pipeline is automatic via Sync-FromNAS. For the 830K backlog of unexported records, run node database/export-datasheets.js --limit N.

2.6 KiB Raw Blame History