Files
claudetools/projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/existing-database/schema-pg.sql
Mike Swanson 45083f4735 Add SCMVAS/SCMHVAS datasheet pipeline extension (Dataforth)
Extends the Test Datasheet Pipeline on AD2:C:\Shares\testdatadb to
generate web-published datasheets for the SCMVAS-Mxxx (obsolete) and
SCMHVAS-Mxxxx (replacement) High Voltage Input Module product lines.
Both are tested either with the existing TESTHV3 software (production
VASLOG .DAT logs) or in Engineering with plain .txt output.

Key changes on AD2 (all deployed 2026-04-12 with dated backups):

- parsers/spec-reader.js: getSpecs() returns a `{_family:'SCMVAS',
  _noSpecs:true}` sentinel for SCMVAS/SCMHVAS/VAS-M/HVAS-M model prefixes
  so the export pipeline does not silently skip them for missing specs.
- templates/datasheet-exact.js: new Accuracy-only template branch
  (generateSCMVASDatasheet + helpers) that mirrors the existing shipped
  format byte-for-byte. Extraction regex covers both QuickBASIC STR$()
  output formats: scientific-with-trailing-status-digit (98.4% of
  records) and plain-decimal (1.6% of records above QB's threshold).
- parsers/vaslog-engtxt.js (new): parses the Engineering-Tested .txt
  files in TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\. Filename SN
  regex strips optional trailing 14-digit timestamp; in-file "SN:"
  header is the authoritative source when the filename is malformed.
- database/import.js: LOG_TYPES grows a VASLOG_ENG entry with
  subfolder + recursive flags. Pre-existing 7 log types keep their
  implicit recursive=true behaviour (config.recursive !== false).
  importFiles() routes VASLOG_ENG paths before the generic loop so a
  VASLOG - Engineering Tested/*.txt path does not mis-dispatch to the
  multiline parser.
- database/export-datasheets.js: VASLOG_ENG records are written
  verbatim via fs.copyFileSync(source_file, For_Web/<SN>.TXT) for true
  byte-level pass-through, with a graceful raw_data fallback when the
  source file is no longer on disk.

Deploy outcome:
- 27,503 SCMVAS/SCMHVAS datasheets rendered (27,065 from scientific +
  438 from plain-decimal PASS lines, post-patch rerun)
- 434 Engineering-Tested .txt files pass-through-copied to For_Web
- 0 errors across both batches

Repo layout added here:
- scmvas-hvas-research/: discovery artifacts (source .BAS, hvin.dat,
  sample .DAT + .txt, binary-format notes, IMPLEMENTATION_PLAN.md)
- implementation/: staged final code + deploy helpers + local test
  harness + per-step verification scripts
- backups/pre-deploy-20260412/: independent local snapshot of the 4
  AD2 files replaced, pulled byte-for-byte before deploy

All helper scripts fetch the AD2 password at runtime from the SOPS
vault (clients/dataforth/ad2.sops.yaml). None of the committed files
contain the plaintext credential. Known vault-entry hygiene issue
(stale shell-escape backslash before the `!`) is documented in the
fetcher comments and stripped at read-time; flagged separately for
cleanup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 07:36:45 -07:00

97 lines
3.4 KiB
PL/PgSQL

-- TestDataDB PostgreSQL Schema
-- Migrated from SQLite schema.sql
-- PostgreSQL 18 on AD2 (192.168.0.6)
-- Main test records table
CREATE TABLE IF NOT EXISTS test_records (
id BIGSERIAL PRIMARY KEY,
log_type TEXT NOT NULL,
model_number TEXT NOT NULL,
serial_number TEXT NOT NULL,
test_date TEXT NOT NULL,
test_station TEXT,
overall_result TEXT,
raw_data TEXT,
source_file TEXT,
import_date TIMESTAMPTZ DEFAULT NOW(),
datasheet_exported_at TIMESTAMPTZ DEFAULT NULL,
forweb_exported_at TIMESTAMPTZ DEFAULT NULL,
work_order TEXT DEFAULT NULL,
search_vector tsvector,
UNIQUE(log_type, model_number, serial_number, test_date, test_station)
);
-- Indexes for fast searching
CREATE INDEX IF NOT EXISTS idx_serial ON test_records(serial_number);
CREATE INDEX IF NOT EXISTS idx_model ON test_records(model_number);
CREATE INDEX IF NOT EXISTS idx_date ON test_records(test_date);
CREATE INDEX IF NOT EXISTS idx_model_serial ON test_records(model_number, serial_number);
CREATE INDEX IF NOT EXISTS idx_result ON test_records(overall_result);
CREATE INDEX IF NOT EXISTS idx_log_type ON test_records(log_type);
CREATE INDEX IF NOT EXISTS idx_test_wo ON test_records(work_order);
-- Partial index for unexported PASS records (speeds up export queries)
CREATE INDEX IF NOT EXISTS idx_unexported_pass ON test_records(overall_result, forweb_exported_at)
WHERE overall_result = 'PASS' AND forweb_exported_at IS NULL;
-- GIN index for full-text search (replaces SQLite FTS5 virtual table)
CREATE INDEX IF NOT EXISTS idx_search_vector ON test_records USING GIN(search_vector);
-- Trigger function to maintain search_vector on INSERT/UPDATE
CREATE OR REPLACE FUNCTION update_search_vector() RETURNS trigger AS $$
BEGIN
NEW.search_vector := to_tsvector('english',
COALESCE(NEW.serial_number, '') || ' ' ||
COALESCE(NEW.model_number, '') || ' ' ||
COALESCE(NEW.raw_data, '')
);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Drop trigger if exists, then create
DROP TRIGGER IF EXISTS trg_search_vector ON test_records;
CREATE TRIGGER trg_search_vector
BEFORE INSERT OR UPDATE ON test_records
FOR EACH ROW
EXECUTE FUNCTION update_search_vector();
-- Work orders table
CREATE TABLE IF NOT EXISTS work_orders (
id BIGSERIAL PRIMARY KEY,
wo_number TEXT NOT NULL,
wo_date TEXT,
program TEXT,
version TEXT,
lib_version TEXT,
test_station TEXT,
source_file TEXT,
import_date TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(wo_number, test_station)
);
CREATE INDEX IF NOT EXISTS idx_wo_number ON work_orders(wo_number);
CREATE INDEX IF NOT EXISTS idx_wo_station ON work_orders(test_station);
-- Work order lines table
CREATE TABLE IF NOT EXISTS work_order_lines (
id BIGSERIAL PRIMARY KEY,
wo_number TEXT NOT NULL,
serial_number TEXT NOT NULL,
status TEXT,
model_number TEXT,
ds_filename TEXT,
test_date TEXT,
test_time TEXT,
test_station TEXT,
UNIQUE(wo_number, serial_number, test_date, test_time)
);
CREATE INDEX IF NOT EXISTS idx_wol_wo ON work_order_lines(wo_number);
CREATE INDEX IF NOT EXISTS idx_wol_serial ON work_order_lines(serial_number);
CREATE INDEX IF NOT EXISTS idx_wol_model ON work_order_lines(model_number);
-- Grant permissions to app role
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO testdatadb_app;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO testdatadb_app;