claudetools/projects/dataforth-dos/TEST-DATASHEET-PROCESS.md

Test Datasheet Pipeline — Process Documentation

Audience: Dataforth Engineering Last Updated: 2026-04-15 System Owner: AZ Computer Guru (Mike Swanson) Environment: Dataforth Tucson production

---

1. Executive Summary

The Test Datasheet Pipeline captures every passing unit tested on Dataforth's test stations, converts the raw test log into a formatted datasheet document, pushes that document to Dataforth's public-facing product website, and gives internal staff a dashboard to search, review, and manually push records as needed.

As of 2026-04-15 the system contains 469,009 unique test records covering 192 SCM7B models, 238 SCM5B models, 129 8B models, 214 DSCA models, and 37 DSCT models. Of those:

• 458,501 are live on the Dataforth website (Hoffman Product API)
• 10,508 are retained internally but not on the website, broken down as:
  • 7,905 — unit tested PASS but no model specs available locally to render the datasheet (waiting on spec data)
  • 2,426 — Hoffman API rejected (data quality issue to investigate per record)
  • 177 — unit failed the test (by engineering policy, FAIL records never appear on the website)

The website total (661,367 records) exceeds our internal database because the website also holds historical units from before the current testdatadb system existed. That legacy data was uploaded by prior tools (DFWDS) and is not reproducible from our current DB.

---

2. System Architecture

2.1 Component Overview

  ┌───────────────────────┐       ┌────────────────────────┐
  │  Test Stations        │       │  Legacy DFWDS (VB6)    │
  │  TS-01 ... TS-27      │       │  (watched .dat files,  │
  │  produce .dat log     │       │   produced For_Web     │
  │  files per test run   │       │   .TXT files)          │
  └──────────┬────────────┘       └──────────┬─────────────┘
             │                                │
             │ writes to                      │ (historical, superseded
             ▼                                 │  by this pipeline)
  ┌──────────────────────────────────────────▼─────────────┐
  │  AD1 HISTLOGS Share                                    │
  │  C:\Shares\test\Ate\HISTLOGS\{log_type}\{model}.DAT    │
  │  Per-station mirrors: \TS-XX\LOGS\{log_type}\          │
  └──────────┬─────────────────────────────────────────────┘
             │ scanned by
             ▼
  ┌────────────────────────────────────────────────────────┐
  │  testdatadb service (AD2, 192.168.0.6:3000)            │
  │  - Node.js + Express API                               │
  │  - PostgreSQL 18 backend                               │
  │  - Web dashboard at http://192.168.0.6:3000/           │
  │  - WinSW service wrapper                               │
  │  - Service account: INTRANET\svc_testdatadb            │
  └──────────┬─────────────────────────────────────────────┘
             │ HTTPS POST
             ▼
  ┌────────────────────────────────────────────────────────┐
  │  Dataforth Hoffman Product API                         │
  │  /api/v1/TestReportDataFiles/bulk                      │
  │  OAuth2 client_credentials                             │
  │  Serves datasheets on product pages (public website)   │
  └────────────────────────────────────────────────────────┘

2.2 Server inventory

Component	Host	Purpose
Test station logs	AD1 (SMB \\ad1\...)	Central HISTLOGS store + per-station mirrors
Application + DB	AD2 (192.168.0.6)	testdatadb Node.js service + PostgreSQL 18
For_Web legacy output	AD2 (C:\Shares\webshare\For_Web)	Historical intermediate; being phased out
Credentials	AD2 (C:\ProgramData\dataforth-uploader\credentials.json)	OAuth creds for Hoffman API, ACL'd to SYSTEM + Administrators + svc_testdatadb
Schedule fallback	AD2 (Task Scheduler: daily 02:30)	Run-as-SYSTEM safety net if real-time upload fails

---

3. Data Flow — Step by Step

3.1 Step 1: Test station produces a log file

A test station (e.g., TS-27) completes a unit test and appends a record to its local .dat file and to the central HISTLOGS .dat. File layout:

C:\Shares\test\Ate\HISTLOGS\5BLOG\48-01.dat       ← central, all TS-XX combined
C:\Shares\test\TS-27\LOGS\5BLOG\48-01.DAT         ← per-station

The .dat files are QuickBASIC random-access binary files containing fixed-width structured records. Each record includes:

• Serial number (e.g., 172789-7)
• Model number (e.g., SCM5B48-01)
• Test date + time
• Measured values (per parameter, e.g. accuracy %, linearity, supply current)
• Overall result (PASS / FAIL)
• Raw readings

Log type naming convention:

Log type	Product family	Spec file
5BLOG	SCM5B (isolated signal conditioning)	5BMAIN.DAT, 5B45DATA.DAT, 5B49_2.DAT, DB5B48.DAT
7BLOG	SCM7B	7BMAIN.DAT
8BLOG	8B	8BMAIN.DAT
DSCLOG	DSCA	DSCMAIN4.DAT, DSCOUT.DAT
SCTLOG	DSCT	SCTMAIN.DAT
PWRLOG	Power supplies	(specs embedded in parser)
VASLOG	SCMVAS (voltage/amplitude sensing)	(specs embedded)
VASLOG_ENG	SCMVAS (customer engineering variants)	(verbatim files, no template)
SHT	Short-form	(specs embedded)

3.2 Step 2: Import to database

The testdatadb service (C:\Shares\testdatadb\database\import.js) scans the HISTLOGS directories, parses the .dat binary files, and inserts each record into the PostgreSQL test_records table.

Key schema:

CREATE TABLE test_records (
    id                  SERIAL PRIMARY KEY,
    log_type            VARCHAR(20)       NOT NULL,
    model_number        VARCHAR(100)      NOT NULL,
    serial_number       VARCHAR(100)      NOT NULL,
    test_date           DATE,
    test_station        VARCHAR(50),
    overall_result      VARCHAR(10),     -- 'PASS' or 'FAIL'
    raw_data            TEXT,            -- decoded record text
    source_file         TEXT,            -- original .dat path
    work_order          VARCHAR(50),
    datasheet_exported_at  TIMESTAMPTZ,
    forweb_exported_at     TIMESTAMPTZ,  -- legacy: when For_Web .TXT was written
    api_uploaded_at        TIMESTAMPTZ,  -- when successfully pushed to Hoffman
    import_date            TIMESTAMPTZ DEFAULT NOW(),
    search_vector          tsvector,
    CONSTRAINT uq_test_records_sn UNIQUE (serial_number)
);

Uniqueness rule: one row per serial number. If a unit is re-tested, the row is updated, not duplicated.

3.3 Step 3: Conflict resolution — FAIL → PASS retest logic

Engineering directive: a unit that initially fails, gets repaired, and retests successfully should appear on the website with the passing datasheet. A unit that already passed and later shows up as failing should not be downgraded (retesting after shipping is treated as informational).

The import INSERT ... ON CONFLICT logic encodes this precisely:

INSERT INTO test_records (log_type, model_number, serial_number, test_date,
                          test_station, overall_result, raw_data, source_file)
VALUES (...)
ON CONFLICT (serial_number) DO UPDATE SET
    log_type       = EXCLUDED.log_type,
    model_number   = EXCLUDED.model_number,
    test_date      = EXCLUDED.test_date,
    test_station   = EXCLUDED.test_station,
    overall_result = EXCLUDED.overall_result,
    raw_data       = EXCLUDED.raw_data,
    source_file    = EXCLUDED.source_file,
    api_uploaded_at    = NULL,            -- force re-push on next upload run
    forweb_exported_at = NULL
WHERE test_records.overall_result = 'FAIL'
   OR (EXCLUDED.overall_result = 'PASS'
       AND EXCLUDED.test_date > test_records.test_date)

Behavior table:

Existing row	New record	Result
FAIL (any date)	PASS (any date)	Row updates to PASS; website push re-queued
FAIL 2026-01-01	FAIL 2026-02-01	Row updates to newer FAIL data
PASS 2026-01-01	PASS 2026-02-01	Row updates to newer PASS; website push re-queued
PASS 2026-02-01	PASS 2026-01-01	Ignored (stale import)
PASS 2026-02-01	FAIL 2026-03-01	Ignored (PASS stays; FAIL not shown)

Verified with 5 scenario tests 2026-04-15.

Important: because api_uploaded_at is cleared on any row update, the next upload run automatically re-pushes the record so the website gets the fresh data.

3.4 Step 4: Render the datasheet (in-memory)

When a record needs to be pushed to the website, the testdatadb service renders the formatted datasheet text in memory from the DB row.

Code: C:\Shares\testdatadb\database\render-datasheet.js (~30 lines)

const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
const { generateExactDatasheet } = require('../templates/datasheet-exact');

function renderContent(record) {
    if (record.log_type === 'VASLOG_ENG') {
        return record.raw_data || null;        // verbatim customer file
    }
    const specs = getSpecs(loadAllSpecs(), record.model_number);
    if (!specs) return null;                    // missing specs → skip
    return generateExactDatasheet(record, specs) || null;
}

The generated text matches the legacy QuickBASIC DATASHEETWRITE output byte-for-byte. Example output header for 172789-7 (SCM5B48-01):

    DATAFORTH CORPORATION                     Phone: (520) 741-1404
    3331 E. Hemisphere Loop                   Fax:   (520) 741-0762
    Tucson, AZ  85706  USA                    email: info@dataforth.com

                            TEST DATA SHEET
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Date: 12-02-2024
    Model: SCM5B48-01
    SN:    172789-7

                             ACCURACY TEST
                Calculated       Measured
    Vin (mV)     Vout (V)        Vout (V)*     Error (%)    Status
    =======     ==========     ==========     =========    ========
    ...

Historical note: Previously, the service wrote these files to C:\Shares\webshare\For_Web\{SN}.TXT via export-datasheets.js. That filesystem intermediate has been retired — datasheets are now rendered directly from the DB at upload time. The For_Web directory still exists for backward compatibility with legacy consumers but the testdatadb upload path no longer depends on it.

3.5 Step 5: Upload to Hoffman API

Code: C:\Shares\testdatadb\database\upload-to-api.js

Endpoint: POST {CF_API_BASE}/api/v1/TestReportDataFiles/bulk

Authentication: OAuth2 client_credentials flow. Token cached for its advertised lifetime minus 60s leeway. Refresh on 401.

Payload shape:

{
  "Items": [
    { "SerialNumber": "172789-7", "Content": "    DATAFORTH CORPORATION ..." },
    { "SerialNumber": "172789-8", "Content": "..." }
  ]
}

Batches of 100 items per HTTP request. Each request has a 120s timeout. HTTP errors and timeouts logged, but the upload path is non-throwing when called from import.js — a Hoffman outage or transient failure must not wedge the import flow.

Response shape:

{
  "TotalReceived": 100,
  "Created": 98,
  "Updated": 0,
  "Unchanged": 0,
  "Errors": [ "SN=12345-7: validation failed: ..." ]
}

Post-response bookkeeping: for every SN that was not in the Errors list, the DB stamps api_uploaded_at = NOW(). This is what drives the "on website" indicator in the dashboard.

Upload triggers — three paths:

1. Automatic on import. After import.js completes a batch insert, it calls uploadNewRecords(filePaths) which picks up PASS records from the just-imported files and pushes them.
2. Manual per-record. The web dashboard shows a PUSH button on every row; clicking it calls POST /api/upload { ids: [123] } which resolves to SNs and pushes.
3. Bulk push. Dashboard has a "PUSH TO WEB" button that pushes all selected records or (internally) all_unuploaded=true to re-run every unpushed PASS.

A fourth safety net — the daily scheduled task at 02:30 — runs C:\ProgramData\dataforth-uploader\run-pipeline.ps1 as SYSTEM to catch anything that slipped through real-time uploads. Introduced as a fallback while the real-time path was stabilizing; now mostly informational.

---

4. Dashboard User Interface

URL: http://192.168.0.6:3000/ (internal LAN only)

4.1 Search and filters

Left panel provides:

• Serial Number — LIKE-match any substring
• Work Order #
• Model Number — LIKE-match
• Result — All / Pass Only / Fail Only
• Product Line — dropdown populated from available log_type values
• Website Status — Any / On Website / Not on Website (new 2026-04-15)
• Advanced: Test Station, Date range, Full-text search in raw_data

4.2 Result rows

Each row shows Serial, Model, Date, Station, Product, Result, Actions. Visual cues:

• Pink tint — record is not on the Dataforth website (api_uploaded_at IS NULL)
• Normal styling — record is live on the website
• Tooltip on mouse-over explains status

4.3 Row actions

• VIEW — expand full record detail modal
• SHEET — preview the rendered datasheet (same text that was/would be sent to the website)
• PUSH / RE-PUSH — send this record to the website. Label says "PUSH" for records never uploaded, "RE-PUSH" for records already on the website (updates the stored copy).
• Button disabled for records where overall_result != 'PASS' (website only accepts PASS units, by policy)

4.4 Bulk actions

• Checkbox on each row + Select Page checkbox
• PUSH TO WEB bulk button — pushes every selected record in one API call

4.5 Export

• EXPORT CSV — exports the current filtered result set as CSV for offline review

---

5. Operational Notes

5.1 Service and account context

• Service name: testdatadb
• Service wrapper: WinSW (C:\Shares\testdatadb\daemon\testdatadb.exe)
• Run-as account: INTRANET\svc_testdatadb
• Auto-start: Automatic (Delayed Start recommended but not required)
• Logs: C:\Shares\testdatadb\logs\testdatadb.out.log, .err.log, .wrapper.log

5.2 Credentials

• File: C:\ProgramData\dataforth-uploader\credentials.json (JSON with CF_TOKEN_URL, CF_API_BASE, CF_CLIENT_ID, CF_CLIENT_SECRET, CF_SCOPE)
• ACL: SYSTEM (FullControl), Administrators (FullControl), INTRANET\svc_testdatadb (Read). Nobody else.
• Source of truth: same file used by both the testdatadb real-time path and the daily scheduled task fallback. Never duplicate creds into the service config.

5.3 Database connection

• Local PostgreSQL on AD2
• Environment variables (with defaults):
  • PGHOST=localhost
  • PGPORT=5432
  • PGUSER=testdatadb_app
  • PGDATABASE=testdatadb
  • PGPASSWORD — set in service config

5.4 Key file locations on AD2

Path	Purpose
C:\Shares\testdatadb\database\import.js	Parses .dat files, inserts to DB
C:\Shares\testdatadb\database\upload-to-api.js	Pushes records to Hoffman
C:\Shares\testdatadb\database\render-datasheet.js	In-memory datasheet rendering
C:\Shares\testdatadb\database\export-datasheets.js	Legacy For_Web writer (retained for compat)
C:\Shares\testdatadb\routes/api.js	HTTP API (search, upload, datasheet preview)
C:\Shares\testdatadb\public\index.html	Dashboard UI
C:\Shares\testdatadb\parsers\	.dat binary parsers per log type
C:\Shares\testdatadb\specdata\	QuickBASIC spec files (5BMAIN.DAT, 7BMAIN.DAT, etc.)
C:\Shares\testdatadb\templates\datasheet-exact.js	Formatter that replicates the QuickBASIC output
C:\Shares\webshare\For_Web\	Legacy intermediate output directory
C:\ProgramData\dataforth-uploader\	Scheduled task payload + credentials

---

6. Record State Diagram

          ┌─────────────────────────────────────────────────────┐
          │                                                     │
    Incoming .dat record                                        │
          │                                                     │
          │ parse, insert (ON CONFLICT (serial_number))         │
          ▼                                                     │
     ┌────────────┐         re-test, new test_date newer        │
     │  In DB     │◀─────────────────────────────────────┐      │
     │  PASS/FAIL │                                      │      │
     └─────┬──────┘                                      │      │
           │                                             │      │
   PASS? ──┴── FAIL?                                     │      │
     │          │                                        │      │
     │          └─▶ Stays internal, never uploaded ──────┘      │
     │              (can become PASS on retest)                 │
     │                                                          │
     ▼                                                          │
  render via render-datasheet.js                                │
     │                                                          │
     │ specs missing? ──▶ skipped; tracked as "unpushable" ─────┘
     │
     ▼
  upload-to-api.js → POST /bulk
     │
     │ Hoffman response
     ▼
  ┌───────────────┬──────────────┬──────────────┐
  │ Created       │ Unchanged    │ Errors       │
  │               │              │              │
  │ stamp         │ stamp        │ no stamp;    │
  │ api_uploaded  │ api_uploaded │ tracked in   │
  │ _at = NOW()   │ _at = NOW()  │ logs; retry  │
  └───────┬───────┴──────┬───────┴──────┬───────┘
          │              │              │
          └──────────┬───┘              │
                     │                  │
                     ▼                  ▼
             ON DATAFORTH WEBSITE    NOT ON WEBSITE
             (pink tint off)         (pink tint on, PUSH button live)

---

7. Troubleshooting & FAQ

7.1 "A unit I know passed isn't showing up on the website"

1. Search the dashboard for the SN. Is it in testdatadb at all?
   • Not in DB: the import hasn't picked up that file. Check the source .dat path and the import logs. Re-run the import manually if needed.
   • In DB but overall_result=FAIL: by engineering policy, FAILs don't go to the website. If the unit has been retested and passes, make sure the retest .dat file has been imported — the FAIL→PASS rule will then update the record.
2. In DB with overall_result=PASS, api_uploaded_at=NULL: the record hasn't been pushed. Check the dashboard: hover on PUSH button for tooltip. If it says "Only PASS records can be pushed", the row is FAIL (verify). Otherwise click PUSH manually.
3. Push attempt fails with "skipped" in service logs: the model has no spec file, so we can't render the datasheet. Either the spec data is missing from C:\Shares\testdatadb\specdata\ or this is a new model variant that hasn't been added to the spec files. Action: get the spec data from engineering and drop into the specdata folder; restart testdatadb service; record will then be pushable.
4. Push attempt fails with Hoffman error: check logs. Usually a data quality issue (malformed field, missing required measurement). Fix the source record or accept that this specific unit won't be on the website.

7.2 "A unit is on the website but shouldn't be"

By engineering policy, only PASS units should be on the website. If a FAIL has made it:

1. Look up the SN history on Hoffman's API. Most likely the unit passed a prior test and was pushed; later failure won't retroactively remove it.
2. If engineering wants the record retracted from the website, that's a direct call to Hoffman's API — outside the testdatadb pipeline.

We also observed about 8% of our local FAIL records are on Hoffman. In every sampled case this is the legitimate pattern: unit passed originally (pushed), was later re-tested and failed (kept locally as FAIL), but the original PASS datasheet is still on the website. If the unit needs retraction, do it at the website end.

7.3 "The dashboard shows counts that don't match Hoffman"

The local dashboard shows our DB state. Hoffman's site contains our local set plus a large pre-testdatadb historical set (202,866 records as of 2026-04-15) that was uploaded by prior tools and is not reproducible from our current DB. This is expected — don't try to reconcile the two totals; check specific SNs instead.

7.4 "How do I force a re-push of an already-on-web record?"

Click RE-PUSH on the row. This sends the record to Hoffman again — useful if:

• The local record was corrected and you want the website copy updated
• You suspect the website copy is wrong
• Engineering asked for a refresh

Hoffman returns Unchanged if the content is identical — so RE-PUSH is idempotent and safe to run casually.

7.5 "How do I check if a specific SN is on the website?"

Three options:

1. Dashboard: Search for the SN. Row styling + tooltip shows status.
2. Direct API: GET https://{CF_API_BASE}/api/v1/TestReportDataFiles/{SN} with a bearer token. Returns 200 + Content if on website, 404 if not.
3. Public product page: navigate to the Dataforth product page for that SN; if the datasheet link works, it's on the site.

---

8. Known Limitations

• No real-time filesystem watcher. The import currently runs on a schedule (or manual trigger). Records appear in the DB minutes after tests complete, not seconds. Good enough for business purposes.
• VASLOG_ENG files are shipped verbatim — the customer-engineering variant has non-standard formatting that varies per customer. We store and push the original file contents rather than reformatting. Implication: the website datasheet for these units matches exactly what engineering produced, no template regeneration.
• Spec data is QuickBASIC binary. New product families or variants require dropping updated .DAT spec files into C:\Shares\testdatadb\specdata\. The modern Node.js code reads the QB binary format directly — no conversion needed — but the files themselves still have to come from the legacy spec authoring tools (or be manually edited with a binary-aware editor).
• Pre-testdatadb historical records. The ~203K units that live on Hoffman from the DFWDS era don't exist in our DB. We can't re-render those datasheets from current state. If one of those legacy records has a data issue, the fix needs to be made directly at Hoffman.

---

9. Recent Changes (2026-04-15 Release)

For context on what's new if engineering has seen earlier documentation:

1. Database deduplicated — one row per serial number (was permitting multiple). Kept the best record per SN by state-priority (on-web > on-filesystem > newer test date).
2. Unique constraint on serial_number added and enforced.
3. FAIL→PASS retest rule formalized in the ON CONFLICT logic. Previously a FAIL record could linger forever even after a successful retest; now the retest correctly replaces it.
4. For_Web filesystem dependency eliminated for the upload path. Datasheets render in memory directly from the DB. Phantom forweb_exported_at stamps (rows that claimed a file existed but the file was gone) are no longer a source of push failures.
5. Dashboard UI upgrades:
   • Row coloring indicates on-website status at a glance
   • Per-row PUSH / RE-PUSH buttons
   • Bulk PUSH TO WEB button
   • Website Status filter (Any / On Website / Not on Website)
6. Manual push end-to-end tested — 170,984 records (mostly SCM7B + DSCT that were historically never pushed) successfully uploaded to Hoffman in this release window.

---

10. Contacts

• System owner / technical questions: Mike Swanson, AZ Computer Guru, mike@azcomputerguru.com, (520) 304-8300
• Dashboard URL (internal): http://192.168.0.6:3000/
• Logs (internal): \\ad2\c-drive\Shares\testdatadb\logs\
• Escalation for Hoffman-side issues: Dataforth web/IT (they own the product API endpoint)