azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dbf4325c46f4fa077ba234c5441e12cd08a4e5d0
claudetools/projects/dataforth-dos/TEST-DATASHEET-PROCESS.md
Mike Swanson f01d9d5538 Add Dataforth process docs + Azure signing attestation letter 
- TEST-DATASHEET-PROCESS.md: comprehensive pipeline documentation for
  Dataforth engineering (10 sections, data flow, state diagram, FAQ)
- signing-attestation/: domain ownership attestation letter with
  in-place signature for Azure Trusted Signing identity validation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 19:33:06 -07:00

474 lines
25 KiB
Markdown
Raw Blame History

# Test Datasheet Pipeline — Process Documentation
**Audience:** Dataforth Engineering
**Last Updated:** 2026-04-15
**System Owner:** AZ Computer Guru (Mike Swanson)
**Environment:** Dataforth Tucson production
---
## 1. Executive Summary
The **Test Datasheet Pipeline** captures every passing unit tested on Dataforth's test stations, converts the raw test log into a formatted datasheet document, pushes that document to Dataforth's public-facing product website, and gives internal staff a dashboard to search, review, and manually push records as needed.
As of 2026-04-15 the system contains **469,009 unique test records** covering **192 SCM7B models, 238 SCM5B models, 129 8B models, 214 DSCA models, and 37 DSCT models**. Of those:
- **458,501 are live on the Dataforth website** (Hoffman Product API)
- **10,508 are retained internally** but not on the website, broken down as:
- 7,905 — unit tested PASS but no model specs available locally to render the datasheet (waiting on spec data)
- 2,426 — Hoffman API rejected (data quality issue to investigate per record)
- 177 — unit failed the test (by engineering policy, FAIL records never appear on the website)
The website total (661,367 records) exceeds our internal database because the website also holds historical units from **before the current testdatadb system existed**. That legacy data was uploaded by prior tools (DFWDS) and is not reproducible from our current DB.
---
## 2. System Architecture
### 2.1 Component Overview
```
┌───────────────────────┐ ┌────────────────────────┐
│ Test Stations │ │ Legacy DFWDS (VB6) │
│ TS-01 ... TS-27 │ │ (watched .dat files, │
│ produce .dat log │ │ produced For_Web │
│ files per test run │ │ .TXT files) │
└──────────┬────────────┘ └──────────┬─────────────┘
│ │
│ writes to │ (historical, superseded
▼ │ by this pipeline)
┌──────────────────────────────────────────▼─────────────┐
│ AD1 HISTLOGS Share │
│ C:\Shares\test\Ate\HISTLOGS\{log_type}\{model}.DAT │
│ Per-station mirrors: \TS-XX\LOGS\{log_type}\ │
└──────────┬─────────────────────────────────────────────┘
│ scanned by
▼
┌────────────────────────────────────────────────────────┐
│ testdatadb service (AD2, 192.168.0.6:3000) │
│ - Node.js + Express API │
│ - PostgreSQL 18 backend │
│ - Web dashboard at http://192.168.0.6:3000/ │
│ - WinSW service wrapper │
│ - Service account: INTRANET\svc_testdatadb │
└──────────┬─────────────────────────────────────────────┘
│ HTTPS POST
▼
┌────────────────────────────────────────────────────────┐
│ Dataforth Hoffman Product API │
│ /api/v1/TestReportDataFiles/bulk │
│ OAuth2 client_credentials │
│ Serves datasheets on product pages (public website) │
└────────────────────────────────────────────────────────┘
```
### 2.2 Server inventory
| Component | Host | Purpose |
|---|---|---|
| Test station logs | AD1 (SMB `\\ad1\...`) | Central HISTLOGS store + per-station mirrors |
| Application + DB | AD2 (192.168.0.6) | testdatadb Node.js service + PostgreSQL 18 |
| For_Web legacy output | AD2 (`C:\Shares\webshare\For_Web`) | Historical intermediate; being phased out |
| Credentials | AD2 (`C:\ProgramData\dataforth-uploader\credentials.json`) | OAuth creds for Hoffman API, ACL'd to SYSTEM + Administrators + svc_testdatadb |
| Schedule fallback | AD2 (Task Scheduler: daily 02:30) | Run-as-SYSTEM safety net if real-time upload fails |
---
## 3. Data Flow — Step by Step
### 3.1 Step 1: Test station produces a log file
A test station (e.g., TS-27) completes a unit test and appends a record to its local `.dat` file *and* to the central HISTLOGS `.dat`. File layout:
```
C:\Shares\test\Ate\HISTLOGS\5BLOG\48-01.dat ← central, all TS-XX combined
C:\Shares\test\TS-27\LOGS\5BLOG\48-01.DAT ← per-station
```
The `.dat` files are **QuickBASIC random-access binary files** containing fixed-width structured records. Each record includes:
- Serial number (e.g., `172789-7`)
- Model number (e.g., `SCM5B48-01`)
- Test date + time
- Measured values (per parameter, e.g. accuracy %, linearity, supply current)
- Overall result (PASS / FAIL)
- Raw readings
Log type naming convention:
| Log type | Product family | Spec file |
|---|---|---|
| 5BLOG | SCM5B (isolated signal conditioning) | 5BMAIN.DAT, 5B45DATA.DAT, 5B49_2.DAT, DB5B48.DAT |
| 7BLOG | SCM7B | 7BMAIN.DAT |
| 8BLOG | 8B | 8BMAIN.DAT |
| DSCLOG | DSCA | DSCMAIN4.DAT, DSCOUT.DAT |
| SCTLOG | DSCT | SCTMAIN.DAT |
| PWRLOG | Power supplies | (specs embedded in parser) |
| VASLOG | SCMVAS (voltage/amplitude sensing) | (specs embedded) |
| VASLOG_ENG | SCMVAS (customer engineering variants) | (verbatim files, no template) |
| SHT | Short-form | (specs embedded) |
### 3.2 Step 2: Import to database
The testdatadb service (`C:\Shares\testdatadb\database\import.js`) scans the HISTLOGS directories, parses the `.dat` binary files, and inserts each record into the PostgreSQL `test_records` table.
**Key schema:**
```sql
CREATE TABLE test_records (
id SERIAL PRIMARY KEY,
log_type VARCHAR(20) NOT NULL,
model_number VARCHAR(100) NOT NULL,
serial_number VARCHAR(100) NOT NULL,
test_date DATE,
test_station VARCHAR(50),
overall_result VARCHAR(10), -- 'PASS' or 'FAIL'
raw_data TEXT, -- decoded record text
source_file TEXT, -- original .dat path
work_order VARCHAR(50),
datasheet_exported_at TIMESTAMPTZ,
forweb_exported_at TIMESTAMPTZ, -- legacy: when For_Web .TXT was written
api_uploaded_at TIMESTAMPTZ, -- when successfully pushed to Hoffman
import_date TIMESTAMPTZ DEFAULT NOW(),
search_vector tsvector,
CONSTRAINT uq_test_records_sn UNIQUE (serial_number)
);
```
**Uniqueness rule:** one row per serial number. If a unit is re-tested, the row is **updated**, not duplicated.
### 3.3 Step 3: Conflict resolution — FAIL → PASS retest logic
Engineering directive: a unit that initially fails, gets repaired, and retests successfully should appear on the website with the **passing** datasheet. A unit that already passed and later shows up as failing should **not** be downgraded (retesting after shipping is treated as informational).
The import `INSERT ... ON CONFLICT` logic encodes this precisely:
```sql
INSERT INTO test_records (log_type, model_number, serial_number, test_date,
test_station, overall_result, raw_data, source_file)
VALUES (...)
ON CONFLICT (serial_number) DO UPDATE SET
log_type = EXCLUDED.log_type,
model_number = EXCLUDED.model_number,
test_date = EXCLUDED.test_date,
test_station = EXCLUDED.test_station,
overall_result = EXCLUDED.overall_result,
raw_data = EXCLUDED.raw_data,
source_file = EXCLUDED.source_file,
api_uploaded_at = NULL, -- force re-push on next upload run
forweb_exported_at = NULL
WHERE test_records.overall_result = 'FAIL'
OR (EXCLUDED.overall_result = 'PASS'
AND EXCLUDED.test_date > test_records.test_date)
```
**Behavior table:**
| Existing row | New record | Result |
|---|---|---|
| FAIL (any date) | PASS (any date) | Row updates to PASS; website push re-queued |
| FAIL 2026-01-01 | FAIL 2026-02-01 | Row updates to newer FAIL data |
| PASS 2026-01-01 | PASS 2026-02-01 | Row updates to newer PASS; website push re-queued |
| PASS 2026-02-01 | PASS 2026-01-01 | Ignored (stale import) |
| PASS 2026-02-01 | FAIL 2026-03-01 | Ignored (PASS stays; FAIL not shown) |
Verified with 5 scenario tests 2026-04-15.
**Important:** because `api_uploaded_at` is cleared on any row update, the next upload run automatically re-pushes the record so the website gets the fresh data.
### 3.4 Step 4: Render the datasheet (in-memory)
When a record needs to be pushed to the website, the testdatadb service renders the formatted datasheet text **in memory** from the DB row.
Code: `C:\Shares\testdatadb\database\render-datasheet.js` (~30 lines)
```javascript
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
const { generateExactDatasheet } = require('../templates/datasheet-exact');
function renderContent(record) {
if (record.log_type === 'VASLOG_ENG') {
return record.raw_data || null; // verbatim customer file
}
const specs = getSpecs(loadAllSpecs(), record.model_number);
if (!specs) return null; // missing specs → skip
return generateExactDatasheet(record, specs) || null;
}
```
The generated text matches the legacy QuickBASIC DATASHEETWRITE output byte-for-byte. Example output header for `172789-7` (SCM5B48-01):
```
DATAFORTH CORPORATION Phone: (520) 741-1404
3331 E. Hemisphere Loop Fax: (520) 741-0762
Tucson, AZ 85706 USA email: info@dataforth.com
TEST DATA SHEET
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Date: 12-02-2024
Model: SCM5B48-01
SN: 172789-7
ACCURACY TEST
Calculated Measured
Vin (mV) Vout (V) Vout (V)* Error (%) Status
======= ========== ========== ========= ========
...
```
**Historical note:** Previously, the service wrote these files to `C:\Shares\webshare\For_Web\{SN}.TXT` via `export-datasheets.js`. That filesystem intermediate has been retired — datasheets are now rendered directly from the DB at upload time. The For_Web directory still exists for backward compatibility with legacy consumers but the testdatadb upload path no longer depends on it.
### 3.5 Step 5: Upload to Hoffman API
Code: `C:\Shares\testdatadb\database\upload-to-api.js`
**Endpoint:** `POST {CF_API_BASE}/api/v1/TestReportDataFiles/bulk`
**Authentication:** OAuth2 client_credentials flow. Token cached for its advertised lifetime minus 60s leeway. Refresh on 401.
**Payload shape:**
```json
{
"Items": [
{ "SerialNumber": "172789-7", "Content": " DATAFORTH CORPORATION ..." },
{ "SerialNumber": "172789-8", "Content": "..." }
]
}
```
Batches of 100 items per HTTP request. Each request has a 120s timeout. HTTP errors and timeouts logged, but the upload path is **non-throwing when called from import.js** — a Hoffman outage or transient failure must not wedge the import flow.
**Response shape:**
```json
{
"TotalReceived": 100,
"Created": 98,
"Updated": 0,
"Unchanged": 0,
"Errors": [ "SN=12345-7: validation failed: ..." ]
}
```
**Post-response bookkeeping:** for every SN that was *not* in the `Errors` list, the DB stamps `api_uploaded_at = NOW()`. This is what drives the "on website" indicator in the dashboard.
**Upload triggers — three paths:**
1. **Automatic on import.** After `import.js` completes a batch insert, it calls `uploadNewRecords(filePaths)` which picks up PASS records from the just-imported files and pushes them.
2. **Manual per-record.** The web dashboard shows a PUSH button on every row; clicking it calls `POST /api/upload { ids: [123] }` which resolves to SNs and pushes.
3. **Bulk push.** Dashboard has a "PUSH TO WEB" button that pushes all selected records or (internally) `all_unuploaded=true` to re-run every unpushed PASS.
A fourth safety net — the **daily scheduled task** at 02:30 — runs `C:\ProgramData\dataforth-uploader\run-pipeline.ps1` as SYSTEM to catch anything that slipped through real-time uploads. Introduced as a fallback while the real-time path was stabilizing; now mostly informational.
---
## 4. Dashboard User Interface
**URL:** `http://192.168.0.6:3000/` (internal LAN only)
### 4.1 Search and filters
Left panel provides:
- **Serial Number** — LIKE-match any substring
- **Work Order #**
- **Model Number** — LIKE-match
- **Result** — All / Pass Only / Fail Only
- **Product Line** — dropdown populated from available `log_type` values
- **Website Status** — **Any / On Website / Not on Website** (new 2026-04-15)
- **Advanced:** Test Station, Date range, Full-text search in raw_data
### 4.2 Result rows
Each row shows Serial, Model, Date, Station, Product, Result, Actions. Visual cues:
- **Pink tint** — record is not on the Dataforth website (`api_uploaded_at IS NULL`)
- **Normal styling** — record is live on the website
- Tooltip on mouse-over explains status
### 4.3 Row actions
- **VIEW** — expand full record detail modal
- **SHEET** — preview the rendered datasheet (same text that was/would be sent to the website)
- **PUSH / RE-PUSH** — send this record to the website. Label says "PUSH" for records never uploaded, "RE-PUSH" for records already on the website (updates the stored copy).
- Button disabled for records where `overall_result != 'PASS'` (website only accepts PASS units, by policy)
### 4.4 Bulk actions
- Checkbox on each row + **Select Page** checkbox
- **PUSH TO WEB** bulk button — pushes every selected record in one API call
### 4.5 Export
- **EXPORT CSV** — exports the current filtered result set as CSV for offline review
---
## 5. Operational Notes
### 5.1 Service and account context
- Service name: `testdatadb`
- Service wrapper: WinSW (`C:\Shares\testdatadb\daemon\testdatadb.exe`)
- Run-as account: `INTRANET\svc_testdatadb`
- Auto-start: Automatic (Delayed Start recommended but not required)
- Logs: `C:\Shares\testdatadb\logs\testdatadb.out.log`, `.err.log`, `.wrapper.log`
### 5.2 Credentials
- **File:** `C:\ProgramData\dataforth-uploader\credentials.json` (JSON with `CF_TOKEN_URL`, `CF_API_BASE`, `CF_CLIENT_ID`, `CF_CLIENT_SECRET`, `CF_SCOPE`)
- **ACL:** SYSTEM (FullControl), Administrators (FullControl), `INTRANET\svc_testdatadb` (Read). Nobody else.
- **Source of truth:** same file used by both the testdatadb real-time path and the daily scheduled task fallback. Never duplicate creds into the service config.
### 5.3 Database connection
- Local PostgreSQL on AD2
- Environment variables (with defaults):
- `PGHOST=localhost`
- `PGPORT=5432`
- `PGUSER=testdatadb_app`
- `PGDATABASE=testdatadb`
- `PGPASSWORD` — set in service config
### 5.4 Key file locations on AD2
| Path | Purpose |
|---|---|
| `C:\Shares\testdatadb\database\import.js` | Parses .dat files, inserts to DB |
| `C:\Shares\testdatadb\database\upload-to-api.js` | Pushes records to Hoffman |
| `C:\Shares\testdatadb\database\render-datasheet.js` | In-memory datasheet rendering |
| `C:\Shares\testdatadb\database\export-datasheets.js` | Legacy For_Web writer (retained for compat) |
| `C:\Shares\testdatadb\routes/api.js` | HTTP API (search, upload, datasheet preview) |
| `C:\Shares\testdatadb\public\index.html` | Dashboard UI |
| `C:\Shares\testdatadb\parsers\` | `.dat` binary parsers per log type |
| `C:\Shares\testdatadb\specdata\` | QuickBASIC spec files (5BMAIN.DAT, 7BMAIN.DAT, etc.) |
| `C:\Shares\testdatadb\templates\datasheet-exact.js` | Formatter that replicates the QuickBASIC output |
| `C:\Shares\webshare\For_Web\` | Legacy intermediate output directory |
| `C:\ProgramData\dataforth-uploader\` | Scheduled task payload + credentials |
---
## 6. Record State Diagram
```
┌─────────────────────────────────────────────────────┐
│ │
Incoming .dat record │
│ │
│ parse, insert (ON CONFLICT (serial_number)) │
▼ │
┌────────────┐ re-test, new test_date newer │
│ In DB │◀─────────────────────────────────────┐ │
│ PASS/FAIL │ │ │
└─────┬──────┘ │ │
│ │ │
PASS? ──┴── FAIL? │ │
│ │ │ │
│ └─▶ Stays internal, never uploaded ──────┘ │
│ (can become PASS on retest) │
│ │
▼ │
render via render-datasheet.js │
│ │
│ specs missing? ──▶ skipped; tracked as "unpushable" ─────┘
│
▼
upload-to-api.js → POST /bulk
│
│ Hoffman response
▼
┌───────────────┬──────────────┬──────────────┐
│ Created │ Unchanged │ Errors │
│ │ │ │
│ stamp │ stamp │ no stamp; │
│ api_uploaded │ api_uploaded │ tracked in │
│ _at = NOW() │ _at = NOW() │ logs; retry │
└───────┬───────┴──────┬───────┴──────┬───────┘
│ │ │
└──────────┬───┘ │
│ │
▼ ▼
ON DATAFORTH WEBSITE NOT ON WEBSITE
(pink tint off) (pink tint on, PUSH button live)
```
---
## 7. Troubleshooting & FAQ
### 7.1 "A unit I know passed isn't showing up on the website"
1. Search the dashboard for the SN. Is it in testdatadb at all?
- **Not in DB:** the import hasn't picked up that file. Check the source `.dat` path and the import logs. Re-run the import manually if needed.
- **In DB but `overall_result=FAIL`:** by engineering policy, FAILs don't go to the website. If the unit has been retested and passes, make sure the retest `.dat` file has been imported — the FAIL→PASS rule will then update the record.
2. **In DB with `overall_result=PASS`, `api_uploaded_at=NULL`:** the record hasn't been pushed. Check the dashboard: hover on PUSH button for tooltip. If it says "Only PASS records can be pushed", the row is FAIL (verify). Otherwise click PUSH manually.
3. **Push attempt fails with "skipped"** in service logs: the model has no spec file, so we can't render the datasheet. Either the spec data is missing from `C:\Shares\testdatadb\specdata\` or this is a new model variant that hasn't been added to the spec files. **Action:** get the spec data from engineering and drop into the specdata folder; restart testdatadb service; record will then be pushable.
4. **Push attempt fails with Hoffman error:** check logs. Usually a data quality issue (malformed field, missing required measurement). Fix the source record or accept that this specific unit won't be on the website.
### 7.2 "A unit is on the website but shouldn't be"
By engineering policy, only PASS units should be on the website. If a FAIL has made it:
1. Look up the SN history on Hoffman's API. Most likely the unit passed a prior test and was pushed; later failure won't retroactively remove it.
2. If engineering wants the record retracted from the website, that's a direct call to Hoffman's API — outside the testdatadb pipeline.
We also observed about 8% of our local FAIL records *are* on Hoffman. In every sampled case this is the legitimate pattern: unit passed originally (pushed), was later re-tested and failed (kept locally as FAIL), but the original PASS datasheet is still on the website. If the unit needs retraction, do it at the website end.
### 7.3 "The dashboard shows counts that don't match Hoffman"
The local dashboard shows our DB state. Hoffman's site contains our local set **plus a large pre-testdatadb historical set** (202,866 records as of 2026-04-15) that was uploaded by prior tools and is not reproducible from our current DB. This is expected — don't try to reconcile the two totals; check specific SNs instead.
### 7.4 "How do I force a re-push of an already-on-web record?"
Click **RE-PUSH** on the row. This sends the record to Hoffman again — useful if:
- The local record was corrected and you want the website copy updated
- You suspect the website copy is wrong
- Engineering asked for a refresh
Hoffman returns `Unchanged` if the content is identical — so RE-PUSH is idempotent and safe to run casually.
### 7.5 "How do I check if a specific SN is on the website?"
Three options:
1. **Dashboard:** Search for the SN. Row styling + tooltip shows status.
2. **Direct API:** `GET https://{CF_API_BASE}/api/v1/TestReportDataFiles/{SN}` with a bearer token. Returns 200 + Content if on website, 404 if not.
3. **Public product page:** navigate to the Dataforth product page for that SN; if the datasheet link works, it's on the site.
---
## 8. Known Limitations
- **No real-time filesystem watcher.** The import currently runs on a schedule (or manual trigger). Records appear in the DB minutes after tests complete, not seconds. Good enough for business purposes.
- **VASLOG_ENG files are shipped verbatim** — the customer-engineering variant has non-standard formatting that varies per customer. We store and push the original file contents rather than reformatting. Implication: the website datasheet for these units matches exactly what engineering produced, no template regeneration.
- **Spec data is QuickBASIC binary.** New product families or variants require dropping updated `.DAT` spec files into `C:\Shares\testdatadb\specdata\`. The modern Node.js code reads the QB binary format directly — no conversion needed — but the files themselves still have to come from the legacy spec authoring tools (or be manually edited with a binary-aware editor).
- **Pre-testdatadb historical records.** The ~203K units that live on Hoffman from the DFWDS era don't exist in our DB. We can't re-render those datasheets from current state. If one of those legacy records has a data issue, the fix needs to be made directly at Hoffman.
---
## 9. Recent Changes (2026-04-15 Release)
For context on what's new if engineering has seen earlier documentation:
1. **Database deduplicated** — one row per serial number (was permitting multiple). Kept the best record per SN by state-priority (on-web > on-filesystem > newer test date).
2. **Unique constraint on `serial_number`** added and enforced.
3. **FAIL→PASS retest rule** formalized in the `ON CONFLICT` logic. Previously a FAIL record could linger forever even after a successful retest; now the retest correctly replaces it.
4. **For_Web filesystem dependency eliminated** for the upload path. Datasheets render in memory directly from the DB. Phantom `forweb_exported_at` stamps (rows that claimed a file existed but the file was gone) are no longer a source of push failures.
5. **Dashboard UI upgrades:**
- Row coloring indicates on-website status at a glance
- Per-row PUSH / RE-PUSH buttons
- Bulk PUSH TO WEB button
- Website Status filter (Any / On Website / Not on Website)
6. **Manual push end-to-end tested** — 170,984 records (mostly SCM7B + DSCT that were historically never pushed) successfully uploaded to Hoffman in this release window.
---
## 10. Contacts
- **System owner / technical questions:** Mike Swanson, AZ Computer Guru, mike@azcomputerguru.com, (520) 304-8300
- **Dashboard URL (internal):** http://192.168.0.6:3000/
- **Logs (internal):** `\\ad2\c-drive\Shares\testdatadb\logs\`
- **Escalation for Hoffman-side issues:** Dataforth web/IT (they own the product API endpoint)
Reference in New Issue View Git Blame Copy Permalink