claudetools/projects/dataforth-dos/documentation/TEST_DATABASE_ARCHITECTURE.md

Dataforth Test Data Database - System Architecture

Location: C:\Shares\testdatadb\ (on AD2 server) Created: 2026-01-13 Purpose: Consolidate and search 1M+ test records from DOS machines Last Retrieved: 2026-01-21

---

Overview

A Node.js web application with SQLite database that consolidates test data from ~30 DOS QC machines. Data flows automatically from DOS machines through the NAS to AD2, where it's imported into the database every 15 minutes.

---

System Architecture

Technology Stack

• Database: SQLite 3 (better-sqlite3)
• Server: Node.js + Express.js (port 3000)
• Import: Automated via Sync-FromNAS.ps1 PowerShell script
• Web UI: HTML/JavaScript search interface
• Parsers: Custom parsers for multiple data formats

Data Flow

DOS Machines (CTONW.BAT)
    ↓
NAS: /data/test/TS-XX/LOGS/[LOG_TYPE]/*.DAT
    ↓ (Every 15 minutes)
AD2: C:\Shares\test\TS-XX\LOGS\[LOG_TYPE]/*.DAT
    ↓ (Automated import via Node.js)
SQLite Database: C:\Shares\testdatadb\database\testdata.db
    ↓
Web Interface: http://localhost:3000

---

Database Details

File Location

C:\Shares\testdatadb\database\testdata.db

Database Type

SQLite 3 (single-file database)

Current Size

1,030,940 records (as of 2026-01-13)

Table Schema

test_records (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    log_type TEXT NOT NULL,           -- DSCLOG, 5BLOG, 7BLOG, 8BLOG, PWRLOG, SCTLOG, VASLOG, SHT
    model_number TEXT NOT NULL,       -- DSCA38-1793, SCM5B30-01, etc.
    serial_number TEXT NOT NULL,      -- 176923-1, 105840-2, etc.
    test_date TEXT NOT NULL,          -- YYYY-MM-DD format
    test_station TEXT,                -- TS-1L, TS-3R, etc.
    overall_result TEXT,              -- PASS/FAIL
    raw_data TEXT,                    -- Full original record
    source_file TEXT,                 -- Original file path
    import_date TEXT DEFAULT (datetime('now')),
    UNIQUE(log_type, model_number, serial_number, test_date, test_station)
)

Indexes

• idx_serial - Fast lookup by serial number
• idx_model - Fast lookup by model number
• idx_date - Fast lookup by test date
• idx_model_serial - Combined model + serial lookup
• idx_result - Filter by PASS/FAIL
• idx_log_type - Filter by log type

Full-Text Search

• FTS5 virtual table: test_records_fts
• Searches: serial_number, model_number, raw_data
• Automatic sync via triggers

---

Import System

Import Script

Location: C:\Shares\testdatadb\database\import.js Language: Node.js Duration: ~30 minutes for full import (1M+ records)

Data Sources (Priority Order)

1. HISTLOGS - C:\Shares\test\Ate\HISTLOGS\ (consolidated history)
   • Authoritative source
   • 576,416 records imported
2. Recovery-TEST - C:\Shares\Recovery-TEST\ (backup dates)
   • Multiple backup dates (12-13-25 to 12-18-25)
   • 454,383 records imported from 12-18-25
3. Live Data - C:\Shares\test\TS-XX\LOGS\
   • Current test station logs
   • 59 records imported (rest are duplicates)

Automated Import (via Sync-FromNAS.ps1)

Configuration in Sync-FromNAS.ps1 (lines 46-48):

$IMPORT_SCRIPT = "C:\Shares\testdatadb\database\import.js"
$NODE_PATH = "node"

Import Function (lines 122-141):

function Import-ToDatabase {
    param([string[]]$FilePaths)

    if ($FilePaths.Count -eq 0) { return }

    Write-Log "Importing $($FilePaths.Count) file(s) to database..."

    # Build argument list
    $args = @("$IMPORT_SCRIPT", "--file") + $FilePaths

    try {
        $output = & $NODE_PATH $args 2>&1
        foreach ($line in $output) {
            Write-Log "  [DB] $line"
        }
        Write-Log "Database import complete"
    } catch {
        Write-Log "ERROR: Database import failed: $_"
    }
}

Trigger: Every 15 minutes when new DAT files are synced from NAS Process: Sync-FromNAS.ps1 → import.js --file [file1] [file2] ... → SQLite insert

---

Supported Log Types

Log Type	Description	Format	Parser	Records
5BLOG	5B product line	Multi-line DAT	multiline	425,378
7BLOG	7B product line	CSV DAT	csvline	262,404
DSCLOG	DSC product line	Multi-line DAT	multiline	181,160
8BLOG	8B product line	Multi-line DAT	multiline	135,858
PWRLOG	Power tests	Multi-line DAT	multiline	12,374
VASLOG	VAS tests	Multi-line DAT	multiline	10,327
SCTLOG	SCT product line	Multi-line DAT	multiline	3,439
SHT	Test sheets	SHT format	shtfile	(varies)

---

Project Structure

C:\Shares\testdatadb/
├── database/
│   ├── testdata.db        # SQLite database (1M+ records)
│   ├── import.js          # Import script (12,774 bytes)
│   └── schema.sql         # Database schema with FTS5
├── parsers/
│   ├── multiline.js       # Parser for multi-line DAT files
│   ├── csvline.js         # Parser for 7BLOG CSV format
│   └── shtfile.js         # Parser for SHT test sheets
├── public/
│   └── index.html         # Web search interface
├── routes/
│   └── api.js             # API endpoints
├── templates/
│   └── datasheet.js       # Datasheet generator
├── node_modules/          # Dependencies
├── package.json           # Node.js project file (342 bytes)
├── package-lock.json      # Dependency lock file (43,983 bytes)
├── server.js              # Express.js server (1,443 bytes)
├── QUICKSTART.md          # Quick start guide (1,019 bytes)
├── SESSION_NOTES.md       # Complete session notes (4,788 bytes)
└── start-server.bat       # Windows startup script (97 bytes)

---

Web Interface

Starting the Server

cd C:\Shares\testdatadb
node server.js

Access: http://localhost:3000 (on AD2)

API Endpoints

Search:

GET /api/search?serial=...&model=...&from=...&to=...&result=...&q=...

• serial - Serial number (partial match)
• model - Model number (partial match)
• from - Start date (YYYY-MM-DD)
• to - End date (YYYY-MM-DD)
• result - PASS or FAIL
• q - Full-text search in raw data

Record Details:

GET /api/record/:id

Generate Datasheet:

GET /api/datasheet/:id

Database Statistics:

GET /api/stats

Export to CSV:

GET /api/export?format=csv

---

Database Statistics (as of 2026-01-13)

Total Records

1,030,940 records

Date Range

1990 to November 2025 (35 years of test data)

Pass/Fail Distribution

• PASS: 1,029,046 (99.82%)
• FAIL: 1,888 (0.18%)
• UNKNOWN: 6 (0.0006%)

Test Stations

TS-1L, TS-3R, TS-4L, TS-4R, TS-8R, TS-10L, TS-11L, and others

---

Manual Operations

Full Re-Import (if needed)

cd C:\Shares\testdatadb
del database\testdata.db
node database\import.js

Duration: ~30 minutes When needed: Parser updates, schema changes, corruption recovery

Incremental Import (single file)

node database\import.js --file C:\Shares\test\TS-4R\LOGS\8BLOG\test.DAT

Incremental Import (multiple files)

node database\import.js --file file1.DAT file2.DAT file3.DAT

This is the method used by Sync-FromNAS.ps1

---

Integration with DOS Update System

CTONW.BAT (v1.2+)

Purpose: Upload test data from DOS machines to NAS

Test Data Upload (lines 234-272):

ECHO [3/3] Uploading test data to LOGS...

REM Create log subdirectories
IF NOT EXIST %LOGSDIR%\8BLOG\NUL MD %LOGSDIR%\8BLOG
IF NOT EXIST %LOGSDIR%\DSCLOG\NUL MD %LOGSDIR%\DSCLOG
IF NOT EXIST %LOGSDIR%\HVLOG\NUL MD %LOGSDIR%\HVLOG
IF NOT EXIST %LOGSDIR%\PWRLOG\NUL MD %LOGSDIR%\PWRLOG
IF NOT EXIST %LOGSDIR%\RMSLOG\NUL MD %LOGSDIR%\RMSLOG
IF NOT EXIST %LOGSDIR%\7BLOG\NUL MD %LOGSDIR%\7BLOG

REM Upload test data files to appropriate log folders
IF EXIST C:\ATE\8BDATA\NUL XCOPY C:\ATE\8BDATA\*.DAT %LOGSDIR%\8BLOG\ /Y /Q
IF EXIST C:\ATE\DSCDATA\NUL XCOPY C:\ATE\DSCDATA\*.DAT %LOGSDIR%\DSCLOG\ /Y /Q
IF EXIST C:\ATE\HVDATA\NUL XCOPY C:\ATE\HVDATA\*.DAT %LOGSDIR%\HVLOG\ /Y /Q
IF EXIST C:\ATE\PWRDATA\NUL XCOPY C:\ATE\PWRDATA\*.DAT %LOGSDIR%\PWRLOG\ /Y /Q
IF EXIST C:\ATE\RMSDATA\NUL XCOPY C:\ATE\RMSDATA\*.DAT %LOGSDIR%\RMSLOG\ /Y /Q
IF EXIST C:\ATE\7BDATA\NUL XCOPY C:\ATE\7BDATA\*.DAT %LOGSDIR%\7BLOG\ /Y /Q

Target: T:\TS-4R\LOGS\8BLOG\ (on NAS)

Sync-FromNAS.ps1

Schedule: Every 15 minutes via Windows Task Scheduler PULL Operation (lines 157-213):

1. Find new DAT files on NAS (modified in last 24 hours)
2. Copy to AD2: C:\Shares\test\TS-XX\LOGS\[LOG_TYPE]\
3. Import to database via import.js --file [files]
4. Delete from NAS after successful import

Result: Test data flows automatically from DOS → NAS → AD2 → Database

---

Deduplication

Unique Key: (log_type, model_number, serial_number, test_date, test_station)

Effect: Same test result imported multiple times (from HISTLOGS, Recovery backups, live data) is stored only once

Example:

Record in HISTLOGS: DSCLOG, DSCA38-1793, 173672-1, 2025-02-15, TS-4R
Same record in Recovery-TEST/12-18-25: DSCLOG, DSCA38-1793, 173672-1, 2025-02-15, TS-4R
Result: Only 1 record in database

---

Search Examples

By Serial Number

http://localhost:3000/api/search?serial=176923

Returns all records with serial numbers containing "176923"

By Model Number

http://localhost:3000/api/search?model=DSCA38-1793

Returns all records for model DSCA38-1793

By Date Range

http://localhost:3000/api/search?from=2025-01-01&to=2025-12-31

Returns all records tested in 2025

By Pass/Fail Status

http://localhost:3000/api/search?result=FAIL

Returns all failed tests (1,888 records)

Full-Text Search

http://localhost:3000/api/search?q=voltage

Searches within raw test data for "voltage"

Combined Search

http://localhost:3000/api/search?model=DSCA38&result=PASS&from=2025-01-01

All passed tests for DSCA38 models since Jan 1, 2025

---

Known Issues

Model Number Parsing

• Parser was updated after initial import
• To fix: Delete testdata.db and re-run full import
• Impact: Model number searches may not be accurate for all records

Performance

• Full import: ~30 minutes
• Database file: ~112 KB (compressed by SQLite)
• Search performance: Very fast (indexed queries)

---

Maintenance

Regular Tasks

• None required - Database updates automatically via Sync-FromNAS.ps1

Occasional Tasks

• Re-import - If parser updates or schema changes (delete testdata.db and re-run)
• Backup - Copy testdata.db to backup location periodically

Monitoring

• Check Sync-FromNAS.ps1 log: C:\Shares\test\scripts\sync-from-nas.log
• Check sync status: C:\Shares\test\_SYNC_STATUS.txt
• View database stats: http://localhost:3000/api/stats (when server running)

---

Original Use Case (2026-01-13)

Request: Search for serial numbers 176923-1 through 176923-26 for model DSCA38-1793

Result: NOT FOUND - These devices haven't been tested yet

Most Recent Serials: 173672-x, 173681-x (February 2025)

Outcome: Database created to enable easy searching of 1M+ test records going back to 1990

---

Dependencies (from package.json)

• better-sqlite3 - Fast SQLite database
• express - Web server framework
• csv-writer - CSV export functionality
• Node.js - Required to run the application

---

Connection Information

Server: AD2 (192.168.0.6) Database Path: C:\Shares\testdatadb\database\testdata.db Web Server Port: 3000 (http://localhost:3000 when running) Access: Local only (on AD2 server)

To Access:

1. SSH or RDP to AD2 (192.168.0.6)
2. Start server: cd C:\Shares\testdatadb && node server.js
3. Open browser: http://localhost:3000

---

Related Documentation

In ClaudeTools:

• CTONW_V1.2_CHANGELOG.md - Test data routing to LOGS folders
• DOS_DEPLOYMENT_STATUS.md - Database import workflow
• Sync-FromNAS-retrieved.ps1 - Complete sync script with database import
• import-js-retrieved.js - Complete import script
• schema-retrieved.sql - Database schema
• QUICKSTART-retrieved.md - Quick start guide
• SESSION_NOTES-retrieved.md - Complete session notes

On AD2:

• C:\Shares\testdatadb\QUICKSTART.md
• C:\Shares\testdatadb\SESSION_NOTES.md
• C:\Shares\test\scripts\Sync-FromNAS.ps1

---

Created: 2026-01-13 Last Updated: 2026-01-21 Status: Production - Operational Automation: Complete (test data imports automatically every 15 minutes)