docs: Document Dataforth test database system and troubleshooting

Investigation and Documentation:
- Discovered and documented test database system on AD2 server
- Created comprehensive TEST_DATABASE_ARCHITECTURE.md with full system details
- Retrieved all key database files from AD2 (import.js, schema.sql, server configs)
- Documented data flow: DOS machines → NAS → AD2 → SQLite → Web interface
- Verified database health: 1,027,517 records, 1075 MB, dates back to 1990

Database System Architecture:
- SQLite database with Node.js/Express.js web server (port 3000)
- Automated import via Sync-FromNAS.ps1 (runs every 15 minutes)
- 8 log types supported: DSCLOG, 5BLOG, 7BLOG, 8BLOG, PWRLOG, SCTLOG, VASLOG, SHT
- FTS5 full-text search, comprehensive indexes for performance
- API endpoints: search, stats, export, datasheet generation

Troubleshooting Scripts Created:
- Database diagnostics: check-db-simple.ps1, test-db-directly.ps1
- Server status checks: check-node-running.ps1, check-db-server.ps1
- Performance analysis: check-db-performance.ps1, check-wal-files.ps1
- API testing: test-api-endpoint.ps1, test-query.js
- Import monitoring: check-new-records.ps1
- Database optimization attempts: api-js-optimized.js, api-js-fixed.js
- Deployment scripts: deploy-db-optimization.ps1, deploy-db-fix.ps1, restore-original.ps1

Key Findings:
- Database file healthy and queryable (verified with test-query.js)
- Node.js server not running (port 3000 closed) - root cause of web interface issues
- Database last updated 8 days ago (01/13/2026) - automated sync may be broken
- Attempted performance optimizations (WAL mode) incompatible with readonly connections
- Original api.js restored from backup after optimization conflicts

Retrieved Documentation:
- QUICKSTART-retrieved.md: Quick start guide for database server
- SESSION_NOTES-retrieved.md: Complete session notes from database creation
- Sync-FromNAS-retrieved.ps1: Full sync script with database import logic
- import-js-retrieved.js: Node.js import script (12,774 bytes)
- schema-retrieved.sql: SQLite schema with FTS5 triggers
- server-js-retrieved.js: Express.js server configuration
- api-js-retrieved.js: API routes and endpoints
- package-retrieved.json: Node.js dependencies

Action Items Identified:
1. Start Node.js server on AD2 to restore web interface functionality
2. Investigate why automated sync hasn't updated database in 8 days
3. Check Windows Task Scheduler for Sync-FromNAS.ps1 scheduled task
4. Run manual import to catch up on 8 days of test data if needed

Technical Details:
- Database path: C:\Shares\testdatadb\database\testdata.db
- Web interface: http://192.168.0.6:3000 (when running)
- Database size: 1075.14 MB (1,127,362,560 bytes)
- Total records: 1,027,517 (slight variance from original 1,030,940)
- Pass rate: 99.82% (1,029,046 passed, 1,888 failed)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-01-21 13:45:27 -07:00
parent 7dc27290fb
commit e4392afce9
33 changed files with 3640 additions and 0 deletions

53
schema-retrieved.sql Normal file
View File

@@ -0,0 +1,53 @@
-- Test Data Database Schema
-- SQLite database for storing and searching test records
-- Main test records table
CREATE TABLE IF NOT EXISTS test_records (
id INTEGER PRIMARY KEY AUTOINCREMENT,
log_type TEXT NOT NULL, -- DSCLOG, 5BLOG, 7BLOG, 8BLOG, PWRLOG, SCTLOG, VASLOG, SHT
model_number TEXT NOT NULL, -- DSCA38-1793, SCM5B30-01, etc.
serial_number TEXT NOT NULL, -- 176923-1, 105840-2, etc.
test_date TEXT NOT NULL, -- Test date (YYYY-MM-DD format)
test_station TEXT, -- TS-1L, TS-3R, etc.
overall_result TEXT, -- PASS/FAIL
raw_data TEXT, -- Full original record
source_file TEXT, -- Original file path
import_date TEXT DEFAULT (datetime('now')),
UNIQUE(log_type, model_number, serial_number, test_date, test_station)
);
-- Indexes for fast searching
CREATE INDEX IF NOT EXISTS idx_serial ON test_records(serial_number);
CREATE INDEX IF NOT EXISTS idx_model ON test_records(model_number);
CREATE INDEX IF NOT EXISTS idx_date ON test_records(test_date);
CREATE INDEX IF NOT EXISTS idx_model_serial ON test_records(model_number, serial_number);
CREATE INDEX IF NOT EXISTS idx_result ON test_records(overall_result);
CREATE INDEX IF NOT EXISTS idx_log_type ON test_records(log_type);
-- Full-text search virtual table
CREATE VIRTUAL TABLE IF NOT EXISTS test_records_fts USING fts5(
serial_number,
model_number,
raw_data,
content='test_records',
content_rowid='id'
);
-- Triggers to keep FTS index in sync
CREATE TRIGGER IF NOT EXISTS test_records_ai AFTER INSERT ON test_records BEGIN
INSERT INTO test_records_fts(rowid, serial_number, model_number, raw_data)
VALUES (new.id, new.serial_number, new.model_number, new.raw_data);
END;
CREATE TRIGGER IF NOT EXISTS test_records_ad AFTER DELETE ON test_records BEGIN
INSERT INTO test_records_fts(test_records_fts, rowid, serial_number, model_number, raw_data)
VALUES ('delete', old.id, old.serial_number, old.model_number, old.raw_data);
END;
CREATE TRIGGER IF NOT EXISTS test_records_au AFTER UPDATE ON test_records BEGIN
INSERT INTO test_records_fts(test_records_fts, rowid, serial_number, model_number, raw_data)
VALUES ('delete', old.id, old.serial_number, old.model_number, old.raw_data);
INSERT INTO test_records_fts(rowid, serial_number, model_number, raw_data)
VALUES (new.id, new.serial_number, new.model_number, new.raw_data);
END;