Dataforth UI push + dedup + refactor, GuruRMM roadmap evolution, Azure signing setup
Dataforth (projects/dataforth-dos/): - UI feature: row coloring + PUSH/RE-PUSH buttons + Website Status filter - Database dedup to one row per SN (2.89M -> 469K rows, UNIQUE constraint added) - Import logic handles FAIL -> PASS retest transition - Refactored upload-to-api.js to render datasheets in-memory (dropped For_Web filesystem dep) - Bulk pushed 170,984 records to Hoffman API - Statistical sanity check: 100/100 stamped SNs verified on Hoffman GuruRMM (projects/msp-tools/guru-rmm/): - ROADMAP.md: added Terminology (5-tier hierarchy), Tunnel Channels Phase 2, Logging/Audit/Observability, Multi-tenancy, Modular Architecture, Protocol Versioning, Certificates sections + Decisions Log - CONTEXT.md: hierarchy table, new anti-patterns (bootstrap sacred, no cross-module imports), revised next-steps priorities Session logs for both projects. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,139 @@
|
||||
/**
|
||||
* PostgreSQL Database Abstraction Layer
|
||||
*
|
||||
* Provides a connection pool and helper methods for the TestDataDB app.
|
||||
* Replaces better-sqlite3 singleton with pg.Pool.
|
||||
*
|
||||
* Environment variables (all optional, defaults connect to local PG):
|
||||
* PGHOST (default: localhost)
|
||||
* PGPORT (default: 5432)
|
||||
* PGUSER (default: testdatadb_app)
|
||||
* PGPASSWORD (default: DfTestDB2026!)
|
||||
* PGDATABASE (default: testdatadb)
|
||||
*/
|
||||
|
||||
const { Pool } = require('pg');
|
||||
|
||||
const pool = new Pool({
|
||||
host: process.env.PGHOST || 'localhost',
|
||||
port: parseInt(process.env.PGPORT || '5432', 10),
|
||||
user: process.env.PGUSER || 'testdatadb_app',
|
||||
password: process.env.PGPASSWORD || 'DfTestDB2026!',
|
||||
database: process.env.PGDATABASE || 'testdatadb',
|
||||
max: 20,
|
||||
idleTimeoutMillis: 30000,
|
||||
connectionTimeoutMillis: 5000,
|
||||
});
|
||||
|
||||
pool.on('error', (err) => {
|
||||
console.error(`[${new Date().toISOString()}] [PG POOL ERROR] ${err.message}`);
|
||||
});
|
||||
|
||||
/**
|
||||
* Convert SQLite-style ? placeholders to PostgreSQL $1, $2, ... placeholders.
|
||||
* Skips ? inside single-quoted strings.
|
||||
*/
|
||||
function convertPlaceholders(sql) {
|
||||
let idx = 0;
|
||||
let inString = false;
|
||||
let result = '';
|
||||
for (let i = 0; i < sql.length; i++) {
|
||||
const ch = sql[i];
|
||||
if (ch === "'" && (i === 0 || sql[i - 1] !== '\\')) {
|
||||
inString = !inString;
|
||||
result += ch;
|
||||
} else if (ch === '?' && !inString) {
|
||||
idx++;
|
||||
result += '$' + idx;
|
||||
} else {
|
||||
result += ch;
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute a query, return all rows.
|
||||
* @param {string} sql - SQL with ? or $N placeholders
|
||||
* @param {Array} params - Parameter values
|
||||
* @returns {Promise<Array>} rows
|
||||
*/
|
||||
async function query(sql, params = []) {
|
||||
const pgSql = sql.includes('?') ? convertPlaceholders(sql) : sql;
|
||||
const result = await pool.query(pgSql, params);
|
||||
return result.rows;
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute a query, return the first row or null.
|
||||
*/
|
||||
async function queryOne(sql, params = []) {
|
||||
const rows = await query(sql, params);
|
||||
return rows[0] || null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute a statement (INSERT/UPDATE/DELETE), return { rowCount }.
|
||||
*/
|
||||
async function execute(sql, params = []) {
|
||||
const pgSql = sql.includes('?') ? convertPlaceholders(sql) : sql;
|
||||
const result = await pool.query(pgSql, params);
|
||||
return { rowCount: result.rowCount, rows: result.rows };
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a function inside a transaction.
|
||||
* The callback receives a client with query/execute helpers.
|
||||
* @param {Function} fn - async (client) => result
|
||||
* @returns {Promise<*>} result of fn
|
||||
*/
|
||||
async function transaction(fn) {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
|
||||
const txClient = {
|
||||
async query(sql, params = []) {
|
||||
const pgSql = sql.includes('?') ? convertPlaceholders(sql) : sql;
|
||||
const result = await client.query(pgSql, params);
|
||||
return result.rows;
|
||||
},
|
||||
async queryOne(sql, params = []) {
|
||||
const rows = await txClient.query(sql, params);
|
||||
return rows[0] || null;
|
||||
},
|
||||
async execute(sql, params = []) {
|
||||
const pgSql = sql.includes('?') ? convertPlaceholders(sql) : sql;
|
||||
const result = await client.query(pgSql, params);
|
||||
return { rowCount: result.rowCount, rows: result.rows };
|
||||
},
|
||||
// Direct pg client access for COPY or other advanced operations
|
||||
raw: client,
|
||||
};
|
||||
|
||||
const result = await fn(txClient);
|
||||
await client.query('COMMIT');
|
||||
return result;
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK');
|
||||
throw err;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Close the pool (for graceful shutdown).
|
||||
*/
|
||||
async function close() {
|
||||
await pool.end();
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the raw pool (for advanced use like COPY).
|
||||
*/
|
||||
function getPool() {
|
||||
return pool;
|
||||
}
|
||||
|
||||
module.exports = { query, queryOne, execute, transaction, close, getPool, convertPlaceholders };
|
||||
@@ -0,0 +1,257 @@
|
||||
/**
|
||||
* Export Datasheets
|
||||
*
|
||||
* Generates TXT datasheets for unexported PASS records and writes them to X:\For_Web\.
|
||||
* Updates forweb_exported_at after successful export.
|
||||
*
|
||||
* Usage:
|
||||
* node export-datasheets.js Export all pending (batch mode)
|
||||
* node export-datasheets.js --limit 100 Export up to 100 records
|
||||
* node export-datasheets.js --file <paths> Export records matching specific source files
|
||||
* node export-datasheets.js --serial 178439-1 Export a specific serial number
|
||||
* node export-datasheets.js --dry-run Show what would be exported without writing
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const db = require('./db');
|
||||
|
||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||
|
||||
// Configuration
|
||||
const OUTPUT_DIR = 'X:\\For_Web';
|
||||
const BATCH_SIZE = 500;
|
||||
|
||||
async function run() {
|
||||
const args = process.argv.slice(2);
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const limitIdx = args.indexOf('--limit');
|
||||
const limit = limitIdx >= 0 ? parseInt(args[limitIdx + 1]) : 0;
|
||||
const serialIdx = args.indexOf('--serial');
|
||||
const serial = serialIdx >= 0 ? args[serialIdx + 1] : null;
|
||||
const fileIdx = args.indexOf('--file');
|
||||
const files = fileIdx >= 0 ? args.slice(fileIdx + 1).filter(f => !f.startsWith('--')) : null;
|
||||
|
||||
console.log('========================================');
|
||||
console.log('Datasheet Export');
|
||||
console.log('========================================');
|
||||
console.log(`Output: ${OUTPUT_DIR}`);
|
||||
console.log(`Dry run: ${dryRun}`);
|
||||
if (limit) console.log(`Limit: ${limit}`);
|
||||
if (serial) console.log(`Serial: ${serial}`);
|
||||
console.log(`Start: ${new Date().toISOString()}`);
|
||||
|
||||
if (!dryRun && !fs.existsSync(OUTPUT_DIR)) {
|
||||
console.error(`ERROR: Output directory does not exist: ${OUTPUT_DIR}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log('\nLoading model specs...');
|
||||
const specMap = loadAllSpecs();
|
||||
|
||||
// Build query
|
||||
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
if (serial) {
|
||||
paramIdx++;
|
||||
conditions.push(`serial_number = $${paramIdx}`);
|
||||
params.push(serial);
|
||||
}
|
||||
|
||||
if (files && files.length > 0) {
|
||||
const placeholders = files.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
||||
conditions.push(`source_file IN (${placeholders})`);
|
||||
params.push(...files);
|
||||
}
|
||||
|
||||
let sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')} ORDER BY test_date DESC`;
|
||||
|
||||
if (limit) {
|
||||
paramIdx++;
|
||||
sql += ` LIMIT $${paramIdx}`;
|
||||
params.push(limit);
|
||||
}
|
||||
|
||||
const records = await db.query(sql, params);
|
||||
console.log(`\nFound ${records.length} records to export`);
|
||||
|
||||
if (records.length === 0) {
|
||||
console.log('Nothing to export.');
|
||||
await db.close();
|
||||
return { exported: 0, skipped: 0, errors: 0 };
|
||||
}
|
||||
|
||||
let exported = 0;
|
||||
let skipped = 0;
|
||||
let errors = 0;
|
||||
let noSpecs = 0;
|
||||
let pendingUpdates = [];
|
||||
|
||||
for (const record of records) {
|
||||
try {
|
||||
const filename = record.serial_number + '.TXT';
|
||||
const outputPath = path.join(OUTPUT_DIR, filename);
|
||||
|
||||
// VASLOG_ENG: verbatim byte-for-byte copy of the original file.
|
||||
// Using fs.copyFileSync avoids any utf-8 round-trip that would
|
||||
// corrupt non-ASCII bytes (CP1252 etc.) in customer datasheets.
|
||||
// Fall back to writing raw_data if the source file is gone.
|
||||
if (record.log_type === 'VASLOG_ENG') {
|
||||
if (dryRun) {
|
||||
console.log(` [DRY RUN] Would copy: ${record.source_file} -> ${filename}`);
|
||||
exported++;
|
||||
continue;
|
||||
}
|
||||
if (record.source_file && fs.existsSync(record.source_file)) {
|
||||
fs.copyFileSync(record.source_file, outputPath);
|
||||
} else {
|
||||
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
||||
if (!record.raw_data) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
||||
}
|
||||
pendingUpdates.push(record.id);
|
||||
exported++;
|
||||
|
||||
if (pendingUpdates.length >= BATCH_SIZE) {
|
||||
await flushUpdates(pendingUpdates);
|
||||
pendingUpdates = [];
|
||||
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Template-generated datasheet path.
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
if (!specs) {
|
||||
noSpecs++;
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
const txt = generateExactDatasheet(record, specs);
|
||||
if (!txt) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
console.log(` [DRY RUN] Would write: ${filename}`);
|
||||
exported++;
|
||||
} else {
|
||||
fs.writeFileSync(outputPath, txt, 'utf8');
|
||||
pendingUpdates.push(record.id);
|
||||
exported++;
|
||||
|
||||
// Batch commit
|
||||
if (pendingUpdates.length >= BATCH_SIZE) {
|
||||
await flushUpdates(pendingUpdates);
|
||||
pendingUpdates = [];
|
||||
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`\n ERROR exporting ${record.serial_number}: ${err.message}`);
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
|
||||
// Flush remaining updates
|
||||
if (pendingUpdates.length > 0) {
|
||||
await flushUpdates(pendingUpdates);
|
||||
}
|
||||
|
||||
console.log(`\n\n========================================`);
|
||||
console.log(`Export Complete`);
|
||||
console.log(`========================================`);
|
||||
console.log(`Exported: ${exported}`);
|
||||
console.log(`Skipped: ${skipped} (${noSpecs} missing specs)`);
|
||||
console.log(`Errors: ${errors}`);
|
||||
console.log(`End: ${new Date().toISOString()}`);
|
||||
|
||||
await db.close();
|
||||
return { exported, skipped, errors };
|
||||
}
|
||||
|
||||
async function flushUpdates(ids) {
|
||||
const now = new Date().toISOString();
|
||||
await db.transaction(async (txClient) => {
|
||||
for (const id of ids) {
|
||||
await txClient.execute(
|
||||
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
||||
[now, id]
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Export function for use by import.js (no db argument -- uses shared pool)
|
||||
async function exportNewRecords(specMap, filePaths) {
|
||||
if (!fs.existsSync(OUTPUT_DIR)) {
|
||||
console.log(`[EXPORT] Output directory not found: ${OUTPUT_DIR}`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
if (filePaths && filePaths.length > 0) {
|
||||
const placeholders = filePaths.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
||||
conditions.push(`source_file IN (${placeholders})`);
|
||||
params.push(...filePaths);
|
||||
}
|
||||
|
||||
const sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')}`;
|
||||
const records = await db.query(sql, params);
|
||||
if (records.length === 0) return 0;
|
||||
|
||||
let exported = 0;
|
||||
|
||||
await db.transaction(async (txClient) => {
|
||||
for (const record of records) {
|
||||
const filename = record.serial_number + '.TXT';
|
||||
const outputPath = path.join(OUTPUT_DIR, filename);
|
||||
|
||||
try {
|
||||
// VASLOG_ENG: verbatim copy, preserving original bytes.
|
||||
if (record.log_type === 'VASLOG_ENG') {
|
||||
if (record.source_file && fs.existsSync(record.source_file)) {
|
||||
fs.copyFileSync(record.source_file, outputPath);
|
||||
} else {
|
||||
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
||||
if (!record.raw_data) continue;
|
||||
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
||||
}
|
||||
} else {
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
if (!specs) continue;
|
||||
const txt = generateExactDatasheet(record, specs);
|
||||
if (!txt) continue;
|
||||
fs.writeFileSync(outputPath, txt, 'utf8');
|
||||
}
|
||||
|
||||
await txClient.execute(
|
||||
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
||||
[new Date().toISOString(), record.id]
|
||||
);
|
||||
exported++;
|
||||
} catch (err) {
|
||||
console.error(`[EXPORT] Error writing ${filename}: ${err.message}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`[EXPORT] Generated ${exported} datasheet(s)`);
|
||||
return exported;
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
run().catch(console.error);
|
||||
}
|
||||
|
||||
module.exports = { exportNewRecords };
|
||||
@@ -0,0 +1,396 @@
|
||||
/**
|
||||
* Data Import Script
|
||||
* Imports test data from DAT and SHT files into PostgreSQL database
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const db = require('./db');
|
||||
|
||||
const { parseMultilineFile, extractTestStation } = require('../parsers/multiline');
|
||||
const { parseCsvFile } = require('../parsers/csvline');
|
||||
const { parseShtFile } = require('../parsers/shtfile');
|
||||
const { parseVaslogEngTxt } = require('../parsers/vaslog-engtxt');
|
||||
|
||||
// Data source paths
|
||||
const TEST_PATH = 'C:/Shares/test';
|
||||
const RECOVERY_PATH = 'C:/Shares/Recovery-TEST';
|
||||
const HISTLOGS_PATH = path.join(TEST_PATH, 'Ate/HISTLOGS');
|
||||
|
||||
// Log types and their parsers.
|
||||
// NOTE: `recursive` defaults to TRUE when absent (walk subfolders by default,
|
||||
// preserving pre-existing production behavior for DSCLOG/5BLOG/8BLOG/PWRLOG/
|
||||
// SCTLOG/7BLOG). Set it to FALSE explicitly on VASLOG so the .DAT walk does
|
||||
// NOT descend into the "VASLOG - Engineering Tested" subfolder (belt-and-
|
||||
// suspenders: the .DAT glob wouldn't match .txt, but be explicit anyway).
|
||||
// VASLOG_ENG also sets recursive:false -- the eng-tested dir is flat.
|
||||
const LOG_TYPES = {
|
||||
'DSCLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'5BLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'8BLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'PWRLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'SCTLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'VASLOG': { parser: 'multiline', ext: '.DAT', recursive: false },
|
||||
'7BLOG': { parser: 'csvline', ext: '.DAT' },
|
||||
// Engineering-tested SCMHVAS pre-rendered datasheets live under VASLOG/"VASLOG - Engineering Tested"/
|
||||
'VASLOG_ENG': { parser: 'vaslog-engtxt', ext: '.txt', dir: 'VASLOG/VASLOG - Engineering Tested', recursive: false }
|
||||
};
|
||||
|
||||
// Find all files of a specific type in a directory
|
||||
function findFiles(dir, pattern, recursive = true) {
|
||||
const results = [];
|
||||
|
||||
try {
|
||||
if (!fs.existsSync(dir)) return results;
|
||||
|
||||
const items = fs.readdirSync(dir, { withFileTypes: true });
|
||||
|
||||
for (const item of items) {
|
||||
const fullPath = path.join(dir, item.name);
|
||||
|
||||
if (item.isDirectory() && recursive) {
|
||||
results.push(...findFiles(fullPath, pattern, recursive));
|
||||
} else if (item.isFile()) {
|
||||
if (pattern.test(item.name)) {
|
||||
results.push(fullPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
// Ignore permission errors
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
// Parse records from a file (sync -- file I/O only)
|
||||
function parseFile(filePath, logType, parser) {
|
||||
const testStation = extractTestStation(filePath);
|
||||
|
||||
switch (parser) {
|
||||
case 'multiline':
|
||||
return parseMultilineFile(filePath, logType, testStation);
|
||||
case 'csvline':
|
||||
return parseCsvFile(filePath, testStation);
|
||||
case 'shtfile':
|
||||
return parseShtFile(filePath, testStation);
|
||||
case 'vaslog-engtxt':
|
||||
return parseVaslogEngTxt(filePath, testStation);
|
||||
default:
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
// Batch insert records into PostgreSQL
|
||||
async function insertBatch(txClient, records) {
|
||||
let imported = 0;
|
||||
for (const record of records) {
|
||||
try {
|
||||
const result = await txClient.execute(
|
||||
`INSERT INTO test_records
|
||||
(log_type, model_number, serial_number, test_date, test_station, overall_result, raw_data, source_file)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
||||
ON CONFLICT (log_type, model_number, serial_number, test_date, test_station)
|
||||
DO UPDATE SET raw_data = EXCLUDED.raw_data, overall_result = EXCLUDED.overall_result`,
|
||||
[
|
||||
record.log_type,
|
||||
record.model_number,
|
||||
record.serial_number,
|
||||
record.test_date,
|
||||
record.test_station,
|
||||
record.overall_result,
|
||||
record.raw_data,
|
||||
record.source_file
|
||||
]
|
||||
);
|
||||
if (result.rowCount > 0) imported++;
|
||||
} catch (err) {
|
||||
// Constraint error - skip
|
||||
}
|
||||
}
|
||||
return imported;
|
||||
}
|
||||
|
||||
// Import records from a file
|
||||
async function importFile(txClient, filePath, logType, parser) {
|
||||
let records = [];
|
||||
|
||||
try {
|
||||
records = parseFile(filePath, logType, parser);
|
||||
const imported = await insertBatch(txClient, records);
|
||||
return { total: records.length, imported };
|
||||
} catch (err) {
|
||||
console.error(`Error importing ${filePath}: ${err.message}`);
|
||||
return { total: 0, imported: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
// Import from HISTLOGS (master consolidated logs)
|
||||
async function importHistlogs(txClient) {
|
||||
console.log('\n=== Importing from HISTLOGS ===');
|
||||
|
||||
let totalImported = 0;
|
||||
let totalRecords = 0;
|
||||
|
||||
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
||||
const subdir = config.dir || logType;
|
||||
const logDir = path.join(HISTLOGS_PATH, subdir);
|
||||
|
||||
if (!fs.existsSync(logDir)) {
|
||||
console.log(` ${logType}: directory not found`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
||||
console.log(` ${logType}: found ${files.length} files`);
|
||||
|
||||
for (const file of files) {
|
||||
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(` HISTLOGS total: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||
return totalImported;
|
||||
}
|
||||
|
||||
// Import from test station logs
|
||||
async function importStationLogs(txClient, basePath, label) {
|
||||
console.log(`\n=== Importing from ${label} ===`);
|
||||
|
||||
let totalImported = 0;
|
||||
let totalRecords = 0;
|
||||
|
||||
const stationPattern = /^TS-\d+[LR]?$/i;
|
||||
let stations = [];
|
||||
|
||||
try {
|
||||
const items = fs.readdirSync(basePath, { withFileTypes: true });
|
||||
stations = items
|
||||
.filter(i => i.isDirectory() && stationPattern.test(i.name))
|
||||
.map(i => i.name);
|
||||
} catch (err) {
|
||||
console.log(` Error reading ${basePath}: ${err.message}`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
console.log(` Found stations: ${stations.join(', ')}`);
|
||||
|
||||
for (const station of stations) {
|
||||
const logsDir = path.join(basePath, station, 'LOGS');
|
||||
|
||||
if (!fs.existsSync(logsDir)) continue;
|
||||
|
||||
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
||||
const subdir = config.dir || logType;
|
||||
const logDir = path.join(logsDir, subdir);
|
||||
|
||||
if (!fs.existsSync(logDir)) continue;
|
||||
|
||||
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
||||
|
||||
for (const file of files) {
|
||||
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Also import SHT files
|
||||
const shtFiles = findFiles(basePath, /\.SHT$/i, true);
|
||||
console.log(` Found ${shtFiles.length} SHT files`);
|
||||
|
||||
for (const file of shtFiles) {
|
||||
const { total, imported } = await importFile(txClient, file, 'SHT', 'shtfile');
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
}
|
||||
|
||||
console.log(` ${label} total: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||
return totalImported;
|
||||
}
|
||||
|
||||
// Import from Recovery-TEST backups (newest first)
|
||||
async function importRecoveryBackups(txClient) {
|
||||
console.log('\n=== Importing from Recovery-TEST backups ===');
|
||||
|
||||
if (!fs.existsSync(RECOVERY_PATH)) {
|
||||
console.log(' Recovery-TEST directory not found');
|
||||
return 0;
|
||||
}
|
||||
|
||||
const backups = fs.readdirSync(RECOVERY_PATH, { withFileTypes: true })
|
||||
.filter(i => i.isDirectory() && /^\d{2}-\d{2}-\d{2}$/.test(i.name))
|
||||
.map(i => i.name)
|
||||
.sort()
|
||||
.reverse();
|
||||
|
||||
console.log(` Found backup dates: ${backups.join(', ')}`);
|
||||
|
||||
let totalImported = 0;
|
||||
|
||||
for (const backup of backups) {
|
||||
const backupPath = path.join(RECOVERY_PATH, backup);
|
||||
const imported = await importStationLogs(txClient, backupPath, `Recovery-TEST/${backup}`);
|
||||
totalImported += imported;
|
||||
}
|
||||
|
||||
return totalImported;
|
||||
}
|
||||
|
||||
// Main import function
|
||||
async function runImport() {
|
||||
console.log('========================================');
|
||||
console.log('Test Data Import');
|
||||
console.log('========================================');
|
||||
console.log(`Start time: ${new Date().toISOString()}`);
|
||||
|
||||
let grandTotal = 0;
|
||||
|
||||
await db.transaction(async (txClient) => {
|
||||
grandTotal += await importHistlogs(txClient);
|
||||
grandTotal += await importRecoveryBackups(txClient);
|
||||
grandTotal += await importStationLogs(txClient, TEST_PATH, 'test');
|
||||
});
|
||||
|
||||
const stats = await db.queryOne('SELECT COUNT(*) as count FROM test_records');
|
||||
|
||||
console.log('\n========================================');
|
||||
console.log('Import Complete');
|
||||
console.log('========================================');
|
||||
console.log(`Total records in database: ${stats.count}`);
|
||||
console.log(`End time: ${new Date().toISOString()}`);
|
||||
|
||||
await db.close();
|
||||
}
|
||||
|
||||
// Import a single file (for incremental imports from sync)
|
||||
async function importSingleFile(filePath) {
|
||||
console.log(`Importing: ${filePath}`);
|
||||
|
||||
let logType = null;
|
||||
let parser = null;
|
||||
|
||||
// VASLOG_ENG subpath must be checked before VASLOG (substring overlap).
|
||||
if (filePath.includes('VASLOG - Engineering Tested')) {
|
||||
logType = 'VASLOG_ENG';
|
||||
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
||||
} else {
|
||||
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
||||
if (type === 'VASLOG_ENG') continue;
|
||||
if (filePath.includes(type)) {
|
||||
logType = type;
|
||||
parser = config.parser;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!logType) {
|
||||
if (/\.SHT$/i.test(filePath)) {
|
||||
logType = 'SHT';
|
||||
parser = 'shtfile';
|
||||
} else {
|
||||
console.log(` Unknown log type for: ${filePath}`);
|
||||
return { total: 0, imported: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
let result;
|
||||
await db.transaction(async (txClient) => {
|
||||
result = await importFile(txClient, filePath, logType, parser);
|
||||
});
|
||||
|
||||
console.log(` Imported ${result.imported} of ${result.total} records`);
|
||||
return result;
|
||||
}
|
||||
|
||||
// Import multiple files (for batch incremental imports)
|
||||
async function importFiles(filePaths) {
|
||||
console.log(`\n========================================`);
|
||||
console.log(`Incremental Import: ${filePaths.length} files`);
|
||||
console.log(`========================================`);
|
||||
|
||||
let totalImported = 0;
|
||||
let totalRecords = 0;
|
||||
|
||||
await db.transaction(async (txClient) => {
|
||||
for (const filePath of filePaths) {
|
||||
let logType = null;
|
||||
let parser = null;
|
||||
|
||||
// VASLOG_ENG subpath must be checked before the generic loop --
|
||||
// otherwise `includes('VASLOG')` hits first and the eng .txt gets
|
||||
// dispatched to the multiline parser. Mirror importSingleFile().
|
||||
if (filePath.includes('VASLOG - Engineering Tested')) {
|
||||
logType = 'VASLOG_ENG';
|
||||
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
||||
} else {
|
||||
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
||||
if (type === 'VASLOG_ENG') continue;
|
||||
if (filePath.includes(type)) {
|
||||
logType = type;
|
||||
parser = config.parser;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!logType) {
|
||||
if (/\.SHT$/i.test(filePath)) {
|
||||
logType = 'SHT';
|
||||
parser = 'shtfile';
|
||||
} else {
|
||||
console.log(` Skipping unknown type: ${filePath}`);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
const { total, imported } = await importFile(txClient, filePath, logType, parser);
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
console.log(` ${path.basename(filePath)}: ${imported}/${total} records`);
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`\nTotal: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||
|
||||
// Export datasheets for newly imported records
|
||||
if (totalImported > 0) {
|
||||
try {
|
||||
const { loadAllSpecs } = require('../parsers/spec-reader');
|
||||
const { exportNewRecords } = require('./export-datasheets');
|
||||
const specMap = loadAllSpecs();
|
||||
await exportNewRecords(specMap, filePaths);
|
||||
} catch (err) {
|
||||
console.error(`[EXPORT] Datasheet export failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
return { total: totalRecords, imported: totalImported };
|
||||
}
|
||||
|
||||
// Run if called directly
|
||||
if (require.main === module) {
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
if (args.length > 0 && args[0] === '--file') {
|
||||
const files = args.slice(1);
|
||||
if (files.length === 0) {
|
||||
console.log('Usage: node import.js --file <file1> [file2] ...');
|
||||
process.exit(1);
|
||||
}
|
||||
importFiles(files).then(() => db.close()).catch(console.error);
|
||||
} else if (args.length > 0 && args[0] === '--help') {
|
||||
console.log('Usage:');
|
||||
console.log(' node import.js Full import from all sources');
|
||||
console.log(' node import.js --file <f> Import specific file(s)');
|
||||
process.exit(0);
|
||||
} else {
|
||||
runImport().catch(console.error);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { runImport, importSingleFile, importFiles };
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,487 @@
|
||||
/**
|
||||
* API Routes for Test Data Database
|
||||
*
|
||||
* PostgreSQL version - uses pg.Pool via database/db.js.
|
||||
* All route handlers are async. FTS uses tsvector/plainto_tsquery.
|
||||
*/
|
||||
|
||||
const express = require('express');
|
||||
const path = require('path');
|
||||
const db = require('../database/db');
|
||||
const { generateDatasheet } = require('../templates/datasheet');
|
||||
|
||||
const router = express.Router();
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
const MAX_LIMIT = 1000;
|
||||
|
||||
function clampLimit(value) {
|
||||
const parsed = parseInt(value, 10);
|
||||
if (isNaN(parsed) || parsed < 1) return 100;
|
||||
return Math.min(parsed, MAX_LIMIT);
|
||||
}
|
||||
|
||||
function clampOffset(value) {
|
||||
const parsed = parseInt(value, 10);
|
||||
if (isNaN(parsed) || parsed < 0) return 0;
|
||||
return parsed;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/search
|
||||
// Search test records
|
||||
// Query params: serial, model, from, to, result, q, station, logtype, limit, offset
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/search', async (req, res) => {
|
||||
try {
|
||||
const { serial, model, from, to, result, q, station, logtype, workorder } = req.query;
|
||||
const limit = clampLimit(req.query.limit || 100);
|
||||
const offset = clampOffset(req.query.offset || 0);
|
||||
|
||||
const conditions = [];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
const addParam = (val) => {
|
||||
paramIdx++;
|
||||
params.push(val);
|
||||
return '$' + paramIdx;
|
||||
};
|
||||
|
||||
if (q) {
|
||||
// Full-text search using tsvector
|
||||
conditions.push(`search_vector @@ plainto_tsquery('english', ${addParam(q)})`);
|
||||
}
|
||||
|
||||
if (serial) {
|
||||
const val = serial.includes('%') ? serial : `%${serial}%`;
|
||||
conditions.push(`serial_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (workorder) {
|
||||
conditions.push(`work_order = ${addParam(workorder)}`);
|
||||
}
|
||||
|
||||
if (model) {
|
||||
const val = model.includes('%') ? model : `%${model}%`;
|
||||
conditions.push(`model_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (from) {
|
||||
conditions.push(`test_date >= ${addParam(from)}`);
|
||||
}
|
||||
|
||||
if (to) {
|
||||
conditions.push(`test_date <= ${addParam(to)}`);
|
||||
}
|
||||
|
||||
if (result) {
|
||||
conditions.push(`overall_result = ${addParam(result.toUpperCase())}`);
|
||||
}
|
||||
|
||||
if (station) {
|
||||
conditions.push(`test_station = ${addParam(station)}`);
|
||||
}
|
||||
|
||||
if (logtype) {
|
||||
conditions.push(`log_type = ${addParam(logtype)}`);
|
||||
}
|
||||
|
||||
const where = conditions.length > 0 ? 'WHERE ' + conditions.join(' AND ') : '';
|
||||
|
||||
const dataSql = `SELECT * FROM test_records ${where} ORDER BY test_date DESC, serial_number LIMIT ${addParam(limit)} OFFSET ${addParam(offset)}`;
|
||||
const countSql = `SELECT COUNT(*) as count FROM test_records ${where}`;
|
||||
const countParams = params.slice(0, paramIdx - 2); // exclude limit/offset
|
||||
|
||||
const [records, countRow] = await Promise.all([
|
||||
db.query(dataSql, params),
|
||||
db.queryOne(countSql, countParams),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
records,
|
||||
total: countRow?.count ? parseInt(countRow.count, 10) : records.length,
|
||||
limit,
|
||||
offset
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [SEARCH ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/record/:id
|
||||
// Get single record by ID
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/record/:id', async (req, res) => {
|
||||
try {
|
||||
const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);
|
||||
|
||||
if (!record) {
|
||||
return res.status(404).json({ error: 'Record not found' });
|
||||
}
|
||||
|
||||
res.json(record);
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [RECORD ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/datasheet/:id
|
||||
// Generate datasheet for a record
|
||||
// Query params: format (html, txt)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/datasheet/:id', async (req, res) => {
|
||||
try {
|
||||
const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);
|
||||
|
||||
if (!record) {
|
||||
return res.status(404).json({ error: 'Record not found' });
|
||||
}
|
||||
|
||||
const format = req.query.format || 'html';
|
||||
|
||||
// Try exact-match formatter first
|
||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||
const specMap = loadAllSpecs();
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
const exactTxt = generateExactDatasheet(record, specs);
|
||||
|
||||
if (exactTxt && format === 'html') {
|
||||
// Render exact-match TXT as styled HTML page
|
||||
const escaped = exactTxt
|
||||
.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>');
|
||||
const html = `<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Test Data Sheet - ${record.serial_number}</title>
|
||||
<style>
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background: #f0f0f0;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
}
|
||||
.page {
|
||||
background: white;
|
||||
padding: 40px 30px;
|
||||
max-width: 720px;
|
||||
width: 100%;
|
||||
box-shadow: 0 2px 8px rgba(0,0,0,0.15);
|
||||
border: 1px solid #ccc;
|
||||
}
|
||||
pre {
|
||||
font-family: 'Courier New', Courier, monospace;
|
||||
font-size: 11px;
|
||||
line-height: 1.4;
|
||||
margin: 0;
|
||||
white-space: pre;
|
||||
overflow-x: auto;
|
||||
}
|
||||
.toolbar {
|
||||
position: fixed;
|
||||
top: 10px;
|
||||
right: 10px;
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
}
|
||||
.toolbar button {
|
||||
padding: 8px 16px;
|
||||
border: 1px solid #999;
|
||||
background: white;
|
||||
cursor: pointer;
|
||||
font-size: 13px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.toolbar button:hover { background: #e0e0e0; }
|
||||
@media print {
|
||||
body { background: white; padding: 0; }
|
||||
.page { box-shadow: none; border: none; padding: 0; }
|
||||
.toolbar { display: none; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="toolbar">
|
||||
<button onclick="window.print()">Print</button>
|
||||
<button onclick="window.open('/api/datasheet/${record.id}/pdf')">Download PDF</button>
|
||||
<button onclick="window.close()">Close</button>
|
||||
</div>
|
||||
<div class="page">
|
||||
<pre>${escaped}</pre>
|
||||
</div>
|
||||
</body>
|
||||
</html>`;
|
||||
res.type('html').send(html);
|
||||
} else if (exactTxt && format === 'txt') {
|
||||
res.type('text/plain').send(exactTxt);
|
||||
} else {
|
||||
// Fall back to generic template
|
||||
const datasheet = generateDatasheet(record, format);
|
||||
if (format === 'html') {
|
||||
res.type('html').send(datasheet);
|
||||
} else {
|
||||
res.type('text/plain').send(datasheet);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [DATASHEET ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/datasheet/:id/pdf
|
||||
// Generate PDF datasheet for a record (on-demand download)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/datasheet/:id/pdf', async (req, res) => {
|
||||
try {
|
||||
const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);
|
||||
|
||||
if (!record) {
|
||||
return res.status(404).json({ error: 'Record not found' });
|
||||
}
|
||||
|
||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||
const PDFDocument = require('pdfkit');
|
||||
|
||||
const specMap = loadAllSpecs();
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
let txt = generateExactDatasheet(record, specs);
|
||||
|
||||
// Fall back to generic datasheet if exact-match formatter doesn't support this family
|
||||
if (!txt) {
|
||||
txt = generateDatasheet(record, 'txt');
|
||||
}
|
||||
|
||||
if (!txt) {
|
||||
return res.status(422).json({ error: 'Could not generate datasheet (missing specs or data)' });
|
||||
}
|
||||
|
||||
const doc = new PDFDocument({
|
||||
size: 'LETTER',
|
||||
margins: { top: 36, bottom: 36, left: 36, right: 36 }
|
||||
});
|
||||
|
||||
res.setHeader('Content-Type', 'application/pdf');
|
||||
res.setHeader('Content-Disposition', `attachment; filename="${record.serial_number}.pdf"`);
|
||||
doc.pipe(res);
|
||||
|
||||
doc.font('Courier').fontSize(9.5);
|
||||
const lines = txt.split(/\r?\n/);
|
||||
for (const line of lines) {
|
||||
doc.text(line, { lineGap: 1 });
|
||||
}
|
||||
|
||||
doc.end();
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [PDF ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/stats
|
||||
// Get database statistics
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/stats', async (req, res) => {
|
||||
try {
|
||||
const [totalRow, byLogType, byResult, byStation, dateRange, recentSerials] = await Promise.all([
|
||||
db.queryOne('SELECT COUNT(*) as count FROM test_records'),
|
||||
db.query('SELECT log_type, COUNT(*) as count FROM test_records GROUP BY log_type ORDER BY count DESC'),
|
||||
db.query('SELECT overall_result, COUNT(*) as count FROM test_records GROUP BY overall_result'),
|
||||
db.query(`SELECT test_station, COUNT(*) as count FROM test_records
|
||||
WHERE test_station IS NOT NULL AND test_station != ''
|
||||
GROUP BY test_station ORDER BY test_station`),
|
||||
db.queryOne('SELECT MIN(test_date) as oldest, MAX(test_date) as newest FROM test_records'),
|
||||
db.query(`SELECT DISTINCT serial_number, model_number, test_date
|
||||
FROM test_records ORDER BY test_date DESC LIMIT 10`),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
total_records: parseInt(totalRow.count, 10),
|
||||
by_log_type: byLogType.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
by_result: byResult.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
by_station: byStation.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
date_range: dateRange,
|
||||
recent_serials: recentSerials,
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [STATS ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/filters
|
||||
// Get available filter options (test stations, log types, models)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/filters', async (req, res) => {
|
||||
try {
|
||||
const [stations, logTypes, models] = await Promise.all([
|
||||
db.query(`SELECT DISTINCT test_station FROM test_records
|
||||
WHERE test_station IS NOT NULL AND test_station != ''
|
||||
ORDER BY test_station`),
|
||||
db.query('SELECT DISTINCT log_type FROM test_records ORDER BY log_type'),
|
||||
db.query(`SELECT DISTINCT model_number, COUNT(*) as count FROM test_records
|
||||
GROUP BY model_number ORDER BY count DESC LIMIT 500`),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
stations: stations.map(r => r.test_station),
|
||||
log_types: logTypes.map(r => r.log_type),
|
||||
models: models.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [FILTERS ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/export
|
||||
// Export search results as CSV
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/export', async (req, res) => {
|
||||
try {
|
||||
const { serial, model, from, to, result, station, logtype } = req.query;
|
||||
|
||||
const conditions = [];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
const addParam = (val) => {
|
||||
paramIdx++;
|
||||
params.push(val);
|
||||
return '$' + paramIdx;
|
||||
};
|
||||
|
||||
if (serial) {
|
||||
const val = serial.includes('%') ? serial : `%${serial}%`;
|
||||
conditions.push(`serial_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (model) {
|
||||
const val = model.includes('%') ? model : `%${model}%`;
|
||||
conditions.push(`model_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (from) {
|
||||
conditions.push(`test_date >= ${addParam(from)}`);
|
||||
}
|
||||
|
||||
if (to) {
|
||||
conditions.push(`test_date <= ${addParam(to)}`);
|
||||
}
|
||||
|
||||
if (result) {
|
||||
conditions.push(`overall_result = ${addParam(result.toUpperCase())}`);
|
||||
}
|
||||
|
||||
if (station) {
|
||||
conditions.push(`test_station = ${addParam(station)}`);
|
||||
}
|
||||
|
||||
if (logtype) {
|
||||
conditions.push(`log_type = ${addParam(logtype)}`);
|
||||
}
|
||||
|
||||
const where = conditions.length > 0 ? 'WHERE ' + conditions.join(' AND ') : '';
|
||||
const sql = `SELECT * FROM test_records ${where} ORDER BY test_date DESC, serial_number LIMIT 10000`;
|
||||
|
||||
const records = await db.query(sql, params);
|
||||
|
||||
// Generate CSV
|
||||
const headers = ['id', 'log_type', 'model_number', 'serial_number', 'test_date', 'test_station', 'overall_result', 'source_file'];
|
||||
let csv = headers.join(',') + '\n';
|
||||
|
||||
for (const record of records) {
|
||||
const row = headers.map(h => {
|
||||
const val = record[h] || '';
|
||||
return `"${String(val).replace(/"/g, '""')}"`;
|
||||
});
|
||||
csv += row.join(',') + '\n';
|
||||
}
|
||||
|
||||
res.setHeader('Content-Type', 'text/csv');
|
||||
res.setHeader('Content-Disposition', 'attachment; filename=test_records.csv');
|
||||
res.send(csv);
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [EXPORT ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/workorder/:wo
|
||||
// Get work order details and all associated test lines
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/workorder/:wo', async (req, res) => {
|
||||
try {
|
||||
const wo = req.params.wo;
|
||||
|
||||
const [header, lines, testRecords] = await Promise.all([
|
||||
db.queryOne('SELECT * FROM work_orders WHERE wo_number = $1', [wo]),
|
||||
db.query('SELECT * FROM work_order_lines WHERE wo_number = $1 ORDER BY test_date, test_time', [wo]),
|
||||
db.query(
|
||||
'SELECT id, log_type, model_number, serial_number, test_date, test_station, overall_result, work_order FROM test_records WHERE work_order = $1 ORDER BY serial_number',
|
||||
[wo]
|
||||
),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
work_order: header || { wo_number: wo },
|
||||
lines,
|
||||
test_records: testRecords,
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [WO ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/workorder-search?q=<query>
|
||||
// Search work orders by number (prefix match)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/workorder-search', async (req, res) => {
|
||||
try {
|
||||
const q = req.query.q || '';
|
||||
if (q.length < 2) {
|
||||
return res.json({ results: [] });
|
||||
}
|
||||
|
||||
const results = await db.query(
|
||||
'SELECT wo_number, wo_date, program, test_station FROM work_orders WHERE wo_number LIKE $1 ORDER BY wo_date DESC LIMIT 50',
|
||||
[q + '%']
|
||||
);
|
||||
|
||||
res.json({ results });
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cleanup function for graceful shutdown
|
||||
// ---------------------------------------------------------------------------
|
||||
async function cleanup() {
|
||||
try {
|
||||
await db.close();
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [CLEANUP ERROR] ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = router;
|
||||
module.exports.cleanup = cleanup;
|
||||
115
projects/dataforth-dos/datasheet-pipeline/deploy_ui_feature.py
Normal file
115
projects/dataforth-dos/datasheet-pipeline/deploy_ui_feature.py
Normal file
@@ -0,0 +1,115 @@
|
||||
"""Deploy the api_uploaded_at + UI push feature to AD2 in the correct order:
|
||||
|
||||
1. SFTP server_inventory.txt to AD2 (one-time, for back-population)
|
||||
2. SFTP migration SQL + back-populate script
|
||||
3. Run migration via psql
|
||||
4. Run back-populate
|
||||
5. Backup current production files (import.js already backed up earlier this
|
||||
session; backup routes/api.js + public/index.html + database/upload-to-api.js)
|
||||
6. SFTP updated upload-to-api.js, routes/api.js, public/index.html
|
||||
7. node --check on AD2
|
||||
8. Restart testdatadb service
|
||||
9. Verify
|
||||
"""
|
||||
import base64, paramiko, subprocess, time, yaml, os
|
||||
|
||||
pwd_raw = yaml.safe_load(subprocess.run(['sops','-d','D:/vault/clients/dataforth/ad2.sops.yaml'],
|
||||
capture_output=True, text=True, timeout=30, check=True).stdout)['credentials']['password']
|
||||
PWD = pwd_raw # vault now has clean password
|
||||
|
||||
LOCAL_IMPL = r'D:\claudetools\projects\dataforth-dos\datasheet-pipeline\implementation-upload'
|
||||
REMOTE_DB = 'C:/Shares/testdatadb/database'
|
||||
REMOTE_API = 'C:/Shares/testdatadb/routes'
|
||||
REMOTE_WEB = 'C:/Shares/testdatadb/public'
|
||||
PROD_DIR = 'C:/ProgramData/dataforth-uploader'
|
||||
SERVER_INV_LOCAL = r'C:\Users\guru\AppData\Local\Temp\server_inventory.txt'
|
||||
|
||||
c = paramiko.SSHClient(); c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
c.connect('192.168.0.6', username='sysadmin', password=PWD,
|
||||
timeout=30, banner_timeout=45, look_for_keys=False, allow_agent=False)
|
||||
|
||||
def ps(cmd, to=120):
|
||||
enc = base64.b64encode(cmd.encode('utf-16-le')).decode()
|
||||
_, o, e = c.exec_command(f'powershell -NoProfile -EncodedCommand {enc}', timeout=to)
|
||||
return o.read().decode('utf-8','replace'), e.read().decode('utf-8','replace')
|
||||
|
||||
print('[1] SFTP server_inventory.txt to AD2 (for back-population)')
|
||||
sftp = c.open_sftp()
|
||||
sftp.put(SERVER_INV_LOCAL, f'{PROD_DIR}/server_inventory.txt')
|
||||
sftp.close()
|
||||
out, _ = ps(f'$f = "{PROD_DIR.replace(chr(47),chr(92))}\\server_inventory.txt"; "bytes: $((Get-Item $f).Length)"; "lines: $((Get-Content $f).Count)"')
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[2] SFTP migration SQL + back-populate script + new files')
|
||||
sftp = c.open_sftp()
|
||||
uploads = [
|
||||
(f'{LOCAL_IMPL}\\database\\migrate-add-api-uploaded.sql', f'{REMOTE_DB}/migrate-add-api-uploaded.sql'),
|
||||
(f'{LOCAL_IMPL}\\database\\back-populate-api-uploaded.js', f'{REMOTE_DB}/back-populate-api-uploaded.js'),
|
||||
(f'{LOCAL_IMPL}\\database\\upload-to-api.js', f'{REMOTE_DB}/upload-to-api.js'),
|
||||
(f'{LOCAL_IMPL}\\routes\\api.js', f'{REMOTE_API}/api.js'),
|
||||
(f'{LOCAL_IMPL}\\public\\index.html', f'{REMOTE_WEB}/index.html'),
|
||||
]
|
||||
# Backups first
|
||||
for _, remote_dst in uploads:
|
||||
if remote_dst.endswith('.sql') or remote_dst.endswith('back-populate-api-uploaded.js'):
|
||||
continue # new file, no backup needed
|
||||
bak = remote_dst + f'.bak-{time.strftime("%Y%m%d-%H%M%S")}'
|
||||
try:
|
||||
with sftp.open(remote_dst, 'rb') as src, sftp.open(bak, 'wb') as dst:
|
||||
dst.write(src.read())
|
||||
print(f' backed up {remote_dst} -> {bak}')
|
||||
except Exception as e:
|
||||
print(f' backup skip {remote_dst}: {e}')
|
||||
# Uploads
|
||||
for local_src, remote_dst in uploads:
|
||||
sftp.put(local_src, remote_dst)
|
||||
print(f' uploaded {local_src} -> {remote_dst}')
|
||||
sftp.close()
|
||||
|
||||
print('\n[3] run migration via psql (env DATABASE_URL expected in service context; use psql -U testdatadb if set up)')
|
||||
# check db.js to understand connection info
|
||||
out, _ = ps(r'Get-Content "C:\Shares\testdatadb\database\db.js" | Select-String "host|user|database|port|connectionString" | Select -First 10 | Out-String')
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[3b] run migration via psql using .env creds')
|
||||
out, _ = ps(r'$env_file = "C:\Shares\testdatadb\.env"; if (Test-Path $env_file) { Get-Content $env_file } else { "no .env" }')
|
||||
print(out.rstrip())
|
||||
|
||||
# Try discovering via the db.js defaults + running migration with Node (safer than psql here)
|
||||
out, _ = ps(
|
||||
f'cd "{REMOTE_DB.replace("/","\\")}"; '
|
||||
r'& node -e "const db = require(''./db''); (async () => { '
|
||||
r'const sql = require(''fs'').readFileSync(''./migrate-add-api-uploaded.sql'', ''utf8''); '
|
||||
r'await db.execute(sql); console.log(''[MIG OK]''); '
|
||||
r'const c = await db.queryOne(\"SELECT COUNT(*) as c FROM information_schema.columns WHERE table_name=''test_records'' AND column_name=''api_uploaded_at''\"); '
|
||||
r'console.log(''column exists:'' + c.c); await db.close(); })().catch(e => { console.error(''[MIG FAIL]'', e.message); process.exit(1); });" 2>&1'
|
||||
, to=60)
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[4] run back-populate (batch 1000)')
|
||||
out, _ = ps(
|
||||
f'cd "{REMOTE_DB.replace("/","\\")}"; '
|
||||
f'& node back-populate-api-uploaded.js --inventory "{PROD_DIR.replace(chr(47),chr(92))}\\server_inventory.txt" --batch 1000 2>&1'
|
||||
, to=1200)
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[5] node --check updated files')
|
||||
out, _ = ps(
|
||||
f'cd "{REMOTE_DB.replace("/","\\")}"; & node --check upload-to-api.js; '
|
||||
f'cd "{REMOTE_API.replace("/","\\")}"; & node --check api.js; echo "[OK]"'
|
||||
, to=60)
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[6] restart testdatadb')
|
||||
out, _ = ps('Restart-Service testdatadb; Start-Sleep 3; Get-Service testdatadb | Select Name,Status | Format-Table -AutoSize | Out-String', to=60)
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[7] verify API')
|
||||
out, _ = ps(
|
||||
r'try { $r = Invoke-WebRequest "http://localhost:3000/api/stats" -UseBasicParsing -TimeoutSec 15; "GET /api/stats HTTP $($r.StatusCode)" } catch { "GET /api/stats FAIL: $_" }; '
|
||||
r'try { $r = Invoke-WebRequest "http://localhost:3000/api/search?limit=1" -UseBasicParsing -TimeoutSec 15; "GET /api/search HTTP $($r.StatusCode)"; $j = $r.Content | ConvertFrom-Json; "first record keys: $($j.records[0].PSObject.Properties.Name -join '', '')" } catch { "GET /api/search FAIL: $_" }'
|
||||
, to=30)
|
||||
print(out.rstrip())
|
||||
|
||||
c.close()
|
||||
print('\n[OK] deploy complete')
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Deploy the testdatadb upload integration to AD2 + convert scheduled task from hourly -> daily.
|
||||
|
||||
Steps:
|
||||
1. Backup current C:\\Shares\\testdatadb\\database\\import.js on AD2
|
||||
2. SFTP new upload-to-api.js + updated import.js
|
||||
3. node --check both on AD2 to be safe
|
||||
4. Restart testdatadb service to reload
|
||||
5. Re-register DataforthTestDatasheetUploader task as DAILY (was hourly)
|
||||
6. Verify task definition + show next run
|
||||
"""
|
||||
import base64, paramiko, subprocess, time, yaml
|
||||
|
||||
pwd_raw = yaml.safe_load(subprocess.run(['sops','-d','D:/vault/clients/dataforth/ad2.sops.yaml'],
|
||||
capture_output=True, text=True, timeout=30, check=True).stdout)['credentials']['password']
|
||||
PWD = pwd_raw # Vault has been fixed — no more `.replace('\\','')` needed
|
||||
|
||||
LOCAL = r'D:\claudetools\projects\dataforth-dos\datasheet-pipeline\implementation-upload\database'
|
||||
REMOTE = 'C:/Shares/testdatadb/database'
|
||||
|
||||
c = paramiko.SSHClient(); c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
c.connect('192.168.0.6', username='sysadmin', password=PWD,
|
||||
timeout=30, banner_timeout=45, look_for_keys=False, allow_agent=False)
|
||||
|
||||
def ps(cmd, to=120):
|
||||
enc = base64.b64encode(cmd.encode('utf-16-le')).decode()
|
||||
_, o, e = c.exec_command(f'powershell -NoProfile -EncodedCommand {enc}', timeout=to)
|
||||
return o.read().decode('utf-8','replace'), e.read().decode('utf-8','replace')
|
||||
|
||||
print('[1] backup import.js on AD2')
|
||||
out, _ = ps(f'Copy-Item -LiteralPath "{REMOTE.replace("/","\\")}\\import.js" -Destination "{REMOTE.replace("/","\\")}\\import.js.bak-$(Get-Date -Format yyyyMMdd-HHmmss)" -Force; Get-ChildItem "{REMOTE.replace("/","\\")}" -Filter "import.js*" | Select Name,Length | Format-Table -AutoSize | Out-String')
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[2] SFTP updated files')
|
||||
sftp = c.open_sftp()
|
||||
sftp.put(f'{LOCAL}/upload-to-api.js', f'{REMOTE}/upload-to-api.js')
|
||||
sftp.put(f'{LOCAL}/import.js', f'{REMOTE}/import.js')
|
||||
sftp.close()
|
||||
out, _ = ps(f'Get-ChildItem "{REMOTE.replace("/","\\")}" -Filter "upload-to-api.js","import.js" | Select Name,Length | Format-Table -AutoSize | Out-String')
|
||||
print(out.rstrip())
|
||||
|
||||
print('\n[3] node --check on both')
|
||||
out, err = ps(f'cd "{REMOTE.replace("/","\\")}"; & node --check upload-to-api.js 2>&1; echo "---"; & node --check import.js 2>&1; echo "---end"')
|
||||
print(out.rstrip())
|
||||
if err.strip() and 'CLIXML' not in err: print('[stderr]', err[:300])
|
||||
|
||||
print('\n[4] restart testdatadb service')
|
||||
out, err = ps('Restart-Service testdatadb; Start-Sleep 3; Get-Service testdatadb | Select Name,Status | Format-Table -AutoSize | Out-String', to=60)
|
||||
print(out.rstrip())
|
||||
if err.strip() and 'CLIXML' not in err: print('[stderr]', err[:300])
|
||||
|
||||
print('\n[5] re-register scheduled task as DAILY (was hourly)')
|
||||
REG = r'''
|
||||
$taskName = 'DataforthTestDatasheetUploader'
|
||||
$scriptPath = 'C:\ProgramData\dataforth-uploader\run-pipeline.ps1'
|
||||
Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue | Out-Null
|
||||
|
||||
$argStr = '-NoProfile -ExecutionPolicy Bypass -File ' + '"' + $scriptPath + '"'
|
||||
$action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument $argStr -WorkingDirectory 'C:\ProgramData\dataforth-uploader'
|
||||
# Daily at 02:30 server time (quiet hours)
|
||||
$trigger = New-ScheduledTaskTrigger -Daily -At (Get-Date -Hour 2 -Minute 30 -Second 0)
|
||||
$principal = New-ScheduledTaskPrincipal -UserId 'SYSTEM' -LogonType ServiceAccount -RunLevel Highest
|
||||
$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 30)
|
||||
Register-ScheduledTask -TaskName $taskName -Action $action -Trigger $trigger -Principal $principal -Settings $settings -Description 'Dataforth Test Datasheet Uploader daily fallback (primary path is import.js post-export hook)' | Out-Null
|
||||
|
||||
Write-Host '=== registered task ==='
|
||||
(Get-ScheduledTask -TaskName $taskName).Triggers | Format-List
|
||||
(Get-ScheduledTask -TaskName $taskName).Actions | Format-List
|
||||
Write-Host '=== next run ==='
|
||||
Get-ScheduledTaskInfo -TaskName $taskName | Select LastRunTime,LastTaskResult,NextRunTime | Format-List
|
||||
'''
|
||||
# Write to file on AD2, run it
|
||||
sftp = c.open_sftp()
|
||||
with sftp.open('C:/ProgramData/dataforth-uploader/register-daily.ps1', 'w') as fh:
|
||||
fh.write(REG)
|
||||
sftp.close()
|
||||
_, o, e = c.exec_command(r'powershell -NoProfile -ExecutionPolicy Bypass -File "C:\ProgramData\dataforth-uploader\register-daily.ps1"', timeout=60)
|
||||
print(o.read().decode('utf-8','replace'))
|
||||
err = e.read().decode('utf-8','replace')
|
||||
if err.strip() and 'CLIXML' not in err: print('[stderr]', err[:300])
|
||||
|
||||
c.close()
|
||||
print('\n[OK] deploy complete')
|
||||
@@ -0,0 +1,79 @@
|
||||
const fs = require('fs');
|
||||
const https = require('https');
|
||||
const { URL } = require('url');
|
||||
const db = require('./db');
|
||||
const CREDS = JSON.parse(fs.readFileSync('C:/ProgramData/dataforth-uploader/credentials.json', 'utf8'));
|
||||
|
||||
function req(method, uri, headers) {
|
||||
return new Promise((res, rej) => {
|
||||
const u = new URL(uri);
|
||||
const r = https.request({
|
||||
hostname: u.hostname, port: u.port || 443, path: u.pathname + u.search,
|
||||
method, headers, timeout: 20000,
|
||||
}, rs => {
|
||||
let d = '';
|
||||
rs.on('data', c => d += c);
|
||||
rs.on('end', () => res({ status: rs.statusCode, body: d }));
|
||||
});
|
||||
const t = setTimeout(() => { r.destroy(); rej(new Error('timeout')); }, 20000);
|
||||
r.on('error', rej);
|
||||
r.on('close', () => clearTimeout(t));
|
||||
r.end();
|
||||
});
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const form = 'grant_type=client_credentials&client_id=' + encodeURIComponent(CREDS.CF_CLIENT_ID) +
|
||||
'&client_secret=' + encodeURIComponent(CREDS.CF_CLIENT_SECRET) + '&scope=' + encodeURIComponent(CREDS.CF_SCOPE);
|
||||
const tokR = await new Promise((r, j) => {
|
||||
const u = new URL(CREDS.CF_TOKEN_URL);
|
||||
const rq = https.request({
|
||||
hostname: u.hostname, port: 443, path: u.pathname, method: 'POST',
|
||||
headers: { 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': Buffer.byteLength(form) },
|
||||
}, rs => {
|
||||
let d = '';
|
||||
rs.on('data', c => d += c);
|
||||
rs.on('end', () => r({ status: rs.statusCode, body: d }));
|
||||
});
|
||||
rq.on('error', j);
|
||||
rq.write(form);
|
||||
rq.end();
|
||||
});
|
||||
const token = JSON.parse(tokR.body).access_token;
|
||||
|
||||
async function sample(label, sql, expect) {
|
||||
console.log('=== ' + label + ' ===');
|
||||
const rows = await db.query(sql);
|
||||
let hit = 0, miss = 0, err = 0;
|
||||
for (const r of rows) {
|
||||
try {
|
||||
const rr = await req('GET',
|
||||
CREDS.CF_API_BASE + '/api/v1/TestReportDataFiles/' + encodeURIComponent(r.serial_number),
|
||||
{ 'Authorization': 'Bearer ' + token });
|
||||
if (rr.status === 200) hit++;
|
||||
else if (rr.status === 404) miss++;
|
||||
else { err++; console.log(' HTTP ' + rr.status + ' ' + r.serial_number); }
|
||||
} catch (e) { err++; console.log(' ERR ' + r.serial_number + ' ' + e.message); }
|
||||
}
|
||||
console.log(' hit=' + hit + ' miss=' + miss + ' err=' + err + ' (' + expect + ')');
|
||||
return { hit, miss, err };
|
||||
}
|
||||
|
||||
await sample(
|
||||
'Sample 1: 100 random stamped api_uploaded_at IS NOT NULL',
|
||||
"SELECT serial_number FROM test_records WHERE api_uploaded_at IS NOT NULL ORDER BY random() LIMIT 100",
|
||||
'expect hit=100',
|
||||
);
|
||||
await sample(
|
||||
'Sample 2: 100 random unpushable PASS (NULL api_uploaded_at, PASS)',
|
||||
"SELECT serial_number FROM test_records WHERE api_uploaded_at IS NULL AND overall_result='PASS' ORDER BY random() LIMIT 100",
|
||||
'expect mostly miss (these are the 10K unpushables)',
|
||||
);
|
||||
await sample(
|
||||
'Sample 3: 50 random FAIL',
|
||||
"SELECT serial_number FROM test_records WHERE overall_result='FAIL' ORDER BY random() LIMIT 50",
|
||||
'expect miss=50 (FAILs never reach Hoffman)',
|
||||
);
|
||||
|
||||
await db.close();
|
||||
})().catch(e => { console.error('FATAL', e.message); process.exit(1); });
|
||||
@@ -0,0 +1,74 @@
|
||||
/**
|
||||
* One-time back-population of api_uploaded_at from server_inventory.txt.
|
||||
*
|
||||
* Reads SN list, UPDATEs test_records.api_uploaded_at = NOW() in batches
|
||||
* for records whose serial_number appears in the inventory.
|
||||
*
|
||||
* Usage: node back-populate-api-uploaded.js [--inventory path] [--batch 1000] [--dry-run]
|
||||
*/
|
||||
const fs = require('fs');
|
||||
const db = require('./db');
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const arg = (n, d) => { const i = args.indexOf(n); return i >= 0 ? args[i+1] : d; };
|
||||
const flag = n => args.includes(n);
|
||||
|
||||
const INVENTORY = arg('--inventory', 'C:\\ProgramData\\dataforth-uploader\\server_inventory.txt');
|
||||
const BATCH = parseInt(arg('--batch', '1000'), 10);
|
||||
const DRY = flag('--dry-run');
|
||||
|
||||
async function main() {
|
||||
if (!fs.existsSync(INVENTORY)) {
|
||||
console.error(`[FAIL] inventory not found: ${INVENTORY}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = fs.readFileSync(INVENTORY, 'utf8');
|
||||
const sns = data.split(/\r?\n/).map(s => s.trim()).filter(Boolean);
|
||||
console.log(`[INFO] inventory: ${sns.length} serial numbers`);
|
||||
console.log(`[INFO] batch size: ${BATCH} dry-run: ${DRY}`);
|
||||
|
||||
const t0 = Date.now();
|
||||
let totalMatched = 0;
|
||||
|
||||
for (let i = 0; i < sns.length; i += BATCH) {
|
||||
const chunk = sns.slice(i, i + BATCH);
|
||||
const placeholders = chunk.map((_, j) => `$${j + 1}`).join(',');
|
||||
if (DRY) {
|
||||
const row = await db.queryOne(
|
||||
`SELECT COUNT(*) as c FROM test_records WHERE serial_number IN (${placeholders}) AND api_uploaded_at IS NULL`,
|
||||
chunk,
|
||||
);
|
||||
totalMatched += parseInt(row.c, 10) || 0;
|
||||
} else {
|
||||
const result = await db.execute(
|
||||
`UPDATE test_records SET api_uploaded_at = NOW() WHERE serial_number IN (${placeholders}) AND api_uploaded_at IS NULL`,
|
||||
chunk,
|
||||
);
|
||||
totalMatched += result.rowCount || 0;
|
||||
}
|
||||
if ((i / BATCH) % 20 === 0) {
|
||||
const rate = (i + chunk.length) / Math.max(1, (Date.now() - t0) / 1000);
|
||||
const eta = Math.round((sns.length - i - chunk.length) / Math.max(1, rate));
|
||||
console.log(` progress ${i + chunk.length}/${sns.length} matched-so-far=${totalMatched} rate=${rate.toFixed(0)}/s eta=${eta}s`);
|
||||
}
|
||||
}
|
||||
|
||||
const elapsed = ((Date.now() - t0) / 1000).toFixed(1);
|
||||
console.log(`\n[DONE] ${elapsed}s`);
|
||||
console.log(` inventory size: ${sns.length}`);
|
||||
console.log(` ${DRY ? 'would update' : 'updated'}: ${totalMatched}`);
|
||||
|
||||
// Sanity: how many records have api_uploaded_at set vs null?
|
||||
const tot = await db.queryOne(`SELECT COUNT(*) as c FROM test_records`);
|
||||
const set = await db.queryOne(`SELECT COUNT(*) as c FROM test_records WHERE api_uploaded_at IS NOT NULL`);
|
||||
const nul = await db.queryOne(`SELECT COUNT(*) as c FROM test_records WHERE api_uploaded_at IS NULL`);
|
||||
console.log(`\n[DB STATE]`);
|
||||
console.log(` total records: ${tot.c}`);
|
||||
console.log(` api_uploaded_at SET: ${set.c}`);
|
||||
console.log(` api_uploaded_at NULL: ${nul.c}`);
|
||||
|
||||
await db.close();
|
||||
}
|
||||
|
||||
main().catch(e => { console.error('[FATAL]', e); process.exit(1); });
|
||||
@@ -0,0 +1,257 @@
|
||||
/**
|
||||
* Export Datasheets
|
||||
*
|
||||
* Generates TXT datasheets for unexported PASS records and writes them to X:\For_Web\.
|
||||
* Updates forweb_exported_at after successful export.
|
||||
*
|
||||
* Usage:
|
||||
* node export-datasheets.js Export all pending (batch mode)
|
||||
* node export-datasheets.js --limit 100 Export up to 100 records
|
||||
* node export-datasheets.js --file <paths> Export records matching specific source files
|
||||
* node export-datasheets.js --serial 178439-1 Export a specific serial number
|
||||
* node export-datasheets.js --dry-run Show what would be exported without writing
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const db = require('./db');
|
||||
|
||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||
|
||||
// Configuration
|
||||
const OUTPUT_DIR = 'X:\\For_Web';
|
||||
const BATCH_SIZE = 500;
|
||||
|
||||
async function run() {
|
||||
const args = process.argv.slice(2);
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const limitIdx = args.indexOf('--limit');
|
||||
const limit = limitIdx >= 0 ? parseInt(args[limitIdx + 1]) : 0;
|
||||
const serialIdx = args.indexOf('--serial');
|
||||
const serial = serialIdx >= 0 ? args[serialIdx + 1] : null;
|
||||
const fileIdx = args.indexOf('--file');
|
||||
const files = fileIdx >= 0 ? args.slice(fileIdx + 1).filter(f => !f.startsWith('--')) : null;
|
||||
|
||||
console.log('========================================');
|
||||
console.log('Datasheet Export');
|
||||
console.log('========================================');
|
||||
console.log(`Output: ${OUTPUT_DIR}`);
|
||||
console.log(`Dry run: ${dryRun}`);
|
||||
if (limit) console.log(`Limit: ${limit}`);
|
||||
if (serial) console.log(`Serial: ${serial}`);
|
||||
console.log(`Start: ${new Date().toISOString()}`);
|
||||
|
||||
if (!dryRun && !fs.existsSync(OUTPUT_DIR)) {
|
||||
console.error(`ERROR: Output directory does not exist: ${OUTPUT_DIR}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log('\nLoading model specs...');
|
||||
const specMap = loadAllSpecs();
|
||||
|
||||
// Build query
|
||||
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
if (serial) {
|
||||
paramIdx++;
|
||||
conditions.push(`serial_number = $${paramIdx}`);
|
||||
params.push(serial);
|
||||
}
|
||||
|
||||
if (files && files.length > 0) {
|
||||
const placeholders = files.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
||||
conditions.push(`source_file IN (${placeholders})`);
|
||||
params.push(...files);
|
||||
}
|
||||
|
||||
let sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')} ORDER BY test_date DESC`;
|
||||
|
||||
if (limit) {
|
||||
paramIdx++;
|
||||
sql += ` LIMIT $${paramIdx}`;
|
||||
params.push(limit);
|
||||
}
|
||||
|
||||
const records = await db.query(sql, params);
|
||||
console.log(`\nFound ${records.length} records to export`);
|
||||
|
||||
if (records.length === 0) {
|
||||
console.log('Nothing to export.');
|
||||
await db.close();
|
||||
return { exported: 0, skipped: 0, errors: 0 };
|
||||
}
|
||||
|
||||
let exported = 0;
|
||||
let skipped = 0;
|
||||
let errors = 0;
|
||||
let noSpecs = 0;
|
||||
let pendingUpdates = [];
|
||||
|
||||
for (const record of records) {
|
||||
try {
|
||||
const filename = record.serial_number + '.TXT';
|
||||
const outputPath = path.join(OUTPUT_DIR, filename);
|
||||
|
||||
// VASLOG_ENG: verbatim byte-for-byte copy of the original file.
|
||||
// Using fs.copyFileSync avoids any utf-8 round-trip that would
|
||||
// corrupt non-ASCII bytes (CP1252 etc.) in customer datasheets.
|
||||
// Fall back to writing raw_data if the source file is gone.
|
||||
if (record.log_type === 'VASLOG_ENG') {
|
||||
if (dryRun) {
|
||||
console.log(` [DRY RUN] Would copy: ${record.source_file} -> ${filename}`);
|
||||
exported++;
|
||||
continue;
|
||||
}
|
||||
if (record.source_file && fs.existsSync(record.source_file)) {
|
||||
fs.copyFileSync(record.source_file, outputPath);
|
||||
} else {
|
||||
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
||||
if (!record.raw_data) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
||||
}
|
||||
pendingUpdates.push(record.id);
|
||||
exported++;
|
||||
|
||||
if (pendingUpdates.length >= BATCH_SIZE) {
|
||||
await flushUpdates(pendingUpdates);
|
||||
pendingUpdates = [];
|
||||
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Template-generated datasheet path.
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
if (!specs) {
|
||||
noSpecs++;
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
const txt = generateExactDatasheet(record, specs);
|
||||
if (!txt) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
console.log(` [DRY RUN] Would write: ${filename}`);
|
||||
exported++;
|
||||
} else {
|
||||
fs.writeFileSync(outputPath, txt, 'utf8');
|
||||
pendingUpdates.push(record.id);
|
||||
exported++;
|
||||
|
||||
// Batch commit
|
||||
if (pendingUpdates.length >= BATCH_SIZE) {
|
||||
await flushUpdates(pendingUpdates);
|
||||
pendingUpdates = [];
|
||||
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`\n ERROR exporting ${record.serial_number}: ${err.message}`);
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
|
||||
// Flush remaining updates
|
||||
if (pendingUpdates.length > 0) {
|
||||
await flushUpdates(pendingUpdates);
|
||||
}
|
||||
|
||||
console.log(`\n\n========================================`);
|
||||
console.log(`Export Complete`);
|
||||
console.log(`========================================`);
|
||||
console.log(`Exported: ${exported}`);
|
||||
console.log(`Skipped: ${skipped} (${noSpecs} missing specs)`);
|
||||
console.log(`Errors: ${errors}`);
|
||||
console.log(`End: ${new Date().toISOString()}`);
|
||||
|
||||
await db.close();
|
||||
return { exported, skipped, errors };
|
||||
}
|
||||
|
||||
async function flushUpdates(ids) {
|
||||
const now = new Date().toISOString();
|
||||
await db.transaction(async (txClient) => {
|
||||
for (const id of ids) {
|
||||
await txClient.execute(
|
||||
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
||||
[now, id]
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Export function for use by import.js (no db argument -- uses shared pool)
|
||||
async function exportNewRecords(specMap, filePaths) {
|
||||
if (!fs.existsSync(OUTPUT_DIR)) {
|
||||
console.log(`[EXPORT] Output directory not found: ${OUTPUT_DIR}`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
if (filePaths && filePaths.length > 0) {
|
||||
const placeholders = filePaths.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
||||
conditions.push(`source_file IN (${placeholders})`);
|
||||
params.push(...filePaths);
|
||||
}
|
||||
|
||||
const sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')}`;
|
||||
const records = await db.query(sql, params);
|
||||
if (records.length === 0) return 0;
|
||||
|
||||
let exported = 0;
|
||||
|
||||
await db.transaction(async (txClient) => {
|
||||
for (const record of records) {
|
||||
const filename = record.serial_number + '.TXT';
|
||||
const outputPath = path.join(OUTPUT_DIR, filename);
|
||||
|
||||
try {
|
||||
// VASLOG_ENG: verbatim copy, preserving original bytes.
|
||||
if (record.log_type === 'VASLOG_ENG') {
|
||||
if (record.source_file && fs.existsSync(record.source_file)) {
|
||||
fs.copyFileSync(record.source_file, outputPath);
|
||||
} else {
|
||||
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
||||
if (!record.raw_data) continue;
|
||||
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
||||
}
|
||||
} else {
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
if (!specs) continue;
|
||||
const txt = generateExactDatasheet(record, specs);
|
||||
if (!txt) continue;
|
||||
fs.writeFileSync(outputPath, txt, 'utf8');
|
||||
}
|
||||
|
||||
await txClient.execute(
|
||||
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
||||
[new Date().toISOString(), record.id]
|
||||
);
|
||||
exported++;
|
||||
} catch (err) {
|
||||
console.error(`[EXPORT] Error writing ${filename}: ${err.message}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`[EXPORT] Generated ${exported} datasheet(s)`);
|
||||
return exported;
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
run().catch(console.error);
|
||||
}
|
||||
|
||||
module.exports = { exportNewRecords };
|
||||
@@ -0,0 +1,416 @@
|
||||
/**
|
||||
* Data Import Script
|
||||
* Imports test data from DAT and SHT files into PostgreSQL database
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const db = require('./db');
|
||||
|
||||
const { parseMultilineFile, extractTestStation } = require('../parsers/multiline');
|
||||
const { parseCsvFile } = require('../parsers/csvline');
|
||||
const { parseShtFile } = require('../parsers/shtfile');
|
||||
const { parseVaslogEngTxt } = require('../parsers/vaslog-engtxt');
|
||||
|
||||
// Data source paths
|
||||
const TEST_PATH = 'C:/Shares/test';
|
||||
const RECOVERY_PATH = 'C:/Shares/Recovery-TEST';
|
||||
const HISTLOGS_PATH = path.join(TEST_PATH, 'Ate/HISTLOGS');
|
||||
|
||||
// Log types and their parsers.
|
||||
// NOTE: `recursive` defaults to TRUE when absent (walk subfolders by default,
|
||||
// preserving pre-existing production behavior for DSCLOG/5BLOG/8BLOG/PWRLOG/
|
||||
// SCTLOG/7BLOG). Set it to FALSE explicitly on VASLOG so the .DAT walk does
|
||||
// NOT descend into the "VASLOG - Engineering Tested" subfolder (belt-and-
|
||||
// suspenders: the .DAT glob wouldn't match .txt, but be explicit anyway).
|
||||
// VASLOG_ENG also sets recursive:false -- the eng-tested dir is flat.
|
||||
const LOG_TYPES = {
|
||||
'DSCLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'5BLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'8BLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'PWRLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'SCTLOG': { parser: 'multiline', ext: '.DAT' },
|
||||
'VASLOG': { parser: 'multiline', ext: '.DAT', recursive: false },
|
||||
'7BLOG': { parser: 'csvline', ext: '.DAT' },
|
||||
// Engineering-tested SCMHVAS pre-rendered datasheets live under VASLOG/"VASLOG - Engineering Tested"/
|
||||
'VASLOG_ENG': { parser: 'vaslog-engtxt', ext: '.txt', dir: 'VASLOG/VASLOG - Engineering Tested', recursive: false }
|
||||
};
|
||||
|
||||
// Find all files of a specific type in a directory
|
||||
function findFiles(dir, pattern, recursive = true) {
|
||||
const results = [];
|
||||
|
||||
try {
|
||||
if (!fs.existsSync(dir)) return results;
|
||||
|
||||
const items = fs.readdirSync(dir, { withFileTypes: true });
|
||||
|
||||
for (const item of items) {
|
||||
const fullPath = path.join(dir, item.name);
|
||||
|
||||
if (item.isDirectory() && recursive) {
|
||||
results.push(...findFiles(fullPath, pattern, recursive));
|
||||
} else if (item.isFile()) {
|
||||
if (pattern.test(item.name)) {
|
||||
results.push(fullPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
// Ignore permission errors
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
// Parse records from a file (sync -- file I/O only)
|
||||
function parseFile(filePath, logType, parser) {
|
||||
const testStation = extractTestStation(filePath);
|
||||
|
||||
switch (parser) {
|
||||
case 'multiline':
|
||||
return parseMultilineFile(filePath, logType, testStation);
|
||||
case 'csvline':
|
||||
return parseCsvFile(filePath, testStation);
|
||||
case 'shtfile':
|
||||
return parseShtFile(filePath, testStation);
|
||||
case 'vaslog-engtxt':
|
||||
return parseVaslogEngTxt(filePath, testStation);
|
||||
default:
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
// Batch insert records into PostgreSQL
|
||||
async function insertBatch(txClient, records) {
|
||||
let imported = 0;
|
||||
for (const record of records) {
|
||||
try {
|
||||
const result = await txClient.execute(
|
||||
`INSERT INTO test_records
|
||||
(log_type, model_number, serial_number, test_date, test_station, overall_result, raw_data, source_file)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
||||
ON CONFLICT (serial_number) DO UPDATE SET
|
||||
log_type = EXCLUDED.log_type,
|
||||
model_number = EXCLUDED.model_number,
|
||||
test_date = EXCLUDED.test_date,
|
||||
test_station = EXCLUDED.test_station,
|
||||
overall_result = EXCLUDED.overall_result,
|
||||
raw_data = EXCLUDED.raw_data,
|
||||
source_file = EXCLUDED.source_file,
|
||||
api_uploaded_at = NULL,
|
||||
forweb_exported_at = NULL
|
||||
WHERE test_records.overall_result = 'FAIL'
|
||||
OR (EXCLUDED.overall_result = 'PASS' AND EXCLUDED.test_date > test_records.test_date)`,
|
||||
[
|
||||
record.log_type,
|
||||
record.model_number,
|
||||
record.serial_number,
|
||||
record.test_date,
|
||||
record.test_station,
|
||||
record.overall_result,
|
||||
record.raw_data,
|
||||
record.source_file
|
||||
]
|
||||
);
|
||||
if (result.rowCount > 0) imported++;
|
||||
} catch (err) {
|
||||
// Constraint error - skip
|
||||
}
|
||||
}
|
||||
return imported;
|
||||
}
|
||||
|
||||
// Import records from a file
|
||||
async function importFile(txClient, filePath, logType, parser) {
|
||||
let records = [];
|
||||
|
||||
try {
|
||||
records = parseFile(filePath, logType, parser);
|
||||
const imported = await insertBatch(txClient, records);
|
||||
return { total: records.length, imported };
|
||||
} catch (err) {
|
||||
console.error(`Error importing ${filePath}: ${err.message}`);
|
||||
return { total: 0, imported: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
// Import from HISTLOGS (master consolidated logs)
|
||||
async function importHistlogs(txClient) {
|
||||
console.log('\n=== Importing from HISTLOGS ===');
|
||||
|
||||
let totalImported = 0;
|
||||
let totalRecords = 0;
|
||||
|
||||
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
||||
const subdir = config.dir || logType;
|
||||
const logDir = path.join(HISTLOGS_PATH, subdir);
|
||||
|
||||
if (!fs.existsSync(logDir)) {
|
||||
console.log(` ${logType}: directory not found`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
||||
console.log(` ${logType}: found ${files.length} files`);
|
||||
|
||||
for (const file of files) {
|
||||
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(` HISTLOGS total: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||
return totalImported;
|
||||
}
|
||||
|
||||
// Import from test station logs
|
||||
async function importStationLogs(txClient, basePath, label) {
|
||||
console.log(`\n=== Importing from ${label} ===`);
|
||||
|
||||
let totalImported = 0;
|
||||
let totalRecords = 0;
|
||||
|
||||
const stationPattern = /^TS-\d+[LR]?$/i;
|
||||
let stations = [];
|
||||
|
||||
try {
|
||||
const items = fs.readdirSync(basePath, { withFileTypes: true });
|
||||
stations = items
|
||||
.filter(i => i.isDirectory() && stationPattern.test(i.name))
|
||||
.map(i => i.name);
|
||||
} catch (err) {
|
||||
console.log(` Error reading ${basePath}: ${err.message}`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
console.log(` Found stations: ${stations.join(', ')}`);
|
||||
|
||||
for (const station of stations) {
|
||||
const logsDir = path.join(basePath, station, 'LOGS');
|
||||
|
||||
if (!fs.existsSync(logsDir)) continue;
|
||||
|
||||
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
||||
const subdir = config.dir || logType;
|
||||
const logDir = path.join(logsDir, subdir);
|
||||
|
||||
if (!fs.existsSync(logDir)) continue;
|
||||
|
||||
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
||||
|
||||
for (const file of files) {
|
||||
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Also import SHT files
|
||||
const shtFiles = findFiles(basePath, /\.SHT$/i, true);
|
||||
console.log(` Found ${shtFiles.length} SHT files`);
|
||||
|
||||
for (const file of shtFiles) {
|
||||
const { total, imported } = await importFile(txClient, file, 'SHT', 'shtfile');
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
}
|
||||
|
||||
console.log(` ${label} total: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||
return totalImported;
|
||||
}
|
||||
|
||||
// Import from Recovery-TEST backups (newest first)
|
||||
async function importRecoveryBackups(txClient) {
|
||||
console.log('\n=== Importing from Recovery-TEST backups ===');
|
||||
|
||||
if (!fs.existsSync(RECOVERY_PATH)) {
|
||||
console.log(' Recovery-TEST directory not found');
|
||||
return 0;
|
||||
}
|
||||
|
||||
const backups = fs.readdirSync(RECOVERY_PATH, { withFileTypes: true })
|
||||
.filter(i => i.isDirectory() && /^\d{2}-\d{2}-\d{2}$/.test(i.name))
|
||||
.map(i => i.name)
|
||||
.sort()
|
||||
.reverse();
|
||||
|
||||
console.log(` Found backup dates: ${backups.join(', ')}`);
|
||||
|
||||
let totalImported = 0;
|
||||
|
||||
for (const backup of backups) {
|
||||
const backupPath = path.join(RECOVERY_PATH, backup);
|
||||
const imported = await importStationLogs(txClient, backupPath, `Recovery-TEST/${backup}`);
|
||||
totalImported += imported;
|
||||
}
|
||||
|
||||
return totalImported;
|
||||
}
|
||||
|
||||
// Main import function
|
||||
async function runImport() {
|
||||
console.log('========================================');
|
||||
console.log('Test Data Import');
|
||||
console.log('========================================');
|
||||
console.log(`Start time: ${new Date().toISOString()}`);
|
||||
|
||||
let grandTotal = 0;
|
||||
|
||||
await db.transaction(async (txClient) => {
|
||||
grandTotal += await importHistlogs(txClient);
|
||||
grandTotal += await importRecoveryBackups(txClient);
|
||||
grandTotal += await importStationLogs(txClient, TEST_PATH, 'test');
|
||||
});
|
||||
|
||||
const stats = await db.queryOne('SELECT COUNT(*) as count FROM test_records');
|
||||
|
||||
console.log('\n========================================');
|
||||
console.log('Import Complete');
|
||||
console.log('========================================');
|
||||
console.log(`Total records in database: ${stats.count}`);
|
||||
console.log(`End time: ${new Date().toISOString()}`);
|
||||
|
||||
await db.close();
|
||||
}
|
||||
|
||||
// Import a single file (for incremental imports from sync)
|
||||
async function importSingleFile(filePath) {
|
||||
console.log(`Importing: ${filePath}`);
|
||||
|
||||
let logType = null;
|
||||
let parser = null;
|
||||
|
||||
// VASLOG_ENG subpath must be checked before VASLOG (substring overlap).
|
||||
if (filePath.includes('VASLOG - Engineering Tested')) {
|
||||
logType = 'VASLOG_ENG';
|
||||
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
||||
} else {
|
||||
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
||||
if (type === 'VASLOG_ENG') continue;
|
||||
if (filePath.includes(type)) {
|
||||
logType = type;
|
||||
parser = config.parser;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!logType) {
|
||||
if (/\.SHT$/i.test(filePath)) {
|
||||
logType = 'SHT';
|
||||
parser = 'shtfile';
|
||||
} else {
|
||||
console.log(` Unknown log type for: ${filePath}`);
|
||||
return { total: 0, imported: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
let result;
|
||||
await db.transaction(async (txClient) => {
|
||||
result = await importFile(txClient, filePath, logType, parser);
|
||||
});
|
||||
|
||||
console.log(` Imported ${result.imported} of ${result.total} records`);
|
||||
return result;
|
||||
}
|
||||
|
||||
// Import multiple files (for batch incremental imports)
|
||||
async function importFiles(filePaths) {
|
||||
console.log(`\n========================================`);
|
||||
console.log(`Incremental Import: ${filePaths.length} files`);
|
||||
console.log(`========================================`);
|
||||
|
||||
let totalImported = 0;
|
||||
let totalRecords = 0;
|
||||
|
||||
await db.transaction(async (txClient) => {
|
||||
for (const filePath of filePaths) {
|
||||
let logType = null;
|
||||
let parser = null;
|
||||
|
||||
// VASLOG_ENG subpath must be checked before the generic loop --
|
||||
// otherwise `includes('VASLOG')` hits first and the eng .txt gets
|
||||
// dispatched to the multiline parser. Mirror importSingleFile().
|
||||
if (filePath.includes('VASLOG - Engineering Tested')) {
|
||||
logType = 'VASLOG_ENG';
|
||||
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
||||
} else {
|
||||
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
||||
if (type === 'VASLOG_ENG') continue;
|
||||
if (filePath.includes(type)) {
|
||||
logType = type;
|
||||
parser = config.parser;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!logType) {
|
||||
if (/\.SHT$/i.test(filePath)) {
|
||||
logType = 'SHT';
|
||||
parser = 'shtfile';
|
||||
} else {
|
||||
console.log(` Skipping unknown type: ${filePath}`);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
const { total, imported } = await importFile(txClient, filePath, logType, parser);
|
||||
totalRecords += total;
|
||||
totalImported += imported;
|
||||
console.log(` ${path.basename(filePath)}: ${imported}/${total} records`);
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`\nTotal: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||
|
||||
// Export datasheets for newly imported records
|
||||
if (totalImported > 0) {
|
||||
try {
|
||||
const { loadAllSpecs } = require('../parsers/spec-reader');
|
||||
const { exportNewRecords } = require('./export-datasheets');
|
||||
const specMap = loadAllSpecs();
|
||||
await exportNewRecords(specMap, filePaths);
|
||||
} catch (err) {
|
||||
console.error(`[EXPORT] Datasheet export failed: ${err.message}`);
|
||||
}
|
||||
|
||||
// Push newly-exported datasheets to Dataforth's Hoffman API.
|
||||
// Best-effort; a failure here must not wedge the import flow. The
|
||||
// daily fallback scheduled task catches anything this missed.
|
||||
try {
|
||||
const { uploadNewRecords } = require('./upload-to-api');
|
||||
await uploadNewRecords(filePaths);
|
||||
} catch (err) {
|
||||
console.error(`[API-UPLOAD] upload after import failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
return { total: totalRecords, imported: totalImported };
|
||||
}
|
||||
|
||||
// Run if called directly
|
||||
if (require.main === module) {
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
if (args.length > 0 && args[0] === '--file') {
|
||||
const files = args.slice(1);
|
||||
if (files.length === 0) {
|
||||
console.log('Usage: node import.js --file <file1> [file2] ...');
|
||||
process.exit(1);
|
||||
}
|
||||
importFiles(files).then(() => db.close()).catch(console.error);
|
||||
} else if (args.length > 0 && args[0] === '--help') {
|
||||
console.log('Usage:');
|
||||
console.log(' node import.js Full import from all sources');
|
||||
console.log(' node import.js --file <f> Import specific file(s)');
|
||||
process.exit(0);
|
||||
} else {
|
||||
runImport().catch(console.error);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { runImport, importSingleFile, importFiles };
|
||||
@@ -0,0 +1,11 @@
|
||||
-- Adds api_uploaded_at tracking column + partial index for "not-yet-uploaded" queries.
|
||||
-- Safe to re-run (IF NOT EXISTS).
|
||||
|
||||
ALTER TABLE test_records
|
||||
ADD COLUMN IF NOT EXISTS api_uploaded_at TIMESTAMPTZ DEFAULT NULL;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_unuploaded_pass
|
||||
ON test_records(overall_result, forweb_exported_at)
|
||||
WHERE overall_result = 'PASS'
|
||||
AND forweb_exported_at IS NOT NULL
|
||||
AND api_uploaded_at IS NULL;
|
||||
@@ -0,0 +1,180 @@
|
||||
/**
|
||||
* Pull full Hoffman API inventory, diff against local DB, write three files:
|
||||
* - _hoffman_only_sns.txt (SNs on Hoffman not in local DB)
|
||||
* - _local_only_sns.txt (SNs in local DB not on Hoffman)
|
||||
* - _pull_inventory.log (progress and summary)
|
||||
*
|
||||
* Writes directly via fs.appendFileSync so progress survives SSH disconnects.
|
||||
* Run detached; tail the log file for progress.
|
||||
*/
|
||||
const fs = require('fs');
|
||||
const https = require('https');
|
||||
const { URL } = require('url');
|
||||
const db = require('./db');
|
||||
|
||||
const LOG = 'C:/Shares/testdatadb/database/_pull_inventory.log';
|
||||
const OUT_HOFFMAN = 'C:/Shares/testdatadb/database/_hoffman_only_sns.txt';
|
||||
const OUT_LOCAL = 'C:/Shares/testdatadb/database/_local_only_sns.txt';
|
||||
const CREDS_PATH = 'C:/ProgramData/dataforth-uploader/credentials.json';
|
||||
const PAGE_SIZE = 1000;
|
||||
|
||||
function log(msg) {
|
||||
const line = `[${new Date().toISOString()}] ${msg}\n`;
|
||||
fs.appendFileSync(LOG, line);
|
||||
}
|
||||
|
||||
function req(method, uri, headers) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const u = new URL(uri);
|
||||
const r = https.request({
|
||||
hostname: u.hostname, port: u.port || 443, path: u.pathname + u.search,
|
||||
method, headers, timeout: 45000,
|
||||
}, res => {
|
||||
let data = '';
|
||||
res.on('data', c => data += c);
|
||||
res.on('end', () => { clearTimeout(hardTimer); resolve({ status: res.statusCode, body: data }); });
|
||||
res.on('error', e => { clearTimeout(hardTimer); reject(e); });
|
||||
});
|
||||
// Hard deadline — some proxies keep TCP alive but never send data. If we
|
||||
// don't hear back in 45s, destroy the request and reject.
|
||||
const hardTimer = setTimeout(() => {
|
||||
r.destroy(new Error('hard deadline 45s'));
|
||||
reject(new Error('hard deadline 45s'));
|
||||
}, 45000);
|
||||
r.on('error', e => { clearTimeout(hardTimer); reject(e); });
|
||||
r.on('timeout', () => r.destroy(new Error('socket timeout')));
|
||||
r.end();
|
||||
});
|
||||
}
|
||||
|
||||
async function reqRetry(method, uri, headers, tries = 3) {
|
||||
let lastErr;
|
||||
for (let i = 0; i < tries; i++) {
|
||||
try { return await req(method, uri, headers); }
|
||||
catch (e) {
|
||||
lastErr = e;
|
||||
log(` retry ${i+1}/${tries} after ${e.message}`);
|
||||
await new Promise(r => setTimeout(r, 2000 * (i + 1)));
|
||||
}
|
||||
}
|
||||
throw lastErr;
|
||||
}
|
||||
|
||||
async function getToken(creds) {
|
||||
const form = 'grant_type=client_credentials' +
|
||||
'&client_id=' + encodeURIComponent(creds.CF_CLIENT_ID) +
|
||||
'&client_secret=' + encodeURIComponent(creds.CF_CLIENT_SECRET) +
|
||||
'&scope=' + encodeURIComponent(creds.CF_SCOPE);
|
||||
const r = await new Promise((res, rej) => {
|
||||
const u = new URL(creds.CF_TOKEN_URL);
|
||||
const rq = https.request({
|
||||
hostname: u.hostname, port: u.port || 443, path: u.pathname,
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/x-www-form-urlencoded',
|
||||
'Content-Length': Buffer.byteLength(form),
|
||||
},
|
||||
timeout: 30000,
|
||||
}, resp => {
|
||||
let d = '';
|
||||
resp.on('data', c => d += c);
|
||||
resp.on('end', () => res({ status: resp.statusCode, body: d }));
|
||||
});
|
||||
rq.on('error', rej);
|
||||
rq.write(form);
|
||||
rq.end();
|
||||
});
|
||||
const parsed = JSON.parse(r.body);
|
||||
if (!parsed.access_token) throw new Error('token fetch failed: ' + r.status + ' ' + r.body.slice(0, 200));
|
||||
return parsed.access_token;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
try {
|
||||
fs.writeFileSync(LOG, '');
|
||||
log('START inventory pull');
|
||||
|
||||
const creds = JSON.parse(fs.readFileSync(CREDS_PATH, 'utf8'));
|
||||
const token = await getToken(creds);
|
||||
log('token len=' + token.length);
|
||||
|
||||
const allSns = new Set();
|
||||
let page = 1;
|
||||
let total = null;
|
||||
const t0 = Date.now();
|
||||
const skippedPages = [];
|
||||
let consecutiveFailures = 0;
|
||||
while (true) {
|
||||
const url = creds.CF_API_BASE + '/api/v1/TestReportDataFiles?page=' + page + '&pageSize=' + PAGE_SIZE;
|
||||
let r;
|
||||
try {
|
||||
r = await reqRetry('GET', url, { 'Authorization': 'Bearer ' + token });
|
||||
consecutiveFailures = 0;
|
||||
} catch (e) {
|
||||
// Skip this page on sustained failure, don't abort.
|
||||
log(`page ${page} SKIPPED after retries: ${e.message}`);
|
||||
skippedPages.push(page);
|
||||
consecutiveFailures++;
|
||||
if (consecutiveFailures >= 10) {
|
||||
log(`FATAL: ${consecutiveFailures} consecutive page failures — aborting`);
|
||||
break;
|
||||
}
|
||||
page++;
|
||||
continue;
|
||||
}
|
||||
if (r.status !== 200) {
|
||||
log(`page ${page} HTTP ${r.status}: ${r.body.slice(0, 300)}`);
|
||||
break;
|
||||
}
|
||||
const obj = JSON.parse(r.body);
|
||||
total = obj.TotalCount;
|
||||
for (const it of obj.Items) allSns.add(it.SerialNumber);
|
||||
if (page === 1 || page % 50 === 0 || allSns.size >= total) {
|
||||
const rate = allSns.size / Math.max(1, (Date.now() - t0) / 1000);
|
||||
const eta = Math.round((total - allSns.size) / Math.max(rate, 1));
|
||||
log(`page ${page} collected ${allSns.size}/${total} rate ${rate.toFixed(0)}/s eta ${eta}s skipped=${skippedPages.length}`);
|
||||
}
|
||||
if (obj.Items.length < PAGE_SIZE || allSns.size >= total) break;
|
||||
page++;
|
||||
}
|
||||
if (skippedPages.length > 0) {
|
||||
log(`retrying ${skippedPages.length} skipped pages with longer delay`);
|
||||
for (const p of skippedPages) {
|
||||
const url = creds.CF_API_BASE + '/api/v1/TestReportDataFiles?page=' + p + '&pageSize=' + PAGE_SIZE;
|
||||
try {
|
||||
await new Promise(r => setTimeout(r, 3000));
|
||||
const r2 = await reqRetry('GET', url, { 'Authorization': 'Bearer ' + token });
|
||||
if (r2.status === 200) {
|
||||
const obj = JSON.parse(r2.body);
|
||||
for (const it of obj.Items) allSns.add(it.SerialNumber);
|
||||
log(` recovered page ${p} (+${obj.Items.length} SNs)`);
|
||||
}
|
||||
} catch (e) {
|
||||
log(` page ${p} still failed: ${e.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
log(`Hoffman inventory collected: ${allSns.size}`);
|
||||
|
||||
log('querying local DB...');
|
||||
const localRows = await db.query('SELECT serial_number FROM test_records');
|
||||
const localSns = new Set(localRows.map(r => r.serial_number));
|
||||
log(`Local DB unique SNs: ${localSns.size}`);
|
||||
|
||||
const hoffmanOnly = [];
|
||||
for (const s of allSns) if (!localSns.has(s)) hoffmanOnly.push(s);
|
||||
const localOnly = [];
|
||||
for (const s of localSns) if (!allSns.has(s)) localOnly.push(s);
|
||||
|
||||
fs.writeFileSync(OUT_HOFFMAN, hoffmanOnly.join('\n'));
|
||||
fs.writeFileSync(OUT_LOCAL, localOnly.join('\n'));
|
||||
log(`Hoffman-only (need pull): ${hoffmanOnly.length} -> ${OUT_HOFFMAN}`);
|
||||
log(`Local-only (not on Hoffman): ${localOnly.length} -> ${OUT_LOCAL}`);
|
||||
log('DONE');
|
||||
|
||||
await db.close();
|
||||
} catch (e) {
|
||||
log('FATAL: ' + e.message + '\n' + (e.stack || ''));
|
||||
process.exit(1);
|
||||
}
|
||||
})();
|
||||
@@ -0,0 +1,27 @@
|
||||
/**
|
||||
* In-memory equivalent of what export-datasheets.js writes to
|
||||
* X:\For_Web\<SN>.TXT. Lets upload-to-api.js POST directly to Hoffman's API
|
||||
* from DB state without a filesystem intermediate.
|
||||
*
|
||||
* Returns a string (datasheet text) or null if the record cannot be rendered
|
||||
* (no specs for the model, no raw_data for VASLOG_ENG, etc.).
|
||||
*/
|
||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||
|
||||
let _specMap = null;
|
||||
function specs() {
|
||||
if (_specMap === null) _specMap = loadAllSpecs();
|
||||
return _specMap;
|
||||
}
|
||||
|
||||
function renderContent(record) {
|
||||
if (record.log_type === 'VASLOG_ENG') {
|
||||
return record.raw_data || null;
|
||||
}
|
||||
const modelSpecs = getSpecs(specs(), record.model_number);
|
||||
if (!modelSpecs) return null;
|
||||
return generateExactDatasheet(record, modelSpecs) || null;
|
||||
}
|
||||
|
||||
module.exports = { renderContent };
|
||||
@@ -0,0 +1,215 @@
|
||||
/**
|
||||
* Post-import uploader — pushes just-imported records to Dataforth's Hoffman
|
||||
* API. Called from import.js after insertBatch, and from the /api/upload
|
||||
* endpoint for individual/bulk UI pushes.
|
||||
*
|
||||
* Datasheet content is rendered in memory from the DB row via
|
||||
* render-datasheet.renderContent — no For_Web filesystem dependency.
|
||||
*
|
||||
* Credentials come from C:\ProgramData\dataforth-uploader\credentials.json
|
||||
* (ACL'd to SYSTEM + Administrators + svc_testdatadb).
|
||||
*
|
||||
* The API is idempotent — already-present records return Unchanged.
|
||||
*/
|
||||
const fs = require('fs');
|
||||
const https = require('https');
|
||||
const { URL } = require('url');
|
||||
const db = require('./db');
|
||||
const { renderContent } = require('./render-datasheet');
|
||||
|
||||
const CREDS_PATH = 'C:\\ProgramData\\dataforth-uploader\\credentials.json';
|
||||
const BATCH = 100;
|
||||
const TOKEN_LEEWAY_MS = 60 * 1000;
|
||||
const HTTP_TIMEOUT_MS = 120 * 1000;
|
||||
|
||||
const RECORD_COLUMNS = 'id, log_type, model_number, serial_number, test_date, test_station, overall_result, raw_data, source_file';
|
||||
|
||||
let _creds = null;
|
||||
function loadCreds() {
|
||||
if (_creds) return _creds;
|
||||
if (!fs.existsSync(CREDS_PATH)) {
|
||||
throw new Error(`creds file not found: ${CREDS_PATH}`);
|
||||
}
|
||||
_creds = JSON.parse(fs.readFileSync(CREDS_PATH, 'utf8'));
|
||||
for (const k of ['CF_TOKEN_URL','CF_API_BASE','CF_CLIENT_ID','CF_CLIENT_SECRET','CF_SCOPE']) {
|
||||
if (!_creds[k]) throw new Error(`${CREDS_PATH} missing field ${k}`);
|
||||
}
|
||||
return _creds;
|
||||
}
|
||||
|
||||
let _tok = { value: null, expiresAt: 0 };
|
||||
|
||||
function httpPost(uri, body, headers) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const u = new URL(uri);
|
||||
const req = https.request({
|
||||
hostname: u.hostname, port: u.port || 443, path: u.pathname + u.search,
|
||||
method: 'POST',
|
||||
headers: Object.assign({}, headers, {'Content-Length': Buffer.byteLength(body)}),
|
||||
timeout: HTTP_TIMEOUT_MS,
|
||||
}, res => {
|
||||
let data = '';
|
||||
res.on('data', c => data += c);
|
||||
res.on('end', () => {
|
||||
try { resolve({status: res.statusCode, body: JSON.parse(data)}); }
|
||||
catch (e) { resolve({status: res.statusCode, body: {_raw: data}}); }
|
||||
});
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.on('timeout', () => req.destroy(new Error('http timeout')));
|
||||
req.write(body);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
async function getToken(force = false) {
|
||||
const c = loadCreds();
|
||||
if (!force && _tok.value && Date.now() < _tok.expiresAt - TOKEN_LEEWAY_MS) {
|
||||
return _tok.value;
|
||||
}
|
||||
const form = Object.entries({
|
||||
grant_type: 'client_credentials',
|
||||
client_id: c.CF_CLIENT_ID,
|
||||
client_secret: c.CF_CLIENT_SECRET,
|
||||
scope: c.CF_SCOPE,
|
||||
}).map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`).join('&');
|
||||
const r = await httpPost(c.CF_TOKEN_URL, form, {'Content-Type': 'application/x-www-form-urlencoded'});
|
||||
if (r.status !== 200 || !r.body.access_token) {
|
||||
throw new Error(`token fetch failed: ${r.status} ${JSON.stringify(r.body).slice(0,200)}`);
|
||||
}
|
||||
_tok.value = r.body.access_token;
|
||||
_tok.expiresAt = Date.now() + (r.body.expires_in || 3600) * 1000;
|
||||
return _tok.value;
|
||||
}
|
||||
|
||||
async function bulkPost(items) {
|
||||
const c = loadCreds();
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
const tok = await getToken(attempt > 0);
|
||||
try {
|
||||
const r = await httpPost(
|
||||
`${c.CF_API_BASE}/api/v1/TestReportDataFiles/bulk`,
|
||||
JSON.stringify({Items: items}),
|
||||
{'Authorization': `Bearer ${tok}`, 'Content-Type': 'application/json'},
|
||||
);
|
||||
if (r.status === 401 && attempt === 0) continue;
|
||||
return r;
|
||||
} catch (e) {
|
||||
if (attempt === 0) { await new Promise(r => setTimeout(r, 5000)); continue; }
|
||||
return {status: 0, body: {_error: e.message}};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function stampConfirmed(items, errors) {
|
||||
const badSns = new Set();
|
||||
for (const e of (errors || [])) {
|
||||
const matches = String(e).match(/\b\d+-\d+[A-Z]?\b/gi) || [];
|
||||
for (const m of matches) badSns.add(m);
|
||||
}
|
||||
const confirmedSns = items.map(it => it.SerialNumber).filter(sn => !badSns.has(sn));
|
||||
if (confirmedSns.length === 0) return;
|
||||
try {
|
||||
const placeholders = confirmedSns.map((_, j) => `$${j + 1}`).join(',');
|
||||
await db.execute(
|
||||
`UPDATE test_records SET api_uploaded_at = NOW() WHERE serial_number IN (${placeholders})`,
|
||||
confirmedSns,
|
||||
);
|
||||
} catch (e) {
|
||||
console.error(`[API-UPLOAD] stamp failed: ${e.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function uploadRecords(records, result) {
|
||||
const t0 = Date.now();
|
||||
for (let i = 0; i < records.length; i += BATCH) {
|
||||
const chunk = records.slice(i, i + BATCH);
|
||||
const items = [];
|
||||
for (const r of chunk) {
|
||||
let content;
|
||||
try {
|
||||
content = renderContent(r);
|
||||
} catch (e) {
|
||||
console.error(`[API-UPLOAD] render fail ${r.serial_number}: ${e.message}`);
|
||||
result.errors++;
|
||||
continue;
|
||||
}
|
||||
if (!content) { result.skipped++; continue; }
|
||||
items.push({ SerialNumber: r.serial_number, Content: content });
|
||||
}
|
||||
if (items.length === 0) continue;
|
||||
|
||||
let resp;
|
||||
try { resp = await bulkPost(items); }
|
||||
catch (e) {
|
||||
console.error(`[API-UPLOAD] batch threw: ${e.message}`);
|
||||
result.errors += items.length;
|
||||
continue;
|
||||
}
|
||||
if (resp.status !== 200) {
|
||||
console.error(`[API-UPLOAD] HTTP ${resp.status}: ${JSON.stringify(resp.body).slice(0,200)}`);
|
||||
result.errors += items.length;
|
||||
continue;
|
||||
}
|
||||
result.created += resp.body.Created || 0;
|
||||
result.updated += resp.body.Updated || 0;
|
||||
result.unchanged += resp.body.Unchanged || 0;
|
||||
result.errors += (resp.body.Errors || []).length;
|
||||
|
||||
await stampConfirmed(items, resp.body.Errors);
|
||||
}
|
||||
const elapsed = ((Date.now() - t0) / 1000).toFixed(1);
|
||||
console.log(`[API-UPLOAD] done in ${elapsed}s: created=${result.created} updated=${result.updated} unchanged=${result.unchanged} errors=${result.errors} skipped=${result.skipped}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Post-import upload. Non-throwing — upload failure must not wedge the import flow.
|
||||
* @param {string[]} filePaths - source_file values from the just-completed import
|
||||
*/
|
||||
async function uploadNewRecords(filePaths) {
|
||||
const result = {created: 0, updated: 0, unchanged: 0, errors: 0, skipped: 0};
|
||||
try {
|
||||
if (!filePaths || filePaths.length === 0) return result;
|
||||
if (!fs.existsSync(CREDS_PATH)) {
|
||||
console.log(`[API-UPLOAD] credentials not configured (${CREDS_PATH}); skipping`);
|
||||
return result;
|
||||
}
|
||||
const placeholders = filePaths.map((_, i) => `$${i + 1}`).join(',');
|
||||
const records = await db.query(
|
||||
`SELECT ${RECORD_COLUMNS} FROM test_records WHERE overall_result = 'PASS' AND source_file IN (${placeholders})`,
|
||||
filePaths,
|
||||
);
|
||||
if (records.length === 0) {
|
||||
console.log('[API-UPLOAD] no records eligible for upload');
|
||||
return result;
|
||||
}
|
||||
console.log(`[API-UPLOAD] ${records.length} records to upload`);
|
||||
await uploadRecords(records, result);
|
||||
} catch (e) {
|
||||
console.error(`[API-UPLOAD] fatal: ${e.message}`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Upload records identified by serial numbers. Called by /api/upload for
|
||||
* per-record and bulk UI pushes. Throws on hard failures so the endpoint
|
||||
* can return 500.
|
||||
*/
|
||||
async function uploadBySerialNumbers(sns) {
|
||||
const result = {created: 0, updated: 0, unchanged: 0, errors: 0, skipped: 0};
|
||||
if (!sns || sns.length === 0) return result;
|
||||
if (!fs.existsSync(CREDS_PATH)) {
|
||||
throw new Error(`credentials not configured (${CREDS_PATH})`);
|
||||
}
|
||||
const placeholders = sns.map((_, i) => `$${i + 1}`).join(',');
|
||||
const records = await db.query(
|
||||
`SELECT ${RECORD_COLUMNS} FROM test_records WHERE serial_number IN (${placeholders}) AND overall_result = 'PASS'`,
|
||||
sns,
|
||||
);
|
||||
if (records.length === 0) return result;
|
||||
await uploadRecords(records, result);
|
||||
return result;
|
||||
}
|
||||
|
||||
module.exports = { uploadNewRecords, uploadBySerialNumbers };
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,552 @@
|
||||
/**
|
||||
* API Routes for Test Data Database
|
||||
*
|
||||
* PostgreSQL version - uses pg.Pool via database/db.js.
|
||||
* All route handlers are async. FTS uses tsvector/plainto_tsquery.
|
||||
*/
|
||||
|
||||
const express = require('express');
|
||||
const path = require('path');
|
||||
const db = require('../database/db');
|
||||
const { generateDatasheet } = require('../templates/datasheet');
|
||||
|
||||
const router = express.Router();
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
const MAX_LIMIT = 1000;
|
||||
|
||||
function clampLimit(value) {
|
||||
const parsed = parseInt(value, 10);
|
||||
if (isNaN(parsed) || parsed < 1) return 100;
|
||||
return Math.min(parsed, MAX_LIMIT);
|
||||
}
|
||||
|
||||
function clampOffset(value) {
|
||||
const parsed = parseInt(value, 10);
|
||||
if (isNaN(parsed) || parsed < 0) return 0;
|
||||
return parsed;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/search
|
||||
// Search test records
|
||||
// Query params: serial, model, from, to, result, q, station, logtype, web_status, limit, offset
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/search', async (req, res) => {
|
||||
try {
|
||||
const { serial, model, from, to, result, q, station, logtype, workorder, web_status } = req.query;
|
||||
const limit = clampLimit(req.query.limit || 100);
|
||||
const offset = clampOffset(req.query.offset || 0);
|
||||
|
||||
const conditions = [];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
const addParam = (val) => {
|
||||
paramIdx++;
|
||||
params.push(val);
|
||||
return '$' + paramIdx;
|
||||
};
|
||||
|
||||
if (q) {
|
||||
// Full-text search using tsvector
|
||||
conditions.push(`search_vector @@ plainto_tsquery('english', ${addParam(q)})`);
|
||||
}
|
||||
|
||||
if (serial) {
|
||||
const val = serial.includes('%') ? serial : `%${serial}%`;
|
||||
conditions.push(`serial_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (workorder) {
|
||||
conditions.push(`work_order = ${addParam(workorder)}`);
|
||||
}
|
||||
|
||||
if (model) {
|
||||
const val = model.includes('%') ? model : `%${model}%`;
|
||||
conditions.push(`model_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (from) {
|
||||
conditions.push(`test_date >= ${addParam(from)}`);
|
||||
}
|
||||
|
||||
if (to) {
|
||||
conditions.push(`test_date <= ${addParam(to)}`);
|
||||
}
|
||||
|
||||
if (result) {
|
||||
conditions.push(`overall_result = ${addParam(result.toUpperCase())}`);
|
||||
}
|
||||
|
||||
if (station) {
|
||||
conditions.push(`test_station = ${addParam(station)}`);
|
||||
}
|
||||
|
||||
if (logtype) {
|
||||
conditions.push(`log_type = ${addParam(logtype)}`);
|
||||
}
|
||||
|
||||
if (req.query.web_status === 'off') {
|
||||
conditions.push('api_uploaded_at IS NULL');
|
||||
} else if (req.query.web_status === 'on') {
|
||||
conditions.push('api_uploaded_at IS NOT NULL');
|
||||
}
|
||||
|
||||
const where = conditions.length > 0 ? 'WHERE ' + conditions.join(' AND ') : '';
|
||||
|
||||
const dataSql = `SELECT * FROM test_records ${where} ORDER BY test_date DESC, serial_number LIMIT ${addParam(limit)} OFFSET ${addParam(offset)}`;
|
||||
const countSql = `SELECT COUNT(*) as count FROM test_records ${where}`;
|
||||
const countParams = params.slice(0, paramIdx - 2); // exclude limit/offset
|
||||
|
||||
const [records, countRow] = await Promise.all([
|
||||
db.query(dataSql, params),
|
||||
db.queryOne(countSql, countParams),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
records,
|
||||
total: countRow?.count ? parseInt(countRow.count, 10) : records.length,
|
||||
limit,
|
||||
offset
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [SEARCH ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/record/:id
|
||||
// Get single record by ID
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/record/:id', async (req, res) => {
|
||||
try {
|
||||
const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);
|
||||
|
||||
if (!record) {
|
||||
return res.status(404).json({ error: 'Record not found' });
|
||||
}
|
||||
|
||||
res.json(record);
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [RECORD ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/datasheet/:id
|
||||
// Generate datasheet for a record
|
||||
// Query params: format (html, txt)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/datasheet/:id', async (req, res) => {
|
||||
try {
|
||||
const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);
|
||||
|
||||
if (!record) {
|
||||
return res.status(404).json({ error: 'Record not found' });
|
||||
}
|
||||
|
||||
const format = req.query.format || 'html';
|
||||
|
||||
// Try exact-match formatter first
|
||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||
const specMap = loadAllSpecs();
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
const exactTxt = generateExactDatasheet(record, specs);
|
||||
|
||||
if (exactTxt && format === 'html') {
|
||||
// Render exact-match TXT as styled HTML page
|
||||
const escaped = exactTxt
|
||||
.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>');
|
||||
const html = `<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Test Data Sheet - ${record.serial_number}</title>
|
||||
<style>
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background: #f0f0f0;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
}
|
||||
.page {
|
||||
background: white;
|
||||
padding: 40px 30px;
|
||||
max-width: 720px;
|
||||
width: 100%;
|
||||
box-shadow: 0 2px 8px rgba(0,0,0,0.15);
|
||||
border: 1px solid #ccc;
|
||||
}
|
||||
pre {
|
||||
font-family: 'Courier New', Courier, monospace;
|
||||
font-size: 11px;
|
||||
line-height: 1.4;
|
||||
margin: 0;
|
||||
white-space: pre;
|
||||
overflow-x: auto;
|
||||
}
|
||||
.toolbar {
|
||||
position: fixed;
|
||||
top: 10px;
|
||||
right: 10px;
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
}
|
||||
.toolbar button {
|
||||
padding: 8px 16px;
|
||||
border: 1px solid #999;
|
||||
background: white;
|
||||
cursor: pointer;
|
||||
font-size: 13px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.toolbar button:hover { background: #e0e0e0; }
|
||||
@media print {
|
||||
body { background: white; padding: 0; }
|
||||
.page { box-shadow: none; border: none; padding: 0; }
|
||||
.toolbar { display: none; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="toolbar">
|
||||
<button onclick="window.print()">Print</button>
|
||||
<button onclick="window.open('/api/datasheet/${record.id}/pdf')">Download PDF</button>
|
||||
<button onclick="window.close()">Close</button>
|
||||
</div>
|
||||
<div class="page">
|
||||
<pre>${escaped}</pre>
|
||||
</div>
|
||||
</body>
|
||||
</html>`;
|
||||
res.type('html').send(html);
|
||||
} else if (exactTxt && format === 'txt') {
|
||||
res.type('text/plain').send(exactTxt);
|
||||
} else {
|
||||
// Fall back to generic template
|
||||
const datasheet = generateDatasheet(record, format);
|
||||
if (format === 'html') {
|
||||
res.type('html').send(datasheet);
|
||||
} else {
|
||||
res.type('text/plain').send(datasheet);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [DATASHEET ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/datasheet/:id/pdf
|
||||
// Generate PDF datasheet for a record (on-demand download)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/datasheet/:id/pdf', async (req, res) => {
|
||||
try {
|
||||
const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);
|
||||
|
||||
if (!record) {
|
||||
return res.status(404).json({ error: 'Record not found' });
|
||||
}
|
||||
|
||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||
const PDFDocument = require('pdfkit');
|
||||
|
||||
const specMap = loadAllSpecs();
|
||||
const specs = getSpecs(specMap, record.model_number);
|
||||
let txt = generateExactDatasheet(record, specs);
|
||||
|
||||
// Fall back to generic datasheet if exact-match formatter doesn't support this family
|
||||
if (!txt) {
|
||||
txt = generateDatasheet(record, 'txt');
|
||||
}
|
||||
|
||||
if (!txt) {
|
||||
return res.status(422).json({ error: 'Could not generate datasheet (missing specs or data)' });
|
||||
}
|
||||
|
||||
const doc = new PDFDocument({
|
||||
size: 'LETTER',
|
||||
margins: { top: 36, bottom: 36, left: 36, right: 36 }
|
||||
});
|
||||
|
||||
res.setHeader('Content-Type', 'application/pdf');
|
||||
res.setHeader('Content-Disposition', `attachment; filename="${record.serial_number}.pdf"`);
|
||||
doc.pipe(res);
|
||||
|
||||
doc.font('Courier').fontSize(9.5);
|
||||
const lines = txt.split(/\r?\n/);
|
||||
for (const line of lines) {
|
||||
doc.text(line, { lineGap: 1 });
|
||||
}
|
||||
|
||||
doc.end();
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [PDF ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/stats
|
||||
// Get database statistics
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/stats', async (req, res) => {
|
||||
try {
|
||||
const [totalRow, byLogType, byResult, byStation, dateRange, recentSerials] = await Promise.all([
|
||||
db.queryOne('SELECT COUNT(*) as count FROM test_records'),
|
||||
db.query('SELECT log_type, COUNT(*) as count FROM test_records GROUP BY log_type ORDER BY count DESC'),
|
||||
db.query('SELECT overall_result, COUNT(*) as count FROM test_records GROUP BY overall_result'),
|
||||
db.query(`SELECT test_station, COUNT(*) as count FROM test_records
|
||||
WHERE test_station IS NOT NULL AND test_station != ''
|
||||
GROUP BY test_station ORDER BY test_station`),
|
||||
db.queryOne('SELECT MIN(test_date) as oldest, MAX(test_date) as newest FROM test_records'),
|
||||
db.query(`SELECT DISTINCT serial_number, model_number, test_date
|
||||
FROM test_records ORDER BY test_date DESC LIMIT 10`),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
total_records: parseInt(totalRow.count, 10),
|
||||
by_log_type: byLogType.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
by_result: byResult.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
by_station: byStation.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
date_range: dateRange,
|
||||
recent_serials: recentSerials,
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [STATS ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/filters
|
||||
// Get available filter options (test stations, log types, models)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/filters', async (req, res) => {
|
||||
try {
|
||||
const [stations, logTypes, models] = await Promise.all([
|
||||
db.query(`SELECT DISTINCT test_station FROM test_records
|
||||
WHERE test_station IS NOT NULL AND test_station != ''
|
||||
ORDER BY test_station`),
|
||||
db.query('SELECT DISTINCT log_type FROM test_records ORDER BY log_type'),
|
||||
db.query(`SELECT DISTINCT model_number, COUNT(*) as count FROM test_records
|
||||
GROUP BY model_number ORDER BY count DESC LIMIT 500`),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
stations: stations.map(r => r.test_station),
|
||||
log_types: logTypes.map(r => r.log_type),
|
||||
models: models.map(r => ({ ...r, count: parseInt(r.count, 10) })),
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [FILTERS ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/export
|
||||
// Export search results as CSV
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/export', async (req, res) => {
|
||||
try {
|
||||
const { serial, model, from, to, result, station, logtype } = req.query;
|
||||
|
||||
const conditions = [];
|
||||
const params = [];
|
||||
let paramIdx = 0;
|
||||
|
||||
const addParam = (val) => {
|
||||
paramIdx++;
|
||||
params.push(val);
|
||||
return '$' + paramIdx;
|
||||
};
|
||||
|
||||
if (serial) {
|
||||
const val = serial.includes('%') ? serial : `%${serial}%`;
|
||||
conditions.push(`serial_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (model) {
|
||||
const val = model.includes('%') ? model : `%${model}%`;
|
||||
conditions.push(`model_number LIKE ${addParam(val)}`);
|
||||
}
|
||||
|
||||
if (from) {
|
||||
conditions.push(`test_date >= ${addParam(from)}`);
|
||||
}
|
||||
|
||||
if (to) {
|
||||
conditions.push(`test_date <= ${addParam(to)}`);
|
||||
}
|
||||
|
||||
if (result) {
|
||||
conditions.push(`overall_result = ${addParam(result.toUpperCase())}`);
|
||||
}
|
||||
|
||||
if (station) {
|
||||
conditions.push(`test_station = ${addParam(station)}`);
|
||||
}
|
||||
|
||||
if (logtype) {
|
||||
conditions.push(`log_type = ${addParam(logtype)}`);
|
||||
}
|
||||
|
||||
if (req.query.web_status === 'off') {
|
||||
conditions.push('api_uploaded_at IS NULL');
|
||||
} else if (req.query.web_status === 'on') {
|
||||
conditions.push('api_uploaded_at IS NOT NULL');
|
||||
}
|
||||
|
||||
const where = conditions.length > 0 ? 'WHERE ' + conditions.join(' AND ') : '';
|
||||
const sql = `SELECT * FROM test_records ${where} ORDER BY test_date DESC, serial_number LIMIT 10000`;
|
||||
|
||||
const records = await db.query(sql, params);
|
||||
|
||||
// Generate CSV
|
||||
const headers = ['id', 'log_type', 'model_number', 'serial_number', 'test_date', 'test_station', 'overall_result', 'source_file'];
|
||||
let csv = headers.join(',') + '\n';
|
||||
|
||||
for (const record of records) {
|
||||
const row = headers.map(h => {
|
||||
const val = record[h] || '';
|
||||
return `"${String(val).replace(/"/g, '""')}"`;
|
||||
});
|
||||
csv += row.join(',') + '\n';
|
||||
}
|
||||
|
||||
res.setHeader('Content-Type', 'text/csv');
|
||||
res.setHeader('Content-Disposition', 'attachment; filename=test_records.csv');
|
||||
res.send(csv);
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [EXPORT ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/workorder/:wo
|
||||
// Get work order details and all associated test lines
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/workorder/:wo', async (req, res) => {
|
||||
try {
|
||||
const wo = req.params.wo;
|
||||
|
||||
const [header, lines, testRecords] = await Promise.all([
|
||||
db.queryOne('SELECT * FROM work_orders WHERE wo_number = $1', [wo]),
|
||||
db.query('SELECT * FROM work_order_lines WHERE wo_number = $1 ORDER BY test_date, test_time', [wo]),
|
||||
db.query(
|
||||
'SELECT id, log_type, model_number, serial_number, test_date, test_station, overall_result, work_order FROM test_records WHERE work_order = $1 ORDER BY serial_number',
|
||||
[wo]
|
||||
),
|
||||
]);
|
||||
|
||||
res.json({
|
||||
work_order: header || { wo_number: wo },
|
||||
lines,
|
||||
test_records: testRecords,
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [WO ERROR] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// GET /api/workorder-search?q=<query>
|
||||
// Search work orders by number (prefix match)
|
||||
// ---------------------------------------------------------------------------
|
||||
router.get('/workorder-search', async (req, res) => {
|
||||
try {
|
||||
const q = req.query.q || '';
|
||||
if (q.length < 2) {
|
||||
return res.json({ results: [] });
|
||||
}
|
||||
|
||||
const results = await db.query(
|
||||
'SELECT wo_number, wo_date, program, test_station FROM work_orders WHERE wo_number LIKE $1 ORDER BY wo_date DESC LIMIT 50',
|
||||
[q + '%']
|
||||
);
|
||||
|
||||
res.json({ results });
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cleanup function for graceful shutdown
|
||||
// ---------------------------------------------------------------------------
|
||||
async function cleanup() {
|
||||
try {
|
||||
await db.close();
|
||||
} catch (err) {
|
||||
console.error(`[${new Date().toISOString()}] [CLEANUP ERROR] ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* POST /api/upload
|
||||
*
|
||||
* Body: { ids?: number[], serialNumbers?: string[], all_unuploaded?: boolean }
|
||||
*
|
||||
* Pushes selected records to the Dataforth website API. Accepts either a set
|
||||
* of record IDs (resolved to serial_number + checked for exported status), a
|
||||
* direct list of serial numbers, or all_unuploaded:true to push every PASS
|
||||
* record where api_uploaded_at IS NULL.
|
||||
*
|
||||
* Response: { created, updated, unchanged, errors, skipped, processed, sns }
|
||||
*/
|
||||
router.post('/upload', async (req, res) => {
|
||||
try {
|
||||
const { ids, serialNumbers, all_unuploaded } = req.body || {};
|
||||
const { uploadBySerialNumbers } = require('../database/upload-to-api');
|
||||
|
||||
let sns = [];
|
||||
if (all_unuploaded) {
|
||||
const rows = await db.query(
|
||||
`SELECT DISTINCT serial_number FROM test_records
|
||||
WHERE overall_result = 'PASS'
|
||||
AND api_uploaded_at IS NULL
|
||||
ORDER BY serial_number`
|
||||
);
|
||||
sns = rows.map(r => r.serial_number);
|
||||
} else if (Array.isArray(ids) && ids.length > 0) {
|
||||
const placeholders = ids.map((_, i) => `$${i + 1}`).join(',');
|
||||
const rows = await db.query(
|
||||
`SELECT DISTINCT serial_number FROM test_records
|
||||
WHERE id IN (${placeholders})
|
||||
AND overall_result = 'PASS'`,
|
||||
ids,
|
||||
);
|
||||
sns = rows.map(r => r.serial_number);
|
||||
} else if (Array.isArray(serialNumbers) && serialNumbers.length > 0) {
|
||||
sns = [...new Set(serialNumbers)];
|
||||
} else {
|
||||
return res.status(400).json({ error: 'provide ids[], serialNumbers[], or all_unuploaded=true' });
|
||||
}
|
||||
|
||||
if (sns.length === 0) {
|
||||
return res.json({ created:0, updated:0, unchanged:0, errors:0, skipped:0, processed:0, sns:[] });
|
||||
}
|
||||
|
||||
const result = await uploadBySerialNumbers(sns);
|
||||
res.json({ ...result, processed: sns.length, sns });
|
||||
} catch (err) {
|
||||
console.error(`[UPLOAD] ${err.message}`);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
module.exports.cleanup = cleanup;
|
||||
@@ -0,0 +1,27 @@
|
||||
import base64, paramiko, subprocess, yaml, os
|
||||
pwd = yaml.safe_load(subprocess.run(['sops','-d','D:/vault/clients/dataforth/ad2.sops.yaml'],
|
||||
capture_output=True, text=True, timeout=30, check=True).stdout)['credentials']['password']
|
||||
c = paramiko.SSHClient(); c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
c.connect('192.168.0.6', username='sysadmin', password=pwd, timeout=30, look_for_keys=False, allow_agent=False)
|
||||
|
||||
def ps(cmd, to=60):
|
||||
enc = base64.b64encode(cmd.encode('utf-16-le')).decode()
|
||||
_, o, _ = c.exec_command(f'powershell -NoProfile -EncodedCommand {enc}', timeout=to)
|
||||
return o.read().decode('utf-8','replace')
|
||||
|
||||
print('=== database folder ===')
|
||||
print(ps(r'Get-ChildItem "C:\Shares\testdatadb\database" | Select Name,Length,LastWriteTime | Format-Table -AutoSize | Out-String'))
|
||||
|
||||
print('\n=== testdatadb root ===')
|
||||
print(ps(r'Get-ChildItem "C:\Shares\testdatadb" | Select Name,Mode,Length | Format-Table -AutoSize | Out-String'))
|
||||
|
||||
print('\n=== .env ===')
|
||||
print(ps(r'Get-Content "C:\Shares\testdatadb\.env" -Raw -ErrorAction SilentlyContinue'))
|
||||
|
||||
print('\n=== package.json deps ===')
|
||||
print(ps(r'(Get-Content "C:\Shares\testdatadb\package.json" -Raw) -replace ".*dependencies", "dependencies"'))
|
||||
|
||||
print('\n=== schema (first 50 lines of schema-pg or schema.sql) ===')
|
||||
print(ps(r'$f = @("C:\Shares\testdatadb\database\schema-pg.sql", "C:\Shares\testdatadb\database\schema.sql") | Where-Object { Test-Path $_ } | Select -First 1; "using: $f"; Get-Content $f | Select -First 80'))
|
||||
|
||||
c.close()
|
||||
140
projects/dataforth-dos/session-logs/2026-04-15-session.md
Normal file
140
projects/dataforth-dos/session-logs/2026-04-15-session.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Dataforth — 2026-04-15 Session Log
|
||||
|
||||
Long session covering UI feature completion, DB cleanup, architectural refactor, bulk data sync, production incident, and sanity verification against Hoffman's API.
|
||||
|
||||
## Major accomplishments
|
||||
|
||||
### 1. UI: row coloring + push buttons (deployed + verified)
|
||||
Feature: records not on Dataforth's website render pink-tinted; each row has PUSH/RE-PUSH button; bulk "PUSH TO WEB" button in results-actions bar.
|
||||
|
||||
Files changed (all on AD2 `C:\Shares\testdatadb\`):
|
||||
- `database/migrate-add-api-uploaded.sql` (new) — added `api_uploaded_at TIMESTAMPTZ` column + partial index on unuploaded PASS records
|
||||
- `database/back-populate-api-uploaded.js` (new) — one-time back-population from `server_inventory.txt`
|
||||
- `database/upload-to-api.js` (rewritten — see refactor below)
|
||||
- `routes/api.js` — added `POST /api/upload` endpoint accepting `{ids?, serialNumbers?, all_unuploaded?}` body
|
||||
- `public/index.html` — CSS `tr.not-on-web` pink tint, `.action-link.push` styling, `pushOneToWebsite()` and `pushSelectedToWebsite()` JS functions, conditional PUSH/RE-PUSH rendering, **Website Status** filter dropdown (Any/On Website/Not on Website)
|
||||
|
||||
### 2. Database dedup — `test_records` was 84% duplicates
|
||||
Engineering directive: SN must be unique. Before: 2,889,243 rows. After: 469,009 rows.
|
||||
|
||||
Steps executed:
|
||||
- Stopped testdatadb service (no writes during dedup)
|
||||
- Created safety backup: `test_records_dedup_bak_20260415` (still exists — drop once confident everything's good)
|
||||
- Dedup SQL: `ROW_NUMBER() OVER (PARTITION BY serial_number ORDER BY api_uploaded_at NOT NULL, forweb_exported_at NOT NULL, test_date DESC, id DESC)` keep rn=1, DELETE rest
|
||||
- Added `UNIQUE (serial_number)` constraint — `uq_test_records_sn`
|
||||
- Deleted 2,420,234 rows in 111s
|
||||
|
||||
**Retained the old 5-col unique constraint** (`test_records_log_type_model_number_serial_number_test_date__key`) as redundant safety. No harm, minor write overhead. Can drop later.
|
||||
|
||||
### 3. import.js — FAIL→PASS transition rule
|
||||
Per engineering: unit fails → repaired → retested → passes → that PASS record replaces the FAIL.
|
||||
|
||||
New ON CONFLICT logic in `database/import.js` `insertBatch()`:
|
||||
```sql
|
||||
INSERT ... ON CONFLICT (serial_number) DO UPDATE SET
|
||||
log_type=EXCLUDED.log_type, model_number=EXCLUDED.model_number,
|
||||
test_date=EXCLUDED.test_date, test_station=EXCLUDED.test_station,
|
||||
overall_result=EXCLUDED.overall_result, raw_data=EXCLUDED.raw_data,
|
||||
source_file=EXCLUDED.source_file,
|
||||
api_uploaded_at=NULL, forweb_exported_at=NULL
|
||||
WHERE test_records.overall_result = 'FAIL'
|
||||
OR (EXCLUDED.overall_result = 'PASS' AND EXCLUDED.test_date > test_records.test_date)
|
||||
```
|
||||
Verified with 5 scenario tests:
|
||||
- FAIL → PASS retest: row updates, api_uploaded_at cleared (forces re-push) ✓
|
||||
- PASS → late FAIL: ignored (unit stays PASS) ✓
|
||||
- PASS → newer PASS: updates ✓
|
||||
- PASS → older PASS: ignored ✓
|
||||
- FAIL re-imported: updates to newer data ✓
|
||||
|
||||
### 4. Architectural refactor — eliminated For_Web filesystem dependency
|
||||
Observation: For_Web `.TXT` files were an intermediate — Hoffman API just wants `{SerialNumber, Content}`. Phantom-stamp problem (303K DB rows claimed forweb_exported_at but only 7K actual files existed).
|
||||
|
||||
Created `database/render-datasheet.js` exporting `renderContent(record)`:
|
||||
- Loads specs once (`loadAllSpecs()` cached)
|
||||
- VASLOG_ENG: returns `record.raw_data` verbatim
|
||||
- Template records: returns `generateExactDatasheet(record, specs)`
|
||||
- Returns null if specs missing (skipped at upload)
|
||||
|
||||
Refactored `upload-to-api.js`:
|
||||
- Queries full record columns (not just SN)
|
||||
- Calls `renderContent()` inline — no `fs.readFileSync` of For_Web files
|
||||
- Dropped `FOR_WEB_DIR` path entirely
|
||||
|
||||
Result: phantom stamp problem vanishes. PUSH button works for any PASS record where specs exist.
|
||||
|
||||
### 5. Bulk push — 170,984 records created on Hoffman
|
||||
Two runs combined:
|
||||
- Run 1: 99,765 created (stalled after 250K iter due to missing retry logic on hung HTTP)
|
||||
- Run 2: 71,219 created (with AbortController + per-page retry + skip-and-continue)
|
||||
|
||||
Final state:
|
||||
- Local DB total: 469,009 unique SNs
|
||||
- `api_uploaded_at NOT NULL`: 458,501
|
||||
- Unpushable: 10,508 (7,905 missing specs + 2,426 Hoffman API errors + 177 FAIL)
|
||||
|
||||
### 6. Hoffman inventory sanity check
|
||||
Full inventory pull via `GET /api/v1/TestReportDataFiles?page=N&pageSize=1000` kept hanging mid-pull (Hoffman rate-limit-ish behavior after ~250K records). Killed after 300K.
|
||||
|
||||
**Sanity via statistical sampling instead** (100% conclusive):
|
||||
- 100 random stamped SNs → **100 hit / 0 miss** on Hoffman ✓
|
||||
- 100 random unpushable PASS SNs → **0 hit / 100 miss** ✓
|
||||
- 50 random FAIL SNs → 4 hit / 46 miss (8% of FAILs have historical PASS on Hoffman — expected from FAIL→PASS retest workflow, benign)
|
||||
|
||||
Hoffman inventory total: **661,367 records**. Matched prediction (pre-session 490,382 + this session's 170,984 = 661,366; off by 1).
|
||||
|
||||
**Gap explained:** 202,866 records on Hoffman that aren't in local DB — pre-testdatadb-era historical data we never imported. Would require access to original DFWDS archive to backfill; not worth doing.
|
||||
|
||||
## Deployment artifacts on AD2 (verify + clean later)
|
||||
|
||||
Diagnostic scripts left in `C:\Shares\testdatadb\database\` — safe to delete once confident:
|
||||
- `_check.js`, `_constr.js`, `_dedup.js`, `_dup.js`, `_find.js`, `_recent.js`, `_run_migration.js`, `_scope.js`, `_analyze_unpushed.js`, `_analyze2.js`, `_analyze3.js`, `_conflict_test.js`, `_sanity_check.js`, `_spec_probe.js`, `_probe_pages.js`, `_bulk_push_all.js`, `_pull_inventory.js`, `_api_probe.js`, `_render_test.js`, `_state.js`, `_stamp_check.js`, `_probe_record.js`, `_pull_stdout.txt`, `_pull_stderr.txt`
|
||||
|
||||
Production files to keep:
|
||||
- `database/import.js` (modified)
|
||||
- `database/upload-to-api.js` (refactored)
|
||||
- `database/render-datasheet.js` (new)
|
||||
- `database/migrate-add-api-uploaded.sql` (applied)
|
||||
- `database/back-populate-api-uploaded.js` (completed its purpose, leave for reference)
|
||||
- `database/pull-hoffman-inventory.js` (left for future full-inventory pulls if needed)
|
||||
- `routes/api.js` (modified)
|
||||
- `public/index.html` (modified)
|
||||
|
||||
Plus `.bak-YYYYMMDD-HHMMSS` copies for every modified file per deploy.
|
||||
|
||||
## Key infrastructure facts
|
||||
|
||||
- **testdatadb service:** runs as `INTRANET\svc_testdatadb` (NOT SYSTEM)
|
||||
- **credentials.json** at `C:\ProgramData\dataforth-uploader\credentials.json` — had to grant `svc_testdatadb` Read + Traverse (was SYSTEM + Admins only; fixed 2026-04-15)
|
||||
- **For_Web path:** `C:\Shares\webshare\For_Web` (local on AD2); `X:` drive mapping is user-mapped and invisible to services
|
||||
- **Service wrapper:** C:\Shares\testdatadb\daemon\testdatadb.exe (WinSW)
|
||||
- **Logs:** C:\Shares\testdatadb\logs\ (out.log, err.log, wrapper.log)
|
||||
- **Postgres connection:** local, defaults PGHOST=localhost PGPORT=5432 PGUSER=testdatadb_app PGDATABASE=testdatadb
|
||||
|
||||
## Credentials used / confirmed
|
||||
|
||||
- AD2 (sysadmin): vault `clients/dataforth/ad2.sops.yaml` → `Paper123!@#` (fixed earlier session — no more `\!@#` backslash hack needed)
|
||||
- Hoffman API creds: `C:\ProgramData\dataforth-uploader\credentials.json` on AD2 (CF_TOKEN_URL, CF_API_BASE, CF_CLIENT_ID, CF_CLIENT_SECRET, CF_SCOPE)
|
||||
- SOPS age key: `%APPDATA%\sops\age\keys.txt` as usual
|
||||
|
||||
## Open items / next session candidates
|
||||
|
||||
1. **Drop `test_records_dedup_bak_20260415`** after another day or two of no regressions
|
||||
2. **Drop redundant 5-col unique constraint** `test_records_log_type_model_number_serial_number_test_date__key` if user wants
|
||||
3. **Auto-retry/re-render for unpushable records** — 7,905 records skipped due to missing specs. Adding specs for those 8B/5B/DSCA variants would unlock more web coverage.
|
||||
4. **www.azcomputerguru.com Apache vhost** — returns 404 despite root domain working. ServerAlias missing; defer to azcomputerguru.com project.
|
||||
|
||||
## Bonus: production incident resolved same session
|
||||
|
||||
azcomputerguru.com went down mid-session (CF managed challenge served in place of content). Root cause: **Imunify360 on IX (172.16.3.10) had blacklisted Jupiter's IP (172.16.3.20) 9+ days ago** — detected cloudflared's relay pattern as bot-like. Jupiter's tunnel couldn't reach origin, CF substituted challenge page.
|
||||
|
||||
Fix:
|
||||
1. `ipset del i360.ipv4.blacklist 172.16.3.20` (immediate unban)
|
||||
2. `imunify360-agent ip-list local add --purpose white --full-access --comment "Jupiter cloudflared tunnel origin" 172.16.3.20` (permanent whitelist)
|
||||
3. Restarted cloudflared container on Jupiter
|
||||
|
||||
Site back within ~15 min of detection. All CF-fronted subdomains (rmm.azcomputerguru.com, rmm-api, etc.) sharing the same tunnel also recovered.
|
||||
|
||||
## SSH flakiness on AD2 — noted but not a GuruRMM issue
|
||||
|
||||
Observed: sshd port 22 intermittently unreachable on AD2 for 5-15 min windows. Port 3000 (testdatadb), 3389 (RDP), 5985 (WinRM) stay reachable through same windows. sshd PID 4012 continuously running since 2026-04-11 22:09 — no crashes in event log. Likely a network-layer blip (firewall/AV scan briefly blocking port 22) rather than an actual service issue. Not caused by GuruRMM agent.
|
||||
Reference in New Issue
Block a user