sync: auto-sync from DESKTOP-0O8A1RL at 2026-04-21 18:46:45
Author: Mike Swanson Machine: DESKTOP-0O8A1RL Timestamp: 2026-04-21 18:46:45
This commit is contained in:
@@ -1,439 +1,450 @@
|
|||||||
# Dataforth DOS Project - Context
|
# Dataforth DOS Project - Context
|
||||||
|
|
||||||
**Last Updated:** 2026-04-14
|
**Last Updated:** 2026-04-14
|
||||||
**Status:** Active - Datasheet Pipeline Extended for SCMVAS/SCMHVAS
|
**Status:** Active - Datasheet Pipeline Extended for SCMVAS/SCMHVAS
|
||||||
|
|
||||||
## Quick Start - Infrastructure Overview
|
## Quick Start - Infrastructure Overview
|
||||||
|
|
||||||
| Component | IP/Location | Access | Notes |
|
| Component | IP/Location | Access | Notes |
|
||||||
|-----------|-------------|--------|-------|
|
|-----------|-------------|--------|-------|
|
||||||
| **AD2** (Primary) | 192.168.0.6 | SSH: sysadmin / vault | Windows Server 2022, hosts testdatadb service |
|
| **AD2** (Primary) | 192.168.0.6 | SSH: sysadmin / vault | Windows Server 2022, hosts testdatadb service |
|
||||||
| **AD1** (Secondary) | 192.168.0.27 | SSH: sysadmin / vault | Hosts Engineering share at \\AD1\Engineering |
|
| **AD1** (Secondary) | 192.168.0.27 | SSH: sysadmin / vault | Hosts Engineering share at \\AD1\Engineering |
|
||||||
| **D2TESTNAS** | 192.168.0.9 | SMB1 only | Bridge for DOS test stations (TS-xx machines) |
|
| **D2TESTNAS** | 192.168.0.9 | SMB1 only | Bridge for DOS test stations (TS-xx machines) |
|
||||||
| **VPN** | Required | FortiClient | Access to 192.168.0.x network |
|
| **VPN** | Required | FortiClient | Access to 192.168.0.x network |
|
||||||
|
|
||||||
**Get credentials:**
|
**Get credentials:**
|
||||||
```bash
|
```bash
|
||||||
# AD2 password (has stale backslash escape - strip it)
|
# AD2 password (has stale backslash escape - strip it)
|
||||||
bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g'
|
bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g'
|
||||||
|
|
||||||
# AD1 password
|
# AD1 password
|
||||||
bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad1.sops.yaml credentials.password
|
bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad1.sops.yaml credentials.password
|
||||||
```
|
```
|
||||||
|
|
||||||
**All passwords:** `Paper123!@#` (stored in vault, note backslash escape issue in ad2.sops.yaml)
|
**All passwords:** `Paper123!@#` (stored in vault, note backslash escape issue in ad2.sops.yaml)
|
||||||
|
|
||||||
## Current State (READ THIS FIRST)
|
## Current State (READ THIS FIRST)
|
||||||
|
|
||||||
### Recent Work (2026-04-11/12)
|
### Recent Work (2026-04-11/12)
|
||||||
**Extended Test Datasheet Pipeline for SCMVAS-Mxxx and SCMHVAS-Mxxxx families**
|
**Extended Test Datasheet Pipeline for SCMVAS-Mxxx and SCMHVAS-Mxxxx families**
|
||||||
- Added VASLOG parser support (multiline CSV .DAT format)
|
- Added VASLOG parser support (multiline CSV .DAT format)
|
||||||
- Created accuracy-only datasheet template (simple format, no hvin.dat lookup)
|
- Created accuracy-only datasheet template (simple format, no hvin.dat lookup)
|
||||||
- Implemented pass-through for Engineering-Tested .txt files
|
- Implemented pass-through for Engineering-Tested .txt files
|
||||||
- **Backfilled 27,503 historical records** (438 required regex patch for QB STR$() format quirk)
|
- **Backfilled 27,503 historical records** (438 required regex patch for QB STR$() format quirk)
|
||||||
- **434 Engineering .txt files** imported and published
|
- **434 Engineering .txt files** imported and published
|
||||||
- Deployed to AD2, service restarted, web publishing verified
|
- Deployed to AD2, service restarted, web publishing verified
|
||||||
|
|
||||||
**Status:** ✅ Complete, production-deployed
|
**Status:** ✅ Complete, production-deployed
|
||||||
|
|
||||||
**Critical Files Changed:** 5 modified, 1 new parser
|
**Critical Files Changed:** 5 modified, 1 new parser
|
||||||
- server/parsers/vaslog.js (new)
|
- server/parsers/vaslog.js (new)
|
||||||
- server/templates/datasheet-exact.js (SCMVAS/SCMHVAS branch added)
|
- server/templates/datasheet-exact.js (SCMVAS/SCMHVAS branch added)
|
||||||
- server/database/import.js (recursive flag fix, VASLOG_ENG support)
|
- server/database/import.js (recursive flag fix, VASLOG_ENG support)
|
||||||
- server/parsers/spec-reader.js (stub for SCMVAS/SCMHVAS)
|
- server/parsers/spec-reader.js (stub for SCMVAS/SCMHVAS)
|
||||||
- deploy/deploy-to-ad2.py (vault-based credentials)
|
- deploy/deploy-to-ad2.py (vault-based credentials)
|
||||||
|
|
||||||
**Session Logs:**
|
**Session Logs:**
|
||||||
- **2026-04-12-session.md** - Implementation, deploy, backfill, patch (DEFINITIVE)
|
- **2026-04-12-session.md** - Implementation, deploy, backfill, patch (DEFINITIVE)
|
||||||
- **2026-04-11-discovery-session.md** - Discovery phase
|
- **2026-04-11-discovery-session.md** - Discovery phase
|
||||||
|
|
||||||
### testdatadb Service (on AD2)
|
### testdatadb Service (on AD2)
|
||||||
- **Service Name:** testdatadb
|
- **Service Name:** testdatadb
|
||||||
- **Status:** Running
|
- **Status:** Running
|
||||||
- **Service Account:** INTRANET\svc_testdatadb
|
- **Service Account:** INTRANET\svc_testdatadb
|
||||||
- **Working Directory:** C:\Shares\testdatadb
|
- **Working Directory:** C:\Shares\testdatadb
|
||||||
- **API Port:** 3000 (http://192.168.0.6:3000)
|
- **API Port:** 3000 (http://192.168.0.6:3000)
|
||||||
- **Database:** SQLite at C:\Shares\testdatadb\database/testdata.db (4.1GB)
|
- **Database:** SQLite at C:\Shares\testdatadb\database/testdata.db (4.1GB)
|
||||||
- **Web Output:** X:\For_Web (= \\ad2\webshare\For_Web UNC path)
|
- **Web Output:** X:\For_Web (= \\ad2\webshare\For_Web UNC path)
|
||||||
|
|
||||||
### File Shares on AD2
|
### File Shares on AD2
|
||||||
```
|
```
|
||||||
C:\Shares\test\ # Mirror of D2TESTNAS test data
|
C:\Shares\test\ # Mirror of D2TESTNAS test data
|
||||||
├── TS-xx\LOGS\ # Test logs from DOS stations
|
├── TS-xx\LOGS\ # Test logs from DOS stations
|
||||||
│ ├── 5BLOG\ # SCM5B family
|
│ ├── 5BLOG\ # SCM5B family
|
||||||
│ ├── 8BLOG\ # 8B family
|
│ ├── 8BLOG\ # 8B family
|
||||||
│ ├── VASLOG\ # SCMVAS/SCMHVAS .DAT files
|
│ ├── VASLOG\ # SCMVAS/SCMHVAS .DAT files
|
||||||
│ │ ├── HVAS-M01.DAT # Production logs
|
│ │ ├── HVAS-M01.DAT # Production logs
|
||||||
│ │ ├── VAS-M100.DAT
|
│ │ ├── VAS-M100.DAT
|
||||||
│ │ └── VASLOG - Engineering Tested\ # 434 .txt files
|
│ │ └── VASLOG - Engineering Tested\ # 434 .txt files
|
||||||
│ └── ...
|
│ └── ...
|
||||||
└── Corrected HVAS Files\ # 200 pre-generated datasheets
|
└── Corrected HVAS Files\ # 200 pre-generated datasheets
|
||||||
|
|
||||||
C:\Shares\testdatadb\ # Node.js application
|
C:\Shares\testdatadb\ # Node.js application
|
||||||
├── server/
|
├── server/
|
||||||
│ ├── parsers/ # Log file parsers
|
│ ├── parsers/ # Log file parsers
|
||||||
│ ├── templates/ # Datasheet formatters
|
│ ├── templates/ # Datasheet formatters
|
||||||
│ └── database/ # Import/export scripts
|
│ └── database/ # Import/export scripts
|
||||||
├── database/
|
├── database/
|
||||||
│ └── testdata.db # SQLite (4.1GB, not in git)
|
│ └── testdata.db # SQLite (4.1GB, not in git)
|
||||||
└── node_modules/
|
└── node_modules/
|
||||||
```
|
```
|
||||||
|
|
||||||
### File Shares on AD1
|
### File Shares on AD1
|
||||||
```
|
```
|
||||||
\\AD1\Engineering\
|
\\AD1\Engineering\
|
||||||
└── ENGR\ATE\High Voltage Input Module Test\
|
└── ENGR\ATE\High Voltage Input Module Test\
|
||||||
├── HVDATA\
|
├── HVDATA\
|
||||||
│ └── hvin.dat # Spec database (33 records, engineering MODNAMEs)
|
│ └── hvin.dat # Spec database (33 records, engineering MODNAMEs)
|
||||||
└── Released\
|
└── Released\
|
||||||
├── TESTHV3.BAS # Primary test program (2020)
|
├── TESTHV3.BAS # Primary test program (2020)
|
||||||
├── TESTHV4.BAS # Alternate test program (2017)
|
├── TESTHV4.BAS # Alternate test program (2017)
|
||||||
├── NLIBATE3.BAS # ATE library
|
├── NLIBATE3.BAS # ATE library
|
||||||
└── DBHV.BAS # Database editor (TYPE DBASE definition)
|
└── DBHV.BAS # Database editor (TYPE DBASE definition)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Anti-Patterns (DON'T DO THIS)
|
## Email / SMTP
|
||||||
|
|
||||||
❌ **DO NOT hardcode Paper123!@#** - Always fetch from vault:
|
Dataforth is **M365 hybrid** — Exchange Online is the mail system. Use SMTP via M365:
|
||||||
```bash
|
|
||||||
bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g'
|
- **SMTP host:** smtp.office365.com **Port:** 587 (STARTTLS)
|
||||||
```
|
- **Auth:** sysadmin@dataforth.com (vault: `clients/dataforth/m365.sops.yaml` → `credentials.password`)
|
||||||
|
- **Tenant ID:** `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584`
|
||||||
❌ **DO NOT use X: drive in SSH sessions** - It's only mapped under service account. Use UNC path instead:
|
- **Neptune Exchange (neptune.acghosting.com):** ACG infrastructure — NOT Dataforth's, do not use
|
||||||
```powershell
|
|
||||||
# Wrong:
|
---
|
||||||
node database/export-datasheets.js # Fails: "X:\For_Web does not exist"
|
|
||||||
|
## Anti-Patterns (DON'T DO THIS)
|
||||||
# Right:
|
|
||||||
$env:OUTPUT_DIR = "\\ad2\webshare\For_Web"
|
❌ **DO NOT hardcode Paper123!@#** - Always fetch from vault:
|
||||||
node database/export-datasheets.js
|
```bash
|
||||||
```
|
bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g'
|
||||||
|
```
|
||||||
❌ **DO NOT assume hvin.dat lookup works** - Marketing names (SCMHVAS-M0100) ≠ engineering MODNAMEs (SCM5B41-1181). SCMVAS/SCMHVAS use simplified accuracy-only template WITHOUT hvin.dat.
|
|
||||||
|
❌ **DO NOT use X: drive in SSH sessions** - It's only mapped under service account. Use UNC path instead:
|
||||||
❌ **DO NOT pass 50+ file paths on PowerShell command line** - Hits "Command line too long". Use inline node script with fs.readdirSync instead.
|
```powershell
|
||||||
|
# Wrong:
|
||||||
❌ **DO NOT commit testdata.db or large samples** - 4.1GB database is in .gitignore. Keep research samples local only.
|
node database/export-datasheets.js # Fails: "X:\For_Web does not exist"
|
||||||
|
|
||||||
❌ **DO NOT use SMB1 on AD2** - Disabled for security. Use SSH/SFTP (port 22) or SMB2+ shares.
|
# Right:
|
||||||
|
$env:OUTPUT_DIR = "\\ad2\webshare\For_Web"
|
||||||
❌ **DO NOT expect immediate output from exec_command** - paramiko buffers stdout. Use progress markers or drain at completion.
|
node database/export-datasheets.js
|
||||||
|
```
|
||||||
❌ **DO NOT assume VPN is stable** - Dataforth VPN can drop mid-session. Save work frequently, use local samples for offline analysis.
|
|
||||||
|
❌ **DO NOT assume hvin.dat lookup works** - Marketing names (SCMHVAS-M0100) ≠ engineering MODNAMEs (SCM5B41-1181). SCMVAS/SCMHVAS use simplified accuracy-only template WITHOUT hvin.dat.
|
||||||
## Where to Find Things
|
|
||||||
|
❌ **DO NOT pass 50+ file paths on PowerShell command line** - Hits "Command line too long". Use inline node script with fs.readdirSync instead.
|
||||||
### Codebase Structure
|
|
||||||
```
|
❌ **DO NOT commit testdata.db or large samples** - 4.1GB database is in .gitignore. Keep research samples local only.
|
||||||
projects/dataforth-dos/
|
|
||||||
├── datasheet-pipeline/
|
❌ **DO NOT use SMB1 on AD2** - Disabled for security. Use SSH/SFTP (port 22) or SMB2+ shares.
|
||||||
│ ├── implementation/ # Staged code (approved by Code Review)
|
|
||||||
│ ├── scmvas-hvas-research/ # Discovery scripts and source files
|
❌ **DO NOT expect immediate output from exec_command** - paramiko buffers stdout. Use progress markers or drain at completion.
|
||||||
│ │ ├── source/ # TESTHV3.BAS, hvin.dat, etc.
|
|
||||||
│ │ ├── samples/ # .DAT and .txt samples (local)
|
❌ **DO NOT assume VPN is stable** - Dataforth VPN can drop mid-session. Save work frequently, use local samples for offline analysis.
|
||||||
│ │ ├── parse_hvin.py # hvin.dat binary parser
|
|
||||||
│ │ └── pull-*.py # SSH download scripts
|
## Where to Find Things
|
||||||
│ └── IMPLEMENTATION_PLAN.md # Approved plan (2026-04-11)
|
|
||||||
├── deploy/
|
### Codebase Structure
|
||||||
│ └── deploy-to-ad2.py # Deployment script (vault-based auth)
|
```
|
||||||
├── session-logs/
|
projects/dataforth-dos/
|
||||||
│ ├── 2026-04-12-session.md # SCMVAS/SCMHVAS implementation (DEFINITIVE)
|
├── datasheet-pipeline/
|
||||||
│ └── 2026-04-11-discovery-session.md
|
│ ├── implementation/ # Staged code (approved by Code Review)
|
||||||
└── CONTEXT.md # This file
|
│ ├── scmvas-hvas-research/ # Discovery scripts and source files
|
||||||
```
|
│ │ ├── source/ # TESTHV3.BAS, hvin.dat, etc.
|
||||||
|
│ │ ├── samples/ # .DAT and .txt samples (local)
|
||||||
### Production Files on AD2
|
│ │ ├── parse_hvin.py # hvin.dat binary parser
|
||||||
```
|
│ │ └── pull-*.py # SSH download scripts
|
||||||
C:\Shares\testdatadb\
|
│ └── IMPLEMENTATION_PLAN.md # Approved plan (2026-04-11)
|
||||||
├── server.js # Main entry point
|
├── deploy/
|
||||||
├── server/
|
│ └── deploy-to-ad2.py # Deployment script (vault-based auth)
|
||||||
│ ├── parsers/
|
├── session-logs/
|
||||||
│ │ ├── multiline.js # Handles VASLOG .DAT (CSV format)
|
│ ├── 2026-04-12-session.md # SCMVAS/SCMHVAS implementation (DEFINITIVE)
|
||||||
│ │ ├── vaslog.js # VASLOG-specific logic (new)
|
│ └── 2026-04-11-discovery-session.md
|
||||||
│ │ └── spec-reader.js # Spec DB loader (stub for SCMVAS/SCMHVAS)
|
└── CONTEXT.md # This file
|
||||||
│ ├── templates/
|
```
|
||||||
│ │ └── datasheet-exact.js # Datasheet formatter (SCMVAS/SCMHVAS branch added)
|
|
||||||
│ └── database/
|
### Production Files on AD2
|
||||||
│ ├── import.js # LOG_TYPES registry, importFiles()
|
```
|
||||||
│ └── export-datasheets.js # Batch export script
|
C:\Shares\testdatadb\
|
||||||
└── database/
|
├── server.js # Main entry point
|
||||||
└── testdata.db # SQLite (27k+ records after backfill)
|
├── server/
|
||||||
```
|
│ ├── parsers/
|
||||||
|
│ │ ├── multiline.js # Handles VASLOG .DAT (CSV format)
|
||||||
## Common Operations
|
│ │ ├── vaslog.js # VASLOG-specific logic (new)
|
||||||
|
│ │ └── spec-reader.js # Spec DB loader (stub for SCMVAS/SCMHVAS)
|
||||||
### Deploy Code to AD2
|
│ ├── templates/
|
||||||
```bash
|
│ │ └── datasheet-exact.js # Datasheet formatter (SCMVAS/SCMHVAS branch added)
|
||||||
# From projects/dataforth-dos/deploy/
|
│ └── database/
|
||||||
python3 deploy-to-ad2.py
|
│ ├── import.js # LOG_TYPES registry, importFiles()
|
||||||
|
│ └── export-datasheets.js # Batch export script
|
||||||
# What it does:
|
└── database/
|
||||||
# 1. Fetches password from vault (D:/vault/scripts/vault.sh)
|
└── testdata.db # SQLite (27k+ records after backfill)
|
||||||
# 2. Connects via paramiko SFTP to 192.168.0.6:22
|
```
|
||||||
# 3. Creates .bak-YYYYMMDD timestamped backups
|
|
||||||
# 4. Uploads modified files from implementation/
|
## Common Operations
|
||||||
# 5. Restarts testdatadb service via SSH exec_command
|
|
||||||
# 6. Verifies API responds 200 OK on port 3000
|
### Deploy Code to AD2
|
||||||
```
|
```bash
|
||||||
|
# From projects/dataforth-dos/deploy/
|
||||||
**Manual deployment (if script unavailable):**
|
python3 deploy-to-ad2.py
|
||||||
```bash
|
|
||||||
# Get password
|
# What it does:
|
||||||
AD2_PASS=$(bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g')
|
# 1. Fetches password from vault (D:/vault/scripts/vault.sh)
|
||||||
|
# 2. Connects via paramiko SFTP to 192.168.0.6:22
|
||||||
# Connect
|
# 3. Creates .bak-YYYYMMDD timestamped backups
|
||||||
sshpass -p "${AD2_PASS}" ssh sysadmin@192.168.0.6
|
# 4. Uploads modified files from implementation/
|
||||||
|
# 5. Restarts testdatadb service via SSH exec_command
|
||||||
# Backup + copy
|
# 6. Verifies API responds 200 OK on port 3000
|
||||||
cd C:\Shares\testdatadb\server\parsers
|
```
|
||||||
copy multiline.js multiline.js.bak-20260414
|
|
||||||
# ... upload new files via SFTP ...
|
**Manual deployment (if script unavailable):**
|
||||||
|
```bash
|
||||||
# Restart service
|
# Get password
|
||||||
Restart-Service -Name testdatadb
|
AD2_PASS=$(bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g')
|
||||||
|
|
||||||
# Verify
|
# Connect
|
||||||
curl http://localhost:3000
|
sshpass -p "${AD2_PASS}" ssh sysadmin@192.168.0.6
|
||||||
```
|
|
||||||
|
# Backup + copy
|
||||||
### Import New Test Data
|
cd C:\Shares\testdatadb\server\parsers
|
||||||
```bash
|
copy multiline.js multiline.js.bak-20260414
|
||||||
# SSH to AD2
|
# ... upload new files via SFTP ...
|
||||||
ssh sysadmin@192.168.0.6
|
|
||||||
|
# Restart service
|
||||||
# Run import for specific log type
|
Restart-Service -Name testdatadb
|
||||||
cd C:\Shares\testdatadb
|
|
||||||
node database/import.js
|
# Verify
|
||||||
|
curl http://localhost:3000
|
||||||
# Import specific files (avoid "Command line too long")
|
```
|
||||||
node -e "
|
|
||||||
const importFiles = require('./server/database/import').importFiles;
|
### Import New Test Data
|
||||||
const fs = require('fs');
|
```bash
|
||||||
const files = fs.readdirSync('C:/Shares/test/TS-3R/LOGS/VASLOG/VASLOG - Engineering Tested')
|
# SSH to AD2
|
||||||
.filter(f => f.endsWith('.txt'))
|
ssh sysadmin@192.168.0.6
|
||||||
.map(f => 'C:/Shares/test/TS-3R/LOGS/VASLOG/VASLOG - Engineering Tested/' + f);
|
|
||||||
importFiles(files, 'VASLOG_ENG').then(() => console.log('Done'));
|
# Run import for specific log type
|
||||||
"
|
cd C:\Shares\testdatadb
|
||||||
```
|
node database/import.js
|
||||||
|
|
||||||
### Export Datasheets for Web
|
# Import specific files (avoid "Command line too long")
|
||||||
```bash
|
node -e "
|
||||||
# SSH to AD2
|
const importFiles = require('./server/database/import').importFiles;
|
||||||
ssh sysadmin@192.168.0.6
|
const fs = require('fs');
|
||||||
|
const files = fs.readdirSync('C:/Shares/test/TS-3R/LOGS/VASLOG/VASLOG - Engineering Tested')
|
||||||
# Export all pending datasheets
|
.filter(f => f.endsWith('.txt'))
|
||||||
cd C:\Shares\testdatadb
|
.map(f => 'C:/Shares/test/TS-3R/LOGS/VASLOG/VASLOG - Engineering Tested/' + f);
|
||||||
$env:OUTPUT_DIR = "\\ad2\webshare\For_Web" # NOT X:\For_Web in SSH
|
importFiles(files, 'VASLOG_ENG').then(() => console.log('Done'));
|
||||||
node database/export-datasheets.js
|
"
|
||||||
|
```
|
||||||
# Export specific model family
|
|
||||||
node database/export-datasheets.js --family SCMHVAS
|
### Export Datasheets for Web
|
||||||
```
|
```bash
|
||||||
|
# SSH to AD2
|
||||||
### Backfill Historical Data
|
ssh sysadmin@192.168.0.6
|
||||||
```bash
|
|
||||||
# SSH to AD2, run as inline script to avoid command-line length limits
|
# Export all pending datasheets
|
||||||
node -e "
|
cd C:\Shares\testdatadb
|
||||||
const db = require('./server/database/db');
|
$env:OUTPUT_DIR = "\\ad2\webshare\For_Web" # NOT X:\For_Web in SSH
|
||||||
const exportDatasheet = require('./server/templates/datasheet-exact');
|
node database/export-datasheets.js
|
||||||
|
|
||||||
db.all(\`
|
# Export specific model family
|
||||||
SELECT * FROM test_records
|
node database/export-datasheets.js --family SCMHVAS
|
||||||
WHERE log_type IN ('VASLOG', 'VASLOG_ENG')
|
```
|
||||||
AND exported_at IS NULL
|
|
||||||
ORDER BY id
|
### Backfill Historical Data
|
||||||
\`, (err, rows) => {
|
```bash
|
||||||
if (err) throw err;
|
# SSH to AD2, run as inline script to avoid command-line length limits
|
||||||
console.log(\`[INFO] Found \${rows.length} records to export\`);
|
node -e "
|
||||||
let count = 0;
|
const db = require('./server/database/db');
|
||||||
rows.forEach(row => {
|
const exportDatasheet = require('./server/templates/datasheet-exact');
|
||||||
try {
|
|
||||||
exportDatasheet(row);
|
db.all(\`
|
||||||
count++;
|
SELECT * FROM test_records
|
||||||
if (count % 100 === 0) console.log(\`[PROGRESS] \${count}/\${rows.length}\`);
|
WHERE log_type IN ('VASLOG', 'VASLOG_ENG')
|
||||||
} catch (e) {
|
AND exported_at IS NULL
|
||||||
console.error(\`[SKIP] \${row.model_name}: \${e.message}\`);
|
ORDER BY id
|
||||||
}
|
\`, (err, rows) => {
|
||||||
});
|
if (err) throw err;
|
||||||
console.log(\`[DONE] Exported \${count} datasheets\`);
|
console.log(\`[INFO] Found \${rows.length} records to export\`);
|
||||||
});
|
let count = 0;
|
||||||
"
|
rows.forEach(row => {
|
||||||
```
|
try {
|
||||||
|
exportDatasheet(row);
|
||||||
### Check Service Status
|
count++;
|
||||||
```powershell
|
if (count % 100 === 0) console.log(\`[PROGRESS] \${count}/\${rows.length}\`);
|
||||||
# On AD2 (via SSH or RDP)
|
} catch (e) {
|
||||||
Get-Service testdatadb
|
console.error(\`[SKIP] \${row.model_name}: \${e.message}\`);
|
||||||
|
}
|
||||||
# View service logs (if logging enabled)
|
});
|
||||||
Get-EventLog -LogName Application -Source testdatadb -Newest 50
|
console.log(\`[DONE] Exported \${count} datasheets\`);
|
||||||
|
});
|
||||||
# Test API
|
"
|
||||||
Invoke-WebRequest http://localhost:3000 | Select-Object StatusCode
|
```
|
||||||
|
|
||||||
# Check process
|
### Check Service Status
|
||||||
Get-Process | Where-Object { $_.ProcessName -like "*node*" }
|
```powershell
|
||||||
```
|
# On AD2 (via SSH or RDP)
|
||||||
|
Get-Service testdatadb
|
||||||
### Access Shares from macOS/Linux
|
|
||||||
```bash
|
# View service logs (if logging enabled)
|
||||||
# Mount AD2 share (SMB2+)
|
Get-EventLog -LogName Application -Source testdatadb -Newest 50
|
||||||
mkdir -p ~/mnt/ad2-testdatadb
|
|
||||||
mount_smbfs //sysadmin:Password@192.168.0.6/testdatadb ~/mnt/ad2-testdatadb
|
# Test API
|
||||||
|
Invoke-WebRequest http://localhost:3000 | Select-Object StatusCode
|
||||||
# Mount AD1 Engineering share
|
|
||||||
mkdir -p ~/mnt/ad1-engineering
|
# Check process
|
||||||
mount_smbfs //sysadmin:Password@192.168.0.27/Engineering ~/mnt/ad1-engineering
|
Get-Process | Where-Object { $_.ProcessName -like "*node*" }
|
||||||
|
```
|
||||||
# Unmount
|
|
||||||
umount ~/mnt/ad2-testdatadb
|
### Access Shares from macOS/Linux
|
||||||
```
|
```bash
|
||||||
|
# Mount AD2 share (SMB2+)
|
||||||
## Key Technical Decisions (ADRs)
|
mkdir -p ~/mnt/ad2-testdatadb
|
||||||
|
mount_smbfs //sysadmin:Password@192.168.0.6/testdatadb ~/mnt/ad2-testdatadb
|
||||||
**2026-04-12:** Use Option C (simple accuracy-only template, no hvin.dat lookup)
|
|
||||||
- Reason: Marketing names (SCMHVAS-M0100) ≠ engineering MODNAMEs (SCM5B41-1181) in hvin.dat
|
# Mount AD1 Engineering share
|
||||||
- Sample datasheets show simple 1-parameter format (Accuracy only)
|
mkdir -p ~/mnt/ad1-engineering
|
||||||
- Spec-reader stub lets SCMVAS/SCMHVAS pass through pipeline without schema changes
|
mount_smbfs //sysadmin:Password@192.168.0.27/Engineering ~/mnt/ad1-engineering
|
||||||
|
|
||||||
**2026-04-12:** Pass-through for VASLOG_ENG .txt files (not re-render)
|
# Unmount
|
||||||
- Reason: Engineering-Tested files already match target format exactly
|
umount ~/mnt/ad2-testdatadb
|
||||||
- fs.copyFileSync() guarantees byte-level fidelity, avoids encoding round-trip
|
```
|
||||||
- Fallback to writeFileSync(raw_data, 'utf8') if source file missing
|
|
||||||
|
## Key Technical Decisions (ADRs)
|
||||||
**2026-04-12:** Fix recursive=false default regression with `config.recursive !== false`
|
|
||||||
- Reason: Adding `recursive` field to LOG_TYPES must not break 7 pre-existing families
|
**2026-04-12:** Use Option C (simple accuracy-only template, no hvin.dat lookup)
|
||||||
- Treats absent/undefined as true (legacy behavior), explicit false as false
|
- Reason: Marketing names (SCMHVAS-M0100) ≠ engineering MODNAMEs (SCM5B41-1181) in hvin.dat
|
||||||
|
- Sample datasheets show simple 1-parameter format (Accuracy only)
|
||||||
**2026-04-12:** Vault-based credentials in deploy script (no hardcoding, no prompts)
|
- Spec-reader stub lets SCMVAS/SCMHVAS pass through pipeline without schema changes
|
||||||
- Reason: Never commit passwords, even to private repo
|
|
||||||
- deploy-to-ad2.py calls vault.sh with 30s timeout, fails loud if unavailable
|
**2026-04-12:** Pass-through for VASLOG_ENG .txt files (not re-render)
|
||||||
- No env-var fallback, no interactive prompt
|
- Reason: Engineering-Tested files already match target format exactly
|
||||||
|
- fs.copyFileSync() guarantees byte-level fidelity, avoids encoding round-trip
|
||||||
**2026-04-12:** MM/DD/YYYY date normalization for datasheet Date field
|
- Fallback to writeFileSync(raw_data, 'utf8') if source file missing
|
||||||
- Reason: Matches newest Engineering-Tested samples
|
|
||||||
- Older "Corrected HVAS Files" used MM-DD-YYYY (hyphens) - backfill rewrites with slashes
|
**2026-04-12:** Fix recursive=false default regression with `config.recursive !== false`
|
||||||
- Intentional visible change, documented in implementation plan
|
- Reason: Adding `recursive` field to LOG_TYPES must not break 7 pre-existing families
|
||||||
|
- Treats absent/undefined as true (legacy behavior), explicit false as false
|
||||||
**2026-04-12:** Patch regex with plain-decimal fallback for QuickBASIC STR$() quirk
|
|
||||||
- Reason: QB STR$() emits scientific notation for most values, plain decimal for ~1.6%
|
**2026-04-12:** Vault-based credentials in deploy script (no hardcoding, no prompts)
|
||||||
- Not a version difference or bug - purely QB float-to-string formatting threshold
|
- Reason: Never commit passwords, even to private repo
|
||||||
- Two-regex approach: try scientific first, fall back to plain decimal
|
- deploy-to-ad2.py calls vault.sh with 30s timeout, fails loud if unavailable
|
||||||
|
- No env-var fallback, no interactive prompt
|
||||||
## QuickBASIC Artifacts & Log Formats
|
|
||||||
|
**2026-04-12:** MM/DD/YYYY date normalization for datasheet Date field
|
||||||
### VASLOG .DAT Structure
|
- Reason: Matches newest Engineering-Tested samples
|
||||||
```
|
- Older "Corrected HVAS Files" used MM-DD-YYYY (hyphens) - backfill rewrites with slashes
|
||||||
"SCMHVAS-M0100 " # Header: model name (marketing, NOT engineering MODNAME)
|
- Intentional visible change, documented in implementation plan
|
||||||
20,0.0034 # CSV line 1: measurement data
|
|
||||||
40,0.0126 # CSV line 2
|
**2026-04-12:** Patch regex with plain-decimal fallback for QuickBASIC STR$() quirk
|
||||||
60,-0.0046 # CSV line 3
|
- Reason: QB STR$() emits scientific notation for most values, plain decimal for ~1.6%
|
||||||
80,0.0141 # CSV line 4
|
- Not a version difference or bug - purely QB float-to-string formatting threshold
|
||||||
100,-0.00325 # CSV line 5
|
- Two-regex approach: try scientific first, fall back to plain decimal
|
||||||
"PASS-7.005501E-033",... # Status line: PASS/FAIL + accuracy (scientific OR plain decimal)
|
|
||||||
"179379-1","04-09-2026" # Footer: serial number, test date (MM-DD-YYYY)
|
## QuickBASIC Artifacts & Log Formats
|
||||||
```
|
|
||||||
|
### VASLOG .DAT Structure
|
||||||
### VASLOG_ENG .txt Structure (Engineering-Tested)
|
```
|
||||||
```
|
"SCMHVAS-M0100 " # Header: model name (marketing, NOT engineering MODNAME)
|
||||||
SCMHVAS - M0100
|
20,0.0034 # CSV line 1: measurement data
|
||||||
SN: 171087-1
|
40,0.0126 # CSV line 2
|
||||||
Date: 04/08/2024
|
60,-0.0046 # CSV line 3
|
||||||
Test: PASS
|
80,0.0141 # CSV line 4
|
||||||
Accuracy: -7.0055E-03 %
|
100,-0.00325 # CSV line 5
|
||||||
```
|
"PASS-7.005501E-033",... # Status line: PASS/FAIL + accuracy (scientific OR plain decimal)
|
||||||
|
"179379-1","04-09-2026" # Footer: serial number, test date (MM-DD-YYYY)
|
||||||
### QuickBASIC STR$() Formatting Quirk
|
```
|
||||||
```basic
|
|
||||||
' QB emits TWO formats for floats:
|
### VASLOG_ENG .txt Structure (Engineering-Tested)
|
||||||
PRINT STR$(-7.005501E-03) ' → "-7.005501E-033" (scientific + status digit)
|
```
|
||||||
PRINT STR$(0.01599373) ' → " .01599373" (plain decimal, leading space)
|
SCMHVAS - M0100
|
||||||
|
SN: 171087-1
|
||||||
' Threshold: ~0.01 magnitude
|
Date: 04/08/2024
|
||||||
' Affects ~1.6% of records (438/27503)
|
Test: PASS
|
||||||
' NOT a bug - documented QB behavior
|
Accuracy: -7.0055E-03 %
|
||||||
```
|
```
|
||||||
|
|
||||||
### hvin.dat Binary Format
|
### QuickBASIC STR$() Formatting Quirk
|
||||||
```
|
```basic
|
||||||
TYPE DBASE (from DBHV.BAS)
|
' QB emits TWO formats for floats:
|
||||||
MODNAME AS STRING * 13 ' Engineering ID: "SCM5B41-1181 "
|
PRINT STR$(-7.005501E-03) ' → "-7.005501E-033" (scientific + status digit)
|
||||||
INTYPE AS STRING * 3
|
PRINT STR$(0.01599373) ' → " .01599373" (plain decimal, leading space)
|
||||||
OUTSIGTYPE AS STRING * 7
|
|
||||||
WAVESHPCAL AS STRING * 8
|
' Threshold: ~0.01 magnitude
|
||||||
' ... 42 SINGLE floats (IEEE 754, 4 bytes each) ...
|
' Affects ~1.6% of records (438/27503)
|
||||||
END TYPE
|
' NOT a bug - documented QB behavior
|
||||||
|
```
|
||||||
' Total: 13+3+7+8 + (42*4) = 199 bytes/record
|
|
||||||
' File size: 6567 bytes = 33 records
|
### hvin.dat Binary Format
|
||||||
```
|
```
|
||||||
|
TYPE DBASE (from DBHV.BAS)
|
||||||
## Troubleshooting
|
MODNAME AS STRING * 13 ' Engineering ID: "SCM5B41-1181 "
|
||||||
|
INTYPE AS STRING * 3
|
||||||
### "Output directory does not exist: X:\For_Web"
|
OUTSIGTYPE AS STRING * 7
|
||||||
- **Cause:** X: drive only mapped under service account, not in SSH session
|
WAVESHPCAL AS STRING * 8
|
||||||
- **Fix:** Use UNC path: `\\ad2\webshare\For_Web`
|
' ... 42 SINGLE floats (IEEE 754, 4 bytes each) ...
|
||||||
```powershell
|
END TYPE
|
||||||
$env:OUTPUT_DIR = "\\ad2\webshare\For_Web"
|
|
||||||
node database/export-datasheets.js
|
' Total: 13+3+7+8 + (42*4) = 199 bytes/record
|
||||||
```
|
' File size: 6567 bytes = 33 records
|
||||||
|
```
|
||||||
### "Command line is too long" (PowerShell)
|
|
||||||
- **Cause:** Passing 50+ file paths as arguments exceeds PowerShell limit
|
## Troubleshooting
|
||||||
- **Fix:** Use inline node script with fs.readdirSync (see Common Operations above)
|
|
||||||
|
### "Output directory does not exist: X:\For_Web"
|
||||||
### VPN Drops Mid-Session
|
- **Cause:** X: drive only mapped under service account, not in SSH session
|
||||||
- **Symptom:** AD2/AD1 become unreachable, SSH hangs
|
- **Fix:** Use UNC path: `\\ad2\webshare\For_Web`
|
||||||
- **Fix:**
|
```powershell
|
||||||
1. Work offline on local samples for analysis
|
$env:OUTPUT_DIR = "\\ad2\webshare\For_Web"
|
||||||
2. Restore VPN (FortiClient)
|
node database/export-datasheets.js
|
||||||
3. Resume deployment/import when connection stable
|
```
|
||||||
|
|
||||||
### Vault Returns `Paper123\!@#` (Backslash)
|
### "Command line is too long" (PowerShell)
|
||||||
- **Cause:** Legacy shell escape stored in ad2.sops.yaml
|
- **Cause:** Passing 50+ file paths as arguments exceeds PowerShell limit
|
||||||
- **Fix:** Strip backslash at read-time: `sed 's/\\//g'`
|
- **Fix:** Use inline node script with fs.readdirSync (see Common Operations above)
|
||||||
- **TODO:** Clean vault entry to remove backslash
|
|
||||||
|
### VPN Drops Mid-Session
|
||||||
### Paramiko "No Output" for Long-Running Commands
|
- **Symptom:** AD2/AD1 become unreachable, SSH hangs
|
||||||
- **Cause:** exec_command buffers stdout until completion
|
- **Fix:**
|
||||||
- **Fix:** Either:
|
1. Work offline on local samples for analysis
|
||||||
1. Accept final output when command completes
|
2. Restore VPN (FortiClient)
|
||||||
2. Add progress markers that flush every N records
|
3. Resume deployment/import when connection stable
|
||||||
3. Drain channel periodically: `while not channel.exit_status_ready(): channel.recv(1024)`
|
|
||||||
|
### Vault Returns `Paper123\!@#` (Backslash)
|
||||||
### 438 Records Skipped During Backfill
|
- **Cause:** Legacy shell escape stored in ad2.sops.yaml
|
||||||
- **Cause:** Plain-decimal format not matching scientific-notation-only regex
|
- **Fix:** Strip backslash at read-time: `sed 's/\\//g'`
|
||||||
- **Fix:** Already patched (2026-04-12). Regex now tries both formats.
|
- **TODO:** Clean vault entry to remove backslash
|
||||||
- **Verification:** Rerun backfill on stragglers → 438/438 rendered
|
|
||||||
|
### Paramiko "No Output" for Long-Running Commands
|
||||||
## Recent Commit History
|
- **Cause:** exec_command buffers stdout until completion
|
||||||
|
- **Fix:** Either:
|
||||||
**2026-04-12 (commit 0dd3d82):** SCMVAS/SCMHVAS pipeline extension
|
1. Accept final output when command completes
|
||||||
- 114 files changed, 35,486 insertions
|
2. Add progress markers that flush every N records
|
||||||
- 5 production files modified, 1 new parser
|
3. Drain channel periodically: `while not channel.exit_status_ready(): channel.recv(1024)`
|
||||||
- All research scripts sanitized (vault-based credentials)
|
|
||||||
- .gitignore updated (exclude testdata.db)
|
### 438 Records Skipped During Backfill
|
||||||
|
- **Cause:** Plain-decimal format not matching scientific-notation-only regex
|
||||||
## Useful Links
|
- **Fix:** Already patched (2026-04-12). Regex now tries both formats.
|
||||||
|
- **Verification:** Rerun backfill on stragglers → 438/438 rendered
|
||||||
- **Latest Session:** session-logs/2026-04-12-session.md (DEFINITIVE)
|
|
||||||
- **Discovery Session:** session-logs/2026-04-11-discovery-session.md
|
## Recent Commit History
|
||||||
- **Implementation Plan:** datasheet-pipeline/scmvas-hvas-research/IMPLEMENTATION_PLAN.md
|
|
||||||
- **Credentials (vault):** D:\vault\clients\dataforth\
|
**2026-04-12 (commit 0dd3d82):** SCMVAS/SCMHVAS pipeline extension
|
||||||
|
- 114 files changed, 35,486 insertions
|
||||||
## Quick Reference - Log Types
|
- 5 production files modified, 1 new parser
|
||||||
|
- All research scripts sanitized (vault-based credentials)
|
||||||
| Family | Log Type | Format | Parser | Location |
|
- .gitignore updated (exclude testdata.db)
|
||||||
|--------|----------|--------|--------|----------|
|
|
||||||
| SCM5B | 5BLOG | Multiline CSV .DAT | multiline.js | TS-xx/LOGS/5BLOG |
|
## Useful Links
|
||||||
| 8B | 8BLOG | Multiline CSV .DAT | multiline.js | TS-xx/LOGS/8BLOG |
|
|
||||||
| DSCA | DSCLOG | Multiline CSV .DAT | multiline.js | TS-xx/LOGS/DSCLOG |
|
- **Latest Session:** session-logs/2026-04-12-session.md (DEFINITIVE)
|
||||||
| SCMVAS | VASLOG | Multiline CSV .DAT | vaslog.js | TS-3R/LOGS/VASLOG |
|
- **Discovery Session:** session-logs/2026-04-11-discovery-session.md
|
||||||
| SCMHVAS (prod) | VASLOG | Multiline CSV .DAT | vaslog.js | TS-3R/LOGS/VASLOG |
|
- **Implementation Plan:** datasheet-pipeline/scmvas-hvas-research/IMPLEMENTATION_PLAN.md
|
||||||
| SCMHVAS (eng) | VASLOG_ENG | .txt (pass-through) | vaslog.js | TS-3R/LOGS/VASLOG/VASLOG - Engineering Tested |
|
- **Credentials (vault):** D:\vault\clients\dataforth\
|
||||||
|
|
||||||
---
|
## Quick Reference - Log Types
|
||||||
|
|
||||||
**Before starting work:** Read session-logs/2026-04-12-session.md for complete context
|
| Family | Log Type | Format | Parser | Location |
|
||||||
**For AD2 access:** Ensure Dataforth VPN connected (FortiClient)
|
|--------|----------|--------|--------|----------|
|
||||||
**For credentials:** Always use vault - never hardcode passwords
|
| SCM5B | 5BLOG | Multiline CSV .DAT | multiline.js | TS-xx/LOGS/5BLOG |
|
||||||
|
| 8B | 8BLOG | Multiline CSV .DAT | multiline.js | TS-xx/LOGS/8BLOG |
|
||||||
|
| DSCA | DSCLOG | Multiline CSV .DAT | multiline.js | TS-xx/LOGS/DSCLOG |
|
||||||
|
| SCMVAS | VASLOG | Multiline CSV .DAT | vaslog.js | TS-3R/LOGS/VASLOG |
|
||||||
|
| SCMHVAS (prod) | VASLOG | Multiline CSV .DAT | vaslog.js | TS-3R/LOGS/VASLOG |
|
||||||
|
| SCMHVAS (eng) | VASLOG_ENG | .txt (pass-through) | vaslog.js | TS-3R/LOGS/VASLOG/VASLOG - Engineering Tested |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Before starting work:** Read session-logs/2026-04-12-session.md for complete context
|
||||||
|
**For AD2 access:** Ensure Dataforth VPN connected (FortiClient)
|
||||||
|
**For credentials:** Always use vault - never hardcode passwords
|
||||||
|
|||||||
@@ -1,8 +1,11 @@
|
|||||||
# Python cache
|
# SMTP credentials written by deploy-to-ad2.py (never commit)
|
||||||
__pycache__/
|
implementation/config/notify.json
|
||||||
*.pyc
|
|
||||||
|
# Python cache
|
||||||
# SQLite snapshot pulled during discovery (4+ GB, customer data)
|
__pycache__/
|
||||||
scmvas-hvas-research/existing-database/testdata.db
|
*.pyc
|
||||||
scmvas-hvas-research/existing-database/testdata.db-shm
|
|
||||||
scmvas-hvas-research/existing-database/testdata.db-wal
|
# SQLite snapshot pulled during discovery (4+ GB, customer data)
|
||||||
|
scmvas-hvas-research/existing-database/testdata.db
|
||||||
|
scmvas-hvas-research/existing-database/testdata.db-shm
|
||||||
|
scmvas-hvas-research/existing-database/testdata.db-wal
|
||||||
|
|||||||
@@ -1,257 +1,273 @@
|
|||||||
/**
|
/**
|
||||||
* Export Datasheets
|
* Export Datasheets
|
||||||
*
|
*
|
||||||
* Generates TXT datasheets for unexported PASS records and writes them to X:\For_Web\.
|
* Generates TXT datasheets for unexported PASS records and writes them to X:\For_Web\.
|
||||||
* Updates forweb_exported_at after successful export.
|
* Updates forweb_exported_at after successful export.
|
||||||
*
|
*
|
||||||
* Usage:
|
* Usage:
|
||||||
* node export-datasheets.js Export all pending (batch mode)
|
* node export-datasheets.js Export all pending (batch mode)
|
||||||
* node export-datasheets.js --limit 100 Export up to 100 records
|
* node export-datasheets.js --limit 100 Export up to 100 records
|
||||||
* node export-datasheets.js --file <paths> Export records matching specific source files
|
* node export-datasheets.js --file <paths> Export records matching specific source files
|
||||||
* node export-datasheets.js --serial 178439-1 Export a specific serial number
|
* node export-datasheets.js --serial 178439-1 Export a specific serial number
|
||||||
* node export-datasheets.js --dry-run Show what would be exported without writing
|
* node export-datasheets.js --dry-run Show what would be exported without writing
|
||||||
*/
|
*/
|
||||||
|
|
||||||
const fs = require('fs');
|
const fs = require('fs');
|
||||||
const path = require('path');
|
const path = require('path');
|
||||||
const db = require('./db');
|
const db = require('./db');
|
||||||
|
|
||||||
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||||
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
const { generateExactDatasheet } = require('../templates/datasheet-exact');
|
||||||
|
const { sendFailureEmail } = require('../server/notify');
|
||||||
// Configuration
|
|
||||||
const OUTPUT_DIR = 'X:\\For_Web';
|
// Configuration
|
||||||
const BATCH_SIZE = 500;
|
const OUTPUT_DIR = 'X:\\For_Web';
|
||||||
|
const BATCH_SIZE = 500;
|
||||||
async function run() {
|
|
||||||
const args = process.argv.slice(2);
|
async function run() {
|
||||||
const dryRun = args.includes('--dry-run');
|
const args = process.argv.slice(2);
|
||||||
const limitIdx = args.indexOf('--limit');
|
const dryRun = args.includes('--dry-run');
|
||||||
const limit = limitIdx >= 0 ? parseInt(args[limitIdx + 1]) : 0;
|
const limitIdx = args.indexOf('--limit');
|
||||||
const serialIdx = args.indexOf('--serial');
|
const limit = limitIdx >= 0 ? parseInt(args[limitIdx + 1]) : 0;
|
||||||
const serial = serialIdx >= 0 ? args[serialIdx + 1] : null;
|
const serialIdx = args.indexOf('--serial');
|
||||||
const fileIdx = args.indexOf('--file');
|
const serial = serialIdx >= 0 ? args[serialIdx + 1] : null;
|
||||||
const files = fileIdx >= 0 ? args.slice(fileIdx + 1).filter(f => !f.startsWith('--')) : null;
|
const fileIdx = args.indexOf('--file');
|
||||||
|
const files = fileIdx >= 0 ? args.slice(fileIdx + 1).filter(f => !f.startsWith('--')) : null;
|
||||||
console.log('========================================');
|
|
||||||
console.log('Datasheet Export');
|
console.log('========================================');
|
||||||
console.log('========================================');
|
console.log('Datasheet Export');
|
||||||
console.log(`Output: ${OUTPUT_DIR}`);
|
console.log('========================================');
|
||||||
console.log(`Dry run: ${dryRun}`);
|
console.log(`Output: ${OUTPUT_DIR}`);
|
||||||
if (limit) console.log(`Limit: ${limit}`);
|
console.log(`Dry run: ${dryRun}`);
|
||||||
if (serial) console.log(`Serial: ${serial}`);
|
if (limit) console.log(`Limit: ${limit}`);
|
||||||
console.log(`Start: ${new Date().toISOString()}`);
|
if (serial) console.log(`Serial: ${serial}`);
|
||||||
|
console.log(`Start: ${new Date().toISOString()}`);
|
||||||
if (!dryRun && !fs.existsSync(OUTPUT_DIR)) {
|
|
||||||
console.error(`ERROR: Output directory does not exist: ${OUTPUT_DIR}`);
|
if (!dryRun && !fs.existsSync(OUTPUT_DIR)) {
|
||||||
process.exit(1);
|
console.error(`ERROR: Output directory does not exist: ${OUTPUT_DIR}`);
|
||||||
}
|
process.exit(1);
|
||||||
|
}
|
||||||
console.log('\nLoading model specs...');
|
|
||||||
const specMap = loadAllSpecs();
|
console.log('\nLoading model specs...');
|
||||||
|
const specMap = loadAllSpecs();
|
||||||
// Build query
|
|
||||||
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
// Build query
|
||||||
const params = [];
|
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
||||||
let paramIdx = 0;
|
const params = [];
|
||||||
|
let paramIdx = 0;
|
||||||
if (serial) {
|
|
||||||
paramIdx++;
|
if (serial) {
|
||||||
conditions.push(`serial_number = $${paramIdx}`);
|
paramIdx++;
|
||||||
params.push(serial);
|
conditions.push(`serial_number = $${paramIdx}`);
|
||||||
}
|
params.push(serial);
|
||||||
|
}
|
||||||
if (files && files.length > 0) {
|
|
||||||
const placeholders = files.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
if (files && files.length > 0) {
|
||||||
conditions.push(`source_file IN (${placeholders})`);
|
const placeholders = files.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
||||||
params.push(...files);
|
conditions.push(`source_file IN (${placeholders})`);
|
||||||
}
|
params.push(...files);
|
||||||
|
}
|
||||||
let sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')} ORDER BY test_date DESC`;
|
|
||||||
|
let sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')} ORDER BY test_date DESC`;
|
||||||
if (limit) {
|
|
||||||
paramIdx++;
|
if (limit) {
|
||||||
sql += ` LIMIT $${paramIdx}`;
|
paramIdx++;
|
||||||
params.push(limit);
|
sql += ` LIMIT $${paramIdx}`;
|
||||||
}
|
params.push(limit);
|
||||||
|
}
|
||||||
const records = await db.query(sql, params);
|
|
||||||
console.log(`\nFound ${records.length} records to export`);
|
const records = await db.query(sql, params);
|
||||||
|
console.log(`\nFound ${records.length} records to export`);
|
||||||
if (records.length === 0) {
|
|
||||||
console.log('Nothing to export.');
|
if (records.length === 0) {
|
||||||
await db.close();
|
console.log('Nothing to export.');
|
||||||
return { exported: 0, skipped: 0, errors: 0 };
|
await db.close();
|
||||||
}
|
return { exported: 0, skipped: 0, errors: 0 };
|
||||||
|
}
|
||||||
let exported = 0;
|
|
||||||
let skipped = 0;
|
let exported = 0;
|
||||||
let errors = 0;
|
let skipped = 0;
|
||||||
let noSpecs = 0;
|
let errors = 0;
|
||||||
let pendingUpdates = [];
|
let noSpecs = 0;
|
||||||
|
let pendingUpdates = [];
|
||||||
for (const record of records) {
|
|
||||||
try {
|
for (const record of records) {
|
||||||
const filename = record.serial_number + '.TXT';
|
try {
|
||||||
const outputPath = path.join(OUTPUT_DIR, filename);
|
const filename = record.serial_number + '.TXT';
|
||||||
|
const outputPath = path.join(OUTPUT_DIR, filename);
|
||||||
// VASLOG_ENG: verbatim byte-for-byte copy of the original file.
|
|
||||||
// Using fs.copyFileSync avoids any utf-8 round-trip that would
|
// VASLOG_ENG: verbatim byte-for-byte copy of the original file.
|
||||||
// corrupt non-ASCII bytes (CP1252 etc.) in customer datasheets.
|
// Using fs.copyFileSync avoids any utf-8 round-trip that would
|
||||||
// Fall back to writing raw_data if the source file is gone.
|
// corrupt non-ASCII bytes (CP1252 etc.) in customer datasheets.
|
||||||
if (record.log_type === 'VASLOG_ENG') {
|
// Fall back to writing raw_data if the source file is gone.
|
||||||
if (dryRun) {
|
if (record.log_type === 'VASLOG_ENG') {
|
||||||
console.log(` [DRY RUN] Would copy: ${record.source_file} -> ${filename}`);
|
if (dryRun) {
|
||||||
exported++;
|
console.log(` [DRY RUN] Would copy: ${record.source_file} -> ${filename}`);
|
||||||
continue;
|
exported++;
|
||||||
}
|
continue;
|
||||||
if (record.source_file && fs.existsSync(record.source_file)) {
|
}
|
||||||
fs.copyFileSync(record.source_file, outputPath);
|
if (record.source_file && fs.existsSync(record.source_file)) {
|
||||||
} else {
|
fs.copyFileSync(record.source_file, outputPath);
|
||||||
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
} else {
|
||||||
if (!record.raw_data) {
|
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
||||||
skipped++;
|
if (!record.raw_data) {
|
||||||
continue;
|
skipped++;
|
||||||
}
|
continue;
|
||||||
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
}
|
||||||
}
|
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
||||||
pendingUpdates.push(record.id);
|
}
|
||||||
exported++;
|
pendingUpdates.push(record.id);
|
||||||
|
exported++;
|
||||||
if (pendingUpdates.length >= BATCH_SIZE) {
|
|
||||||
await flushUpdates(pendingUpdates);
|
if (pendingUpdates.length >= BATCH_SIZE) {
|
||||||
pendingUpdates = [];
|
await flushUpdates(pendingUpdates);
|
||||||
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
pendingUpdates = [];
|
||||||
}
|
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
||||||
continue;
|
}
|
||||||
}
|
continue;
|
||||||
|
}
|
||||||
// Template-generated datasheet path.
|
|
||||||
const specs = getSpecs(specMap, record.model_number);
|
// Template-generated datasheet path.
|
||||||
if (!specs) {
|
const specs = getSpecs(specMap, record.model_number);
|
||||||
noSpecs++;
|
if (!specs) {
|
||||||
skipped++;
|
noSpecs++;
|
||||||
continue;
|
skipped++;
|
||||||
}
|
continue;
|
||||||
const txt = generateExactDatasheet(record, specs);
|
}
|
||||||
if (!txt) {
|
const txt = generateExactDatasheet(record, specs);
|
||||||
skipped++;
|
if (!txt) {
|
||||||
continue;
|
skipped++;
|
||||||
}
|
continue;
|
||||||
|
}
|
||||||
if (dryRun) {
|
|
||||||
console.log(` [DRY RUN] Would write: ${filename}`);
|
if (dryRun) {
|
||||||
exported++;
|
console.log(` [DRY RUN] Would write: ${filename}`);
|
||||||
} else {
|
exported++;
|
||||||
fs.writeFileSync(outputPath, txt, 'utf8');
|
} else {
|
||||||
pendingUpdates.push(record.id);
|
fs.writeFileSync(outputPath, txt, 'utf8');
|
||||||
exported++;
|
pendingUpdates.push(record.id);
|
||||||
|
exported++;
|
||||||
// Batch commit
|
|
||||||
if (pendingUpdates.length >= BATCH_SIZE) {
|
// Batch commit
|
||||||
await flushUpdates(pendingUpdates);
|
if (pendingUpdates.length >= BATCH_SIZE) {
|
||||||
pendingUpdates = [];
|
await flushUpdates(pendingUpdates);
|
||||||
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
pendingUpdates = [];
|
||||||
}
|
process.stdout.write(`\r Exported: ${exported} / ${records.length}`);
|
||||||
}
|
}
|
||||||
} catch (err) {
|
}
|
||||||
console.error(`\n ERROR exporting ${record.serial_number}: ${err.message}`);
|
} catch (err) {
|
||||||
errors++;
|
console.error(`\n ERROR exporting ${record.serial_number}: ${err.message}`);
|
||||||
}
|
errors++;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
// Flush remaining updates
|
|
||||||
if (pendingUpdates.length > 0) {
|
// Flush remaining updates
|
||||||
await flushUpdates(pendingUpdates);
|
if (pendingUpdates.length > 0) {
|
||||||
}
|
await flushUpdates(pendingUpdates);
|
||||||
|
}
|
||||||
console.log(`\n\n========================================`);
|
|
||||||
console.log(`Export Complete`);
|
console.log(`\n\n========================================`);
|
||||||
console.log(`========================================`);
|
console.log(`Export Complete`);
|
||||||
console.log(`Exported: ${exported}`);
|
console.log(`========================================`);
|
||||||
console.log(`Skipped: ${skipped} (${noSpecs} missing specs)`);
|
console.log(`Exported: ${exported}`);
|
||||||
console.log(`Errors: ${errors}`);
|
console.log(`Skipped: ${skipped} (${noSpecs} missing specs)`);
|
||||||
console.log(`End: ${new Date().toISOString()}`);
|
console.log(`Errors: ${errors}`);
|
||||||
|
console.log(`End: ${new Date().toISOString()}`);
|
||||||
await db.close();
|
|
||||||
return { exported, skipped, errors };
|
await db.close();
|
||||||
}
|
return { exported, skipped, errors };
|
||||||
|
}
|
||||||
async function flushUpdates(ids) {
|
|
||||||
const now = new Date().toISOString();
|
async function flushUpdates(ids) {
|
||||||
await db.transaction(async (txClient) => {
|
const now = new Date().toISOString();
|
||||||
for (const id of ids) {
|
await db.transaction(async (txClient) => {
|
||||||
await txClient.execute(
|
for (const id of ids) {
|
||||||
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
await txClient.execute(
|
||||||
[now, id]
|
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
||||||
);
|
[now, id]
|
||||||
}
|
);
|
||||||
});
|
}
|
||||||
}
|
});
|
||||||
|
}
|
||||||
// Export function for use by import.js (no db argument -- uses shared pool)
|
|
||||||
async function exportNewRecords(specMap, filePaths) {
|
// Export function for use by import.js (no db argument -- uses shared pool)
|
||||||
if (!fs.existsSync(OUTPUT_DIR)) {
|
async function exportNewRecords(specMap, filePaths) {
|
||||||
console.log(`[EXPORT] Output directory not found: ${OUTPUT_DIR}`);
|
if (!fs.existsSync(OUTPUT_DIR)) {
|
||||||
return 0;
|
console.log(`[EXPORT] Output directory not found: ${OUTPUT_DIR}`);
|
||||||
}
|
return 0;
|
||||||
|
}
|
||||||
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
|
||||||
const params = [];
|
const conditions = [`overall_result = 'PASS'`, `forweb_exported_at IS NULL`];
|
||||||
let paramIdx = 0;
|
const params = [];
|
||||||
|
let paramIdx = 0;
|
||||||
if (filePaths && filePaths.length > 0) {
|
|
||||||
const placeholders = filePaths.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
if (filePaths && filePaths.length > 0) {
|
||||||
conditions.push(`source_file IN (${placeholders})`);
|
const placeholders = filePaths.map(() => { paramIdx++; return `$${paramIdx}`; }).join(',');
|
||||||
params.push(...filePaths);
|
conditions.push(`source_file IN (${placeholders})`);
|
||||||
}
|
params.push(...filePaths);
|
||||||
|
}
|
||||||
const sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')}`;
|
|
||||||
const records = await db.query(sql, params);
|
const sql = `SELECT * FROM test_records WHERE ${conditions.join(' AND ')}`;
|
||||||
if (records.length === 0) return 0;
|
const records = await db.query(sql, params);
|
||||||
|
if (records.length === 0) return 0;
|
||||||
let exported = 0;
|
|
||||||
|
let exported = 0;
|
||||||
await db.transaction(async (txClient) => {
|
|
||||||
for (const record of records) {
|
await db.transaction(async (txClient) => {
|
||||||
const filename = record.serial_number + '.TXT';
|
for (const record of records) {
|
||||||
const outputPath = path.join(OUTPUT_DIR, filename);
|
const filename = record.serial_number + '.TXT';
|
||||||
|
const outputPath = path.join(OUTPUT_DIR, filename);
|
||||||
try {
|
|
||||||
// VASLOG_ENG: verbatim copy, preserving original bytes.
|
try {
|
||||||
if (record.log_type === 'VASLOG_ENG') {
|
// VASLOG_ENG: verbatim copy, preserving original bytes.
|
||||||
if (record.source_file && fs.existsSync(record.source_file)) {
|
if (record.log_type === 'VASLOG_ENG') {
|
||||||
fs.copyFileSync(record.source_file, outputPath);
|
if (record.source_file && fs.existsSync(record.source_file)) {
|
||||||
} else {
|
fs.copyFileSync(record.source_file, outputPath);
|
||||||
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
} else {
|
||||||
if (!record.raw_data) continue;
|
console.warn(`[WARN] source file missing, writing decoded raw_data for ${record.serial_number}`);
|
||||||
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
if (!record.raw_data) continue;
|
||||||
}
|
fs.writeFileSync(outputPath, record.raw_data, 'utf8');
|
||||||
} else {
|
}
|
||||||
const specs = getSpecs(specMap, record.model_number);
|
} else {
|
||||||
if (!specs) continue;
|
const specs = getSpecs(specMap, record.model_number);
|
||||||
const txt = generateExactDatasheet(record, specs);
|
if (!specs) continue;
|
||||||
if (!txt) continue;
|
const txt = generateExactDatasheet(record, specs);
|
||||||
fs.writeFileSync(outputPath, txt, 'utf8');
|
if (!txt) continue;
|
||||||
}
|
fs.writeFileSync(outputPath, txt, 'utf8');
|
||||||
|
}
|
||||||
await txClient.execute(
|
|
||||||
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
await txClient.execute(
|
||||||
[new Date().toISOString(), record.id]
|
'UPDATE test_records SET forweb_exported_at = $1 WHERE id = $2',
|
||||||
);
|
[new Date().toISOString(), record.id]
|
||||||
exported++;
|
);
|
||||||
} catch (err) {
|
exported++;
|
||||||
console.error(`[EXPORT] Error writing ${filename}: ${err.message}`);
|
} catch (err) {
|
||||||
}
|
console.error(`[EXPORT] Error writing ${filename}: ${err.message}`);
|
||||||
}
|
}
|
||||||
});
|
}
|
||||||
|
});
|
||||||
console.log(`[EXPORT] Generated ${exported} datasheet(s)`);
|
|
||||||
return exported;
|
console.log(`[EXPORT] Generated ${exported} datasheet(s)`);
|
||||||
}
|
return exported;
|
||||||
|
}
|
||||||
if (require.main === module) {
|
|
||||||
run().catch(console.error);
|
if (require.main === module) {
|
||||||
}
|
run()
|
||||||
|
.then(({ exported, skipped, errors }) => {
|
||||||
module.exports = { exportNewRecords };
|
if (errors > 0) {
|
||||||
|
return sendFailureEmail(
|
||||||
|
`[testdatadb] Datasheet export completed with ${errors} error(s)`,
|
||||||
|
`Export finished but ${errors} record(s) failed to write to the web directory.\n\nExported: ${exported}\nSkipped: ${skipped}\nErrors: ${errors}\n\nCheck the service log on AD2 for details.`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.catch(async (err) => {
|
||||||
|
console.error(err);
|
||||||
|
await sendFailureEmail(
|
||||||
|
'[testdatadb] Datasheet export failed',
|
||||||
|
`Export task crashed before completion.\n\nError: ${err.message}\n\nStack:\n${err.stack || '(none)'}`
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { exportNewRecords };
|
||||||
|
|||||||
@@ -1,396 +1,407 @@
|
|||||||
/**
|
/**
|
||||||
* Data Import Script
|
* Data Import Script
|
||||||
* Imports test data from DAT and SHT files into PostgreSQL database
|
* Imports test data from DAT and SHT files into PostgreSQL database
|
||||||
*/
|
*/
|
||||||
|
|
||||||
const fs = require('fs');
|
const fs = require('fs');
|
||||||
const path = require('path');
|
const path = require('path');
|
||||||
const db = require('./db');
|
const db = require('./db');
|
||||||
|
|
||||||
const { parseMultilineFile, extractTestStation } = require('../parsers/multiline');
|
const { parseMultilineFile, extractTestStation } = require('../parsers/multiline');
|
||||||
const { parseCsvFile } = require('../parsers/csvline');
|
const { parseCsvFile } = require('../parsers/csvline');
|
||||||
const { parseShtFile } = require('../parsers/shtfile');
|
const { parseShtFile } = require('../parsers/shtfile');
|
||||||
const { parseVaslogEngTxt } = require('../parsers/vaslog-engtxt');
|
const { parseVaslogEngTxt } = require('../parsers/vaslog-engtxt');
|
||||||
|
const { sendFailureEmail } = require('../server/notify');
|
||||||
// Data source paths
|
|
||||||
const TEST_PATH = 'C:/Shares/test';
|
// Data source paths
|
||||||
const RECOVERY_PATH = 'C:/Shares/Recovery-TEST';
|
const TEST_PATH = 'C:/Shares/test';
|
||||||
const HISTLOGS_PATH = path.join(TEST_PATH, 'Ate/HISTLOGS');
|
const RECOVERY_PATH = 'C:/Shares/Recovery-TEST';
|
||||||
|
const HISTLOGS_PATH = path.join(TEST_PATH, 'Ate/HISTLOGS');
|
||||||
// Log types and their parsers.
|
|
||||||
// NOTE: `recursive` defaults to TRUE when absent (walk subfolders by default,
|
// Log types and their parsers.
|
||||||
// preserving pre-existing production behavior for DSCLOG/5BLOG/8BLOG/PWRLOG/
|
// NOTE: `recursive` defaults to TRUE when absent (walk subfolders by default,
|
||||||
// SCTLOG/7BLOG). Set it to FALSE explicitly on VASLOG so the .DAT walk does
|
// preserving pre-existing production behavior for DSCLOG/5BLOG/8BLOG/PWRLOG/
|
||||||
// NOT descend into the "VASLOG - Engineering Tested" subfolder (belt-and-
|
// SCTLOG/7BLOG). Set it to FALSE explicitly on VASLOG so the .DAT walk does
|
||||||
// suspenders: the .DAT glob wouldn't match .txt, but be explicit anyway).
|
// NOT descend into the "VASLOG - Engineering Tested" subfolder (belt-and-
|
||||||
// VASLOG_ENG also sets recursive:false -- the eng-tested dir is flat.
|
// suspenders: the .DAT glob wouldn't match .txt, but be explicit anyway).
|
||||||
const LOG_TYPES = {
|
// VASLOG_ENG also sets recursive:false -- the eng-tested dir is flat.
|
||||||
'DSCLOG': { parser: 'multiline', ext: '.DAT' },
|
const LOG_TYPES = {
|
||||||
'5BLOG': { parser: 'multiline', ext: '.DAT' },
|
'DSCLOG': { parser: 'multiline', ext: '.DAT' },
|
||||||
'8BLOG': { parser: 'multiline', ext: '.DAT' },
|
'5BLOG': { parser: 'multiline', ext: '.DAT' },
|
||||||
'PWRLOG': { parser: 'multiline', ext: '.DAT' },
|
'8BLOG': { parser: 'multiline', ext: '.DAT' },
|
||||||
'SCTLOG': { parser: 'multiline', ext: '.DAT' },
|
'PWRLOG': { parser: 'multiline', ext: '.DAT' },
|
||||||
'VASLOG': { parser: 'multiline', ext: '.DAT', recursive: false },
|
'SCTLOG': { parser: 'multiline', ext: '.DAT' },
|
||||||
'7BLOG': { parser: 'csvline', ext: '.DAT' },
|
'VASLOG': { parser: 'multiline', ext: '.DAT', recursive: false },
|
||||||
// Engineering-tested SCMHVAS pre-rendered datasheets live under VASLOG/"VASLOG - Engineering Tested"/
|
'7BLOG': { parser: 'csvline', ext: '.DAT' },
|
||||||
'VASLOG_ENG': { parser: 'vaslog-engtxt', ext: '.txt', dir: 'VASLOG/VASLOG - Engineering Tested', recursive: false }
|
// Engineering-tested SCMHVAS pre-rendered datasheets live under VASLOG/"VASLOG - Engineering Tested"/
|
||||||
};
|
'VASLOG_ENG': { parser: 'vaslog-engtxt', ext: '.txt', dir: 'VASLOG/VASLOG - Engineering Tested', recursive: false }
|
||||||
|
};
|
||||||
// Find all files of a specific type in a directory
|
|
||||||
function findFiles(dir, pattern, recursive = true) {
|
// Find all files of a specific type in a directory
|
||||||
const results = [];
|
function findFiles(dir, pattern, recursive = true) {
|
||||||
|
const results = [];
|
||||||
try {
|
|
||||||
if (!fs.existsSync(dir)) return results;
|
try {
|
||||||
|
if (!fs.existsSync(dir)) return results;
|
||||||
const items = fs.readdirSync(dir, { withFileTypes: true });
|
|
||||||
|
const items = fs.readdirSync(dir, { withFileTypes: true });
|
||||||
for (const item of items) {
|
|
||||||
const fullPath = path.join(dir, item.name);
|
for (const item of items) {
|
||||||
|
const fullPath = path.join(dir, item.name);
|
||||||
if (item.isDirectory() && recursive) {
|
|
||||||
results.push(...findFiles(fullPath, pattern, recursive));
|
if (item.isDirectory() && recursive) {
|
||||||
} else if (item.isFile()) {
|
results.push(...findFiles(fullPath, pattern, recursive));
|
||||||
if (pattern.test(item.name)) {
|
} else if (item.isFile()) {
|
||||||
results.push(fullPath);
|
if (pattern.test(item.name)) {
|
||||||
}
|
results.push(fullPath);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} catch (err) {
|
}
|
||||||
// Ignore permission errors
|
} catch (err) {
|
||||||
}
|
// Ignore permission errors
|
||||||
|
}
|
||||||
return results;
|
|
||||||
}
|
return results;
|
||||||
|
}
|
||||||
// Parse records from a file (sync -- file I/O only)
|
|
||||||
function parseFile(filePath, logType, parser) {
|
// Parse records from a file (sync -- file I/O only)
|
||||||
const testStation = extractTestStation(filePath);
|
function parseFile(filePath, logType, parser) {
|
||||||
|
const testStation = extractTestStation(filePath);
|
||||||
switch (parser) {
|
|
||||||
case 'multiline':
|
switch (parser) {
|
||||||
return parseMultilineFile(filePath, logType, testStation);
|
case 'multiline':
|
||||||
case 'csvline':
|
return parseMultilineFile(filePath, logType, testStation);
|
||||||
return parseCsvFile(filePath, testStation);
|
case 'csvline':
|
||||||
case 'shtfile':
|
return parseCsvFile(filePath, testStation);
|
||||||
return parseShtFile(filePath, testStation);
|
case 'shtfile':
|
||||||
case 'vaslog-engtxt':
|
return parseShtFile(filePath, testStation);
|
||||||
return parseVaslogEngTxt(filePath, testStation);
|
case 'vaslog-engtxt':
|
||||||
default:
|
return parseVaslogEngTxt(filePath, testStation);
|
||||||
return [];
|
default:
|
||||||
}
|
return [];
|
||||||
}
|
}
|
||||||
|
}
|
||||||
// Batch insert records into PostgreSQL
|
|
||||||
async function insertBatch(txClient, records) {
|
// Batch insert records into PostgreSQL
|
||||||
let imported = 0;
|
async function insertBatch(txClient, records) {
|
||||||
for (const record of records) {
|
let imported = 0;
|
||||||
try {
|
for (const record of records) {
|
||||||
const result = await txClient.execute(
|
try {
|
||||||
`INSERT INTO test_records
|
const result = await txClient.execute(
|
||||||
(log_type, model_number, serial_number, test_date, test_station, overall_result, raw_data, source_file)
|
`INSERT INTO test_records
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
(log_type, model_number, serial_number, test_date, test_station, overall_result, raw_data, source_file)
|
||||||
ON CONFLICT (log_type, model_number, serial_number, test_date, test_station)
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
||||||
DO UPDATE SET raw_data = EXCLUDED.raw_data, overall_result = EXCLUDED.overall_result`,
|
ON CONFLICT (log_type, model_number, serial_number, test_date, test_station)
|
||||||
[
|
DO UPDATE SET raw_data = EXCLUDED.raw_data, overall_result = EXCLUDED.overall_result`,
|
||||||
record.log_type,
|
[
|
||||||
record.model_number,
|
record.log_type,
|
||||||
record.serial_number,
|
record.model_number,
|
||||||
record.test_date,
|
record.serial_number,
|
||||||
record.test_station,
|
record.test_date,
|
||||||
record.overall_result,
|
record.test_station,
|
||||||
record.raw_data,
|
record.overall_result,
|
||||||
record.source_file
|
record.raw_data,
|
||||||
]
|
record.source_file
|
||||||
);
|
]
|
||||||
if (result.rowCount > 0) imported++;
|
);
|
||||||
} catch (err) {
|
if (result.rowCount > 0) imported++;
|
||||||
// Constraint error - skip
|
} catch (err) {
|
||||||
}
|
// Constraint error - skip
|
||||||
}
|
}
|
||||||
return imported;
|
}
|
||||||
}
|
return imported;
|
||||||
|
}
|
||||||
// Import records from a file
|
|
||||||
async function importFile(txClient, filePath, logType, parser) {
|
// Import records from a file
|
||||||
let records = [];
|
async function importFile(txClient, filePath, logType, parser) {
|
||||||
|
let records = [];
|
||||||
try {
|
|
||||||
records = parseFile(filePath, logType, parser);
|
try {
|
||||||
const imported = await insertBatch(txClient, records);
|
records = parseFile(filePath, logType, parser);
|
||||||
return { total: records.length, imported };
|
const imported = await insertBatch(txClient, records);
|
||||||
} catch (err) {
|
return { total: records.length, imported };
|
||||||
console.error(`Error importing ${filePath}: ${err.message}`);
|
} catch (err) {
|
||||||
return { total: 0, imported: 0 };
|
console.error(`Error importing ${filePath}: ${err.message}`);
|
||||||
}
|
return { total: 0, imported: 0 };
|
||||||
}
|
}
|
||||||
|
}
|
||||||
// Import from HISTLOGS (master consolidated logs)
|
|
||||||
async function importHistlogs(txClient) {
|
// Import from HISTLOGS (master consolidated logs)
|
||||||
console.log('\n=== Importing from HISTLOGS ===');
|
async function importHistlogs(txClient) {
|
||||||
|
console.log('\n=== Importing from HISTLOGS ===');
|
||||||
let totalImported = 0;
|
|
||||||
let totalRecords = 0;
|
let totalImported = 0;
|
||||||
|
let totalRecords = 0;
|
||||||
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
|
||||||
const subdir = config.dir || logType;
|
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
||||||
const logDir = path.join(HISTLOGS_PATH, subdir);
|
const subdir = config.dir || logType;
|
||||||
|
const logDir = path.join(HISTLOGS_PATH, subdir);
|
||||||
if (!fs.existsSync(logDir)) {
|
|
||||||
console.log(` ${logType}: directory not found`);
|
if (!fs.existsSync(logDir)) {
|
||||||
continue;
|
console.log(` ${logType}: directory not found`);
|
||||||
}
|
continue;
|
||||||
|
}
|
||||||
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
|
||||||
console.log(` ${logType}: found ${files.length} files`);
|
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
||||||
|
console.log(` ${logType}: found ${files.length} files`);
|
||||||
for (const file of files) {
|
|
||||||
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
for (const file of files) {
|
||||||
totalRecords += total;
|
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
||||||
totalImported += imported;
|
totalRecords += total;
|
||||||
}
|
totalImported += imported;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
console.log(` HISTLOGS total: ${totalImported} records imported (${totalRecords} parsed)`);
|
|
||||||
return totalImported;
|
console.log(` HISTLOGS total: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||||
}
|
return totalImported;
|
||||||
|
}
|
||||||
// Import from test station logs
|
|
||||||
async function importStationLogs(txClient, basePath, label) {
|
// Import from test station logs
|
||||||
console.log(`\n=== Importing from ${label} ===`);
|
async function importStationLogs(txClient, basePath, label) {
|
||||||
|
console.log(`\n=== Importing from ${label} ===`);
|
||||||
let totalImported = 0;
|
|
||||||
let totalRecords = 0;
|
let totalImported = 0;
|
||||||
|
let totalRecords = 0;
|
||||||
const stationPattern = /^TS-\d+[LR]?$/i;
|
|
||||||
let stations = [];
|
const stationPattern = /^TS-\d+[LR]?$/i;
|
||||||
|
let stations = [];
|
||||||
try {
|
|
||||||
const items = fs.readdirSync(basePath, { withFileTypes: true });
|
try {
|
||||||
stations = items
|
const items = fs.readdirSync(basePath, { withFileTypes: true });
|
||||||
.filter(i => i.isDirectory() && stationPattern.test(i.name))
|
stations = items
|
||||||
.map(i => i.name);
|
.filter(i => i.isDirectory() && stationPattern.test(i.name))
|
||||||
} catch (err) {
|
.map(i => i.name);
|
||||||
console.log(` Error reading ${basePath}: ${err.message}`);
|
} catch (err) {
|
||||||
return 0;
|
console.log(` Error reading ${basePath}: ${err.message}`);
|
||||||
}
|
return 0;
|
||||||
|
}
|
||||||
console.log(` Found stations: ${stations.join(', ')}`);
|
|
||||||
|
console.log(` Found stations: ${stations.join(', ')}`);
|
||||||
for (const station of stations) {
|
|
||||||
const logsDir = path.join(basePath, station, 'LOGS');
|
for (const station of stations) {
|
||||||
|
const logsDir = path.join(basePath, station, 'LOGS');
|
||||||
if (!fs.existsSync(logsDir)) continue;
|
|
||||||
|
if (!fs.existsSync(logsDir)) continue;
|
||||||
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
|
||||||
const subdir = config.dir || logType;
|
for (const [logType, config] of Object.entries(LOG_TYPES)) {
|
||||||
const logDir = path.join(logsDir, subdir);
|
const subdir = config.dir || logType;
|
||||||
|
const logDir = path.join(logsDir, subdir);
|
||||||
if (!fs.existsSync(logDir)) continue;
|
|
||||||
|
if (!fs.existsSync(logDir)) continue;
|
||||||
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
|
||||||
|
const files = findFiles(logDir, new RegExp(`\\${config.ext}$`, 'i'), config.recursive !== false);
|
||||||
for (const file of files) {
|
|
||||||
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
for (const file of files) {
|
||||||
totalRecords += total;
|
const { total, imported } = await importFile(txClient, file, logType, config.parser);
|
||||||
totalImported += imported;
|
totalRecords += total;
|
||||||
}
|
totalImported += imported;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
// Also import SHT files
|
|
||||||
const shtFiles = findFiles(basePath, /\.SHT$/i, true);
|
// Also import SHT files
|
||||||
console.log(` Found ${shtFiles.length} SHT files`);
|
const shtFiles = findFiles(basePath, /\.SHT$/i, true);
|
||||||
|
console.log(` Found ${shtFiles.length} SHT files`);
|
||||||
for (const file of shtFiles) {
|
|
||||||
const { total, imported } = await importFile(txClient, file, 'SHT', 'shtfile');
|
for (const file of shtFiles) {
|
||||||
totalRecords += total;
|
const { total, imported } = await importFile(txClient, file, 'SHT', 'shtfile');
|
||||||
totalImported += imported;
|
totalRecords += total;
|
||||||
}
|
totalImported += imported;
|
||||||
|
}
|
||||||
console.log(` ${label} total: ${totalImported} records imported (${totalRecords} parsed)`);
|
|
||||||
return totalImported;
|
console.log(` ${label} total: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||||
}
|
return totalImported;
|
||||||
|
}
|
||||||
// Import from Recovery-TEST backups (newest first)
|
|
||||||
async function importRecoveryBackups(txClient) {
|
// Import from Recovery-TEST backups (newest first)
|
||||||
console.log('\n=== Importing from Recovery-TEST backups ===');
|
async function importRecoveryBackups(txClient) {
|
||||||
|
console.log('\n=== Importing from Recovery-TEST backups ===');
|
||||||
if (!fs.existsSync(RECOVERY_PATH)) {
|
|
||||||
console.log(' Recovery-TEST directory not found');
|
if (!fs.existsSync(RECOVERY_PATH)) {
|
||||||
return 0;
|
console.log(' Recovery-TEST directory not found');
|
||||||
}
|
return 0;
|
||||||
|
}
|
||||||
const backups = fs.readdirSync(RECOVERY_PATH, { withFileTypes: true })
|
|
||||||
.filter(i => i.isDirectory() && /^\d{2}-\d{2}-\d{2}$/.test(i.name))
|
const backups = fs.readdirSync(RECOVERY_PATH, { withFileTypes: true })
|
||||||
.map(i => i.name)
|
.filter(i => i.isDirectory() && /^\d{2}-\d{2}-\d{2}$/.test(i.name))
|
||||||
.sort()
|
.map(i => i.name)
|
||||||
.reverse();
|
.sort()
|
||||||
|
.reverse();
|
||||||
console.log(` Found backup dates: ${backups.join(', ')}`);
|
|
||||||
|
console.log(` Found backup dates: ${backups.join(', ')}`);
|
||||||
let totalImported = 0;
|
|
||||||
|
let totalImported = 0;
|
||||||
for (const backup of backups) {
|
|
||||||
const backupPath = path.join(RECOVERY_PATH, backup);
|
for (const backup of backups) {
|
||||||
const imported = await importStationLogs(txClient, backupPath, `Recovery-TEST/${backup}`);
|
const backupPath = path.join(RECOVERY_PATH, backup);
|
||||||
totalImported += imported;
|
const imported = await importStationLogs(txClient, backupPath, `Recovery-TEST/${backup}`);
|
||||||
}
|
totalImported += imported;
|
||||||
|
}
|
||||||
return totalImported;
|
|
||||||
}
|
return totalImported;
|
||||||
|
}
|
||||||
// Main import function
|
|
||||||
async function runImport() {
|
// Main import function
|
||||||
console.log('========================================');
|
async function runImport() {
|
||||||
console.log('Test Data Import');
|
console.log('========================================');
|
||||||
console.log('========================================');
|
console.log('Test Data Import');
|
||||||
console.log(`Start time: ${new Date().toISOString()}`);
|
console.log('========================================');
|
||||||
|
console.log(`Start time: ${new Date().toISOString()}`);
|
||||||
let grandTotal = 0;
|
|
||||||
|
let grandTotal = 0;
|
||||||
await db.transaction(async (txClient) => {
|
|
||||||
grandTotal += await importHistlogs(txClient);
|
await db.transaction(async (txClient) => {
|
||||||
grandTotal += await importRecoveryBackups(txClient);
|
grandTotal += await importHistlogs(txClient);
|
||||||
grandTotal += await importStationLogs(txClient, TEST_PATH, 'test');
|
grandTotal += await importRecoveryBackups(txClient);
|
||||||
});
|
grandTotal += await importStationLogs(txClient, TEST_PATH, 'test');
|
||||||
|
});
|
||||||
const stats = await db.queryOne('SELECT COUNT(*) as count FROM test_records');
|
|
||||||
|
const stats = await db.queryOne('SELECT COUNT(*) as count FROM test_records');
|
||||||
console.log('\n========================================');
|
|
||||||
console.log('Import Complete');
|
console.log('\n========================================');
|
||||||
console.log('========================================');
|
console.log('Import Complete');
|
||||||
console.log(`Total records in database: ${stats.count}`);
|
console.log('========================================');
|
||||||
console.log(`End time: ${new Date().toISOString()}`);
|
console.log(`Total records in database: ${stats.count}`);
|
||||||
|
console.log(`End time: ${new Date().toISOString()}`);
|
||||||
await db.close();
|
|
||||||
}
|
await db.close();
|
||||||
|
}
|
||||||
// Import a single file (for incremental imports from sync)
|
|
||||||
async function importSingleFile(filePath) {
|
// Import a single file (for incremental imports from sync)
|
||||||
console.log(`Importing: ${filePath}`);
|
async function importSingleFile(filePath) {
|
||||||
|
console.log(`Importing: ${filePath}`);
|
||||||
let logType = null;
|
|
||||||
let parser = null;
|
let logType = null;
|
||||||
|
let parser = null;
|
||||||
// VASLOG_ENG subpath must be checked before VASLOG (substring overlap).
|
|
||||||
if (filePath.includes('VASLOG - Engineering Tested')) {
|
// VASLOG_ENG subpath must be checked before VASLOG (substring overlap).
|
||||||
logType = 'VASLOG_ENG';
|
if (filePath.includes('VASLOG - Engineering Tested')) {
|
||||||
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
logType = 'VASLOG_ENG';
|
||||||
} else {
|
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
||||||
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
} else {
|
||||||
if (type === 'VASLOG_ENG') continue;
|
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
||||||
if (filePath.includes(type)) {
|
if (type === 'VASLOG_ENG') continue;
|
||||||
logType = type;
|
if (filePath.includes(type)) {
|
||||||
parser = config.parser;
|
logType = type;
|
||||||
break;
|
parser = config.parser;
|
||||||
}
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
if (!logType) {
|
|
||||||
if (/\.SHT$/i.test(filePath)) {
|
if (!logType) {
|
||||||
logType = 'SHT';
|
if (/\.SHT$/i.test(filePath)) {
|
||||||
parser = 'shtfile';
|
logType = 'SHT';
|
||||||
} else {
|
parser = 'shtfile';
|
||||||
console.log(` Unknown log type for: ${filePath}`);
|
} else {
|
||||||
return { total: 0, imported: 0 };
|
console.log(` Unknown log type for: ${filePath}`);
|
||||||
}
|
return { total: 0, imported: 0 };
|
||||||
}
|
}
|
||||||
|
}
|
||||||
let result;
|
|
||||||
await db.transaction(async (txClient) => {
|
let result;
|
||||||
result = await importFile(txClient, filePath, logType, parser);
|
await db.transaction(async (txClient) => {
|
||||||
});
|
result = await importFile(txClient, filePath, logType, parser);
|
||||||
|
});
|
||||||
console.log(` Imported ${result.imported} of ${result.total} records`);
|
|
||||||
return result;
|
console.log(` Imported ${result.imported} of ${result.total} records`);
|
||||||
}
|
return result;
|
||||||
|
}
|
||||||
// Import multiple files (for batch incremental imports)
|
|
||||||
async function importFiles(filePaths) {
|
// Import multiple files (for batch incremental imports)
|
||||||
console.log(`\n========================================`);
|
async function importFiles(filePaths) {
|
||||||
console.log(`Incremental Import: ${filePaths.length} files`);
|
console.log(`\n========================================`);
|
||||||
console.log(`========================================`);
|
console.log(`Incremental Import: ${filePaths.length} files`);
|
||||||
|
console.log(`========================================`);
|
||||||
let totalImported = 0;
|
|
||||||
let totalRecords = 0;
|
let totalImported = 0;
|
||||||
|
let totalRecords = 0;
|
||||||
await db.transaction(async (txClient) => {
|
|
||||||
for (const filePath of filePaths) {
|
await db.transaction(async (txClient) => {
|
||||||
let logType = null;
|
for (const filePath of filePaths) {
|
||||||
let parser = null;
|
let logType = null;
|
||||||
|
let parser = null;
|
||||||
// VASLOG_ENG subpath must be checked before the generic loop --
|
|
||||||
// otherwise `includes('VASLOG')` hits first and the eng .txt gets
|
// VASLOG_ENG subpath must be checked before the generic loop --
|
||||||
// dispatched to the multiline parser. Mirror importSingleFile().
|
// otherwise `includes('VASLOG')` hits first and the eng .txt gets
|
||||||
if (filePath.includes('VASLOG - Engineering Tested')) {
|
// dispatched to the multiline parser. Mirror importSingleFile().
|
||||||
logType = 'VASLOG_ENG';
|
if (filePath.includes('VASLOG - Engineering Tested')) {
|
||||||
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
logType = 'VASLOG_ENG';
|
||||||
} else {
|
parser = LOG_TYPES['VASLOG_ENG'].parser;
|
||||||
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
} else {
|
||||||
if (type === 'VASLOG_ENG') continue;
|
for (const [type, config] of Object.entries(LOG_TYPES)) {
|
||||||
if (filePath.includes(type)) {
|
if (type === 'VASLOG_ENG') continue;
|
||||||
logType = type;
|
if (filePath.includes(type)) {
|
||||||
parser = config.parser;
|
logType = type;
|
||||||
break;
|
parser = config.parser;
|
||||||
}
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
if (!logType) {
|
|
||||||
if (/\.SHT$/i.test(filePath)) {
|
if (!logType) {
|
||||||
logType = 'SHT';
|
if (/\.SHT$/i.test(filePath)) {
|
||||||
parser = 'shtfile';
|
logType = 'SHT';
|
||||||
} else {
|
parser = 'shtfile';
|
||||||
console.log(` Skipping unknown type: ${filePath}`);
|
} else {
|
||||||
continue;
|
console.log(` Skipping unknown type: ${filePath}`);
|
||||||
}
|
continue;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
const { total, imported } = await importFile(txClient, filePath, logType, parser);
|
|
||||||
totalRecords += total;
|
const { total, imported } = await importFile(txClient, filePath, logType, parser);
|
||||||
totalImported += imported;
|
totalRecords += total;
|
||||||
console.log(` ${path.basename(filePath)}: ${imported}/${total} records`);
|
totalImported += imported;
|
||||||
}
|
console.log(` ${path.basename(filePath)}: ${imported}/${total} records`);
|
||||||
});
|
}
|
||||||
|
});
|
||||||
console.log(`\nTotal: ${totalImported} records imported (${totalRecords} parsed)`);
|
|
||||||
|
console.log(`\nTotal: ${totalImported} records imported (${totalRecords} parsed)`);
|
||||||
// Export datasheets for newly imported records
|
|
||||||
if (totalImported > 0) {
|
// Export datasheets for newly imported records
|
||||||
try {
|
if (totalImported > 0) {
|
||||||
const { loadAllSpecs } = require('../parsers/spec-reader');
|
try {
|
||||||
const { exportNewRecords } = require('./export-datasheets');
|
const { loadAllSpecs } = require('../parsers/spec-reader');
|
||||||
const specMap = loadAllSpecs();
|
const { exportNewRecords } = require('./export-datasheets');
|
||||||
await exportNewRecords(specMap, filePaths);
|
const specMap = loadAllSpecs();
|
||||||
} catch (err) {
|
await exportNewRecords(specMap, filePaths);
|
||||||
console.error(`[EXPORT] Datasheet export failed: ${err.message}`);
|
} catch (err) {
|
||||||
}
|
console.error(`[EXPORT] Datasheet export failed: ${err.message}`);
|
||||||
}
|
await sendFailureEmail(
|
||||||
|
'[testdatadb] Datasheet export failed after import',
|
||||||
return { total: totalRecords, imported: totalImported };
|
`Export step failed after importing ${totalImported} record(s).\n\nError: ${err.message}\n\nStack:\n${err.stack || '(none)'}`
|
||||||
}
|
);
|
||||||
|
}
|
||||||
// Run if called directly
|
}
|
||||||
if (require.main === module) {
|
|
||||||
const args = process.argv.slice(2);
|
return { total: totalRecords, imported: totalImported };
|
||||||
|
}
|
||||||
if (args.length > 0 && args[0] === '--file') {
|
|
||||||
const files = args.slice(1);
|
// Run if called directly
|
||||||
if (files.length === 0) {
|
if (require.main === module) {
|
||||||
console.log('Usage: node import.js --file <file1> [file2] ...');
|
const args = process.argv.slice(2);
|
||||||
process.exit(1);
|
|
||||||
}
|
if (args.length > 0 && args[0] === '--file') {
|
||||||
importFiles(files).then(() => db.close()).catch(console.error);
|
const files = args.slice(1);
|
||||||
} else if (args.length > 0 && args[0] === '--help') {
|
if (files.length === 0) {
|
||||||
console.log('Usage:');
|
console.log('Usage: node import.js --file <file1> [file2] ...');
|
||||||
console.log(' node import.js Full import from all sources');
|
process.exit(1);
|
||||||
console.log(' node import.js --file <f> Import specific file(s)');
|
}
|
||||||
process.exit(0);
|
importFiles(files).then(() => db.close()).catch(console.error);
|
||||||
} else {
|
} else if (args.length > 0 && args[0] === '--help') {
|
||||||
runImport().catch(console.error);
|
console.log('Usage:');
|
||||||
}
|
console.log(' node import.js Full import from all sources');
|
||||||
}
|
console.log(' node import.js --file <f> Import specific file(s)');
|
||||||
|
process.exit(0);
|
||||||
module.exports = { runImport, importSingleFile, importFiles };
|
} else {
|
||||||
|
runImport().catch(async (err) => {
|
||||||
|
console.error(err);
|
||||||
|
await sendFailureEmail(
|
||||||
|
'[testdatadb] DB import failed',
|
||||||
|
`The scheduled import job crashed.\n\nError: ${err.message}\n\nStack:\n${err.stack || '(none)'}`
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { runImport, importSingleFile, importFiles };
|
||||||
|
|||||||
@@ -1,224 +1,300 @@
|
|||||||
"""
|
"""
|
||||||
Deploy staged pipeline changes to AD2:C:\\Shares\\testdatadb\\.
|
Deploy staged pipeline changes to AD2:C:\\Shares\\testdatadb\\.
|
||||||
|
|
||||||
Backs up each existing target to <name>.bak-YYYYMMDD before overwriting.
|
Backs up each existing target to <name>.bak-YYYYMMDD before overwriting.
|
||||||
Fails if a target file does not exist on AD2 (excluding brand-new files
|
Fails if a target file does not exist on AD2 (excluding brand-new files
|
||||||
declared in NEW_FILES below).
|
declared in NEW_FILES below).
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
python deploy-to-ad2.py --dry-run
|
python deploy-to-ad2.py --dry-run
|
||||||
python deploy-to-ad2.py
|
python deploy-to-ad2.py
|
||||||
|
|
||||||
Credentials: fetched at runtime from the SOPS vault
|
Credentials: fetched at runtime from the SOPS vault
|
||||||
(clients/dataforth/ad2.sops.yaml -> credentials.password). No hardcoded
|
(clients/dataforth/ad2.sops.yaml -> credentials.password). No hardcoded
|
||||||
password; no env-var / prompt fallback. Fails loud if the vault read fails.
|
password; no env-var / prompt fallback. Fails loud if the vault read fails.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import datetime
|
import datetime
|
||||||
import os
|
import os
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
import paramiko
|
import paramiko
|
||||||
|
|
||||||
HOST = '192.168.0.6'
|
HOST = '192.168.0.6'
|
||||||
USER = 'sysadmin'
|
USER = 'sysadmin'
|
||||||
|
|
||||||
VAULT_SH = 'D:/vault/scripts/vault.sh'
|
VAULT_SH = 'D:/vault/scripts/vault.sh'
|
||||||
VAULT_ENTRY = 'clients/dataforth/ad2.sops.yaml'
|
VAULT_ENTRY = 'clients/dataforth/ad2.sops.yaml'
|
||||||
VAULT_FIELD = 'credentials.password'
|
VAULT_FIELD = 'credentials.password'
|
||||||
|
|
||||||
|
SMTP_VAULT_ENTRY = 'clients/dataforth/m365.sops.yaml'
|
||||||
def get_ad2_password() -> str:
|
SMTP_VAULT_FIELD = 'credentials.password'
|
||||||
"""Fetch the AD2 sysadmin password from the SOPS vault.
|
SMTP_USER = 'sysadmin@dataforth.com'
|
||||||
|
SMTP_HOST = 'smtp.office365.com'
|
||||||
Fails loud (raises) on any error: missing vault, decryption failure,
|
SMTP_PORT = 587
|
||||||
empty value. Do NOT fall back to env vars or prompts -- per CLAUDE.md
|
NOTIFY_TO = 'mike@azcomputerguru.com'
|
||||||
deploy scripts must not hold credentials.
|
|
||||||
"""
|
|
||||||
try:
|
def get_ad2_password() -> str:
|
||||||
result = subprocess.run(
|
"""Fetch the AD2 sysadmin password from the SOPS vault.
|
||||||
['bash', VAULT_SH, 'get-field', VAULT_ENTRY, VAULT_FIELD],
|
|
||||||
capture_output=True, text=True, timeout=30, check=False,
|
Fails loud (raises) on any error: missing vault, decryption failure,
|
||||||
)
|
empty value. Do NOT fall back to env vars or prompts -- per CLAUDE.md
|
||||||
except FileNotFoundError as e:
|
deploy scripts must not hold credentials.
|
||||||
raise RuntimeError(
|
"""
|
||||||
f'[FAIL] vault helper not runnable: {VAULT_SH} ({e})'
|
try:
|
||||||
) from e
|
result = subprocess.run(
|
||||||
except subprocess.TimeoutExpired as e:
|
['bash', VAULT_SH, 'get-field', VAULT_ENTRY, VAULT_FIELD],
|
||||||
raise RuntimeError(
|
capture_output=True, text=True, timeout=30, check=False,
|
||||||
f'[FAIL] vault read timed out after 30s for {VAULT_ENTRY}'
|
)
|
||||||
) from e
|
except FileNotFoundError as e:
|
||||||
|
raise RuntimeError(
|
||||||
if result.returncode != 0:
|
f'[FAIL] vault helper not runnable: {VAULT_SH} ({e})'
|
||||||
stderr = (result.stderr or '').strip()
|
) from e
|
||||||
raise RuntimeError(
|
except subprocess.TimeoutExpired as e:
|
||||||
f'[FAIL] vault read failed (rc={result.returncode}) for '
|
raise RuntimeError(
|
||||||
f'{VAULT_ENTRY}:{VAULT_FIELD}: {stderr}'
|
f'[FAIL] vault read timed out after 30s for {VAULT_ENTRY}'
|
||||||
)
|
) from e
|
||||||
|
|
||||||
pwd = (result.stdout or '').strip()
|
if result.returncode != 0:
|
||||||
if not pwd:
|
stderr = (result.stderr or '').strip()
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
f'[FAIL] vault returned empty value for {VAULT_ENTRY}:{VAULT_FIELD}'
|
f'[FAIL] vault read failed (rc={result.returncode}) for '
|
||||||
)
|
f'{VAULT_ENTRY}:{VAULT_FIELD}: {stderr}'
|
||||||
return pwd
|
)
|
||||||
|
|
||||||
REMOTE_ROOT = 'C:/Shares/testdatadb'
|
pwd = (result.stdout or '').strip()
|
||||||
LOCAL_ROOT = os.path.dirname(os.path.abspath(__file__))
|
if not pwd:
|
||||||
|
raise RuntimeError(
|
||||||
# ---------------------------------------------------------------------------
|
f'[FAIL] vault returned empty value for {VAULT_ENTRY}:{VAULT_FIELD}'
|
||||||
# Deployment file lists. Each list has different semantics:
|
)
|
||||||
#
|
return pwd
|
||||||
# UPDATE_FILES -- file MUST already exist on AD2. Backup-then-overwrite.
|
|
||||||
# Fails loud if the remote file is missing (that's a drift
|
|
||||||
# signal -- something changed on the box we didn't expect).
|
def get_smtp_password() -> str:
|
||||||
#
|
try:
|
||||||
# NEW_FILES -- file must NOT already exist on AD2. Creates it.
|
result = subprocess.run(
|
||||||
# Fails loud if the remote file is already present (we would
|
['bash', VAULT_SH, 'get-field', SMTP_VAULT_ENTRY, SMTP_VAULT_FIELD],
|
||||||
# otherwise silently clobber something we didn't back up).
|
capture_output=True, text=True, timeout=30, check=False,
|
||||||
# ---------------------------------------------------------------------------
|
)
|
||||||
|
except (FileNotFoundError, subprocess.TimeoutExpired) as e:
|
||||||
# Files that already exist on AD2 and will be backed up + overwritten.
|
raise RuntimeError(f'[FAIL] vault read failed for SMTP creds: {e}') from e
|
||||||
UPDATE_FILES = [
|
|
||||||
('parsers/spec-reader.js', 'parsers/spec-reader.js'),
|
if result.returncode != 0:
|
||||||
('templates/datasheet-exact.js', 'templates/datasheet-exact.js'),
|
raise RuntimeError(
|
||||||
('database/import.js', 'database/import.js'),
|
f'[FAIL] vault read failed (rc={result.returncode}) for '
|
||||||
('database/export-datasheets.js', 'database/export-datasheets.js'),
|
f'{SMTP_VAULT_ENTRY}:{SMTP_VAULT_FIELD}: {result.stderr.strip()}'
|
||||||
]
|
)
|
||||||
|
|
||||||
# Files that do NOT yet exist on AD2 and must be created fresh.
|
pwd = (result.stdout or '').strip().replace('\\', '')
|
||||||
NEW_FILES = [
|
if not pwd:
|
||||||
('parsers/vaslog-engtxt.js', 'parsers/vaslog-engtxt.js'),
|
raise RuntimeError(f'[FAIL] vault returned empty SMTP password')
|
||||||
]
|
return pwd
|
||||||
|
|
||||||
|
REMOTE_ROOT = 'C:/Shares/testdatadb'
|
||||||
def connect() -> paramiko.SSHClient:
|
LOCAL_ROOT = os.path.dirname(os.path.abspath(__file__))
|
||||||
pwd = get_ad2_password()
|
|
||||||
c = paramiko.SSHClient()
|
# ---------------------------------------------------------------------------
|
||||||
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
# Deployment file lists. Each list has different semantics:
|
||||||
c.connect(
|
#
|
||||||
HOST, username=USER, password=pwd,
|
# UPDATE_FILES -- file MUST already exist on AD2. Backup-then-overwrite.
|
||||||
timeout=15, look_for_keys=False, allow_agent=False, banner_timeout=30,
|
# Fails loud if the remote file is missing (that's a drift
|
||||||
)
|
# signal -- something changed on the box we didn't expect).
|
||||||
return c
|
#
|
||||||
|
# NEW_FILES -- file must NOT already exist on AD2. Creates it.
|
||||||
|
# Fails loud if the remote file is already present (we would
|
||||||
def remote_exists(sftp: paramiko.SFTPClient, path: str) -> bool:
|
# otherwise silently clobber something we didn't back up).
|
||||||
try:
|
# ---------------------------------------------------------------------------
|
||||||
sftp.stat(path)
|
|
||||||
return True
|
# Files that already exist on AD2 and will be backed up + overwritten.
|
||||||
except IOError:
|
UPDATE_FILES = [
|
||||||
return False
|
('parsers/spec-reader.js', 'parsers/spec-reader.js'),
|
||||||
|
('templates/datasheet-exact.js', 'templates/datasheet-exact.js'),
|
||||||
|
('database/import.js', 'database/import.js'),
|
||||||
def to_remote(rel: str) -> str:
|
('database/export-datasheets.js', 'database/export-datasheets.js'),
|
||||||
return f'{REMOTE_ROOT}/{rel}'
|
]
|
||||||
|
|
||||||
|
# Files that do NOT yet exist on AD2 and must be created fresh.
|
||||||
def backup_and_copy(sftp: paramiko.SFTPClient, ssh: paramiko.SSHClient,
|
NEW_FILES = [
|
||||||
local_rel: str, remote_rel: str, dry_run: bool, stamp: str) -> None:
|
('parsers/vaslog-engtxt.js', 'parsers/vaslog-engtxt.js'),
|
||||||
local_path = os.path.join(LOCAL_ROOT, local_rel.replace('/', os.sep))
|
('server/notify.js', 'server/notify.js'),
|
||||||
remote_path = to_remote(remote_rel)
|
]
|
||||||
backup_path = f'{remote_path}.bak-{stamp}'
|
|
||||||
|
|
||||||
if not os.path.isfile(local_path):
|
def connect() -> paramiko.SSHClient:
|
||||||
raise FileNotFoundError(f'[FAIL] local file missing: {local_path}')
|
pwd = get_ad2_password()
|
||||||
|
c = paramiko.SSHClient()
|
||||||
if not remote_exists(sftp, remote_path):
|
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||||
raise FileNotFoundError(f'[FAIL] remote file missing on AD2: {remote_path}')
|
c.connect(
|
||||||
|
HOST, username=USER, password=pwd,
|
||||||
print(f'[INFO] {remote_rel}')
|
timeout=15, look_for_keys=False, allow_agent=False, banner_timeout=30,
|
||||||
if dry_run:
|
)
|
||||||
print(f' would back up to: {backup_path}')
|
return c
|
||||||
print(f' would upload: {local_path} -> {remote_path}')
|
|
||||||
return
|
|
||||||
|
def remote_exists(sftp: paramiko.SFTPClient, path: str) -> bool:
|
||||||
# Backup via SFTP copy (read + re-upload). Paramiko has no server-side copy.
|
try:
|
||||||
with sftp.open(remote_path, 'rb') as src:
|
sftp.stat(path)
|
||||||
data = src.read()
|
return True
|
||||||
with sftp.open(backup_path, 'wb') as dst:
|
except IOError:
|
||||||
dst.write(data)
|
return False
|
||||||
print(f' backup: {backup_path} ({len(data)} bytes)')
|
|
||||||
|
|
||||||
sftp.put(local_path, remote_path)
|
def to_remote(rel: str) -> str:
|
||||||
size = os.path.getsize(local_path)
|
return f'{REMOTE_ROOT}/{rel}'
|
||||||
print(f' uploaded: {local_path} -> {remote_path} ({size} bytes)')
|
|
||||||
|
|
||||||
|
def backup_and_copy(sftp: paramiko.SFTPClient, ssh: paramiko.SSHClient,
|
||||||
def create_new(sftp: paramiko.SFTPClient, local_rel: str, remote_rel: str,
|
local_rel: str, remote_rel: str, dry_run: bool, stamp: str) -> None:
|
||||||
dry_run: bool) -> None:
|
local_path = os.path.join(LOCAL_ROOT, local_rel.replace('/', os.sep))
|
||||||
"""Create a file that is expected to be NEW on AD2.
|
remote_path = to_remote(remote_rel)
|
||||||
|
backup_path = f'{remote_path}.bak-{stamp}'
|
||||||
Fails loud if the remote file already exists -- NEW_FILES declares this
|
|
||||||
is a brand-new file, so pre-existence is a drift signal. If a previous
|
if not os.path.isfile(local_path):
|
||||||
deploy partially ran, clean up manually or move the entry to
|
raise FileNotFoundError(f'[FAIL] local file missing: {local_path}')
|
||||||
UPDATE_FILES.
|
|
||||||
"""
|
if not remote_exists(sftp, remote_path):
|
||||||
local_path = os.path.join(LOCAL_ROOT, local_rel.replace('/', os.sep))
|
raise FileNotFoundError(f'[FAIL] remote file missing on AD2: {remote_path}')
|
||||||
remote_path = to_remote(remote_rel)
|
|
||||||
|
print(f'[INFO] {remote_rel}')
|
||||||
if not os.path.isfile(local_path):
|
if dry_run:
|
||||||
raise FileNotFoundError(f'[FAIL] local file missing: {local_path}')
|
print(f' would back up to: {backup_path}')
|
||||||
|
print(f' would upload: {local_path} -> {remote_path}')
|
||||||
print(f'[INFO] {remote_rel} (NEW)')
|
return
|
||||||
|
|
||||||
if remote_exists(sftp, remote_path):
|
# Backup via SFTP copy (read + re-upload). Paramiko has no server-side copy.
|
||||||
raise FileExistsError(
|
with sftp.open(remote_path, 'rb') as src:
|
||||||
f'[FAIL] remote target already exists but is declared NEW: {remote_path} '
|
data = src.read()
|
||||||
f'-- move to UPDATE_FILES or remove remote manually'
|
with sftp.open(backup_path, 'wb') as dst:
|
||||||
)
|
dst.write(data)
|
||||||
|
print(f' backup: {backup_path} ({len(data)} bytes)')
|
||||||
if dry_run:
|
|
||||||
print(f' would create: {local_path} -> {remote_path}')
|
sftp.put(local_path, remote_path)
|
||||||
return
|
size = os.path.getsize(local_path)
|
||||||
|
print(f' uploaded: {local_path} -> {remote_path} ({size} bytes)')
|
||||||
sftp.put(local_path, remote_path)
|
|
||||||
size = os.path.getsize(local_path)
|
|
||||||
print(f' created: {remote_path} ({size} bytes)')
|
def create_new(sftp: paramiko.SFTPClient, local_rel: str, remote_rel: str,
|
||||||
|
dry_run: bool) -> None:
|
||||||
|
"""Create a file that is expected to be NEW on AD2.
|
||||||
def main() -> int:
|
|
||||||
ap = argparse.ArgumentParser(description=__doc__)
|
Fails loud if the remote file already exists -- NEW_FILES declares this
|
||||||
ap.add_argument('--dry-run', action='store_true', help='print actions without writing')
|
is a brand-new file, so pre-existence is a drift signal. If a previous
|
||||||
args = ap.parse_args()
|
deploy partially ran, clean up manually or move the entry to
|
||||||
|
UPDATE_FILES.
|
||||||
stamp = datetime.date.today().strftime('%Y%m%d')
|
"""
|
||||||
|
local_path = os.path.join(LOCAL_ROOT, local_rel.replace('/', os.sep))
|
||||||
print('=' * 72)
|
remote_path = to_remote(remote_rel)
|
||||||
print('Deploy staged pipeline changes to AD2')
|
|
||||||
print('=' * 72)
|
if not os.path.isfile(local_path):
|
||||||
print(f'Host: {HOST}')
|
raise FileNotFoundError(f'[FAIL] local file missing: {local_path}')
|
||||||
print(f'Remote root: {REMOTE_ROOT}')
|
|
||||||
print(f'Local root: {LOCAL_ROOT}')
|
print(f'[INFO] {remote_rel} (NEW)')
|
||||||
print(f'Dry run: {args.dry_run}')
|
|
||||||
print(f'Backup tag: .bak-{stamp}')
|
if remote_exists(sftp, remote_path):
|
||||||
print('')
|
raise FileExistsError(
|
||||||
|
f'[FAIL] remote target already exists but is declared NEW: {remote_path} '
|
||||||
ssh = connect()
|
f'-- move to UPDATE_FILES or remove remote manually'
|
||||||
try:
|
)
|
||||||
sftp = ssh.open_sftp()
|
|
||||||
try:
|
if dry_run:
|
||||||
for local_rel, remote_rel in UPDATE_FILES:
|
print(f' would create: {local_path} -> {remote_path}')
|
||||||
backup_and_copy(sftp, ssh, local_rel, remote_rel, args.dry_run, stamp)
|
return
|
||||||
|
|
||||||
for local_rel, remote_rel in NEW_FILES:
|
sftp.put(local_path, remote_path)
|
||||||
create_new(sftp, local_rel, remote_rel, args.dry_run)
|
size = os.path.getsize(local_path)
|
||||||
finally:
|
print(f' created: {remote_path} ({size} bytes)')
|
||||||
sftp.close()
|
|
||||||
finally:
|
|
||||||
ssh.close()
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__)
|
||||||
print('')
|
ap.add_argument('--dry-run', action='store_true', help='print actions without writing')
|
||||||
print('[OK] done' if not args.dry_run else '[OK] dry-run complete (no changes made)')
|
args = ap.parse_args()
|
||||||
return 0
|
|
||||||
|
stamp = datetime.date.today().strftime('%Y%m%d')
|
||||||
|
|
||||||
if __name__ == '__main__':
|
print('=' * 72)
|
||||||
try:
|
print('Deploy staged pipeline changes to AD2')
|
||||||
sys.exit(main())
|
print('=' * 72)
|
||||||
except Exception as e:
|
print(f'Host: {HOST}')
|
||||||
print(f'[FAIL] {e}', file=sys.stderr)
|
print(f'Remote root: {REMOTE_ROOT}')
|
||||||
sys.exit(1)
|
print(f'Local root: {LOCAL_ROOT}')
|
||||||
|
print(f'Dry run: {args.dry_run}')
|
||||||
|
print(f'Backup tag: .bak-{stamp}')
|
||||||
|
print('')
|
||||||
|
|
||||||
|
smtp_pass = get_smtp_password()
|
||||||
|
|
||||||
|
ssh = connect()
|
||||||
|
try:
|
||||||
|
sftp = ssh.open_sftp()
|
||||||
|
try:
|
||||||
|
for local_rel, remote_rel in UPDATE_FILES:
|
||||||
|
backup_and_copy(sftp, ssh, local_rel, remote_rel, args.dry_run, stamp)
|
||||||
|
|
||||||
|
for local_rel, remote_rel in NEW_FILES:
|
||||||
|
create_new(sftp, local_rel, remote_rel, args.dry_run)
|
||||||
|
|
||||||
|
# Write notify config (creds fetched from vault, never committed to git)
|
||||||
|
import json
|
||||||
|
notify_cfg = json.dumps({
|
||||||
|
'smtp': {
|
||||||
|
'host': SMTP_HOST,
|
||||||
|
'port': SMTP_PORT,
|
||||||
|
'user': SMTP_USER,
|
||||||
|
'pass': smtp_pass,
|
||||||
|
},
|
||||||
|
'from': SMTP_USER,
|
||||||
|
'to': NOTIFY_TO,
|
||||||
|
}, indent=2)
|
||||||
|
notify_remote = f'{REMOTE_ROOT}/config/notify.json'
|
||||||
|
print(f'[INFO] config/notify.json (SMTP creds)')
|
||||||
|
if not args.dry_run:
|
||||||
|
# Ensure config dir exists
|
||||||
|
stdin, stdout, stderr = ssh.exec_command(
|
||||||
|
f'powershell -Command "New-Item -ItemType Directory -Force -Path '
|
||||||
|
f'C:\\Shares\\testdatadb\\config | Out-Null"'
|
||||||
|
)
|
||||||
|
stdout.channel.recv_exit_status()
|
||||||
|
with sftp.open(notify_remote, 'w') as f:
|
||||||
|
f.write(notify_cfg)
|
||||||
|
print(f' written: {notify_remote}')
|
||||||
|
else:
|
||||||
|
print(f' would write: {notify_remote}')
|
||||||
|
|
||||||
|
finally:
|
||||||
|
sftp.close()
|
||||||
|
|
||||||
|
# Install nodemailer if not already present
|
||||||
|
print('[INFO] npm install nodemailer')
|
||||||
|
if not args.dry_run:
|
||||||
|
cmd = 'cd C:\\Shares\\testdatadb && npm list nodemailer --depth=0 2>nul || npm install nodemailer'
|
||||||
|
stdin, stdout, stderr = ssh.exec_command(f'cmd /c "{cmd}"')
|
||||||
|
exit_code = stdout.channel.recv_exit_status()
|
||||||
|
out = stdout.read().decode(errors='replace').strip()
|
||||||
|
if out:
|
||||||
|
print(f' {out}')
|
||||||
|
if exit_code != 0:
|
||||||
|
err = stderr.read().decode(errors='replace').strip()
|
||||||
|
raise RuntimeError(f'[FAIL] npm install nodemailer failed: {err}')
|
||||||
|
print('[OK] nodemailer ready')
|
||||||
|
else:
|
||||||
|
print(' would run: npm install nodemailer (if not already installed)')
|
||||||
|
|
||||||
|
finally:
|
||||||
|
ssh.close()
|
||||||
|
|
||||||
|
print('')
|
||||||
|
print('[OK] done' if not args.dry_run else '[OK] dry-run complete (no changes made)')
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
try:
|
||||||
|
sys.exit(main())
|
||||||
|
except Exception as e:
|
||||||
|
print(f'[FAIL] {e}', file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
/**
|
||||||
|
* Failure notification via email.
|
||||||
|
*
|
||||||
|
* Reads SMTP config from config/notify.json (gitignored, written by deploy-to-ad2.py).
|
||||||
|
* Silently swallows send errors so a notification failure never masks the real error.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
|
||||||
|
const CONFIG_PATH = path.join(__dirname, '..', 'config', 'notify.json');
|
||||||
|
|
||||||
|
function loadConfig() {
|
||||||
|
try {
|
||||||
|
return JSON.parse(fs.readFileSync(CONFIG_PATH, 'utf8'));
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[NOTIFY] Could not read ${CONFIG_PATH}: ${err.message}`);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Send a failure notification email.
|
||||||
|
* @param {string} subject
|
||||||
|
* @param {string} body - plain text
|
||||||
|
*/
|
||||||
|
async function sendFailureEmail(subject, body) {
|
||||||
|
const cfg = loadConfig();
|
||||||
|
if (!cfg) return;
|
||||||
|
|
||||||
|
let nodemailer;
|
||||||
|
try {
|
||||||
|
nodemailer = require('nodemailer');
|
||||||
|
} catch (err) {
|
||||||
|
console.error('[NOTIFY] nodemailer not installed — skipping email');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const transporter = nodemailer.createTransport({
|
||||||
|
host: cfg.smtp.host,
|
||||||
|
port: cfg.smtp.port,
|
||||||
|
secure: false,
|
||||||
|
requireTLS: true,
|
||||||
|
auth: {
|
||||||
|
user: cfg.smtp.user,
|
||||||
|
pass: cfg.smtp.pass,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
try {
|
||||||
|
await transporter.sendMail({
|
||||||
|
from: cfg.from,
|
||||||
|
to: cfg.to,
|
||||||
|
subject,
|
||||||
|
text: body,
|
||||||
|
});
|
||||||
|
console.log(`[NOTIFY] Failure email sent: ${subject}`);
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[NOTIFY] Failed to send email: ${err.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { sendFailureEmail };
|
||||||
Reference in New Issue
Block a user