azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Session log 2026-04-03: WO import, 7B support, PG migration started

Browse Source 
- 33K work orders imported, 2.27M records linked
- 7B exact-match formatter added (31 params, 120VAC, Packing Check List)
- TXT formatting refined to match QB TAB positions exactly
- PostgreSQL 18 installed on AD2, database created
- SQL Server Express uninstalled
- Full Dataforth audit document generated

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
sysadmin
2026-04-03 09:47:30 -07:00
parent 7431bfd52b
commit 88dc431cfa
2 changed files with 257 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

80
Test Datasheets/weekend-update-draft.md Normal file
View File

@@ -0,0 +1,80 @@
Subject: Test Datasheets - Weekend Update: All 73 Quatronix Sheets Generated, Work Order Search Live
John, Ken,
Quick update on progress since Friday's email. The pipeline is significantly further along.
## Quatronix Customer Issue - RESOLVED
All 73 requested datasheets have been generated (TXT + PDF). The last holdout was SCM5B49-05 (SN 177000-15) — the 5B49DATA.DAT spec file was empty, but John pointed us to 5B49_2.DAT which had the data. All 73 files are ready to send to Peter/Ginger.
## Model Spec Coverage Expanded
We went from 751 model specs to 1,470+ by loading additional spec databases:
| Spec File | Family | Models |
|-----------|--------|--------|
| 5BMAIN.DAT | SCM5B | 481 |
| 5B45DATA.DAT | SCM5B (freq/counter) | 56 |
| DB5B48.DAT | SCM5B (multi-bandwidth) | 3 |
| 5B49_2.DAT | SCM5B (sample & hold) | 15 |
| 8BMAIN.DAT | 8B | 148 |
| DSCOUT.DAT | DSCA (output) | 23 |
| DSCMAIN4.DAT | DSCA (input) | 391 |
| SCTMAIN.DAT | DSCT | 103 |
| 7BMAIN.DAT | SCM7B | 276 |
If there are additional spec files we're missing, let me know the paths and we'll add them.
## SCM7B Support Added
The 7B product family is now fully supported in the datasheet generator:
- 31 test parameters (vs 20 for SCM5B)
- Correct header ("SCM" prefix prepended to model name)
- 120VAC Withstand / Hi-Pot (skipped for 7BPT models)
- "Packing Check List" with blank fields (vs pre-marked checkboxes on 5B/8B)
- "Tested by" and "QC" signature lines
- Note: The 7B DAT format (single CSV line) doesn't include individual accuracy test points, so the accuracy table is omitted. Only the Final Test Results section is generated from DAT data.
## Work Order Search & Linking
Imported all 33,745 work order status reports from the test station Reports folders:
- 63,263 individual test lines parsed (serial number, model, pass/fail, date/time, station)
- 2.27 million test records linked to their work orders
In the web app (http://192.168.0.6:3000):
- New "Work Order #" search field — enter a WO number to find all associated test records
- Click the WO number in any record's detail view to see the full work order:
- All serial numbers tested under that WO
- Pass/fail status for each (including retests)
- Test program and version used
- Test station and timestamps
- New work order reports are automatically imported when synced from the NAS
## Datasheet Formatting Refined
Compared generated datasheets against originals from the DFWDS archive and fixed column alignment to match the QuickBASIC output:
- TAB positions match exactly (parameter names, measured values, spec limits, pass/fail)
- Number formatting matches QB PRINT USING (right-justified, correct decimal places)
- STR$() behavior replicated (leading space for positive numbers, dropped leading zeros)
- Spec limit formatting matches (e.g., "+/- .03 %" not "+/- 0.03 %")
## View Button Updated
The "SHEET" button in the web app now shows a styled HTML page that matches the PDF/TXT layout — white page, monospace font, same column alignment. Includes Print and Download PDF buttons.
## Infrastructure
- Created domain service account (INTRANET\svc_testdatadb) for the TestDataDB Windows service — resolves the file permission issues we were hitting
- Added STAGE folder sync to the NAS sync script — TXT datasheets from DOS machines will now be pulled to AD2 automatically
- Work order report import added to sync script — new reports are ingested automatically every 15 minutes
## Still Open
1. **Website upload** — The old Uploader.aspx endpoints are dead. Need to determine the new upload mechanism for dataforth.com.
2. **STAGE backlog** — ~8,100 TXT files on the NAS from DOS machines need to be processed (script ready, haven't run it yet).
3. **Pending ForWeb export** — ~845K records in the database don't have TXT files in For_Web yet (mostly 7B and older records). Can batch-export as needed.
Let me know if you need anything else.
Mike

177
session-logs/2026-04-03-session-ad2.md Normal file
View File

@@ -0,0 +1,177 @@
# Session Log - AD2 - 2026-04-03
## Session Summary
Continued test datasheet pipeline work from the 03-27/28/29 session. Major focus on:
1. Fixing formatting issues in generated TXT datasheets (column alignment matching QuickBASIC)
2. Adding 7B product family support to the exact-match formatter
3. Loading additional spec files (5B49_2.DAT for the last missing Quatronix datasheet)
4. Work order report ingestion (33K WOs, 63K lines, 2.27M records linked)
5. UI improvements (WO search, WO detail popup, View button styled HTML)
6. Generating full Dataforth audit document
7. Planning and beginning PostgreSQL migration (SQLite hitting 4.4GB, queries timing out)
8. Uninstalling SQL Server Express, installing PostgreSQL 18
## Key Decisions
- PostgreSQL over SQL Server Express (no 10GB limit, better FTS, no licensing)
- Install PG locally on AD2 (not SAGE-SQL or new VM)
- Clean cutover (no parallel run with SQLite)
- MSSQL Express uninstalled to free resources
## Credentials
### PostgreSQL (newly installed)
- Host: localhost (AD2, 192.168.0.6)
- Port: 5432
- Superuser: postgres / DfPgSQL2026!
- App role: testdatadb_app / DfTestDB2026!
- Database: testdatadb
- Service: postgresql-18 (runs as INTRANET\sysadmin)
- Install path: C:\Program Files\PostgreSQL\18\
- Data dir: C:\Program Files\PostgreSQL\18\data\
### TestDataDB (existing)
- Service: testdatadb (runs as INTRANET\svc_testdatadb / DfSvcTDB2026!)
- URL: http://192.168.0.6:3000
- SQLite DB: C:\Shares\testdatadb\database\testdata.db (4.4 GB)
### Existing (from CLAUDE.md)
- AD Sysadmin: INTRANET\sysadmin / Paper123!@#
- D2TESTNAS SSH: root@192.168.0.9 / Paper123!@#-nas
- Rsync: port 873, user rsync / IQ203s32119
- M365 Tenant: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
- Gitea: mike@azcomputerguru.com / Gptf*77ttb123!@#-git
- Gitea API Token: 9b1da4b79a38ef782268341d25a4b6880572063f
## Work Completed
### Formatting Fixes (datasheet-exact.js)
- Compared generated TXT against originals from X:\For_Web\2025\
- Fixed Final Test Results column alignment to match QB TAB positions:
- TAB(5) param name, TAB(31) measured value, TAB(60-speclen) spec, TAB(61) unit, TAB(71) PASS/FAIL
- Added setCol() and padToCol() helpers for exact column positioning
- Fixed STR$() emulation: leading space for positive, drops leading zero (.03 not 0.03)
- Fixed header spacing (Date, Model, SN fields)
- Fixed checklist spacing to match QB TAB(45)
- Fixed separator line length (23 chars not 24)
### 7B Product Family Support
- Added SCM7B DATA_LINES (31 parameters vs 20 for SCM5B)
- Added SCM7B TSPECS (supply current, linearity, accuracy, excitation, CJC, noise, etc.)
- Added 7B raw_data parser (single CSV line format vs multi-line for other families)
- 7B-specific footer: 120VAC (not 240), "Packing Check List" (not "Check List"), blank checkmarks, Tested by/QC lines
- Accuracy section suppressed for 7B (DAT format doesn't include individual test points)
- 7B model names prepend "SCM" in header
### Additional Spec File: 5B49_2.DAT
- 5B49DATA.DAT was 0 bytes (empty), John pointed to 5B49_2.DAT
- 15 models, 93 bytes/record, added SCM5B49_FIELDS to spec-reader.js
- Completed final missing Quatronix datasheet (177000-15, SCM5B49-05)
- All 73/73 Quatronix datasheets now generated
### View Button Enhancement
- /api/datasheet/:id now uses exact-match formatter with styled HTML
- White page on gray background, monospace font, print-optimized
- Includes Print and Download PDF buttons
- Falls back to generic template if exact-match not available
### PDF Endpoint Fallback
- /api/datasheet/:id/pdf falls back to generic template when exact-match fails
- Prevents 422 errors for unsupported families
### Work Order Reports
- Created parsers/wo-report.js (parses WO status report TXT format)
- Created database/import-work-orders.js (imports WOs, creates tables, links to test records)
- Created work_orders table (33,745 records) and work_order_lines table (63,263 records)
- Added work_order column to test_records, 2,277,183 records linked via serial number pattern
- Added /api/workorder/:wo and /api/workorder-search endpoints
- Added WO# search field and clickable WO link in UI detail view
- WO detail popup shows all test lines, pass/fail history, program version
- Added auto-import hooks to Sync-FromNAS-rsync.ps1 for Reports and STAGE folders
### Sync Script Updates (Sync-FromNAS-rsync.ps1)
- Added STAGE folder sync (NAS STAGE -> AD2)
- Added $syncedReportFiles and $syncedStageFiles tracking
- Added WO report auto-import after sync
- Added STAGE TXT auto-import after sync
- Updated status file with WO and STAGE counts
### Dataforth Audit Document
- Generated C:\Users\sysadmin\Desktop\Dataforth-Audit-2026-04-02.txt
- 22 sections: company info, network, AD computers/users/groups, GPO, DNS, shares, tasks, services, firewall, credentials, M365, security incidents, manufacturing infrastructure, pipeline, applications, backups, known issues, contacts
### PostgreSQL Migration (IN PROGRESS)
- Plan approved: PostgreSQL 18 on AD2, clean cutover
- PostgreSQL 18.3 installed via Chocolatey
- Database cluster initialized (C:\Program Files\PostgreSQL\18\data\)
- Service registered as postgresql-18 (runs as INTRANET\sysadmin)
- Database created: testdatadb owned by testdatadb_app
- SQL Server Express 2022 uninstalled
### SQLite Issues Documented
- Database grown to 4.4GB
- Single search query took 5,157 seconds (85 minutes)
- Stats query took 216 seconds
- WAL file grew to 454MB
- ANALYZE ran for 30+ minutes without completing
- SHM/WAL permission conflicts between SYSTEM and sysadmin
### Service Account (from previous session, still relevant)
- INTRANET\svc_testdatadb created in OU=ServiceAccounts
- testdatadb Windows service runs under this account
- SeServiceLogonRight granted
- Permissions on C:\Shares\testdatadb and C:\Shares\webshare
## Files Created
- C:\Shares\testdatadb\fix-sysadmin-logon.ps1
- C:\ClaudeTools\Test Datasheets\weekend-update-draft.md
- C:\Users\sysadmin\Desktop\Dataforth-Audit-2026-04-02.txt
- C:\Users\sysadmin\.claude\plans\generic-petting-lovelace.md (PG migration plan)
## Files Modified
- C:\Shares\testdatadb\templates\datasheet-exact.js — Column alignment, 7B support, STR$() emulation
- C:\Shares\testdatadb\parsers\spec-reader.js — Added 5B49 TYPE, fuzzy model matching
- C:\Shares\testdatadb\parsers\wo-report.js — Created (WO report parser)
- C:\Shares\testdatadb\database\import-work-orders.js — Created (WO importer)
- C:\Shares\testdatadb\database\schema.sql — Added work_order column
- C:\Shares\testdatadb\routes\api.js — Added WO endpoints, PDF fallback, view button exact-match
- C:\Shares\testdatadb\public\index.html — WO search, WO popup, view button styling
- C:\Shares\test\scripts\Sync-FromNAS-rsync.ps1 — STAGE sync, WO auto-import
## Database Stats (as of session end)
- Test records: 2,294,877
- Work orders: 33,745
- Work order lines: 63,263
- Records with WO linked: 2,277,183
- ForWeb exported: ~1,436,000
- Model specs loaded: 1,470+
## Pending / Next Steps
1. **PostgreSQL Migration (Step 2-8):**
- Create PG schema (schema-pg.sql) with tsvector FTS
- Create database abstraction layer (db.js)
- Migrate routes/api.js to async
- Migrate import/export scripts
- Build data migration script
- Migrate 2.28M records
- Test and cutover
2. **Tune PostgreSQL config:**
- shared_buffers=1GB, work_mem=64MB, maintenance_work_mem=256MB
3. **MSSQL cleanup:**
- May need reboot to fully remove SQL Server services
4. **Website upload:**
- Old ASP.NET endpoints still dead (404)
- Need to determine new upload mechanism
5. **Joel Lohr account:**
- Retired March 31 — account needs to be disabled
## Infrastructure
- AD2: 192.168.0.6 (Windows Server 2016)
- PostgreSQL 18: localhost:5432 (newly installed, service running)
- TestDataDB: localhost:3000 (SQLite, service running)
- NAS: 192.168.0.9 (rsync port 873)
- STAGE backlog: cleared (0 files remaining)