Files

Mike Swanson 000ee3da5c Session log: D2TESTNAS VM build, NAS migration, rsync sync fix

Built Debian 13 VM replacement for aging ReadyNAS, deployed rsync-based
sync script to AD2, transferred data, completed IP cutover to 192.168.0.9.
Includes setup scripts, sync fixes, and comprehensive session logs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-13 06:08:32 -07:00

3.2 KiB

Raw Blame History

TestDataDB Investigation - Missing Recent Records

Date: 2026-03-11 Status: BLOCKED - VPN down, need parser source code from AD2

Problem

The test database on AD2 has newest test_date of 2026-01-19 despite daily tests being run every weekday. After a full re-import (1,028,275 -> 1,632,793 records), the max date did not change.

Key Evidence

DAT files with TODAY's timestamps (2026-03-11 13:29-13:30) exist on AD2 at C:\Shares\test\TS-1L\LOGS\5BLOG\
These files were processed by the full import (604,518 new records added)
But MAX(test_date) is still 2026-01-19

Sample DAT File Content (33-06D.DAT from 5BLOG)

"SCM5B33-06D  "
9.528068E-02,.9528068,.9696456,.1683873,"PASS"
.3142081,3.142081,3.155015,.1293349,"PASS"
.5374944,5.374944,5.380653,5.708694E-02,"PASS"
.7651215,7.651215,7.651233,1.811981E-04,"PASS"
.997809,9.97809,9.967015,-.1107502,"PASS"
"PASS 92.941","","PASS .16838733","PASS 2.045663E-023","PASS-2.213427E-023"
"","PASS .29938753","","PASS .79032473","PASS .05982453"
"","PASS 117.35150","","PASS-.1008325","PASS 91.444891"
"PASS 1.024068E-023",""

No date anywhere in the content or filename.

Root Cause Hypothesis

The multiline.js parser at C:\Shares\testdatadb\parsers\multiline.js determines test_date from some source. Possibilities:

File modification time (mtime) - most likely since there's no date in content
A date field elsewhere in larger files - maybe we only saw a partial file
A hardcoded date or fallback - parser might have a bug

If parser uses mtime, the question is whether mtime is preserved when:

DOS machines XCOPY files to NAS
Sync-FromNAS.ps1 SCPs files from NAS to AD2
SCP with -O flag may or may not preserve timestamps

HISTLOGS vs Station Files

HISTLOGS files at C:\Shares\test\Ate\HISTLOGS\ are the authoritative consolidated source. These may have a DIFFERENT format than the per-station DAT files. The initial import (1,030,940 records) came mostly from HISTLOGS (576K) and Recovery-TEST (454K), with only 59 from live station data.

The 604K new records from the re-import might all be from HISTLOGS/Recovery with dates up to Jan 19, while the per-station files might be producing 0 records or records with the same old dates.

Next Steps (when VPN reconnects)

READ THE PARSER: ssh sysadmin@AD2 "type C:\Shares\testdatadb\parsers\multiline.js"
Check a specific record: Query DB for records from 33-06D.DAT source file to see what test_date was assigned
Check import logs: ssh sysadmin@AD2 "type C:\Shares\test\scripts\sync-from-nas.log" for any import errors
Verify HISTLOGS content: Check if HISTLOGS files have different format than station files

Deployed Fixes This Session

Sync-FromNAS.ps1: Get-NASFileList fix (stdout deadlock), 8.3 filename filtering, SCP path escaping
import.js: Changed INSERT OR IGNORE to INSERT OR REPLACE
Both deployed to AD2 at C:\Shares\test\scripts\ and C:\Shares\testdatadb\database\
Commit dd4086d (local only, not pushed - Gitea unreachable)

Session Context

Worked on from Windows machine (ACG-M-L5090)
VPN went down during investigation
Previous session summary in conversation compaction
User said "continue to work this problem - we need to find those records"

3.2 KiB Raw Blame History