Session log: D2TESTNAS VM build, NAS migration, rsync sync fix
Built Debian 13 VM replacement for aging ReadyNAS, deployed rsync-based sync script to AD2, transferred data, completed IP cutover to 192.168.0.9. Includes setup scripts, sync fixes, and comprehensive session logs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# TestDataDB Investigation - Missing Recent Records
|
||||
**Date:** 2026-03-11
|
||||
**Status:** BLOCKED - VPN down, need parser source code from AD2
|
||||
|
||||
## Problem
|
||||
The test database on AD2 has newest `test_date` of 2026-01-19 despite daily tests being run every weekday. After a full re-import (1,028,275 -> 1,632,793 records), the max date did not change.
|
||||
|
||||
## Key Evidence
|
||||
1. DAT files with TODAY's timestamps (2026-03-11 13:29-13:30) exist on AD2 at `C:\Shares\test\TS-1L\LOGS\5BLOG\`
|
||||
2. These files were processed by the full import (604,518 new records added)
|
||||
3. But `MAX(test_date)` is still 2026-01-19
|
||||
|
||||
## Sample DAT File Content (33-06D.DAT from 5BLOG)
|
||||
```
|
||||
"SCM5B33-06D "
|
||||
9.528068E-02,.9528068,.9696456,.1683873,"PASS"
|
||||
.3142081,3.142081,3.155015,.1293349,"PASS"
|
||||
.5374944,5.374944,5.380653,5.708694E-02,"PASS"
|
||||
.7651215,7.651215,7.651233,1.811981E-04,"PASS"
|
||||
.997809,9.97809,9.967015,-.1107502,"PASS"
|
||||
"PASS 92.941","","PASS .16838733","PASS 2.045663E-023","PASS-2.213427E-023"
|
||||
"","PASS .29938753","","PASS .79032473","PASS .05982453"
|
||||
"","PASS 117.35150","","PASS-.1008325","PASS 91.444891"
|
||||
"PASS 1.024068E-023",""
|
||||
```
|
||||
|
||||
**No date anywhere in the content or filename.**
|
||||
|
||||
## Root Cause Hypothesis
|
||||
The `multiline.js` parser at `C:\Shares\testdatadb\parsers\multiline.js` determines `test_date` from some source. Possibilities:
|
||||
1. **File modification time (mtime)** - most likely since there's no date in content
|
||||
2. **A date field elsewhere in larger files** - maybe we only saw a partial file
|
||||
3. **A hardcoded date or fallback** - parser might have a bug
|
||||
|
||||
If parser uses mtime, the question is whether mtime is preserved when:
|
||||
- DOS machines XCOPY files to NAS
|
||||
- Sync-FromNAS.ps1 SCPs files from NAS to AD2
|
||||
- SCP with -O flag may or may not preserve timestamps
|
||||
|
||||
## HISTLOGS vs Station Files
|
||||
HISTLOGS files at `C:\Shares\test\Ate\HISTLOGS\` are the authoritative consolidated source. These may have a DIFFERENT format than the per-station DAT files. The initial import (1,030,940 records) came mostly from HISTLOGS (576K) and Recovery-TEST (454K), with only 59 from live station data.
|
||||
|
||||
The 604K new records from the re-import might all be from HISTLOGS/Recovery with dates up to Jan 19, while the per-station files might be producing 0 records or records with the same old dates.
|
||||
|
||||
## Next Steps (when VPN reconnects)
|
||||
1. **READ THE PARSER:** `ssh sysadmin@AD2 "type C:\Shares\testdatadb\parsers\multiline.js"`
|
||||
2. **Check a specific record:** Query DB for records from `33-06D.DAT` source file to see what test_date was assigned
|
||||
3. **Check import logs:** `ssh sysadmin@AD2 "type C:\Shares\test\scripts\sync-from-nas.log"` for any import errors
|
||||
4. **Verify HISTLOGS content:** Check if HISTLOGS files have different format than station files
|
||||
|
||||
## Deployed Fixes This Session
|
||||
- Sync-FromNAS.ps1: Get-NASFileList fix (stdout deadlock), 8.3 filename filtering, SCP path escaping
|
||||
- import.js: Changed INSERT OR IGNORE to INSERT OR REPLACE
|
||||
- Both deployed to AD2 at C:\Shares\test\scripts\ and C:\Shares\testdatadb\database\
|
||||
- Commit dd4086d (local only, not pushed - Gitea unreachable)
|
||||
|
||||
## Session Context
|
||||
- Worked on from Windows machine (ACG-M-L5090)
|
||||
- VPN went down during investigation
|
||||
- Previous session summary in conversation compaction
|
||||
- User said "continue to work this problem - we need to find those records"
|
||||
Reference in New Issue
Block a user