Mike Swanson
000ee3da5c
Built Debian 13 VM replacement for aging ReadyNAS, deployed rsync-based sync script to AD2, transferred data, completed IP cutover to 192.168.0.9. Includes setup scripts, sync fixes, and comprehensive session logs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
62 lines
3.2 KiB
Markdown
62 lines
3.2 KiB
Markdown
# TestDataDB Investigation - Missing Recent Records
|
|
**Date:** 2026-03-11
|
|
**Status:** BLOCKED - VPN down, need parser source code from AD2
|
|
|
|
## Problem
|
|
The test database on AD2 has newest `test_date` of 2026-01-19 despite daily tests being run every weekday. After a full re-import (1,028,275 -> 1,632,793 records), the max date did not change.
|
|
|
|
## Key Evidence
|
|
1. DAT files with TODAY's timestamps (2026-03-11 13:29-13:30) exist on AD2 at `C:\Shares\test\TS-1L\LOGS\5BLOG\`
|
|
2. These files were processed by the full import (604,518 new records added)
|
|
3. But `MAX(test_date)` is still 2026-01-19
|
|
|
|
## Sample DAT File Content (33-06D.DAT from 5BLOG)
|
|
```
|
|
"SCM5B33-06D "
|
|
9.528068E-02,.9528068,.9696456,.1683873,"PASS"
|
|
.3142081,3.142081,3.155015,.1293349,"PASS"
|
|
.5374944,5.374944,5.380653,5.708694E-02,"PASS"
|
|
.7651215,7.651215,7.651233,1.811981E-04,"PASS"
|
|
.997809,9.97809,9.967015,-.1107502,"PASS"
|
|
"PASS 92.941","","PASS .16838733","PASS 2.045663E-023","PASS-2.213427E-023"
|
|
"","PASS .29938753","","PASS .79032473","PASS .05982453"
|
|
"","PASS 117.35150","","PASS-.1008325","PASS 91.444891"
|
|
"PASS 1.024068E-023",""
|
|
```
|
|
|
|
**No date anywhere in the content or filename.**
|
|
|
|
## Root Cause Hypothesis
|
|
The `multiline.js` parser at `C:\Shares\testdatadb\parsers\multiline.js` determines `test_date` from some source. Possibilities:
|
|
1. **File modification time (mtime)** - most likely since there's no date in content
|
|
2. **A date field elsewhere in larger files** - maybe we only saw a partial file
|
|
3. **A hardcoded date or fallback** - parser might have a bug
|
|
|
|
If parser uses mtime, the question is whether mtime is preserved when:
|
|
- DOS machines XCOPY files to NAS
|
|
- Sync-FromNAS.ps1 SCPs files from NAS to AD2
|
|
- SCP with -O flag may or may not preserve timestamps
|
|
|
|
## HISTLOGS vs Station Files
|
|
HISTLOGS files at `C:\Shares\test\Ate\HISTLOGS\` are the authoritative consolidated source. These may have a DIFFERENT format than the per-station DAT files. The initial import (1,030,940 records) came mostly from HISTLOGS (576K) and Recovery-TEST (454K), with only 59 from live station data.
|
|
|
|
The 604K new records from the re-import might all be from HISTLOGS/Recovery with dates up to Jan 19, while the per-station files might be producing 0 records or records with the same old dates.
|
|
|
|
## Next Steps (when VPN reconnects)
|
|
1. **READ THE PARSER:** `ssh sysadmin@AD2 "type C:\Shares\testdatadb\parsers\multiline.js"`
|
|
2. **Check a specific record:** Query DB for records from `33-06D.DAT` source file to see what test_date was assigned
|
|
3. **Check import logs:** `ssh sysadmin@AD2 "type C:\Shares\test\scripts\sync-from-nas.log"` for any import errors
|
|
4. **Verify HISTLOGS content:** Check if HISTLOGS files have different format than station files
|
|
|
|
## Deployed Fixes This Session
|
|
- Sync-FromNAS.ps1: Get-NASFileList fix (stdout deadlock), 8.3 filename filtering, SCP path escaping
|
|
- import.js: Changed INSERT OR IGNORE to INSERT OR REPLACE
|
|
- Both deployed to AD2 at C:\Shares\test\scripts\ and C:\Shares\testdatadb\database\
|
|
- Commit dd4086d (local only, not pushed - Gitea unreachable)
|
|
|
|
## Session Context
|
|
- Worked on from Windows machine (ACG-M-L5090)
|
|
- VPN went down during investigation
|
|
- Previous session summary in conversation compaction
|
|
- User said "continue to work this problem - we need to find those records"
|