- UNREGISTERED_MODELS set: 9 model numbers not in Hoffman API catalog; skipped
silently instead of generating errors
- batch-500 fallback: when a bulk batch returns HTTP 500, retry each record
individually so good records get stamped and only truly-bad records count
as errors
- FAIL-parameter filter: records with any FAIL on a parameter line are excluded
from the push before the batch is assembled
- notify.js integration: wired in existing notification module
Files added:
- projects/dataforth-dos/database/upload-to-api.js
- projects/dataforth-dos/database/notify.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Establishes inter-session coordination for 29 projects/clients:
- Full lock/component format for active projects (dataforth-dos,
radio-show, cascades-tucson, valleywide, instrumental-music-center,
lens-auto-brokerage, msp-audit-scripts)
- Light format for complete/stalled/planning (msp-pricing, pavon,
wrightstown-*, gururmm-agent, community-forum, glaztech, etc.)
- Onboarding stubs for recently added clients
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- TEST-DATASHEET-PROCESS.md: comprehensive pipeline documentation for
Dataforth engineering (10 sections, data flow, state diagram, FAQ)
- signing-attestation/: domain ownership attestation letter with
in-place signature for Azure Trusted Signing identity validation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Installed C:\ProgramData\dataforth-uploader\ on AD2 with:
- credentials.json (SYSTEM+Administrators ACL only)
- run-pipeline.ps1 (DFWDS-process -> enumerate For_Web -> upload-delta)
- dfwds-process.js + upload-delta.js (copied from prior install dir)
- logs/ with 60-day retention
Scheduled Task 'DataforthTestDatasheetUploader' registered as SYSTEM,
hourly trigger, 30-min execution limit. First SYSTEM-context run verified:
received=7061 unchanged=7061 errors=0 in 8.7s.
Initial registration via inline base64 mangled the backslashes in the -File
argument (resulted in ERROR_DIRECTORY 0x8007010B). Fixed by running the
registration PowerShell from a file rather than an encoded command string.
Also deleted throwaway tmp/list_amtransit.py + tmp/reset_cansley.py which
had hardcoded ACG admin password.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Built the missing piece between the test datasheet pipeline and Dataforth's
new product API. End-to-end:
- Pulled DFWDS (Dataforth Web Datasheet System) VB6 source from
AD1\Engineering\ENGR\ATE\Test Datasheets\DFWDS to local for analysis
- Decoded its filename validation: A-J prefix decodes (A=10..J=19), all-
numeric WO# valid (no leading 0), anything else bad
- Ported the validation + move logic to Node (dfwds-process.js)
- Built bulk uploader (upload-delta.js) for Hoffman's Swagger API
(POST /api/v1/TestReportDataFiles/bulk with OAuth client_credentials)
Sanitized 3 prior reference scripts (fetch-server-inventory, test-scenarios,
test-upload-two) to read CF_* env vars instead of hardcoded creds.
Live drain results:
- 897 files moved Test_Datasheets -> For_Web (all valid, no renames, no
bad), DFWDS port summary in 1.1s
- Pushed entire For_Web (7,061 files) to Hoffman API in 49.7s @ 142/s:
Created=803 Updated=114 Unchanged=6,144 Errors=0
- Server count: 489,579 -> 490,382 (+803 net new)
Also:
- Added clients/dataforth/.gitignore to exclude plaintext Oauth.txt note
- Added clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md
(ticket write-up of 2026-04-11/12/13 IMC1 RDS removal/SQL migration work)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive record of 2026-04-11/12 work extending the Dataforth Test
Datasheet Pipeline: discovery, implementation, deploy to AD2, full
backfill of 27,937 datasheets, post-deploy regex patch for QB plain-
decimal PASS lines, and repo commit 0dd3d82.
Includes credentials, infrastructure paths, commit reference, open
items (vault hygiene, rsync coverage), and accuracy-extraction
reference logic for future sessions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extends the Test Datasheet Pipeline on AD2:C:\Shares\testdatadb to
generate web-published datasheets for the SCMVAS-Mxxx (obsolete) and
SCMHVAS-Mxxxx (replacement) High Voltage Input Module product lines.
Both are tested either with the existing TESTHV3 software (production
VASLOG .DAT logs) or in Engineering with plain .txt output.
Key changes on AD2 (all deployed 2026-04-12 with dated backups):
- parsers/spec-reader.js: getSpecs() returns a `{_family:'SCMVAS',
_noSpecs:true}` sentinel for SCMVAS/SCMHVAS/VAS-M/HVAS-M model prefixes
so the export pipeline does not silently skip them for missing specs.
- templates/datasheet-exact.js: new Accuracy-only template branch
(generateSCMVASDatasheet + helpers) that mirrors the existing shipped
format byte-for-byte. Extraction regex covers both QuickBASIC STR$()
output formats: scientific-with-trailing-status-digit (98.4% of
records) and plain-decimal (1.6% of records above QB's threshold).
- parsers/vaslog-engtxt.js (new): parses the Engineering-Tested .txt
files in TS-3R\LOGS\VASLOG\VASLOG - Engineering Tested\. Filename SN
regex strips optional trailing 14-digit timestamp; in-file "SN:"
header is the authoritative source when the filename is malformed.
- database/import.js: LOG_TYPES grows a VASLOG_ENG entry with
subfolder + recursive flags. Pre-existing 7 log types keep their
implicit recursive=true behaviour (config.recursive !== false).
importFiles() routes VASLOG_ENG paths before the generic loop so a
VASLOG - Engineering Tested/*.txt path does not mis-dispatch to the
multiline parser.
- database/export-datasheets.js: VASLOG_ENG records are written
verbatim via fs.copyFileSync(source_file, For_Web/<SN>.TXT) for true
byte-level pass-through, with a graceful raw_data fallback when the
source file is no longer on disk.
Deploy outcome:
- 27,503 SCMVAS/SCMHVAS datasheets rendered (27,065 from scientific +
438 from plain-decimal PASS lines, post-patch rerun)
- 434 Engineering-Tested .txt files pass-through-copied to For_Web
- 0 errors across both batches
Repo layout added here:
- scmvas-hvas-research/: discovery artifacts (source .BAS, hvin.dat,
sample .DAT + .txt, binary-format notes, IMPLEMENTATION_PLAN.md)
- implementation/: staged final code + deploy helpers + local test
harness + per-step verification scripts
- backups/pre-deploy-20260412/: independent local snapshot of the 4
AD2 files replaced, pulled byte-for-byte before deploy
All helper scripts fetch the AD2 password at runtime from the SOPS
vault (clients/dataforth/ad2.sops.yaml). None of the committed files
contain the plaintext credential. Known vault-entry hygiene issue
(stale shell-escape backslash before the `!`) is documented in the
fetcher comments and stripped at read-time; flagged separately for
cleanup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DOS 6.22 MD fails on existing dirs, then COPY creates a file instead of
copying into a directory. Dirs now pre-created on NAS server side.
Also removed trailing backslash from COPY target path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mixed-case paths caused "invalid directory" on TS-27. All paths now uppercase
to match DOS convention and Samba default case = upper setting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
No conditionals, no DEL. Just MD dirs and COPY. Deployed to AD2 and synced to D2TESTNAS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CTONWTXT.BAT was never being called. Now invoked from CTONW.BAT on every boot.
Destination changed from X: (broken Novell serve.sys check) to T:\Stage\%MACHINE%\.
Removed IF NOT clauses for DOS 6.22 compatibility. Deployed to AD2 and synced to D2TESTNAS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added EXE copy from Ate\ProdSW to C:\ATE in NWTOC.BAT
- Added /Y overwrite flag to all COPY commands
- Removed cyclic DATA folder copies from NWTOC
- Session log for 2026-03-16 DF DOS troubleshooting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Problem: Data syncs to AD2 but import.js not importing to database
- Newest DB record stuck at 2026-01-19 (2 months stale)
- rsync transfers working (9.8MB at 06:30 & 07:15)
- import.js not being called or failing silently
Instructions for PC to diagnose and fix.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
DEPLOY v4.1 fixes critical bug where ECHO >> redirects included
trailing space in MACHINE variable, causing "Too many parameters"
on all COPY commands with subdirectory paths. TS-4L data upload
confirmed working - 84 test files + 90 reports on NAS.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DOS 6.22 IF EXIST with wildcards on SMB1 network paths causes
"Bad command" and "Too many parameters" errors. Rewrote CTONW,
NWTOC, and AUTOEXEC to v4.0 with direct COPY/MD commands.
Pre-created all station LOGS directories on new D2TESTNAS.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Built Debian 13 VM replacement for aging ReadyNAS, deployed rsync-based
sync script to AD2, transferred data, completed IP cutover to 192.168.0.9.
Includes setup scripts, sync fixes, and comprehensive session logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SolverBot:
- Inject active project path into agent system prompts so agents
know which directory to scope file operations to
GuruRMM:
- Bump agent version to 0.6.0
- Add serde aliases for PowerShell/ClaudeTask command types
- Add typed CommandType enum on server for proper serialization
- Support claude_task command type in send_command API
Dataforth:
- Fix SCP space-escaping in Sync-FromNAS.ps1
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Video Analysis Agent (.claude/agents/video-analysis.md):
- Frame extraction with ffmpeg
- DOS console text recognition
- Boot sequence documentation
- Integration with Photo Agent and DOS Coding Agent
Debug batch files for video recording:
- ATESYNC-DEBUG.BAT: Orchestrator with PAUSE at each step
- CTONW-DEBUG.BAT: Upload with 10 step-by-step pauses
- NWTOC-DEBUG.BAT: Download with 11 step-by-step pauses
Each step clearly labeled with ECHO for video analysis.
Run ATESYNC-DEBUG TS-3R to capture boot process.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ATESYNC.BAT v1.0:
- Boot-time orchestrator (ARCHBAT equivalent from TS-27)
- Calls CTONW (upload) then NWTOC (download)
- Creates machine folder structure if missing
- Accepts machine name as parameter or MACHINE env var
CTONW.BAT v3.1:
- Fixed upload path: now uploads to T:\%MACHINE%\LOGS\*LOG
- Added safeguards to prevent data overwriting:
- Refuses to run if MACHINE not set
- Refuses to run if T:\%MACHINE% folder missing
- Logs machine name, date/time, target path
- Uploads all 8 LOG folders plus Reports
Based on analysis of TS-27 golden example machine backup.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Investigation and Documentation:
- Discovered and documented test database system on AD2 server
- Created comprehensive TEST_DATABASE_ARCHITECTURE.md with full system details
- Retrieved all key database files from AD2 (import.js, schema.sql, server configs)
- Documented data flow: DOS machines → NAS → AD2 → SQLite → Web interface
- Verified database health: 1,027,517 records, 1075 MB, dates back to 1990
Database System Architecture:
- SQLite database with Node.js/Express.js web server (port 3000)
- Automated import via Sync-FromNAS.ps1 (runs every 15 minutes)
- 8 log types supported: DSCLOG, 5BLOG, 7BLOG, 8BLOG, PWRLOG, SCTLOG, VASLOG, SHT
- FTS5 full-text search, comprehensive indexes for performance
- API endpoints: search, stats, export, datasheet generation
Troubleshooting Scripts Created:
- Database diagnostics: check-db-simple.ps1, test-db-directly.ps1
- Server status checks: check-node-running.ps1, check-db-server.ps1
- Performance analysis: check-db-performance.ps1, check-wal-files.ps1
- API testing: test-api-endpoint.ps1, test-query.js
- Import monitoring: check-new-records.ps1
- Database optimization attempts: api-js-optimized.js, api-js-fixed.js
- Deployment scripts: deploy-db-optimization.ps1, deploy-db-fix.ps1, restore-original.ps1
Key Findings:
- Database file healthy and queryable (verified with test-query.js)
- Node.js server not running (port 3000 closed) - root cause of web interface issues
- Database last updated 8 days ago (01/13/2026) - automated sync may be broken
- Attempted performance optimizations (WAL mode) incompatible with readonly connections
- Original api.js restored from backup after optimization conflicts
Retrieved Documentation:
- QUICKSTART-retrieved.md: Quick start guide for database server
- SESSION_NOTES-retrieved.md: Complete session notes from database creation
- Sync-FromNAS-retrieved.ps1: Full sync script with database import logic
- import-js-retrieved.js: Node.js import script (12,774 bytes)
- schema-retrieved.sql: SQLite schema with FTS5 triggers
- server-js-retrieved.js: Express.js server configuration
- api-js-retrieved.js: API routes and endpoints
- package-retrieved.json: Node.js dependencies
Action Items Identified:
1. Start Node.js server on AD2 to restore web interface functionality
2. Investigate why automated sync hasn't updated database in 8 days
3. Check Windows Task Scheduler for Sync-FromNAS.ps1 scheduled task
4. Run manual import to catch up on 8 days of test data if needed
Technical Details:
- Database path: C:\Shares\testdatadb\database\testdata.db
- Web interface: http://192.168.0.6:3000 (when running)
- Database size: 1075.14 MB (1,127,362,560 bytes)
- Total records: 1,027,517 (slight variance from original 1,030,940)
- Pass rate: 99.82% (1,029,046 passed, 1,888 failed)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
NWTOC.BAT v3.5:
- Switch from XCOPY to COPY (more reliable in DOS 6.22)
- Remove all >NUL redirects that cause issues
- Add IF NOT EXIST checks before MD to avoid errors
- Add 8 ATE data folder copies (5BDATA, 7BDATA, 8BDATA, DSCDATA,
HVDATA, PWRDATA, RMSDATA, SCTDATA)
- Remove machine-specific section (no longer needed)
- Remove MACHINE variable requirement
DEPLOY.BAT v2.4:
- Switch all XCOPY to COPY for DOS 6.22 compatibility
- Simplify output messages
Also fixed AD2->NAS sync issue:
- Ate/ProdSW folder was not being synced to NAS
- DOS machines were getting outdated DSCDATA files (Dec 2025 vs Jan 2026)
- Updated Sync-FromNAS.ps1 on AD2 to include Ate/ProdSW folder
- Manually synced correct files to NAS (DSCMAIN4.DAT 65508 bytes)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Session log updates for 2026-01-20 with additional work documentation.
This checkpoint also marks completion of GuruRMM security remediation:
- Phase 1: 10 critical security fixes deployed
- Phase 2: 8 major fixes deployed
- Production server updated at 172.16.3.30
- Gitea tracking issue #1 updated
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
DOS 6.22 does not support stderr redirection (2>NUL), only stdout (>NUL).
Added /I flag to XCOPY to assume destination is directory.
Added CD \ATE and menux to AUTOEXEC.BAT generation.
Changes:
- CTONW.BAT v2.5: Removed 2>NUL from MD commands, added /I to XCOPY
- NWTOC.BAT v2.8: Removed 2>NUL from MD commands, added /I to XCOPY
- DEPLOY.BAT v2.3: Removed 2>NUL, added CD \ATE and menux to AUTOEXEC
Tested successfully on TS-4R and TS-3R DOS machines.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major DOS 6.22 compatibility fixes for the Dataforth update system:
Changes Made:
- Replace COPY /Y with XCOPY /Y (COPY doesn't support /Y in DOS 6.22)
- Remove all trailing backslashes from XCOPY destinations (causes "Too many parameters")
- Remove %%~dpnF and %~nx1 syntax (Windows NT only, not DOS 6.22)
- Remove \NUL directory existence checks (unreliable in DOS 6.22)
- Simplify all batch files to minimal, reliable DOS 6.22 patterns
- Use MD >NUL 2>NUL for directory creation (ignore errors)
Files Updated:
- NWTOC.BAT v2.7: Simplified download with XCOPY /Y
- CTONW.BAT v2.4: Simplified upload with XCOPY /Y
- DEPLOY.BAT v2.2: Simplified deployment with XCOPY /Y
- CHECKUPD.BAT v1.3: Removed %~nx1 syntax
- UPDATE-ROOT.BAT: Root redirect script
- UPDATE-PRODSW.BAT v2.3: Backup utility (new file, was UPDATE.BAT in ProdSW)
Why:
- Previous versions caused infinite loops due to COPY /Y not existing in DOS 6.22
- Trailing backslashes on XCOPY destinations caused "Too many parameters" errors
- Complex variable syntax like %%~dpnF is NT-only and breaks on DOS 6.22
- Simplified scripts are more reliable and easier to debug
Testing:
- Deployed to AD2 (192.168.0.6) and D2TESTNAS (192.168.0.9)
- Ready for testing on TS-4R and TS-3R DOS machines
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
