Session log + DFWDS Node port + Hoffman API uploader pipeline
Built the missing piece between the test datasheet pipeline and Dataforth's new product API. End-to-end: - Pulled DFWDS (Dataforth Web Datasheet System) VB6 source from AD1\Engineering\ENGR\ATE\Test Datasheets\DFWDS to local for analysis - Decoded its filename validation: A-J prefix decodes (A=10..J=19), all- numeric WO# valid (no leading 0), anything else bad - Ported the validation + move logic to Node (dfwds-process.js) - Built bulk uploader (upload-delta.js) for Hoffman's Swagger API (POST /api/v1/TestReportDataFiles/bulk with OAuth client_credentials) Sanitized 3 prior reference scripts (fetch-server-inventory, test-scenarios, test-upload-two) to read CF_* env vars instead of hardcoded creds. Live drain results: - 897 files moved Test_Datasheets -> For_Web (all valid, no renames, no bad), DFWDS port summary in 1.1s - Pushed entire For_Web (7,061 files) to Hoffman API in 49.7s @ 142/s: Created=803 Updated=114 Unchanged=6,144 Errors=0 - Server count: 489,579 -> 490,382 (+803 net new) Also: - Added clients/dataforth/.gitignore to exclude plaintext Oauth.txt note - Added clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md (ticket write-up of 2026-04-11/12/13 IMC1 RDS removal/SQL migration work) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
74
projects/dataforth-dos/datasheet-pipeline/compute-delta.py
Normal file
74
projects/dataforth-dos/datasheet-pipeline/compute-delta.py
Normal file
@@ -0,0 +1,74 @@
|
||||
"""Compare local For_Web serials vs server-side serials; emit delta list."""
|
||||
import os
|
||||
|
||||
LOCAL = r"C:\Users\guru\AppData\Local\Temp\for_web_inventory.txt"
|
||||
SERVER = r"C:\Users\guru\AppData\Local\Temp\server_inventory.txt"
|
||||
DELTA_OUT = r"C:\Users\guru\AppData\Local\Temp\delta_to_upload.txt"
|
||||
|
||||
# Load server set (case-sensitive exact; mirror what we'd POST as SerialNumber)
|
||||
server = set()
|
||||
with open(SERVER) as f:
|
||||
for line in f:
|
||||
s = line.strip()
|
||||
if s:
|
||||
server.add(s)
|
||||
|
||||
# Also load a case-insensitive lookup just to see if anything only differs by case
|
||||
server_ci = {s.lower() for s in server}
|
||||
|
||||
# Walk local inventory
|
||||
local_total = 0
|
||||
local_sns = set()
|
||||
path_by_sn = {}
|
||||
with open(LOCAL) as f:
|
||||
for line in f:
|
||||
parts = line.rstrip("\n").split("|")
|
||||
if len(parts) < 4:
|
||||
continue
|
||||
full, sn, size, mtime = parts[0], parts[1], parts[2], parts[3]
|
||||
local_total += 1
|
||||
local_sns.add(sn)
|
||||
path_by_sn[sn] = (full, int(size), mtime)
|
||||
|
||||
# Diff
|
||||
missing_on_server = sorted(sn for sn in local_sns if sn not in server)
|
||||
missing_ci_only = sorted(sn for sn in local_sns if sn not in server and sn.lower() in server_ci)
|
||||
already_present = sorted(sn for sn in local_sns if sn in server)
|
||||
|
||||
print(f"Local total: {local_total}")
|
||||
print(f"Local unique serials: {len(local_sns)}")
|
||||
print(f"Server total: {len(server)}")
|
||||
print(f"")
|
||||
print(f"Already on server (case-sensitive exact match): {len(already_present)}")
|
||||
print(f"Missing on server (case-sensitive): {len(missing_on_server)}")
|
||||
print(f" ...of which only differ by case: {len(missing_ci_only)}")
|
||||
print(f"")
|
||||
|
||||
# Write delta list (paths) to upload
|
||||
with open(DELTA_OUT, "w") as f:
|
||||
for sn in missing_on_server:
|
||||
full, size, mtime = path_by_sn[sn]
|
||||
f.write(f"{sn}|{full}|{size}|{mtime}\n")
|
||||
|
||||
print(f"Wrote delta list ({len(missing_on_server)} entries) -> {DELTA_OUT}")
|
||||
|
||||
# Show sample of delta
|
||||
if missing_on_server:
|
||||
print("\nFirst 10 missing SNs:")
|
||||
for sn in missing_on_server[:10]:
|
||||
full, size, mtime = path_by_sn[sn]
|
||||
print(f" {sn} ({size} bytes, {mtime[:19]})")
|
||||
print("\nLast 10 missing SNs:")
|
||||
for sn in missing_on_server[-10:]:
|
||||
full, size, mtime = path_by_sn[sn]
|
||||
print(f" {sn} ({size} bytes, {mtime[:19]})")
|
||||
|
||||
# Show counts by year-month for the delta
|
||||
from collections import Counter
|
||||
by_month = Counter()
|
||||
for sn in missing_on_server:
|
||||
_, _, mtime = path_by_sn[sn]
|
||||
by_month[mtime[:7]] += 1
|
||||
print("\nDelta by year-month:")
|
||||
for k in sorted(by_month):
|
||||
print(f" {k}: {by_month[k]}")
|
||||
Reference in New Issue
Block a user