Merge remote-tracking branch 'origin/ad2'
This commit is contained in:
@@ -160,3 +160,4 @@
|
|||||||
- [CT Thoughts backlog](feedback_ct_thoughts_backlog.md) — ClaudeTools harness ideas go in docs/CT_THOUGHTS.md (trigger "ct thought:"); CT analogue of RMM_THOUGHTS. Don't build until explicit go. First entry = ClaudeTools 3.0 web co-work vision.
|
- [CT Thoughts backlog](feedback_ct_thoughts_backlog.md) — ClaudeTools harness ideas go in docs/CT_THOUGHTS.md (trigger "ct thought:"); CT analogue of RMM_THOUGHTS. Don't build until explicit go. First entry = ClaudeTools 3.0 web co-work vision.
|
||||||
- [AI-auth product boundary](project_ai_auth_product_boundary.md) — ClaudeTools/ClaudeTools 3.0 = internal-only, per-person subscription OAuth ok; GuruRMM = sellable, customer brings own API key (never ACG's subscription); backend dev = internal. Anthropic ToS bans subscription auth in third-party products.
|
- [AI-auth product boundary](project_ai_auth_product_boundary.md) — ClaudeTools/ClaudeTools 3.0 = internal-only, per-person subscription OAuth ok; GuruRMM = sellable, customer brings own API key (never ACG's subscription); backend dev = internal. Anthropic ToS bans subscription auth in third-party products.
|
||||||
- [RMM SYSTEM context can't see user mapped drives](feedback_rmm_system_context_mapped_drives.md) — RMM runs as SYSTEM; `Test-Path F:\` etc. is False even when the user's mapped/redirected drive exists. Diagnose mapped-drive/redirect issues in `context:user_session`. Elevated apps (e.g. QB DB Server Manager "unable to retrieve root folder") need `EnableLinkedConnections=1` + reboot.
|
- [RMM SYSTEM context can't see user mapped drives](feedback_rmm_system_context_mapped_drives.md) — RMM runs as SYSTEM; `Test-Path F:\` etc. is False even when the user's mapped/redirected drive exists. Diagnose mapped-drive/redirect issues in `context:user_session`. Elevated apps (e.g. QB DB Server Manager "unable to retrieve root folder") need `EnableLinkedConnections=1` + reboot.
|
||||||
|
- [AD2 = Dataforth-ops fork](project_ad2_dataforth_fork.md) — branch ad2 = main + thin Dataforth layer; keep fork edits ADDITIVE (Dataforth context in clients/dataforth/CLAUDE.dataforth.md, NOT .claude/CLAUDE.md); rebase onto main directly when sync.sh self-lock hits; no vault/jq/sops/age on this box.
|
||||||
|
|||||||
19
.claude/memory/project_ad2_dataforth_fork.md
Normal file
19
.claude/memory/project_ad2_dataforth_fork.md
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
---
|
||||||
|
name: project_ad2_dataforth_fork
|
||||||
|
description: AD2 is the Dataforth-operations fork (branch ad2); how it relates to main and how to sync it cleanly
|
||||||
|
metadata:
|
||||||
|
type: project
|
||||||
|
---
|
||||||
|
|
||||||
|
AD2 (192.168.0.6, Dataforth domain controller) runs the **`ad2` branch** — a fork dedicated to Dataforth operations. Its `/sync` model is: `git rebase origin/main` then `git push origin ad2`, so `ad2` = `main` + a thin layer of Dataforth-specific commits.
|
||||||
|
|
||||||
|
**Keep the fork layer ADDITIVE** — never edit shared fleet files (especially `.claude/CLAUDE.md`) on `ad2`, or every sync re-conflicts. On 2026-06-17 the old fork edit of `.claude/CLAUDE.md` was relocated: `.claude/CLAUDE.md` now tracks the lean fleet doc (from main), and Dataforth operational context (network, credentials, pipeline, batch-file versions) lives in **`clients/dataforth/CLAUDE.dataforth.md`**. Read that file for Dataforth context.
|
||||||
|
|
||||||
|
**Sync gotchas on AD2:**
|
||||||
|
- `sync.sh` is NOT fork-aware on the PUSH step: it pushes `main` (`error: src refspec main does not match any`), so `/sync` and `/save` auto-sync ALWAYS fail at Phase 4 on this fork. It DOES rebase + commit fine. After running it, push manually: `git push --force-with-lease origin ad2` (force needed because the rebase rewrote history). The documented fork flow (fetch → rebase origin/main → push origin ad2) is the reliable path.
|
||||||
|
- `sync.sh` cannot overwrite itself while it is the executing script (Windows file lock) → the rebase step fails mid-run with "unable to create file .claude/scripts/sync.sh: Permission denied" the FIRST time main has a new sync.sh. When that happens: `git reset --hard HEAD`, `git clean -fd` (removes the partial origin/main checkout debris), then run `git rebase origin/main` **directly** (not via sync.sh), and force-push with `--force-with-lease`.
|
||||||
|
- Submodule `projects/msp-tools/guru-connect` fails to clone from 172.16.3.20 (ACG-internal, unreachable from Dataforth LAN) — non-fatal sync warning.
|
||||||
|
- Python was absent until 2026-06-17 (installed 3.12.8, `py` launcher); identity.json was created then too (`user=mike`, `machine=AD2`).
|
||||||
|
- Missing tooling: `jq`, `sops`, `age`. No vault cloned (`D:/vault` absent) → vault sync is N/A here. coord_api (172.16.3.30) is unreachable from the Dataforth LAN.
|
||||||
|
|
||||||
|
Related: [[project_ad2_context.md]]
|
||||||
@@ -17,11 +17,28 @@ Mined **56 of 58 models** straight from Hoffman into `projects/dataforth-dos/dsc
|
|||||||
Coverage: ~7,157 units already correct + live on Hoffman (no action); ~1,580 not-yet-uploaded units
|
Coverage: ~7,157 units already correct + live on Hoffman (no action); ~1,580 not-yet-uploaded units
|
||||||
need rendering from the mined templates + AD2's already-derived slotMaps.
|
need rendering from the mined templates + AD2's already-derived slotMaps.
|
||||||
|
|
||||||
AD2 handoff + the critical gate: `projects/dataforth-dos/DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md`.
|
**STATUS — DONE on AD2 (2026-06-18, commits up to `4615877f` on `ad2`):** the AD2 session wired the
|
||||||
**Critical:** validate each model's render byte-for-byte against its Hoffman original BEFORE enabling
|
mined templates into the renderer, reverse-engineered the DSCA33/45 accuracy-block formatting, and
|
||||||
live DSCA33/45 rendering — once the renderer returns non-null, the pipeline stops skipping these and
|
published the gap. **54 of 56 models validated byte-for-byte vs the live Hoffman originals** and
|
||||||
will re-push/UPDATE the 7,157 good originals on the next cycle (safe only if the render matches).
|
marked `validated:true` in `dsca33-45-templates.json` (the per-model render gate — a model renders
|
||||||
|
only when validated). **1,452 truly-new certs published to Hoffman (created=1452, updated=0,
|
||||||
|
errors=0)** — published absent-only (probed each serial with a GET; the stale inventory means
|
||||||
|
`api_uploaded_at IS NULL` does NOT mean "absent from Hoffman", so never trust it for this) so ZERO
|
||||||
|
originals were overwritten. Still null/unpublished: **DSCA33-04A** + **DSCA33-1891** (each one accuracy
|
||||||
|
cert at a rounding boundary where `Math.fround` rounds opposite the original — left UNvalidated), and
|
||||||
|
the 2 no-original models **DSCA33-1948 / DSCA45-1746** (24 units). The 126 already-live gap units were
|
||||||
|
left untouched.
|
||||||
|
|
||||||
Hoffman read API: `GET https://www.dataforth.com/api/v1/TestReportDataFiles/{serial}` (returns
|
Render specifics now in `templates/datasheet-exact.js`: DSCA3345_TEMPLATES takes precedence over the
|
||||||
`{SerialNumber,Content,CreatedAtUtc,UpdatedAtUtc}`); creds vault `clients/dataforth/hoffman-product-api`.
|
stale staged-extraction entries; `formatAccuracyLineDSCA3345` (mADC calc stored in A -> x1000; DSCA33
|
||||||
Miner: `projects/dataforth-dos/tools/mine-hoffman-dsca.py`. AD2 access notes: [[ad2-ssh-mtu-blackhole]].
|
meas unscaled, DSCA45 meas scaled; DSCA33 unsigned stim/calc/meas, DSCA45 signed + integer-Hz stim;
|
||||||
|
fround); leading-zero drop only when the value overflows the 6-char field; DSCA33 "Check List" header;
|
||||||
|
section sub-heads (Zero-Crossing/TTL) get no status. Tools: `validate-dsca3345.js` (oracle gate),
|
||||||
|
`slotmap-from-hoffman.js`, `publish-dsca3345-gap.js`.
|
||||||
|
|
||||||
|
Handoff doc (historical): `projects/dataforth-dos/DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md`. Hoffman
|
||||||
|
read API: `GET https://www.dataforth.com/api/v1/TestReportDataFiles/{serial}`; creds vault
|
||||||
|
`clients/dataforth/hoffman-product-api` (on AD2 the deployed uploader token at
|
||||||
|
`C:\ProgramData\dataforth-uploader\credentials.json` works for reads too — no vault needed). Miner:
|
||||||
|
`projects/dataforth-dos/tools/mine-hoffman-dsca.py`. AD2 access notes: [[ad2-ssh-mtu-blackhole]],
|
||||||
|
[[ad2-comms-via-sync-only]].
|
||||||
|
|||||||
65
Prompt617.txt
Normal file
65
Prompt617.txt
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
TASK: Diagnose + fully document the Dataforth test-datasheet bug. You are running ON AD2
|
||||||
|
(192.168.0.6), the host of the testdatadb generator + PostgreSQL. Do NOT fix anything yet <20>
|
||||||
|
trace, document, and pinpoint root cause.
|
||||||
|
|
||||||
|
=== THE PROBLEM (reported today, 2026-06-17) ===
|
||||||
|
John Lehman (jlehman@dataforth.com) to Mike: "We are experiencing problems with test
|
||||||
|
datasheets. Column headers are wrong. Some Final Test lines are missing."
|
||||||
|
|
||||||
|
From the thread (apps engineer Peter Iliya):
|
||||||
|
- For the 8B35: the STAGED datasheet "H9553-13" (T: drive, TS-4L/STAGE) has the CORRECT
|
||||||
|
input column label "Temp" <20> but the WEBSITE version "179553-13" shows the WRONG label
|
||||||
|
"resistance". DSCA38 is flagged with the same issue.
|
||||||
|
- Peter CANNOT find 179553-13 as a file on the X: drive or in the DFWDS report <20> because the
|
||||||
|
website version is generated from the DB, not stored as a file.
|
||||||
|
- Core question (Peter): "Is there a reason we list the input for our RTD modules as
|
||||||
|
resistance, but show the temperature instead?"
|
||||||
|
- Trigger: customer Wellbore Integrity (Joseph Swinehart) raised a cal-cert AUDIT discrepancy
|
||||||
|
on 8B35 4-wire RTD certs.
|
||||||
|
|
||||||
|
=== THE KEY POINT (Mike) ===
|
||||||
|
The rendered test sheet EXISTS AS A FILE BEFORE it is ingested into the database. That
|
||||||
|
original/staged file is the ground truth (correct Temp label). Our DB-based regeneration is
|
||||||
|
what introduces the error. FOLLOW THE TRAIL UPSTREAM and FIND WHERE THE ORIGINAL FILE IS.
|
||||||
|
|
||||||
|
=== ARCHITECTURE (per TEST-DATASHEET-PROCESS.md) ===
|
||||||
|
Test stations (TS-01..27, DOS) write .dat logs
|
||||||
|
-> AD1 HISTLOGS: \\ad1\... C:\Shares\test\Ate\HISTLOGS\{log_type}\{model}.DAT (+ per-station \TS-XX\LOGS\)
|
||||||
|
-> Legacy DFWDS (VB6) historically rendered "For_Web" .TXT datasheets (the ORIGINAL rendered files)
|
||||||
|
-> testdatadb (THIS host AD2): Node.js + Express + PostgreSQL 18, service on :3000, dashboard
|
||||||
|
http://localhost:3000/, WinSW wrapper, service account INTRANET\svc_testdatadb
|
||||||
|
-> Hoffman Product API (/api/v1/TestReportDataFiles/bulk, OAuth2) -> public website.
|
||||||
|
|
||||||
|
=== WHAT WE ALREADY FOUND IN THE CODE (verify the DEPLOYED copy matches) ===
|
||||||
|
Generator template: datasheet-exact.js (find the deployed copy on AD2; repo copy is
|
||||||
|
projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js).
|
||||||
|
- getSensorNum(): RTD -> 7. Input header logic: sensorNum 3-6 (thermocouples) -> " Temp. (C)";
|
||||||
|
sensorNum 7 (RTD) -> " Rin (ohms)"; so RTD is rendered as RESISTANCE. The accuracy VALUE
|
||||||
|
formatting is ALSO sensorNum-driven (temperature format vs resistance format) - so values may
|
||||||
|
be wrong too, not just the label.
|
||||||
|
- Final Test loop: `for (i < dataLines.length && i < parsed.statusEntries.length)` with
|
||||||
|
`if (status.length <= 4) continue` -> rows DROP when statusEntries are misaligned/short.
|
||||||
|
|
||||||
|
=== YOUR DELIVERABLES ===
|
||||||
|
1. FULLY DOCUMENT THE END-TO-END PROCESS (write it to a markdown file), especially the UPSTREAM
|
||||||
|
half: exactly what the test station emits, what the ORIGINAL rendered datasheet file is,
|
||||||
|
WHERE it physically lives (find TS-4L/STAGE, the staged H9553-13, and/or the DFWDS For_Web
|
||||||
|
.TXT), and exactly how testdatadb ingests it (parser, which fields, the .DAT format).
|
||||||
|
2. LOCATE THE ORIGINAL FILE for the 8B35 example (H9553-13 / its For_Web .TXT / the .dat it came
|
||||||
|
from) and for a DSCA38 example.
|
||||||
|
3. DIFF original (correct) vs DB-generated (wrong) for 8B35 RTD and DSCA38: the exact column
|
||||||
|
header difference, any VALUE differences, and which Final Test lines are missing and WHY.
|
||||||
|
4. ROOT-CAUSE LOCALIZATION: is the defect in (a) ingestion/.DAT parsing, (b) the DB data itself,
|
||||||
|
or (c) datasheet-exact.js rendering? Confirm against the original file, not assumptions.
|
||||||
|
5. State the CORRECT RTD output (label = Temp (C); values = temperature, derived how?) per the
|
||||||
|
original, and the precise code change(s) that would fix both the header/values and the
|
||||||
|
dropped Final Test lines. Note any modules where resistance IS the correct label (don't
|
||||||
|
over-correct).
|
||||||
|
|
||||||
|
Resources you have locally: PostgreSQL on AD2 (query testdatadb directly), the deployed Node
|
||||||
|
service + its code, SMB to AD1 (\\ad1\...) and D2TESTNAS (\\192.168.0.9\test, /data/test has
|
||||||
|
DFWDS, Ate, 8BDATA, DSCDATA, etc.). The ClaudeTools repo has the pipeline source under
|
||||||
|
projects/dataforth-dos/.
|
||||||
|
|
||||||
|
Output: the process doc (file) + a diagnosis summary + the proposed fix. Diagnose only <20> no
|
||||||
|
changes to the generator or DB until reviewed.
|
||||||
68
STAGE-IMPORT-INSTRUCTIONS.md
Normal file
68
STAGE-IMPORT-INSTRUCTIONS.md
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
# Stage TXT Import Task
|
||||||
|
# Date: 2026-03-28
|
||||||
|
# Context: CTONWTXT.BAT now uploads C:\STAGE\*.TXT from DOS machines to T:\STAGE\%MACHINE%\
|
||||||
|
|
||||||
|
## What happened
|
||||||
|
|
||||||
|
1. CTONWTXT.BAT was never being called -- fixed, now called from CTONW.BAT on every boot
|
||||||
|
2. Destination changed from broken X: (Novell serve.sys check) to T:\STAGE\%MACHINE%\
|
||||||
|
3. DOS 6.22 can't MD on existing dirs without error, so dirs are pre-created on NAS
|
||||||
|
4. All TS-* machine folders pre-created under /data/test/STAGE/ on D2TESTNAS
|
||||||
|
|
||||||
|
## What needs to run
|
||||||
|
|
||||||
|
Save the script below as C:\Shares\testdatadb\import-all-stage.js and run it:
|
||||||
|
|
||||||
|
cd C:\Shares\testdatadb
|
||||||
|
node import-all-stage.js
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
- Scans \\D2TESTNAS\test\STAGE\TS-*\*.TXT (~8,100 files across 10 machines)
|
||||||
|
- Parses each TXT datasheet (Date, Model, SN)
|
||||||
|
- Decodes hex-prefix serial numbers for 8.3 filename encoding:
|
||||||
|
- Letter prefix = hex digit: A=10, B=11, C=12, ..., H=17, etc.
|
||||||
|
- Example: H8236-12.TXT has SN: 178236-12 inside the file
|
||||||
|
- Example: A819-1.TXT has SN: A819-1 inside -> decoded to 10819-1
|
||||||
|
- The SN line inside H-prefix files already has the full numeric serial
|
||||||
|
- The SN line inside A-prefix files still has the encoded serial
|
||||||
|
- Cross-references against testdata.db by (serial_number, model_number)
|
||||||
|
- Inserts MISSING records as log_type='SHT' with test_station from folder name
|
||||||
|
- Copies ALL files to X:\For_Web\{decoded_serial}.TXT (the web share)
|
||||||
|
|
||||||
|
## Machines with data
|
||||||
|
|
||||||
|
TS-4L: 3,082 files (largest)
|
||||||
|
TS-4R: 2,741 files
|
||||||
|
TS-1R: 509 files
|
||||||
|
TS-8R: 478 files
|
||||||
|
TS-3R: 435 files
|
||||||
|
TS-11R: 325 files
|
||||||
|
TS-8L: 285 files
|
||||||
|
TS-11L: 248 files
|
||||||
|
TS-27: 10 files (already imported this session)
|
||||||
|
TS-1L: 1 file
|
||||||
|
|
||||||
|
## Serial number encoding (8.3 filename scheme)
|
||||||
|
|
||||||
|
The QuickBASIC ATE software encodes long serial numbers to fit DOS 8.3 filenames.
|
||||||
|
The first two digits get replaced with a hex letter if the serial is too long:
|
||||||
|
|
||||||
|
178236-12 -> H8236-12.TXT (17 -> H, which is char code 72, 72-55=17)
|
||||||
|
10819-1 -> A819-1.TXT (10 -> A, which is char code 65, 65-55=10)
|
||||||
|
|
||||||
|
Decode: letter.charCodeAt(0) - 55 = numeric prefix
|
||||||
|
Only applies if filename starts with [A-Z] followed by digits.
|
||||||
|
|
||||||
|
## TS-27 already done
|
||||||
|
|
||||||
|
10 files from TS-27 were already imported earlier this session into the DB as SHT records.
|
||||||
|
The import script uses INSERT OR REPLACE so re-running is safe.
|
||||||
|
|
||||||
|
## Previous CTONWTXT.BAT issues (resolved)
|
||||||
|
|
||||||
|
- v1.0: Never called, checked for Novell serve.sys, used X: drive parameter
|
||||||
|
- v2.0: Called from CTONW, but used mixed-case "Stage" path -> failed on DOS
|
||||||
|
- v2.1: All uppercase STAGE, but had MD commands that fail on existing dirs
|
||||||
|
- v2.2: Same issue
|
||||||
|
- v2.3: Removed MD entirely, dirs pre-created on NAS. CURRENT VERSION.
|
||||||
80
Test Datasheets/weekend-update-draft.md
Normal file
80
Test Datasheets/weekend-update-draft.md
Normal file
@@ -0,0 +1,80 @@
|
|||||||
|
Subject: Test Datasheets - Weekend Update: All 73 Quatronix Sheets Generated, Work Order Search Live
|
||||||
|
|
||||||
|
John, Ken,
|
||||||
|
|
||||||
|
Quick update on progress since Friday's email. The pipeline is significantly further along.
|
||||||
|
|
||||||
|
## Quatronix Customer Issue - RESOLVED
|
||||||
|
|
||||||
|
All 73 requested datasheets have been generated (TXT + PDF). The last holdout was SCM5B49-05 (SN 177000-15) — the 5B49DATA.DAT spec file was empty, but John pointed us to 5B49_2.DAT which had the data. All 73 files are ready to send to Peter/Ginger.
|
||||||
|
|
||||||
|
## Model Spec Coverage Expanded
|
||||||
|
|
||||||
|
We went from 751 model specs to 1,470+ by loading additional spec databases:
|
||||||
|
|
||||||
|
| Spec File | Family | Models |
|
||||||
|
|-----------|--------|--------|
|
||||||
|
| 5BMAIN.DAT | SCM5B | 481 |
|
||||||
|
| 5B45DATA.DAT | SCM5B (freq/counter) | 56 |
|
||||||
|
| DB5B48.DAT | SCM5B (multi-bandwidth) | 3 |
|
||||||
|
| 5B49_2.DAT | SCM5B (sample & hold) | 15 |
|
||||||
|
| 8BMAIN.DAT | 8B | 148 |
|
||||||
|
| DSCOUT.DAT | DSCA (output) | 23 |
|
||||||
|
| DSCMAIN4.DAT | DSCA (input) | 391 |
|
||||||
|
| SCTMAIN.DAT | DSCT | 103 |
|
||||||
|
| 7BMAIN.DAT | SCM7B | 276 |
|
||||||
|
|
||||||
|
If there are additional spec files we're missing, let me know the paths and we'll add them.
|
||||||
|
|
||||||
|
## SCM7B Support Added
|
||||||
|
|
||||||
|
The 7B product family is now fully supported in the datasheet generator:
|
||||||
|
- 31 test parameters (vs 20 for SCM5B)
|
||||||
|
- Correct header ("SCM" prefix prepended to model name)
|
||||||
|
- 120VAC Withstand / Hi-Pot (skipped for 7BPT models)
|
||||||
|
- "Packing Check List" with blank fields (vs pre-marked checkboxes on 5B/8B)
|
||||||
|
- "Tested by" and "QC" signature lines
|
||||||
|
- Note: The 7B DAT format (single CSV line) doesn't include individual accuracy test points, so the accuracy table is omitted. Only the Final Test Results section is generated from DAT data.
|
||||||
|
|
||||||
|
## Work Order Search & Linking
|
||||||
|
|
||||||
|
Imported all 33,745 work order status reports from the test station Reports folders:
|
||||||
|
- 63,263 individual test lines parsed (serial number, model, pass/fail, date/time, station)
|
||||||
|
- 2.27 million test records linked to their work orders
|
||||||
|
|
||||||
|
In the web app (http://192.168.0.6:3000):
|
||||||
|
- New "Work Order #" search field — enter a WO number to find all associated test records
|
||||||
|
- Click the WO number in any record's detail view to see the full work order:
|
||||||
|
- All serial numbers tested under that WO
|
||||||
|
- Pass/fail status for each (including retests)
|
||||||
|
- Test program and version used
|
||||||
|
- Test station and timestamps
|
||||||
|
- New work order reports are automatically imported when synced from the NAS
|
||||||
|
|
||||||
|
## Datasheet Formatting Refined
|
||||||
|
|
||||||
|
Compared generated datasheets against originals from the DFWDS archive and fixed column alignment to match the QuickBASIC output:
|
||||||
|
- TAB positions match exactly (parameter names, measured values, spec limits, pass/fail)
|
||||||
|
- Number formatting matches QB PRINT USING (right-justified, correct decimal places)
|
||||||
|
- STR$() behavior replicated (leading space for positive numbers, dropped leading zeros)
|
||||||
|
- Spec limit formatting matches (e.g., "+/- .03 %" not "+/- 0.03 %")
|
||||||
|
|
||||||
|
## View Button Updated
|
||||||
|
|
||||||
|
The "SHEET" button in the web app now shows a styled HTML page that matches the PDF/TXT layout — white page, monospace font, same column alignment. Includes Print and Download PDF buttons.
|
||||||
|
|
||||||
|
## Infrastructure
|
||||||
|
|
||||||
|
- Created domain service account (INTRANET\svc_testdatadb) for the TestDataDB Windows service — resolves the file permission issues we were hitting
|
||||||
|
- Added STAGE folder sync to the NAS sync script — TXT datasheets from DOS machines will now be pulled to AD2 automatically
|
||||||
|
- Work order report import added to sync script — new reports are ingested automatically every 15 minutes
|
||||||
|
|
||||||
|
## Still Open
|
||||||
|
|
||||||
|
1. **Website upload** — The old Uploader.aspx endpoints are dead. Need to determine the new upload mechanism for dataforth.com.
|
||||||
|
2. **STAGE backlog** — ~8,100 TXT files on the NAS from DOS machines need to be processed (script ready, haven't run it yet).
|
||||||
|
3. **Pending ForWeb export** — ~845K records in the database don't have TXT files in For_Web yet (mostly 7B and older records). Can batch-export as needed.
|
||||||
|
|
||||||
|
Let me know if you need anything else.
|
||||||
|
|
||||||
|
Mike
|
||||||
3
clients/dataforth/.gitignore
vendored
3
clients/dataforth/.gitignore
vendored
@@ -1,3 +1,6 @@
|
|||||||
|
|
||||||
# plaintext credential note - never commit
|
# plaintext credential note - never commit
|
||||||
Oauth.txt
|
Oauth.txt
|
||||||
|
|
||||||
|
# Large local-only binary artifacts (WizTree disk scans, dumps) - keep on the machine, never commit
|
||||||
|
local-artifacts/
|
||||||
|
|||||||
246
clients/dataforth/CLAUDE.dataforth.md
Normal file
246
clients/dataforth/CLAUDE.dataforth.md
Normal file
@@ -0,0 +1,246 @@
|
|||||||
|
# ClaudeTools on AD2 (Dataforth Domain Controller)
|
||||||
|
|
||||||
|
## Identity
|
||||||
|
|
||||||
|
This is the AD2 workstation instance of ClaudeTools. This machine is a Windows Server on the Dataforth LAN (192.168.0.6). Your scope is Dataforth-only -- you do not need context about other clients.
|
||||||
|
|
||||||
|
## NO EMOJIS
|
||||||
|
|
||||||
|
Use ASCII markers: [OK], [ERROR], [WARNING], [SUCCESS], [INFO]
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Git & Sync
|
||||||
|
|
||||||
|
### Gitea Credentials (no 1Password on this machine)
|
||||||
|
- URL: https://git.azcomputerguru.com
|
||||||
|
- Username: mike@azcomputerguru.com
|
||||||
|
- Password: Gptf*77ttb123!@#-git
|
||||||
|
- URL-encoded password: Gptf%2A77ttb123%21%40%23-git
|
||||||
|
- API Token: 9b1da4b79a38ef782268341d25a4b6880572063f
|
||||||
|
- Remote: https://mike%40azcomputerguru.com:Gptf%2A77ttb123%21%40%23-git@git.azcomputerguru.com/azcomputerguru/claudetools.git
|
||||||
|
|
||||||
|
### Branch: ad2
|
||||||
|
This machine operates on the `ad2` branch. The main workstation merges into main.
|
||||||
|
|
||||||
|
### /save behavior
|
||||||
|
Save session logs to `session-logs/YYYY-MM-DD-session-ad2.md` (note the -ad2 suffix).
|
||||||
|
After saving, commit and push to origin/ad2.
|
||||||
|
|
||||||
|
### /sync behavior
|
||||||
|
```
|
||||||
|
git fetch origin
|
||||||
|
git rebase origin/main
|
||||||
|
git push origin ad2
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dataforth Network
|
||||||
|
|
||||||
|
| Host | IP | Role | Notes |
|
||||||
|
|------|-----|------|-------|
|
||||||
|
| AD1 | 192.168.0.27 | Primary DC | Disk at 90%, C:\Engineering = 787 GB |
|
||||||
|
| **AD2** | **192.168.0.6** | **This machine** | Secondary DC, TestDataDB, file shares |
|
||||||
|
| D2TESTNAS | 192.168.0.9 | SMB1 proxy for DOS | Debian 13, Samba, SSH root/Paper123!@#-nas |
|
||||||
|
| UDM | 192.168.0.254 | Gateway/Router | UniFi Dream Machine |
|
||||||
|
| ESXi-122 | 192.168.0.122 | Hypervisor | ESXi |
|
||||||
|
| ESXi-124 | 192.168.0.124 | Hypervisor | ESXi |
|
||||||
|
| DOS stations | TS-01 to TS-30+ | Test stations | DOS 6.22, QuickBASIC ATE software |
|
||||||
|
|
||||||
|
### Credentials
|
||||||
|
- AD Sysadmin: INTRANET\sysadmin / Paper123!@#
|
||||||
|
- D2TESTNAS SSH: root@192.168.0.9 / Paper123!@#-nas
|
||||||
|
- D2TESTNAS Samba: guest access (no password)
|
||||||
|
- WINS/NPS: 192.168.0.27:1812/1813
|
||||||
|
- M365 Tenant: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
|
||||||
|
- Rsync daemon (NAS): port 873, module "test", user rsync / IQ203s32119
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Local Resources
|
||||||
|
|
||||||
|
| Resource | Path |
|
||||||
|
|----------|------|
|
||||||
|
| TestDataDB app | C:\Shares\testdatadb\ |
|
||||||
|
| Test database | C:\Shares\testdatadb\database\testdata.db (SQLite, 2.2M+ records) |
|
||||||
|
| TestDataDB API | http://localhost:3000 |
|
||||||
|
| Parsers | C:\Shares\testdatadb\parsers\ (multiline.js, csvline.js, shtfile.js, spec-reader.js) |
|
||||||
|
| Templates | C:\Shares\testdatadb\templates\datasheet-exact.js |
|
||||||
|
| Import script | C:\Shares\testdatadb\database\import.js |
|
||||||
|
| Export script | C:\Shares\testdatadb\database\export-datasheets.js |
|
||||||
|
| Stage import | C:\Shares\testdatadb\import-all-stage.js |
|
||||||
|
| NAS share | \\D2TESTNAS\test (mapped as T:) |
|
||||||
|
| Datasheets share | X:\For_Web |
|
||||||
|
| ProdSW (BAT files) | C:\Shares\test\COMMON\ProdSW\ |
|
||||||
|
| Sync script | C:\Shares\test\scripts\Sync-FromNAS.ps1 (bidirectional, 15-min schedule) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## DOS Update System - Batch Files
|
||||||
|
|
||||||
|
### Boot Sequence on DOS Machines
|
||||||
|
```
|
||||||
|
AUTOEXEC.BAT (v4.1)
|
||||||
|
-> STARTNET.BAT (v2.0) -- init network, map T: and X: drives
|
||||||
|
-> ATESYNC.BAT
|
||||||
|
-> CTONW.BAT (v5.0) -- upload test data to network
|
||||||
|
-> CTONWTXT.BAT (v2.3) -- upload C:\STAGE\*.TXT to T:\STAGE\%MACHINE%
|
||||||
|
-> NWTOC.BAT (v5.0) -- download updates from network
|
||||||
|
```
|
||||||
|
|
||||||
|
### Current Production Versions (on AD2 & NAS)
|
||||||
|
| File | Version | Last Update | Purpose |
|
||||||
|
|------|---------|-------------|---------|
|
||||||
|
| AUTOEXEC.BAT | v4.1 | 2026-03-12 | Startup config |
|
||||||
|
| STARTNET.BAT | v2.0 | 2026-01-20 | Network init |
|
||||||
|
| NWTOC.BAT | v5.0 | 2026-03-16 | Download updates from network |
|
||||||
|
| CTONW.BAT | v5.0 | 2026-03-28 | Upload test data (5 steps with echo) |
|
||||||
|
| CTONWTXT.BAT | v2.3 | 2026-03-28 | Upload Stage TXT files (no MD, dirs pre-created) |
|
||||||
|
| CHECKUPD.BAT | v1.3 | 2026-01-20 | Check for updates |
|
||||||
|
| UPDATE.BAT | v2.3 | 2026-01-20 | Full system backup |
|
||||||
|
| STAGE.BAT | v1.0 | Original | Stage system file updates |
|
||||||
|
| DEPLOY.BAT | v1.0 | 2026-01-20 | One-time deployment installer |
|
||||||
|
|
||||||
|
### DOS 6.22 Compatibility Rules
|
||||||
|
- NO `IF NOT` -- unreliable on DOS 6.22. Use positive `IF EXIST` with GOTO
|
||||||
|
- NO `IF /I` (case-insensitive compare)
|
||||||
|
- NO `FOR /F` loops
|
||||||
|
- NO `%COMPUTERNAME%` -- use `%MACHINE%` (set during DEPLOY)
|
||||||
|
- `XCOPY /D` requires date parameter (`/D:mm-dd-yy`)
|
||||||
|
- `MD` fails with error on existing directories -- pre-create dirs server-side
|
||||||
|
- `COPY` without `/Y` hangs on overwrite prompts
|
||||||
|
- All paths UPPERCASE for Samba compatibility
|
||||||
|
- Line endings MUST be CRLF (0D 0A)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Serial Number Encoding (DOS 8.3 filenames)
|
||||||
|
|
||||||
|
QuickBASIC ATE encodes long serial numbers for 8.3 filenames:
|
||||||
|
```
|
||||||
|
First 2 digits replaced with hex letter if serial too long:
|
||||||
|
178236-12 -> H8236-12.TXT (17 -> H, charCode 72 - 55 = 17)
|
||||||
|
10819-1 -> A819-1.TXT (10 -> A, charCode 65 - 55 = 10)
|
||||||
|
|
||||||
|
Decode: letter.charCodeAt(0) - 55 = numeric prefix
|
||||||
|
Only applies when filename starts with [A-Z] followed by digits.
|
||||||
|
|
||||||
|
H-prefix files have decoded SN inside the file (SN: 178236-12)
|
||||||
|
A-prefix files have encoded SN inside the file (SN: A819-1) -- must decode to 10819-1
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test Datasheet Pipeline
|
||||||
|
|
||||||
|
### 5-Stage Architecture
|
||||||
|
1. **DOS Test Programs** -> Write DAT files to C:\ATE\*LOG\ and TXT to C:\STAGE\
|
||||||
|
2. **Boot Upload** -> CTONW.BAT copies DAT to T:\%MACHINE%\LOGS\, CTONWTXT copies TXT to T:\STAGE\%MACHINE%
|
||||||
|
3. **NAS <-> AD2 Sync** -> Rsync every 15 min (Sync-FromNAS.ps1 scheduled task)
|
||||||
|
4. **TestDataDB Import** -> import.js parses DAT into SQLite; export-datasheets.js generates TXT to X:\For_Web
|
||||||
|
5. **Web Share** -> X:\For_Web\ holds validated datasheets (501K+ files)
|
||||||
|
|
||||||
|
### import-all-stage.js (ready to run)
|
||||||
|
Located at `C:\Shares\testdatadb\import-all-stage.js`. Processes ~8,100 TXT files:
|
||||||
|
- Scans \\D2TESTNAS\test\STAGE\TS-*\*.TXT
|
||||||
|
- Decodes hex-prefix serial numbers
|
||||||
|
- Cross-references testdata.db by (serial_number, model_number)
|
||||||
|
- Inserts missing records as log_type='SHT'
|
||||||
|
- Copies to X:\For_Web\{decoded_serial}.TXT
|
||||||
|
|
||||||
|
```
|
||||||
|
cd C:\Shares\testdatadb
|
||||||
|
node import-all-stage.js
|
||||||
|
```
|
||||||
|
|
||||||
|
### Machine data volumes in STAGE
|
||||||
|
| Machine | Files |
|
||||||
|
|---------|-------|
|
||||||
|
| TS-4L | 3,082 |
|
||||||
|
| TS-4R | 2,741 |
|
||||||
|
| TS-1R | 509 |
|
||||||
|
| TS-8R | 478 |
|
||||||
|
| TS-3R | 435 |
|
||||||
|
| TS-11R | 325 |
|
||||||
|
| TS-8L | 285 |
|
||||||
|
| TS-11L | 248 |
|
||||||
|
| TS-27 | 10 (already imported) |
|
||||||
|
| TS-1L | 1 |
|
||||||
|
|
||||||
|
### Web Share Layout (X:\)
|
||||||
|
- X:\For_Web -- Validated datasheets (production)
|
||||||
|
- X:\For_Web_PDF -- PDF versions (4.7K files)
|
||||||
|
- X:\Test_Datasheets -- Incoming/staging
|
||||||
|
- X:\Bad_Datasheets -- Invalid files (18K)
|
||||||
|
- X:\Datasheets_Log -- Processing logs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Known Issues & Pending Work
|
||||||
|
|
||||||
|
### HIGH PRIORITY
|
||||||
|
1. **Run import-all-stage.js** -- 8,100 TXT files need cross-referencing and ingestion
|
||||||
|
2. **Website Upload Replacement** -- Old ASP.NET endpoints (Uploader.aspx) return 404. Need new approach.
|
||||||
|
3. **7B Series Datasheets** -- ~830K records can't generate datasheets (missing 7BMAIN.DAT spec file). Check ENGR share.
|
||||||
|
4. **Service Permissions** -- testdatadb runs as SYSTEM, causing file permission issues. Change to INTRANET\sysadmin.
|
||||||
|
|
||||||
|
### MEDIUM PRIORITY
|
||||||
|
5. **C2 IP Blocking** -- iptables rules added to UDM for 80.76.49.18 and 45.88.91.99. Need permanent rules in UniFi UI.
|
||||||
|
6. **MFA Enforcement** -- 19/38 users ready. Report-only until April 4, 2026. Monitor registration.
|
||||||
|
7. **Joel Lohr Account** -- Retiring March 31. Disable account post-retirement. Auto-reply set to Dan Center.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Security Incident (2026-03-27)
|
||||||
|
|
||||||
|
**DF-JOEL2 (192.168.0.143) compromised via phishing:**
|
||||||
|
- Joel Lohr clicked phishing link in personal Yahoo email
|
||||||
|
- ScreenConnect C2 installed, "Angel Raya" connected remotely
|
||||||
|
- Two C2 backdoors deployed via PowerShell
|
||||||
|
- C2 IPs: 80.76.49.18, 45.88.91.99 (AS399486, suspended by host)
|
||||||
|
- IC3 Complaint: 1c32ade367084be9acd548f23705736f
|
||||||
|
- ConnectWise Case: 03464184
|
||||||
|
- **Remediation complete:** IPs blocked, 3 rogue clients removed, password reset, sessions revoked
|
||||||
|
- **No lateral movement detected** (32 machines scanned clean)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Contacts
|
||||||
|
|
||||||
|
| Person | Email | Role |
|
||||||
|
|--------|-------|------|
|
||||||
|
| John Lehman | jlehman@dataforth.com | Engineering, QB code, test specs |
|
||||||
|
| Dan Center | dcenter@dataforth.com | Operations (replacing Joel) |
|
||||||
|
| Peter Iliya | pIliya@dataforth.com | Applications Engineer |
|
||||||
|
| AJ | dataforthgit@... | Engineering contact |
|
||||||
|
| Ken Hoffman | (unresponsive) | TestDataSheetUploader author |
|
||||||
|
| Georg Haubner | ghaubner@dataforth.com | Has pre-crypto backup on D: drive |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Reference Commands
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# Check BAT files on NAS
|
||||||
|
ssh root@192.168.0.9 'ls -la /data/test/COMMON/ProdSW/'
|
||||||
|
|
||||||
|
# Trigger NAS sync
|
||||||
|
Start-ScheduledTask -TaskName 'Sync-FromNAS'
|
||||||
|
|
||||||
|
# Check sync log
|
||||||
|
Get-Content 'C:\Shares\test\scripts\sync-from-nas.log' -Tail 20
|
||||||
|
|
||||||
|
# Check TestDataDB health
|
||||||
|
curl http://localhost:3000/health
|
||||||
|
|
||||||
|
# Query test records
|
||||||
|
node -e "const db=require('better-sqlite3')('C:\\Shares\\testdatadb\\database\\testdata.db',{readonly:true});console.log(db.prepare('SELECT COUNT(*) as cnt FROM test_records').get())"
|
||||||
|
|
||||||
|
# Check Stage files on NAS
|
||||||
|
ssh root@192.168.0.9 'find /data/test/STAGE -name "*.TXT" | wc -l'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Last Updated:** 2026-03-29
|
||||||
@@ -37,6 +37,10 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
|
|||||||
|
|
||||||
2026-06-18 | Howard-Home | pfsense-ssh/logs | [friction] used clog on pfSense 25.07 logs (now plain-text ASCII) -> empty output -> wrongly concluded DHCP log was empty / dhcpd not serving; cost a hypothesis. Read pfSense 25.07 logs with tail/grep/cat directly, NOT clog [ctx: ref=reference_pfsense_25_07_ops client=cascades-tucson]
|
2026-06-18 | Howard-Home | pfsense-ssh/logs | [friction] used clog on pfSense 25.07 logs (now plain-text ASCII) -> empty output -> wrongly concluded DHCP log was empty / dhcpd not serving; cost a hypothesis. Read pfSense 25.07 logs with tail/grep/cat directly, NOT clog [ctx: ref=reference_pfsense_25_07_ops client=cascades-tucson]
|
||||||
|
|
||||||
|
2026-06-18 | AD2 | vault | real vault.sh not found at resolved vault_path; vault read failed [ctx: path=D:/vault/scripts/vault.sh]
|
||||||
|
|
||||||
|
2026-06-18 | AD2 | vault | real vault.sh not found at resolved vault_path; vault read failed [ctx: path=D:/vault/scripts/vault.sh]
|
||||||
|
|
||||||
2026-06-17 | GURU-5070 | mailbox/365-mail | [correction] claimed in a prior session that /mailbox skill + memories were repointed off the deleted fabb3421 to the 365-mail suite, but mailbox.md still hardwired fabb3421 (token 401 AADSTS700016). Correct app is the dedicated ComputerGuru Mailbox app 1873b1b0 via get-token.sh 'mailbox' tier (cert auth); repointed mailbox.md + feedback_365_remediation_tool.md 2026-06-17. Lesson: verify the edit actually landed before reporting it done.
|
2026-06-17 | GURU-5070 | mailbox/365-mail | [correction] claimed in a prior session that /mailbox skill + memories were repointed off the deleted fabb3421 to the 365-mail suite, but mailbox.md still hardwired fabb3421 (token 401 AADSTS700016). Correct app is the dedicated ComputerGuru Mailbox app 1873b1b0 via get-token.sh 'mailbox' tier (cert auth); repointed mailbox.md + feedback_365_remediation_tool.md 2026-06-17. Lesson: verify the edit actually landed before reporting it done.
|
||||||
|
|
||||||
2026-06-17 | Howard-Home | wiki-compile/coord | [friction] skill doc Phase 6 shows 'lock release claudetools wiki/<type>/<slug>' but coord.py takes 'lock release <id>'; wasted a round-trip. Capture the lock id from claim output and release by id. [ctx: ref=wiki-compile-skill]
|
2026-06-17 | Howard-Home | wiki-compile/coord | [friction] skill doc Phase 6 shows 'lock release claudetools wiki/<type>/<slug>' but coord.py takes 'lock release <id>'; wasted a round-trip. Capture the lock id from claim output and release by id. [ctx: ref=wiki-compile-skill]
|
||||||
|
|||||||
54
generated-129093-16.TXT
Normal file
54
generated-129093-16.TXT
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
DATAFORTH CORPORATION Phone: (520) 741-1404
|
||||||
|
3331 E. Hemisphere Loop Fax: (520) 741-0762
|
||||||
|
Tucson, AZ 85706 USA email: info@dataforth.com
|
||||||
|
|
||||||
|
TEST DATA SHEET
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
Date: 05-15-2018
|
||||||
|
Model: SCM5B38-05
|
||||||
|
SN: 129093-16
|
||||||
|
|
||||||
|
ACCURACY TEST
|
||||||
|
|
||||||
|
Calculated Measured
|
||||||
|
Vin (mV) Vout (V) Vout (V)* Error (%) Status
|
||||||
|
========== ========== ========== ========= ========
|
||||||
|
-20.000 -5.000 -5.000 -0.005 PASS
|
||||||
|
-9.999 -2.500 -2.499 +0.002 PASS
|
||||||
|
-0.001 -0.000 +0.001 +0.016 PASS
|
||||||
|
+10.001 +2.500 +2.502 +0.018 PASS
|
||||||
|
+20.000 +5.000 +5.000 +0.003 PASS
|
||||||
|
|
||||||
|
FINAL TEST RESULTS
|
||||||
|
|
||||||
|
Parameter Measured Value Specification Status
|
||||||
|
======================== =============== ===================== ======
|
||||||
|
Supply Current, Nom 29.0 mA < 75 mA PASS
|
||||||
|
Supply Current, Max 130.6 mA < 194 mA PASS
|
||||||
|
Output Resistance 22 ohms < 55 ohms PASS
|
||||||
|
Exc. Voltage 10.001 V 10.0+/-0.003 V PASS
|
||||||
|
Exc. Load Reg. 5 ppm/mA +/-11 ppm/mA PASS
|
||||||
|
Vout Reg. w/ Load 0.0 % +/-0.1 % PASS
|
||||||
|
Exc. Current Limit 54.4 mA < 63 mA PASS
|
||||||
|
Linearity 0.010 % +/- 0.03 % PASS
|
||||||
|
Accuracy 0.018 % +/- 0.08 % PASS
|
||||||
|
Supply Sensitivity 0.8 uV/% +/- 2.4 uV/% PASS
|
||||||
|
Frequency Response 29.2 dB 27+/-7 dB PASS
|
||||||
|
Output Noise 566 uVrms < 4000 uVrms PASS
|
||||||
|
240 VAC Withstand PASS
|
||||||
|
Hi-Pot PASS
|
||||||
|
_______________________________________________________________________
|
||||||
|
Check List
|
||||||
|
|
||||||
|
Module Appearance: __X__ Mounting Screw: __X__
|
||||||
|
|
||||||
|
Pins Straight: __X__ Module Header: __X__
|
||||||
|
|
||||||
|
|
||||||
|
It is hereby certified that the above product is in conformance with
|
||||||
|
all requirements to the extent specified. This product is not
|
||||||
|
authorized or warranted for use in life support devices and/or systems.
|
||||||
|
|
||||||
|
* NIST traceable calibration certificates support Measured Value data.
|
||||||
|
Calibration services are available through ANSI/NCSL Z540-1 and
|
||||||
|
ISO Guide 25 Certified Metrology Labs.
|
||||||
53
generated-v2-129093-16.TXT
Normal file
53
generated-v2-129093-16.TXT
Normal file
@@ -0,0 +1,53 @@
|
|||||||
|
DATAFORTH CORPORATION Phone: (520) 741-1404
|
||||||
|
3331 E. Hemisphere Loop Fax: (520) 741-0762
|
||||||
|
Tucson, AZ 85706 USA email: info@dataforth.com
|
||||||
|
|
||||||
|
TEST DATA SHEET
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
Date: 05-15-2018
|
||||||
|
Model: SCM5B38-05
|
||||||
|
SN: 129093-16
|
||||||
|
|
||||||
|
ACCURACY TEST
|
||||||
|
|
||||||
|
Calculated Measured
|
||||||
|
Vin (mV) Vout (V) Vout (V)* Error (%) Status
|
||||||
|
========== ========== ========== ========= ========
|
||||||
|
-20.000 -5.000 -5.000 -0.005 PASS
|
||||||
|
-9.999 -2.500 -2.499 +0.002 PASS
|
||||||
|
-0.001 -0.000 +0.001 +0.016 PASS
|
||||||
|
+10.001 +2.500 +2.502 +0.018 PASS
|
||||||
|
+20.000 +5.000 +5.000 +0.003 PASS
|
||||||
|
|
||||||
|
FINAL TEST RESULTS
|
||||||
|
|
||||||
|
Parameter Measured Value Specification Status
|
||||||
|
======================= =============== ===================== ======
|
||||||
|
Supply Current, Nom 29.0 mA < 75 mA PASS
|
||||||
|
Supply Current, Max 130.6 mA < 194 mA PASS
|
||||||
|
Output Resistance 22 ohms < 55 ohms PASS
|
||||||
|
Exc. Voltage 10.001 V 10.0+/-0.003 V PASS
|
||||||
|
Exc. Load Reg. 5 ppm/mA +/-11 ppm/mA PASS
|
||||||
|
Vout Reg. w/ Load 0.0 % +/-0.1 % PASS
|
||||||
|
Exc. Current Limit 54.4 mA < 63 mA PASS
|
||||||
|
Linearity 0.010 % +/- 0.03 % PASS
|
||||||
|
Accuracy 0.018 % +/- 0.08 % PASS
|
||||||
|
Supply Sensitivity 0.8 uV/% +/- 2.4 uV/% PASS
|
||||||
|
Frequency Response 29.2 dB 27+/-7 dB PASS
|
||||||
|
Output Noise 566 uVrms < 4000 uVrms PASS
|
||||||
|
240 VAC Withstand PASS
|
||||||
|
Hi-Pot PASS
|
||||||
|
_______________________________________________________________________
|
||||||
|
Check List
|
||||||
|
|
||||||
|
Module Appearance: __X__ Mounting Screw: __X__
|
||||||
|
|
||||||
|
Pins Straight: __X__ Module Header: __X__
|
||||||
|
|
||||||
|
It is hereby certified that the above product is in conformance with
|
||||||
|
all requirements to the extent specified. This product is not
|
||||||
|
authorized or warranted for use in life support devices and/or systems.
|
||||||
|
|
||||||
|
* NIST traceable calibration certificates support Measured Value data.
|
||||||
|
Calibration services are available through ANSI/NCSL Z540-1 and
|
||||||
|
ISO Guide 25 Certified Metrology Labs.
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# Proposal — make the DB hold the LATEST test run (same-day retest fix)
|
||||||
|
|
||||||
|
**Date:** 2026-06-17 · **Host:** AD2 · **Status:** PROPOSAL — diagnose-only, review before deploying
|
||||||
|
**File to change:** `C:\Shares\testdatadb\database\import.js` (repo: `projects/dataforth-dos/database/import.js`)
|
||||||
|
**Evidence:** `PARSING-FIDELITY-VERDICT-2026-06-17.md`, `SAMEDAY-RETEST-EXPOSURE-2026-06-17.txt`
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
`test_records` is one row per serial number. On re-import, the `INSERT ... ON CONFLICT (serial_number)` updates only when:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
WHERE test_records.overall_result = 'FAIL'
|
||||||
|
OR (EXCLUDED.overall_result = 'PASS' AND EXCLUDED.test_date > test_records.test_date)
|
||||||
|
```
|
||||||
|
|
||||||
|
The date comparison is **strictly greater**, and the `.DAT` serial/date line carries **date only** (no time). So when a unit is tested two or more times on the **same date**, the first same-day run to be imported wins and no later same-day run can replace it. The DB — and therefore the website datasheet — can show a **non-final** run.
|
||||||
|
|
||||||
|
This is the documented audit failure mode: same-day runs are usually trim / re-test iterations, and the **last** run is the accepted certificate result.
|
||||||
|
|
||||||
|
## Exposure (whole-source sweep, 2026-06-17)
|
||||||
|
|
||||||
|
981,716 records parsed across 26,815 `.DAT` files (406,549 serials):
|
||||||
|
|
||||||
|
- Same-day multi-run events (distinct values): **6,515** across **5,977 serials**
|
||||||
|
- DB already on the latest same-day run: 3,803
|
||||||
|
- Superseded by a later-date retest (fine): 984
|
||||||
|
- **DB on a non-latest run (the defect): 311**
|
||||||
|
- Serial absent from DB (collisions/completeness): 1,417
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
|
||||||
|
1. **Strictly-greater date** (`>`) in the conflict `WHERE` — rejects all same-date updates.
|
||||||
|
2. **Date-only granularity** — no intra-day timestamp in the `.DAT` to order same-day runs.
|
||||||
|
|
||||||
|
## Proposed fix (minimal, guarded)
|
||||||
|
|
||||||
|
Allow a same-date PASS to overwrite **only when the data actually differs**, so the last differing same-day run processed wins (imports run in chronological append order, and the live station logs are scanned last — so the last-processed run is the latest):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
ON CONFLICT (serial_number) DO UPDATE SET
|
||||||
|
log_type = EXCLUDED.log_type,
|
||||||
|
model_number = EXCLUDED.model_number,
|
||||||
|
test_date = EXCLUDED.test_date,
|
||||||
|
test_station = EXCLUDED.test_station,
|
||||||
|
overall_result = EXCLUDED.overall_result,
|
||||||
|
raw_data = EXCLUDED.raw_data,
|
||||||
|
source_file = EXCLUDED.source_file,
|
||||||
|
api_uploaded_at = NULL,
|
||||||
|
forweb_exported_at = NULL
|
||||||
|
WHERE test_records.overall_result = 'FAIL'
|
||||||
|
OR (EXCLUDED.overall_result = 'PASS' AND EXCLUDED.test_date > test_records.test_date)
|
||||||
|
OR (EXCLUDED.overall_result = 'PASS' AND EXCLUDED.test_date = test_records.test_date
|
||||||
|
AND EXCLUDED.raw_data IS DISTINCT FROM test_records.raw_data) -- NEW: latest same-day run wins
|
||||||
|
```
|
||||||
|
|
||||||
|
The added clause only fires on a genuine same-date data change, so identical re-imports do **not** needlessly clear `api_uploaded_at` (avoids re-push churn).
|
||||||
|
|
||||||
|
### Behavior after fix
|
||||||
|
|
||||||
|
| Existing | Incoming | Before | After |
|
||||||
|
|---|---|---|---|
|
||||||
|
| PASS date D | PASS date D, different data | ignored (stale) | **updated → latest run** |
|
||||||
|
| PASS date D | PASS date D, identical | ignored | ignored (no churn) |
|
||||||
|
| PASS date D | PASS date D+1 | updated | updated (unchanged) |
|
||||||
|
| PASS date D+1 | PASS date D | ignored | ignored (unchanged) |
|
||||||
|
| FAIL | PASS (any date) | updated | updated (unchanged) |
|
||||||
|
|
||||||
|
## Caveats / assumptions
|
||||||
|
|
||||||
|
- **Relies on chronological append order** within a `.DAT` and on the live station logs being scanned **last** (they are: `runImport` does HISTLOGS → Recovery → station `TEST_PATH`). If a serial's latest run existed only in HISTLOGS (scanned first) and an older copy in a station log (scanned last), the older copy would win. Rare, but possible. For a hard guarantee, add a monotonic tiebreaker (ingest sequence, or a per-run timestamp if the test program can emit one) — a larger change.
|
||||||
|
- **Re-push impact:** the 311 corrected rows (plus any future same-day retests) will clear `api_uploaded_at` and re-upload to Hoffman on the next run. Expected and desired (the website gets the final result), but it is outbound API traffic — run deliberately.
|
||||||
|
- **Does NOT fix** generic reused serials (`1-1`, `1-2`, …) that collide across different products, nor the 608 units absent from the DB. Those are separate items (serial-uniqueness model / ingestion completeness).
|
||||||
|
|
||||||
|
## Stronger alternative (larger migration)
|
||||||
|
|
||||||
|
If full per-run archival is required (every test sheet reproducible), replace the `UNIQUE (serial_number)` model with a composite key **`(serial_number, test_date, run_sequence)`** (or store all runs and select the latest at render time). This preserves every run and removes the same-day ambiguity entirely, but is a schema migration + dedupe + render/upload changes — propose separately if desired.
|
||||||
|
|
||||||
|
## Rollout (after approval)
|
||||||
|
|
||||||
|
1. Apply the `WHERE`-clause change to `database/import.js` (repo copy first, review, then deploy).
|
||||||
|
2. Re-run the import so the 311 same-day cases settle on the latest run.
|
||||||
|
3. Let the upload path re-push the cleared rows; confirm counts.
|
||||||
|
4. Re-run `tools/validate-parsing.js` to confirm same-day violations drop to ~0.
|
||||||
285
projects/dataforth-dos/DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md
Normal file
285
projects/dataforth-dos/DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md
Normal file
@@ -0,0 +1,285 @@
|
|||||||
|
# Test-Datasheet Bug — End-to-End Trace & Diagnosis
|
||||||
|
|
||||||
|
**Date:** 2026-06-17
|
||||||
|
**Host:** AD2 (192.168.0.6) — testdatadb generator + PostgreSQL 18
|
||||||
|
**Author:** Mike Swanson / AZ Computer Guru
|
||||||
|
**Status:** DIAGNOSIS ONLY — no code or DB changes made
|
||||||
|
**Reported by:** John Lehman / Peter Iliya (Dataforth) — customer Wellbore Integrity (Joseph Swinehart) cal-cert audit on 8B35 4-wire RTD certs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. TL;DR
|
||||||
|
|
||||||
|
There are **two independent defects, both in the datasheet *renderer*** (`templates/datasheet-exact.js`). Ingestion (.DAT parsing), the database contents, and the spec files are all **correct** — the raw test data in the DB holds the right values; the renderer mislabels and mis-maps them.
|
||||||
|
|
||||||
|
| # | Defect | Symptom on cert | Scope | Severity |
|
||||||
|
|---|--------|-----------------|-------|----------|
|
||||||
|
| **A** | RTD input column rendered as **resistance** instead of **temperature** | Header reads `Rin (ohms)`, should read `Temp. (C)`; positive input values lost their leading `+` | ~24,000 RTD certs (8B35, DSCA34, SCM5B34/35, and any RTD variant) | **HIGH** — this is the audit finding |
|
||||||
|
| **B** | **Entire DSCA Final-Test parameter list is wrong** | Wrong parameter names, garbage specs (`< 0 mA`, `+/- 0 %`), values aligned to the wrong rows, output column mislabeled (`Vout (V)` vs `Output (mA)`), lines missing/added | up to 78,343 DSCA certs (all DSCLOG) | **HIGH** |
|
||||||
|
|
||||||
|
Defect A is a small, surgical fix. Defect B requires rebuilding the DSCA template against the legacy spec (`DSCFIN.DAT`) and is a larger effort.
|
||||||
|
|
||||||
|
**Root cause is confirmed against the original ground-truth files** (the DOS-station-generated staged `.TXT`), not assumptions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The Original File IS the Ground Truth — and We Found It
|
||||||
|
|
||||||
|
Mike's key point was correct: **the rendered datasheet exists as a file BEFORE it ever reaches the database.** That original file is produced by the DOS test station itself and is the source of truth. Our DB-based regeneration is what introduces the error.
|
||||||
|
|
||||||
|
### Where the original files physically live
|
||||||
|
|
||||||
|
The DOS station's QuickBASIC ATE program writes a fully-rendered `.TXT` datasheet to `C:\STAGE\` on the station, then `CTONWTXT.BAT` uploads it to the NAS `STAGE` share. It is mirrored to AD2:
|
||||||
|
|
||||||
|
```
|
||||||
|
ORIGINAL (ground truth) staged datasheets:
|
||||||
|
AD2: C:\Shares\test\STAGE\<TS-station>\<encoded-SN>.TXT
|
||||||
|
NAS: /data/test/STAGE/<TS-station>/<encoded-SN>.TXT (\\192.168.0.9\test\STAGE)
|
||||||
|
```
|
||||||
|
|
||||||
|
Filenames use the 8.3 hex-prefix serial encoding (first two digits → letter, `55 + n`):
|
||||||
|
`179553-13` → `17`→`H` → **`H9553-13.TXT`**; `180224-7` → `18`→`I` → **`I0224-7.TXT`**.
|
||||||
|
|
||||||
|
**Confirmed files for this investigation:**
|
||||||
|
|
||||||
|
| Module | SN | Original staged file (ground truth) | Source `.DAT` |
|
||||||
|
|--------|----|--------------------------------------|---------------|
|
||||||
|
| 8B35-04 (4-wire RTD) | 179553-13 | `C:\Shares\test\STAGE\TS-4L\H9553-13.TXT` | `C:\Shares\test\TS-4L\LOGS\8BLOG\35-04.DAT` |
|
||||||
|
| DSCA38-05 (full bridge) | 180224-7 | `C:\Shares\test\STAGE\TS-11R\I0224-7.TXT` | `C:\Shares\test\TS-11R\LOGS\DSCLOG\38-05.DAT` |
|
||||||
|
| DSCA34-05C (3-wire RTD) | 180007-8 | `C:\Shares\test\STAGE\TS-4R\I0007-8.TXT` | `C:\Shares\test\TS-4R\LOGS\DSCLOG\34-05C.DAT` |
|
||||||
|
|
||||||
|
> Note re Peter: `179553-13` cannot be found on X: / DFWDS / For_Web because the **website copy is regenerated from the DB at upload time** — it is never written to disk. The on-disk ground truth is the **STAGE** copy above, not For_Web. (For_Web on AD2 holds only ~7,500 legacy files and does not contain this SN.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. End-to-End Pipeline (upstream emphasis)
|
||||||
|
|
||||||
|
```
|
||||||
|
[1] DOS Test Station (TS-xx, DOS 6.22, QuickBASIC ATE)
|
||||||
|
- Runs the unit test, measures everything.
|
||||||
|
- Writes TWO artifacts:
|
||||||
|
(a) C:\ATE\...\<model>.DAT raw CSV-ish multi-line test log (-> network LOGS)
|
||||||
|
(b) C:\STAGE\<encSN>.TXT FULLY RENDERED datasheet <-- GROUND TRUTH
|
||||||
|
- The station ALREADY knows the sensor type, so it prints the correct
|
||||||
|
input column ("Temp. (C)" for RTD) and the correct Final-Test parameter
|
||||||
|
list for that exact model. This is the format we must reproduce.
|
||||||
|
|
||||||
|
[2] Boot upload (CTONW.BAT / CTONWTXT.BAT)
|
||||||
|
- .DAT -> \\NAS\test\<TS>\LOGS\<logtype>\<model>.DAT
|
||||||
|
- .TXT -> \\NAS\test\STAGE\<TS>\<encSN>.TXT
|
||||||
|
|
||||||
|
[3] NAS <-> AD2 sync (Sync-FromNAS.ps1, 15 min)
|
||||||
|
- Mirrors to C:\Shares\test\... on AD2.
|
||||||
|
|
||||||
|
[4] testdatadb ingest (THIS host) ----- the original .TXT is IGNORED here -----
|
||||||
|
- import.js scans .DAT files (NOT the staged .TXT).
|
||||||
|
- parsers/multiline.js parses the .DAT into a record:
|
||||||
|
{ log_type, model_number, serial_number, test_date, test_station,
|
||||||
|
overall_result, raw_data (the verbatim .DAT block), source_file }
|
||||||
|
- INSERT ... ON CONFLICT(serial_number) into PostgreSQL test_records.
|
||||||
|
raw_data = the exact .DAT text block (this is faithful & correct).
|
||||||
|
|
||||||
|
[5] Render + upload (the bug lives here)
|
||||||
|
- render-datasheet.js -> templates/datasheet-exact.js regenerates the
|
||||||
|
datasheet text FROM raw_data + spec files, in memory.
|
||||||
|
- upload-to-api.js POSTs {SerialNumber, Content} to Hoffman bulk API.
|
||||||
|
- Hoffman serves it on the public product page.
|
||||||
|
```
|
||||||
|
|
||||||
|
**The pivotal architectural fact:** step [4] throws away the already-correct rendered `.TXT` and keeps only the raw `.DAT` block (`raw_data`). Step [5] then *re-renders* from scratch. Every rendering defect is introduced in step [5]; the upstream data is fine.
|
||||||
|
|
||||||
|
### 2.1 What the `.DAT` / `raw_data` actually contains (8B35-04, SN 179553-13)
|
||||||
|
|
||||||
|
```
|
||||||
|
"8B35-04 " <- model
|
||||||
|
-1.461694,-1.218078E-02,-.014174,-3.986431E-02,"PASS" <- accuracy pt 1: stim,calc,meas,err,status
|
||||||
|
151.5394,1.262828,1.26273,-1.966953E-03,"PASS" <- pt 2
|
||||||
|
303.6477,2.530397,2.531,1.204967E-02,"PASS" <- pt 3
|
||||||
|
448.7633,3.739694,3.7414,.0341177,"PASS" <- pt 4
|
||||||
|
598.0475,4.983729,4.9824,-2.658844E-02,"PASS" <- pt 5
|
||||||
|
"0","0",0 <- step-response placeholder
|
||||||
|
"PASS 28.424741","PASS","PASS 252.21681","PASS","PASS" <- final-test STATUS groups (5 per line)
|
||||||
|
"PASS","","PASS","PASS","PASS"
|
||||||
|
"PASS","PASS 3.478871E-023","PASS 3.986431E-023","PASS","PASS 26.328911"
|
||||||
|
"PASS","PASS","PASS 40.429370","PASS","PASS 140.50"
|
||||||
|
```
|
||||||
|
|
||||||
|
The **first column of each accuracy point is the stimulus**. For SN 179553-13 it is `-1.46, 151.5, 303.6, 448.8, 598.0` — clearly **temperatures in °C** (model MAXIN = 600 °C), **not** ohms. The spec record confirms `SENTYPE = "P1RTD4W"`, `MAXIN = 600`. So the DB has the right numbers and the right sensor type. Nothing upstream is wrong.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Diffs — Original (correct) vs DB-Generated (wrong)
|
||||||
|
|
||||||
|
### 3.1 8B35-04 RTD (SN 179553-13) — Defect A only
|
||||||
|
|
||||||
|
ACCURACY block header + first/last rows:
|
||||||
|
|
||||||
|
```
|
||||||
|
ORIGINAL (H9553-13.TXT, ground truth) GENERATED (current testdatadb)
|
||||||
|
----------------------------------- ------------------------------
|
||||||
|
Temp. (C) Vout (V) ... Rin (ohms) Vout (V) ... <-- WRONG LABEL
|
||||||
|
-1.46 ... -1.46 ... (same)
|
||||||
|
+151.54 ... 151.54 ... <-- lost '+'
|
||||||
|
+598.05 ... 598.05 ... <-- lost '+'
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Header:** `Temp. (C)` → rendered as `Rin (ohms)`. **This is the audit discrepancy.**
|
||||||
|
- **Values:** numerically identical (correct temperatures). Only difference: positive values lose the leading `+` because the resistance formatter omits the sign.
|
||||||
|
- **Final Test Results: byte-for-byte IDENTICAL** to the original (7 lines: Supply Current Nom, Exc. Current #1, Linearity, Accuracy, Supply Sensitivity, Frequency Response, Output Noise). **No missing lines for 8B35.** The 8B/5B Final-Test rendering is correct.
|
||||||
|
|
||||||
|
So for the Wellbore Integrity audit, the **only** defect on the 8B35 cert is the input column header label (and the cosmetic `+` sign). The measured data is right.
|
||||||
|
|
||||||
|
### 3.2 DSCA38-05 full-bridge (SN 180224-7) — Defect B
|
||||||
|
|
||||||
|
This is not a label tweak — the **whole Final Test table is wrong**:
|
||||||
|
|
||||||
|
```
|
||||||
|
ORIGINAL (I0224-7.TXT) GENERATED (current)
|
||||||
|
Supply Current 23.8 mA < 30 mA Supply Current, Nom 23.8 mA < 0 mA <- spec garbage
|
||||||
|
Supply Curr. w/ EXC Load 53.8 mA < 80 mA Supply Current @ Max Load 53.8 mA < 0 mA
|
||||||
|
Excitation Voltage 10.000 V 10+/-.003V Linearity, 50mA Load 10.000 % +/- 0 % <- wrong name+unit
|
||||||
|
Exc. Load Regulation -6 ppm/mA ... Accuracy, 50mA Load 6 % +/- 0 % <- wrong row
|
||||||
|
Output Reg. w/ EXC Load 0.00 % +/- .05 % Positive Current Limit 0.0 mA < 0 mA
|
||||||
|
Excitation Current Limit 54 mA < 65 mA Negative Current Limit 54 mA > 0 mA
|
||||||
|
Linearity 0.002 % +/- .02 % Overrange 0.002 % > 0 %
|
||||||
|
Accuracy -0.008 % +/- .05 % Power Supply Sensitivity 0.008 %/% +/-.0006
|
||||||
|
Power Supply Sens. 0.0000 %/% +/-.0006 Frequency Response 0.0000 dB 25+/-5 dB <- value=0
|
||||||
|
Frequency Response 25.0 dB 25+/-5 dB Compliance 25.0 % +/- 0 % <- 25.0 is freq!
|
||||||
|
Output Noise 1205 uVrms <=2000 (Output Noise line dropped entirely)
|
||||||
|
```
|
||||||
|
|
||||||
|
Also in the ACCURACY block: original column titles are `Output (V)` and separator dashes `----------`; the generator emits `Vout (V)` and `==========`. The input header happens to read `Vin (mV)` in both (bridge module), so the bridge input label is OK, but everything below it is misaligned.
|
||||||
|
|
||||||
|
### 3.3 DSCA34-05C 3-wire RTD (SN 180007-8) — Defect A **and** B together
|
||||||
|
|
||||||
|
```
|
||||||
|
ORIGINAL (I0007-8.TXT) GENERATED (current)
|
||||||
|
Temp. (C) Output (mA) ... Rin (ohms) Vout (V) ... <-- A: label + wrong out-unit
|
||||||
|
Supply Current 50.7 mA < 65 mA Supply Current, Nom 50.7 mA < 0 mA
|
||||||
|
Exc. Current @ -f.s. 264.0 uA 261 uA Linearity, 0mA Load 264.0 % +/- 0 % <-- 264.0 is uA, not %
|
||||||
|
Exc. Current @ +f.s. 281.0 uA 278 uA Accuracy, 0mA Load 281.0 % +/- 0 %
|
||||||
|
Linearity 0.017 % +/-.03% Overrange 0.017 % > 0 %
|
||||||
|
Accuracy -0.034 % +/-.05% Power Supply Sens. 0.034 %/% +/-.0005
|
||||||
|
... (9 real params) ... (wrong names, garbage specs, lines dropped)
|
||||||
|
```
|
||||||
|
|
||||||
|
The excitation currents (264.0 uA, 281.0 uA) get printed under the labels "Linearity, 0mA Load" / "Accuracy, 0mA Load" as **percentages** — visibly nonsensical.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Root-Cause Localization
|
||||||
|
|
||||||
|
Tested against the original files, the defect is **(c) the renderer** — `templates/datasheet-exact.js`. Ingestion/parsing (a) and the DB data (b) are correct.
|
||||||
|
|
||||||
|
### Defect A — RTD treated as resistance
|
||||||
|
|
||||||
|
```js
|
||||||
|
// getSensorNum(): RTD maps to 7
|
||||||
|
if (s.includes('RTD')) return 7; // line ~150
|
||||||
|
|
||||||
|
// Accuracy input-column header (generateExactDatasheet):
|
||||||
|
} else if (sensorNum === 7) {
|
||||||
|
inputHeader = ' Rin (ohms)'; // line ~564 <-- WRONG for Dataforth RTD
|
||||||
|
}
|
||||||
|
|
||||||
|
// Accuracy value formatting (formatAccuracyLine):
|
||||||
|
} else if (sensorNum === 7) {
|
||||||
|
stimStr = point.stim.toFixed(2).padStart(8); // line ~447 <-- resistance format, no sign
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Dataforth RTD modules always express the input as **temperature** on the datasheet (the RTD curve converts resistance→°C; the `.DAT` already stores °C). `sensorNum === 7` is reached **only** by RTD sentypes (`P1RTD3W`, `P1RTD4W`, `NIRTD3W`, …). There is currently **no module for which `Rin (ohms)` is correct** — true resistance/potentiometer inputs are not routed to 7 (they fall through to the voltage default). So fixing the `7` branch is safe and will not over-correct.
|
||||||
|
|
||||||
|
### Defect B — DSCA parameter list mismatch
|
||||||
|
|
||||||
|
`DATA_LINES['DSCA']` is a **single hardcoded list** (Supply Current Nom / @ Max Load / Linearity 0mA / Accuracy 0mA / Linearity 5mA / … / Compliance / Accuracy @ 5 ohm). Real DSCA modules use **different Final-Test layouts per subtype** (bridge/excitation modules list Excitation Voltage / Exc. Load Reg. / Output Reg.; RTD/TC list Exc. Current @ ±f.s.; etc.). The hardcoded list does not match, so:
|
||||||
|
|
||||||
|
- `raw_data` STATUS groups are mapped positionally onto the **wrong** parameter names;
|
||||||
|
- `buildTSpecs()` for DSCA reads spec fields (`ILIMIT`, `PERCOVER`, `COMPLIANCE`, `ACCURACY1/2/3`, `LINEAR1/2/3`) that are zero/absent for these modules → specs print as `< 0`, `+/- 0`;
|
||||||
|
- the skip rule `if (status.length <= 4) continue` drops every bare-`PASS` slot, but because the list is misaligned the *wrong* lines drop and the survivors land on wrong rows;
|
||||||
|
- the ACCURACY block also uses the 5B/8B titles (`Vout (V)` / `==========`) instead of DSCA's (`Output (V|mA)` / `----------`).
|
||||||
|
|
||||||
|
The "missing Final Test lines" complaint is a **symptom of Defect B** (DSCA misalignment + skip rule), **not** a separate bug. The 8B/5B skip rule is correct — the legacy station also prints only tested parameters (verified: 8B35 matches exactly).
|
||||||
|
|
||||||
|
> Clarification for the thread: **DSCA38 is a bridge/strain-gauge module (FBRIDGE/HBRIDGE), not an RTD.** The DSCA RTD analog is **DSCA34**. So DSCA38's problem is Defect B; DSCA34's problem is A+B. If the audit specifically concerns RTD resistance-vs-temperature, the DSCA part to verify with the customer is **DSCA34**, while DSCA38 demonstrates the broader DSCA table breakage.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Correct Output & Proposed Fix
|
||||||
|
|
||||||
|
### 5.1 Correct RTD output (Defect A)
|
||||||
|
|
||||||
|
Per the originals, RTD modules must render:
|
||||||
|
- **Input column header:** `Temp. (C)` (same column/format as thermocouples, `sensorNum` 3–6).
|
||||||
|
- **Input values:** the stimulus value straight from `raw_data` (already °C — **no conversion needed**), signed format (`+598.05`, `-1.46`).
|
||||||
|
|
||||||
|
Surgical change in `templates/datasheet-exact.js` (fold RTD into the temperature path):
|
||||||
|
|
||||||
|
```js
|
||||||
|
// (1) Header — replace the sensorNum===7 branch:
|
||||||
|
if ((sensorNum >= 3 && sensorNum <= 6) || sensorNum === 7) {
|
||||||
|
inputHeader = ' Temp. (C)';
|
||||||
|
} else if (sensorNum === 2 || sensorNum === 9) {
|
||||||
|
inputHeader = ' Iin (mA)';
|
||||||
|
} else {
|
||||||
|
inputHeader = (maxIn != null && maxIn < 1) ? ' Vin (mV)' : ' Vin (V)';
|
||||||
|
}
|
||||||
|
|
||||||
|
// (2) Value format — in formatAccuracyLine, treat 7 like 3–6:
|
||||||
|
if ((sensorNum >= 3 && sensorNum <= 6) || sensorNum === 7) {
|
||||||
|
stimStr = formatSigned(point.stim, 2, 8); // temperature, signed
|
||||||
|
} else { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
Leave the existing `i===13 && sensorNum===7 → 'ohm/ohm'` unit override (Lead-R-Effect) as-is; it is a separate, correct detail. **Verify exact leading-space alignment** against a known-good thermocouple original before pushing (the header column should byte-match; this is the same polish already in progress in `generated-v2-*.TXT`).
|
||||||
|
|
||||||
|
This fix corrects **header + values** for all RTD modules (8B35, DSCA34, SCM5B34/35, etc.) at once. **Do not** add any resistance→temperature math — the data is already temperature.
|
||||||
|
|
||||||
|
### 5.2 DSCA template (Defect B)
|
||||||
|
|
||||||
|
Not a one-liner. The fix is to drive the DSCA Final-Test parameter list (and ACCURACY column titles/units) **per module subtype**, matching the legacy QuickBASIC DSC writer. The legacy parameter selection lives in **`specdata\DSCFIN.DAT`** (DSC final-test definitions) alongside `DSCMAIN4.DAT`/`DSCOUT.DAT`. Recommended approach:
|
||||||
|
1. Reverse the DSCFIN.DAT layout (or read the QB DSC datasheet source) to get the per-subtype parameter name/unit/spec list.
|
||||||
|
2. Replace the single `DATA_LINES['DSCA']` + DSCA branch of `buildTSpecs()` with subtype-aware selection keyed on SENTYPE / output-signal type.
|
||||||
|
3. Fix the DSCA ACCURACY block to use `Output (V|mA)` (per `OUTSIGTYPE`) and dash separators.
|
||||||
|
4. Validate byte-for-byte against staged originals across DSCA subtypes (bridge, RTD, TC, current-out, voltage-out).
|
||||||
|
|
||||||
|
Until B is fixed, **all DSCA (DSCLOG) website datasheets should be treated as unreliable** for Final-Test content.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Impact (records currently on the website)
|
||||||
|
|
||||||
|
| Group | On-web count | Defect |
|
||||||
|
|-------|-------------:|--------|
|
||||||
|
| 8B35* | 5,476 | A |
|
||||||
|
| DSCA34* (RTD) | 3,573 | A + B |
|
||||||
|
| SCM5B34/35* (RTD) | 14,887 | A |
|
||||||
|
| All DSCA (DSCLOG) | 78,343 | B (RTD subset also A) |
|
||||||
|
| **Total on website** | 464,671 | — |
|
||||||
|
|
||||||
|
RTD-label exposure (Defect A) ≈ **24,000 certs**; DSCA table exposure (Defect B) ≈ **78,000 certs**.
|
||||||
|
|
||||||
|
After fixes are reviewed and deployed, affected records can be re-pushed by clearing `api_uploaded_at` for the affected models and letting the upload path re-render (RE-PUSH is idempotent; Hoffman returns `Unchanged` when content matches).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. How to Reproduce / Verify (read-only)
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# Render current generator output for a SN and compare to the staged original:
|
||||||
|
cd C:\Shares\testdatadb
|
||||||
|
node -e "const db=require('./database/db');const {renderContent}=require('./database/render-datasheet');(async()=>{const r=await db.queryOne('SELECT * FROM test_records WHERE serial_number=$1',['179553-13']);if(r.test_date&&r.test_date.toISOString)r.test_date=r.test_date.toISOString().slice(0,10);console.log(renderContent(r));await db.close();})()"
|
||||||
|
|
||||||
|
# Ground truth:
|
||||||
|
type C:\Shares\test\STAGE\TS-4L\H9553-13.TXT
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Files Referenced
|
||||||
|
|
||||||
|
- Generator: `C:\Shares\testdatadb\templates\datasheet-exact.js` (repo: `projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js`)
|
||||||
|
- Render glue: `C:\Shares\testdatadb\database\render-datasheet.js`
|
||||||
|
- Specs: `C:\Shares\testdatadb\parsers\spec-reader.js`, `C:\Shares\testdatadb\specdata\*.DAT` (incl. `DSCFIN.DAT`)
|
||||||
|
- Ingest: `C:\Shares\testdatadb\database\import.js`, `C:\Shares\testdatadb\parsers\multiline.js`
|
||||||
|
- Ground-truth originals: `C:\Shares\test\STAGE\<TS>\<encSN>.TXT`
|
||||||
58
projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt
Normal file
58
projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
Fix 2 STAGE 3 — DSCA Final-Test render vs staged-original content validation
|
||||||
|
GATE = FINAL TEST RESULTS section, content-strict (rule lines canonicalized,
|
||||||
|
whitespace collapsed). Accuracy-section diffs reported separately (deferred
|
||||||
|
cosmetic spacing + any pre-existing calc rounding — NOT a Fix 2 gate).
|
||||||
|
Corpus: 2806 staged DSCA originals across 126 models.
|
||||||
|
==============================================================================
|
||||||
|
|
||||||
|
SUMMARY
|
||||||
|
models with staged originals: 126
|
||||||
|
models FINAL-TEST CLEAN (>=1 compared, 0 mismatch): 92
|
||||||
|
models with FINAL-TEST mismatches: 6
|
||||||
|
certs compared: 2450
|
||||||
|
Final-Test match: 2412
|
||||||
|
Final-Test mismatch: 38
|
||||||
|
(certs with accuracy-section diffs: 1369 — informational)
|
||||||
|
staged serials not in DB: 40
|
||||||
|
in DB but not rendered (skipped/null): 316
|
||||||
|
|
||||||
|
MODELS WITH FINAL-TEST CONTENT MISMATCHES (investigate before re-push):
|
||||||
|
DSCA38-05 compared=129 ftMatch=97 ftMismatch=32
|
||||||
|
[finaltest L3] SN 180257-1
|
||||||
|
render: "Supply Current 19.8 mA < 30 mA PASS"
|
||||||
|
golden: "Supply Current 20.0 mA < 30 mA PASS"
|
||||||
|
[finaltest L3] SN 180257-10
|
||||||
|
render: "Supply Current 19.6 mA < 30 mA PASS"
|
||||||
|
golden: "Supply Current 20.3 mA < 30 mA PASS"
|
||||||
|
[finaltest L3] SN 180257-11
|
||||||
|
render: "Supply Current 20.1 mA < 30 mA PASS"
|
||||||
|
golden: "Supply Current 19.8 mA < 30 mA PASS"
|
||||||
|
DSCA38-1793 compared=33 ftMatch=32 ftMismatch=1
|
||||||
|
[finaltest L3] SN 176923-11
|
||||||
|
render: "Supply Current 22.3 mA < 25 mA PASS"
|
||||||
|
golden: "Supply Current 23.1 mA < 25 mA PASS"
|
||||||
|
DSCA38-19C compared=248 ftMatch=247 ftMismatch=1
|
||||||
|
[finaltest L4] SN 180036-8
|
||||||
|
render: "Supply Curr. w/ EXC Load 54.5 mA < 100 mA PASS"
|
||||||
|
golden: "Supply Curr. w/ EXC Load 55.0 mA < 100 mA PASS"
|
||||||
|
DSCA38-19E compared=28 ftMatch=27 ftMismatch=1
|
||||||
|
[finaltest L3] SN 179698-16
|
||||||
|
render: "Supply Current 42.7 mA < 60 mA PASS"
|
||||||
|
golden: "Supply Current 45.7 mA < 60 mA PASS"
|
||||||
|
DSCA39-05 compared=17 ftMatch=16 ftMismatch=1
|
||||||
|
[finaltest L3] SN A276-2
|
||||||
|
render: "Supply Current, Nom 61.7 mA < 75 mA PASS"
|
||||||
|
golden: "Supply Current, Nom 60.0 mA < 75 mA PASS"
|
||||||
|
DSCA39-1950 compared=17 ftMatch=15 ftMismatch=2
|
||||||
|
[finaltest L4] SN 178236-12
|
||||||
|
render: "Linearity 0.013 % +/- .05 % PASS"
|
||||||
|
golden: "Linearity 0.009 % +/- .05 % PASS"
|
||||||
|
[finaltest L4] SN 178236-13
|
||||||
|
render: "Linearity 0.017 % +/- .05 % PASS"
|
||||||
|
golden: "Linearity 0.012 % +/- .05 % PASS"
|
||||||
|
|
||||||
|
FINAL-TEST CLEAN MODELS (92):
|
||||||
|
DSCA30-01, DSCA30-02, DSCA30-03, DSCA30-06, DSCA30-07, DSCA30-08, DSCA30-08C, DSCA30-09, DSCA30-09C, DSCA30-1944, DSCA30-1945, DSCA30-1946, DSCA31-02, DSCA31-03, DSCA31-06, DSCA31-07, DSCA31-11, DSCA31-12, DSCA31-1273, DSCA31-12C, DSCA31-13, DSCA31-13C, DSCA31-15, DSCA31-1918, DSCA32-01, DSCA32-01C, DSCA32-01E, DSCA34-01, DSCA34-02C, DSCA34-04, DSCA34-04C, DSCA34-05, DSCA34-05C, DSCA34-1858, DSCA36-01, DSCA36-02, DSCA36-03, DSCA36-04, DSCA36-04C, DSCA36-1949, DSCA38-02, DSCA38-03, DSCA38-07, DSCA38-08C, DSCA38-09, DSCA38-09E, DSCA38-12C, DSCA38-12E, DSCA38-1468, DSCA38-1544, DSCA38-15C, DSCA38-16, DSCA38-16C, DSCA38-18C, DSCA38-19, DSCA39-01, DSCA39-02, DSCA39-07, DSCA40-03, DSCA40-05, DSCA40-05C, DSCA40-06, DSCA40-1951, DSCA40-1952, DSCA41-01, DSCA41-02, DSCA41-03, DSCA41-05C, DSCA41-06, DSCA41-09, DSCA41-13, DSCA41-14, DSCA41-15, DSCA41-15E, DSCA42-01, DSCA42-01C, DSCA42-02, DSCA43-10, DSCA43-20E, DSCA47E-08C, DSCA47J-01C, DSCA47J-03, DSCA47K-05, DSCA47K-13, DSCA47K-14, DSCA47N-15, DSCA47T-06, DSCA47T-1928, DSCA49-04, DSCA49-05, DSCA49-1601, DSCA49-1895
|
||||||
|
|
||||||
|
MODELS WITH NO COMPARABLE CERT (no staged serial in DB, or all skipped/null):
|
||||||
|
DSCA31-1947(null), DSCA33-01(null), DSCA33-01A(null), DSCA33-02(null), DSCA33-02C(null), DSCA33-03(null), DSCA33-03A(null), DSCA33-03C(null), DSCA33-04(null), DSCA33-04C(null), DSCA33-05(null), DSCA33-05C(null), DSCA33-07C(null), DSCA33-1917(null), DSCA33-1919(null), DSCA33-1948(null), DSCA45-01(null), DSCA45-01C(null), DSCA45-02(null), DSCA45-03(null), DSCA45-03C(null), DSCA45-04(null), DSCA45-04C(null), DSCA45-05C(null), DSCA45-06(null), DSCA45-07(null), DSCA45-08(null), DSCA47N-15C(null)
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Email draft — to John Lehman, 2026-06-17
|
||||||
|
|
||||||
|
**To:** John Lehman <jlehman@dataforth.com>
|
||||||
|
**From:** Mike Swanson <mike@azcomputerguru.com>
|
||||||
|
**Subject:** Test Datasheet Problems — What's Going On and How We're Fixing It
|
||||||
|
**Status:** DRAFT (not yet sent)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
John,
|
||||||
|
|
||||||
|
I dug all the way into the datasheet issues you and Peter flagged (the 8B35 RTD certs the Wellbore Integrity audit caught), and I found the original problem plus a few related things worth knowing about. Good news up front: **your test data and original datasheets are accurate — nothing is corrupting your measurements.** The problems are all on our side, in the system that loads the data and re-creates the datasheets for the website, and **we'll be making the fixes** — that piece is our responsibility.
|
||||||
|
|
||||||
|
**First, how the system works (so the rest makes sense)**
|
||||||
|
|
||||||
|
When a unit is tested, the DOS test station produces **two** things:
|
||||||
|
1. The **datasheet** you're used to seeing — a text file the station writes at test time. This is the original, and it's correct.
|
||||||
|
2. A **raw data log** (the `.DAT` file) — the underlying numbers.
|
||||||
|
|
||||||
|
The website doesn't store that original datasheet. Our system loads the raw log into a database and then **rebuilds** the datasheet from the database when it's needed. That rebuild step is where the errors are. So the unit you tested is fine and the original sheet is fine — the *regenerated* copy on the website is what's off.
|
||||||
|
|
||||||
|
**Problem 1 — RTD modules say "resistance" when they should say "temperature" (this is the audit finding)**
|
||||||
|
|
||||||
|
An RTD senses temperature by changing its resistance, but on a Dataforth datasheet the input column is shown as **Temperature (°C)** — exactly what your original sheets do. Our rebuild has a bug: for RTD modules it labels that column **"Rin (ohms)"** even though it's printing the temperature numbers underneath, so the header and the values disagree. That's precisely what the auditor caught on the 8B35.
|
||||||
|
|
||||||
|
Important: on the 8B35 cert the **measured values are correct** — it's the column label that's wrong (and the positive numbers lost their leading "+"). This affects every RTD product, not just the 8B35 — roughly 24,000 certificates (8B35, DSCA34, the SCM5B34/35 family). We've already written and tested the fix against your original sheets; it produces the correct "Temp. (°C)" column with no effect on any non-RTD product.
|
||||||
|
|
||||||
|
**Problem 2 — DSCA datasheets have the wrong Final Test lines (the "missing lines / wrong headers")**
|
||||||
|
|
||||||
|
For the DSCA line, our rebuild is using the **wrong master list of test parameters** — so you get wrong parameter names, wrong spec limits, values on the wrong rows, and some lines dropping off. This is the "Final Test lines are missing" part of your report, and it affects the DSCA line broadly (up to ~78,000 certs). One clarification: DSCA38 is a bridge/strain-gauge module — its issue is this one; the DSCA *RTD* module is DSCA34, which has both this and Problem 1. This one's a real rebuild on our end (matching each DSCA sub-type to its spec file), so it'll take longer than the others.
|
||||||
|
|
||||||
|
**Problem 3 — some units show an earlier same-day test instead of the final one**
|
||||||
|
|
||||||
|
If a unit was tested more than once on the **same day** (a re-trim, say), our system can keep an *earlier* run instead of the final accepted one, so the website cert may show non-final numbers. It affects about **311 units** across your history. It matters for audits because the **last** same-day run is the one that belongs on the cert. We'll change our database rule so the latest same-day run wins.
|
||||||
|
|
||||||
|
**Problem 4 — two batches of units never made it into the system**
|
||||||
|
|
||||||
|
I also found units that were tested and have an original datasheet but **no** record in the database/website. Two separate reasons:
|
||||||
|
|
||||||
|
- **A serial-number format our importer can't read (~9,500 records).** In the DOS days, when a serial was too long for the old 8-character filename limit, the software shortened it with a letter code — e.g. `10243-1` is written as `A243-1`. Our importer only recognizes serials that start with a number, so it **silently skips** every letter-coded unit. The data is sitting right there in the logs; we just aren't reading it. Across all your logs that's about **840 units / 9,500 records / 141 models** never imported. We'll update the importer to recognize and decode those serials, which recovers the whole backlog.
|
||||||
|
|
||||||
|
- **A one-time casualty of the cryptolocker incident (~379 units).** This is *not* an ongoing gap — we import every 15 minutes and everything since has come through cleanly. Here's what happened: the DOS stations append all of a given model's results into a single, shared log file. During the incident, stations were failing to sync, and in the recovery we didn't yet realize they were appending to one filename — so for a couple of weeks, fresh logs coming off the DOS machines overwrote the accumulated server-side logs, wiping the earlier history those files held. The data backs this up: the affected units are confined to **October 2025 through January 2026** (most in December 2025) and **stop cold after January**, on three stations (TS-4L, TS-4R, TS-1R) — the exact fingerprint of a one-time overwrite during the incident, not a recurring problem. The good news: the **original datasheet text files for these units still exist** on the server, so we can backfill them from those.
|
||||||
|
|
||||||
|
**The bottom line**
|
||||||
|
|
||||||
|
- Your **test data and original datasheets are intact and accurate** — I verified the database matches the originals to the number across ~11,000 units, with zero cases of data being read in wrong.
|
||||||
|
- Everything above is in **our** load-and-rebuild software, or was a one-time incident casualty — not a measurement or test problem.
|
||||||
|
- For the **audit specifically**: the 8B35 measured values are right; it's the column heading, and that fix is ready to go.
|
||||||
|
|
||||||
|
**What we're doing next**
|
||||||
|
|
||||||
|
We'll tackle them in this order unless you'd rather reprioritize:
|
||||||
|
1. RTD column-label fix — clears the audit issue (ready now).
|
||||||
|
2. Missing-units importer fix — recovers the ~9,500-record backlog.
|
||||||
|
3. Same-day-run rule — so certs show the final run.
|
||||||
|
4. DSCA Final Test rebuild — the larger one.
|
||||||
|
5. Backfill the ~379 incident-era units from their original datasheet files.
|
||||||
|
|
||||||
|
The only thing I need from you is a thumbs-up on that priority order (and on backfilling the 379 from their original sheets). Everything else is on us.
|
||||||
|
|
||||||
|
I've documented all of this in detail and am glad to walk your team through any of it on a call.
|
||||||
|
|
||||||
|
Thanks,
|
||||||
|
Mike Swanson
|
||||||
|
AZ Computer Guru
|
||||||
|
(520) 304-8300
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Supporting detail: `DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md` (Problems 1 & 2), `PARSING-FIDELITY-VERDICT-2026-06-17.md` + `CONFLICT-RULE-FIX-PROPOSAL-2026-06-17.md` (Problem 3), `MISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md` (Problem 4).*
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
# Test Datasheets Missing From the Database/Website — Findings
|
||||||
|
|
||||||
|
**To:** John Lehman (Engineering)
|
||||||
|
**From:** Mike Swanson, AZ Computer Guru
|
||||||
|
**Date:** 2026-06-17
|
||||||
|
**Scope:** Why some tested units have a staged datasheet but no record in testdatadb / on the website.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
I cross-checked **all 11,921 staged datasheet files** (the `.TXT` the test stations produce) against the database. **608 had no matching database record.** They fall into two distinct causes:
|
||||||
|
|
||||||
|
| Cause | Units | Recoverable? |
|
||||||
|
|---|---:|---|
|
||||||
|
| **1. Encoded / non-standard serial numbers the importer skips** | **229** | **Yes** — the data exists, the importer just doesn't read it |
|
||||||
|
| **2. One-time log loss during the cryptolocker incident/recovery** | **379** | Yes — from the staged `.TXT`, which still exists on disk |
|
||||||
|
|
||||||
|
The first cause is the important one: it is a **software limitation we can fix**, and its true reach is far larger than these 608 — see "Full scope" below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cause 1 — Encoded serial numbers are silently skipped (229 units, fixable)
|
||||||
|
|
||||||
|
**What happens:** When a serial number is too long for the DOS 8.3 filename, the test program encodes the first two digits as a letter (e.g. `10243-1` is written as `A243-1`; `10` -> `A`). For these units, the serial is stored **with a leading letter** inside the log file:
|
||||||
|
|
||||||
|
```
|
||||||
|
"A243-1","01-21-2025" <- real serial 10243-1, model 5B45-25D
|
||||||
|
```
|
||||||
|
|
||||||
|
The database importer recognizes a record only when the serial **starts with a digit**. A serial that starts with a letter never matches, so the whole record is **silently dropped** — it is never imported, never rendered, never sent to the website.
|
||||||
|
|
||||||
|
**Confirmed:** the encoded serials are present in the `.DAT` logs (e.g. `5BLOG\45-25D.DAT` contains `"A243-1","01-21-2025"`), and the decoded form (`10243-1`) appears in no log and in no database row. So the data exists; the importer simply can't read it.
|
||||||
|
|
||||||
|
**Of the 608 missing, 229 are this case:**
|
||||||
|
- 212 are hex-encoded serials (all `A`-prefix, i.e. `10xxx` serials)
|
||||||
|
- 17 are other non-standard serial formats the same rule rejects (e.g. `TEST-1`, `178540-A1`, `A-1`)
|
||||||
|
- Stations: TS-11R (142), TS-11L (59), TS-8R (11), TS-8L (17)
|
||||||
|
- Dates: late 2025 through 2026
|
||||||
|
- Example models: SCM5B47K-05, SCM5B38-04, SCM5B34-01, 8B45-02, SCM5B36-02, 8B32-01, 5B45-25D, DSCA45-01
|
||||||
|
|
||||||
|
**Examples (encoded -> real serial):**
|
||||||
|
```
|
||||||
|
A243-1 -> 10243-1 5B45-25D 02-03-2026 TS-11L
|
||||||
|
A244-1 -> 10244-1 SCM5B30-02 03-26-2026 TS-11L
|
||||||
|
A276-1 -> 10276-1 DSCA39-05 05-07-2026 TS-11L
|
||||||
|
A328-1 -> 10328-1 DSCA45-08 02-17-2026 TS-11L
|
||||||
|
```
|
||||||
|
|
||||||
|
### Full scope of this bug (beyond the 608)
|
||||||
|
|
||||||
|
The 229 above are only the units that *also* still have a staged `.TXT` on disk. Scanning **every** `.DAT` log across all stations and the central history logs, the importer is dropping:
|
||||||
|
|
||||||
|
- **840 distinct encoded serial numbers**
|
||||||
|
- **9,510 individual test records**
|
||||||
|
- across **141 models**
|
||||||
|
- of which **831 of 840 serials are absent from the database**
|
||||||
|
|
||||||
|
So this single serial-format limitation is keeping on the order of **~9,500 test results out of the database and off the website.**
|
||||||
|
|
||||||
|
**Fix:** teach the importer to (a) accept a serial that starts with a letter and (b) decode it back to its real number (`A243-1` -> `10243-1`) before storing — matching how the longer serials (the `H`-prefix range) are already handled. This is a one-function change to the import parser. It would recover the 229 units here plus the ~9,500-record backlog. (I will write this up as a separate proposed change for review; no code has been changed.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cause 2 — One-time log loss during the cryptolocker incident/recovery (379 units)
|
||||||
|
|
||||||
|
These have ordinary numeric serials (no encoding issue), but their raw test data is **no longer in any log file** we import from. **This is not an ongoing gap** — the import runs every 15 minutes and everything since has come through cleanly.
|
||||||
|
|
||||||
|
**What happened:** the DOS stations append all of a given model's results into a single shared per-model `.DAT` file. During the crypto incident, stations were failing to sync, and in the recovery the appended-to-one-filename behavior wasn't yet understood — so for ~2 weeks, freshly-created logs coming off the DOS machines overwrote the accumulated server-side logs, wiping the earlier history those files held.
|
||||||
|
|
||||||
|
**Evidence (date/station fingerprint of a one-time overwrite):** the 379 affected units are confined to **2025-10 → 2026-01 and stop cold after January 2026**, concentrated on three stations:
|
||||||
|
|
||||||
|
| Test month | Units | | Station | Units |
|
||||||
|
|---|---:|---|---|---:|
|
||||||
|
| 2025-10 | 95 | | TS-4L | 185 |
|
||||||
|
| 2025-11 | 66 | | TS-4R | 171 |
|
||||||
|
| 2025-12 | 210 | | TS-1R | 23 |
|
||||||
|
| 2026-01 | 8 | | | |
|
||||||
|
|
||||||
|
**Confirmed example:** units `177097-1 ... 177097-16` (model DSCA33-05, tested 10-17-2025, TS-1R) appear in **no** log file anywhere under the test share. Their model's log (`DSCLOG\33-05.DAT`) now holds a later work order (`178644-*`, tested 02-26-2026) — the older appended history was overwritten; only the staged `.TXT` datasheet survived.
|
||||||
|
|
||||||
|
**Recovery:** the rendered datasheet `.TXT` still exists on disk for these units, so they can be backfilled directly from the `.TXT` (store the existing sheet as-is). The raw `.DAT` history is gone (the `Recovery-TEST` backup area the importer references does not exist on this server). Backfilling from the staged `.TXT` is the practical path and recovers all 379.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended actions
|
||||||
|
|
||||||
|
1. **Fix the importer's serial handling** (Cause 1) — recovers 229 staged units and ~9,500 total dropped records across 141 models. Highest value, single code change. Proposal to follow for review.
|
||||||
|
2. **Backfill the 379 incident-era units (Cause 2) from their staged `.TXT` files** — recovers the datasheets that still exist on disk.
|
||||||
|
3. **Recurrence:** Cause 2 was a one-time incident artifact (the 15-minute import has captured everything since January 2026), so no ongoing process change is required. Worth confirming the current sync no longer overwrites accumulated server-side logs with fresh DOS-side copies.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How this was determined (for the record)
|
||||||
|
|
||||||
|
- Compared every `C:\Shares\test\STAGE\**\*.TXT` against `test_records` (serial, model, date, measured results).
|
||||||
|
- For each missing unit, searched all `.DAT` sources (central HISTLOGS + every station's LOGS, 26,815 files) for the encoded and decoded serial.
|
||||||
|
- Confirmed encoded serials are present in the logs but skipped by the import regex; confirmed overwritten units are absent from all logs and their model log now holds a newer work order.
|
||||||
|
- Tools (read-only) committed under `projects/dataforth-dos/datasheet-pipeline/implementation/tools/`; raw data in `MISSING-UNITS-ROOTCAUSE-2026-06-17.txt`.
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
========== MISSING-UNITS ROOT CAUSE & SCOPE ==========
|
||||||
|
Staged .TXT with SN : 11921
|
||||||
|
Staged units MISSING from DB : 608
|
||||||
|
|
||||||
|
ROOT-CAUSE CATEGORIES (of the missing):
|
||||||
|
Parser-drop (encoded serial w/ leading letter present in .DAT, regex rejects): 212
|
||||||
|
Source .DAT has no record for this unit (data absent) : 379
|
||||||
|
Other (decoded present, still missing - investigate) : 17
|
||||||
|
|
||||||
|
|
||||||
|
by leading letter: A=212
|
||||||
|
by station : TS-11R=142, TS-11L=59, TS-8R=11
|
||||||
|
by model (top 12): SCM5B47K-05=18, SCM5B38-04=14, SCM5B34-01=13, 8B45-02=12, SCM5B36-02=11, 8B32-01=10, 5B45-25D=9, SCM5B49-05=8, 5B45-01=6, SCM5B39-05=6, DSCA45-01=5, SCM5B36-04=5
|
||||||
|
date range : 01-13-2026 .. 12-11-2025
|
||||||
|
samples :
|
||||||
|
A243-1 -> 10243-1 5B45-25D 02-03-2026 TS-11L
|
||||||
|
A243-2 -> 10243-2 5B45-25D 02-03-2026 TS-11L
|
||||||
|
A244-1 -> 10244-1 SCM5B30-02 03-26-2026 TS-11L
|
||||||
|
A255-1 -> 10255-1 5B45-25D 02-03-2026 TS-11L
|
||||||
|
A255-2 -> 10255-2 5B45-25D 02-03-2026 TS-11L
|
||||||
|
A276-1 -> 10276-1 DSCA39-05 05-07-2026 TS-11L
|
||||||
|
A276-2 -> 10276-2 DSCA39-05 05-07-2026 TS-11L
|
||||||
|
A328-1 -> 10328-1 DSCA45-08 02-17-2026 TS-11L
|
||||||
|
A328-2 -> 10328-2 DSCA45-01C 02-17-2026 TS-11L
|
||||||
|
A376-1 -> 10376-1 DSCA45-08 02-17-2026 TS-11L
|
||||||
|
A376-2 -> 10376-2 DSCA45-08 02-17-2026 TS-11L
|
||||||
|
A376-3 -> 10376-3 DSCA45-02 02-17-2026 TS-11L
|
||||||
|
|
||||||
|
DATA-ABSENT samples:
|
||||||
|
177097-1 -> 177097-1 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-10 -> 177097-10 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-11 -> 177097-11 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-12 -> 177097-12 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-13 -> 177097-13 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-14 -> 177097-14 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-16 -> 177097-16 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-2 -> 177097-2 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-3 -> 177097-3 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
177097-4 -> 177097-4 DSCA33-05 10-17-2025 TS-1R
|
||||||
|
|
||||||
|
OTHER samples:
|
||||||
|
A-1 -> A-1 SCM5B38-37 11-20-2025 TS-11R
|
||||||
|
TEST-1 -> TEST-1 SCM5B392-04 11-12-2025 TS-11R
|
||||||
|
TEST-2 -> TEST-2 SCM5B392-04 11-12-2025 TS-11R
|
||||||
|
178540-A1 -> 178540-A1 SCM5B40-03 02-26-2026 TS-8L
|
||||||
|
178540-A2 -> 178540-A2 SCM5B40-03 02-26-2026 TS-8L
|
||||||
|
178540-A3 -> 178540-A3 SCM5B40-03 02-26-2026 TS-8L
|
||||||
|
178540-A4 -> 178540-A4 SCM5B40-03 02-26-2026 TS-8L
|
||||||
|
178540-B1 -> 178540-B1 SCM5B40-03 02-26-2026 TS-8L
|
||||||
|
178540-B2 -> 178540-B2 SCM5B40-03 02-26-2026 TS-8L
|
||||||
|
178540-B3 -> 178540-B3 SCM5B40-03 02-26-2026 TS-8L
|
||||||
|
|
||||||
|
FULL .DAT BLIND SPOT (all letter-prefixed serials the importer skips, not just staged):
|
||||||
|
distinct letter-prefixed serials in .DAT : 840
|
||||||
|
total letter-prefixed records (dropped) : 9510
|
||||||
|
distinct models affected : 141
|
||||||
|
of those serials, DECODED form absent from DB: 831 / 840
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
========== PARSING FIDELITY REPORT ==========
|
||||||
|
Staged .TXT files scanned : 11922
|
||||||
|
- no SN line (non-standard fmt): 1
|
||||||
|
- SN found / compared : 11921
|
||||||
|
- .TXT w/o 5 accuracy rows : 239
|
||||||
|
Unique SNs looked up in DB : 11811
|
||||||
|
SNs present in DB : 11239
|
||||||
|
|
||||||
|
EXPLAINED (not parsing faults):
|
||||||
|
Consistent (SN+model+date+5 error% match) : 11226
|
||||||
|
Retest, DB newer date than .TXT : 35
|
||||||
|
Retest same-day (stim matches, run differs): 42
|
||||||
|
VAS/single-point fmt (no 5-row block) : 5
|
||||||
|
Serial collision (generic SN, diff family): 2
|
||||||
|
|
||||||
|
NEEDS REVIEW (potential genuine issues):
|
||||||
|
Missing from DB (after hex-decode) : 608
|
||||||
|
Model variant mismatch (same family) : 2
|
||||||
|
DB OLDER than .TXT (stale DB?) : 1
|
||||||
|
GENUINE error% fault (stim ALSO differs) : 0
|
||||||
|
Accuracy-row-count diff : 0
|
||||||
|
|
||||||
|
COLLISION (informational) (first 20):
|
||||||
|
1-1: txt=SCM5B34-02 db=SCMVAS-M300
|
||||||
|
1-2: txt=SCM5B34-02 db=SCMVAS-M300
|
||||||
|
|
||||||
|
MODEL VARIANT MISMATCH (first 20):
|
||||||
|
A819-1: txt=8B35-01 db=8B36-04
|
||||||
|
A821-2: txt=8B35-04 db=8B36-01
|
||||||
|
|
||||||
|
DB OLDER THAN .TXT (first 20):
|
||||||
|
A821-1: txt=02-25-2026 db=2026-01-13
|
||||||
|
|
||||||
|
MISSING-FROM-DB (first 30): A243-1 (dec 10243-1), A243-2 (dec 10243-2), A244-1 (dec 10244-1), A255-1 (dec 10255-1), A255-2 (dec 10255-2), A276-1 (dec 10276-1), A276-2 (dec 10276-2), A328-1 (dec 10328-1), A328-2 (dec 10328-2), A376-1 (dec 10376-1), A376-2 (dec 10376-2), A376-3 (dec 10376-3), A377-1 (dec 10377-1), A377-2 (dec 10377-2), A377-3 (dec 10377-3), A405-1 (dec 10405-1), A405-2 (dec 10405-2), A405-3 (dec 10405-3), A405-4 (dec 10405-4), A417-1 (dec 10417-1), A417-2 (dec 10417-2), A561-1 (dec 10561-1), A561-2 (dec 10561-2), A561-3 (dec 10561-3), A561-4 (dec 10561-4), A561-5 (dec 10561-5), A561-6 (dec 10561-6), A601-1 (dec 10601-1), A601-2 (dec 10601-2), A602-1 (dec 10602-1)
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# Parsing Fidelity Verdict — testdatadb ingestion vs original staged datasheets
|
||||||
|
|
||||||
|
**Date:** 2026-06-17 · **Host:** AD2 · **Scope:** all 11,922 staged original `.TXT` datasheets vs the PostgreSQL `test_records`
|
||||||
|
**Raw report:** `PARSING-FIDELITY-REPORT-2026-06-17.txt` · **Tool:** `datasheet-pipeline/implementation/tools/validate-parsing.js`
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
|
||||||
|
**Two distinct questions, two answers:**
|
||||||
|
|
||||||
|
1. **Is the parser faithful to the `.DAT` record it reads?** YES — 0 genuine parse faults across 11,239 comparable records. Every value the importer stores is byte-exact; no misreads, no mis-segmentation.
|
||||||
|
2. **Does each DB row faithfully reproduce the unit's *final* test sheet?** NOT always. The DB is one-row-per-serial, and for units re-tested **on the same calendar date** the conflict rule (strictly-greater date) freezes on an arbitrary same-day run instead of the latest. Whole-source sweep: **311 (serial,date) groups where the DB holds a non-latest same-day run** (see `SAMEDAY-RETEST-EXPOSURE-2026-06-17.txt`). This is a data-model / conflict-rule defect, not a parser fault — fix proposed in `CONFLICT-RULE-FIX-PROPOSAL-2026-06-17.md`.
|
||||||
|
|
||||||
|
The remaining staged-sample "mismatches" were explained by legitimate later-date retests (latest-wins working), reused generic serials, VAS format, or legacy out-of-scope units.
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
Compared each staged original `.TXT` (the DOS-station ground truth, written *before* ingestion) against the DB record's `raw_data` (parsed from the `.DAT`). The cross-check keyed on **scale-invariant data**:
|
||||||
|
- **Error (%)** — dimensionless, identical in `.DAT` and `.TXT` for every family (immune to mV scaling and current-output V→mA conversion). The primary fidelity signal.
|
||||||
|
- **Stim setpoints** (scale-aware) — used to confirm the *same unit/test* when error values differed (retest vs wrong-record).
|
||||||
|
- Serial (with hex-prefix decode), model (SCM-prefix normalized), date.
|
||||||
|
|
||||||
|
## Results (11,921 staged files with an SN)
|
||||||
|
|
||||||
|
| Outcome | Count | Meaning |
|
||||||
|
|---|---:|---|
|
||||||
|
| **Consistent** (SN+model+date+5×error%) | **11,226** | Faithful parse, confirmed |
|
||||||
|
| Retest — DB date newer than `.TXT` | 35 | ON-CONFLICT updated DB to a later test (expected) |
|
||||||
|
| Retest — same date, stim matches, run differs | 42 | **FAITHFULNESS VIOLATION** — unit tested 2+ times same day; DB froze on a non-latest run (strictly-greater-date rule). Staged subset of the 311 whole-source cases. |
|
||||||
|
| VAS/single-point format | 5 | No 5-row accuracy block (SCMVAS) — not comparable by this method |
|
||||||
|
| Serial collision (generic SN, diff family) | 2 | `1-1`/`1-2` reused across products; unique-on-serial keeps one |
|
||||||
|
| **Genuine parse fault** | **0** | — |
|
||||||
|
| Model variant mismatch (same family) | 2 | `A819-1`/`A821-2` — reused serial across 8B35/8B36 (collision) |
|
||||||
|
| DB older than `.TXT` | 1 | `A821-1` — same collision pair |
|
||||||
|
| Accuracy-row-count diff | 0 | — |
|
||||||
|
|
||||||
|
### Why the "genuine" bucket collapsed to 0
|
||||||
|
The last 16 suspects were all `SCM5B37K-1530` (K-thermocouple). Their stim values matched the same 5 nominal setpoints (-50/112.5/275/437.5/600 °C) but differed by ~0.06 °C run-to-run — because the thermocouple input is a *measured analog value*, not an exact setpoint. A scale+relative stim tolerance correctly classifies them as same-day retests. A real segmentation fault would show a different setpoint *structure*; none did.
|
||||||
|
|
||||||
|
## Two follow-up items (NOT parsing-correctness bugs)
|
||||||
|
|
||||||
|
1. **608 staged originals have no DB record** (mostly A-prefix `10xxx` serials, e.g. `A243-1` = `10243-1`, model `5B45-25D`). These exist as staged `.TXT` but are absent from the DB under both decoded and encoded serial. This is an **ingestion-completeness** question (the source `.DAT` for these units appears to be out of the import scan scope, or these are custom `-NND` variants), separate from parsing fidelity. Worth a completeness pass: confirm which `.DAT` paths the importer scans and whether these models' `.DAT` files are present.
|
||||||
|
2. **Same-day retests don't apply "latest wins" (PRIMARY DEFECT).** The `ON CONFLICT` rule updates only when `EXCLUDED.test_date > test_records.test_date` (strictly greater). For a unit tested 2+ times on one date, the rule freezes on whichever same-day run the import processed first and never advances to the latest — so the DB (and the website cert) can show non-final measured values. Whole-source exposure (981,716 records / 406,549 serials): **6,515 same-day multi-run events across 5,977 serials; the DB holds a non-latest run for 311 of them** (3,803 already on latest, 984 superseded by a later-date retest, 1,417 serial absent). Fix proposed in `CONFLICT-RULE-FIX-PROPOSAL-2026-06-17.md`. Directly audit-relevant: same-day runs are typically trim/re-test iterations and the **last** run is the accepted cert result.
|
||||||
|
|
||||||
|
## How to re-run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd C:\Shares\testdatadb
|
||||||
|
node <path-to>/validate-parsing.js [optional-report-path]
|
||||||
|
# Reads C:\Shares\test\STAGE\**\*.TXT and compares to test_records. Read-only.
|
||||||
|
```
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
|
||||||
|
========== SAME-DAY RETEST EXPOSURE (whole source) ==========
|
||||||
|
Records parsed : 981716
|
||||||
|
Distinct serials in source : 406549
|
||||||
|
Serial+date with same-day multi-runs : 6515
|
||||||
|
Distinct serials affected : 5977
|
||||||
|
|
||||||
|
Of those same-day multi-run (serial,date) groups, the DB row:
|
||||||
|
matches the LATEST same-day run : 3803
|
||||||
|
does NOT hold the latest run : 311 <-- faithfulness violations
|
||||||
|
holds an even newer-date test (ok) : 984
|
||||||
|
serial absent from DB : 1417
|
||||||
|
|
||||||
|
Examples (not-latest):
|
||||||
|
4321-1 (2020-07-16, 2 runs): DB sig != latest
|
||||||
|
82001-1 (2012-09-05, 6 runs): DB sig != latest
|
||||||
|
608-55 (2018-01-28, 2 runs): DB sig != latest
|
||||||
|
610-7 (2020-03-05, 2 runs): DB sig != latest
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-07-07
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-08-19
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-08-23
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-08-16
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-08-22
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-08-26
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2017-08-31
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-06-23
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-08-02
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2021-08-03
|
||||||
|
1-2: DB date 2017-02-06 < multirun 2022-06-22
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
/**
|
||||||
|
* In-memory equivalent of what export-datasheets.js writes to
|
||||||
|
* X:\For_Web\<SN>.TXT. Lets upload-to-api.js POST directly to Hoffman's API
|
||||||
|
* from DB state without a filesystem intermediate.
|
||||||
|
*
|
||||||
|
* Returns a string (datasheet text) or null if the record cannot be rendered
|
||||||
|
* (no specs for the model, no raw_data for VASLOG_ENG, etc.).
|
||||||
|
*/
|
||||||
|
const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
|
||||||
|
const { generateExactDatasheet, rendersWithoutSpecs } = require('../templates/datasheet-exact');
|
||||||
|
|
||||||
|
let _specMap = null;
|
||||||
|
function specs() {
|
||||||
|
if (_specMap === null) _specMap = loadAllSpecs();
|
||||||
|
return _specMap;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderContent(record) {
|
||||||
|
if (record.log_type === 'VASLOG_ENG') {
|
||||||
|
return record.raw_data || null;
|
||||||
|
}
|
||||||
|
const modelSpecs = getSpecs(specs(), record.model_number);
|
||||||
|
// DSCA33/45 lost their spec files in the wipe but render from the Hoffman-mined
|
||||||
|
// templates (template-driven names/specs + verbatim accuracy header), so allow a
|
||||||
|
// null-specs render for them. generateExactDatasheet still gates on per-model
|
||||||
|
// validation and returns null until the model is verified.
|
||||||
|
if (!modelSpecs && !rendersWithoutSpecs(record.model_number)) return null;
|
||||||
|
return generateExactDatasheet(record, modelSpecs) || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { renderContent };
|
||||||
File diff suppressed because one or more lines are too long
@@ -7,6 +7,29 @@
|
|||||||
|
|
||||||
const { getFamily } = require('../parsers/spec-reader');
|
const { getFamily } = require('../parsers/spec-reader');
|
||||||
|
|
||||||
|
// DSCA per-model Final-Test templates (Fix 2 STAGE 1 output). Each entry is
|
||||||
|
// { accOut: 'Output (V)'|'Output (mA)', rows: [{name, spec}, ...] }, extracted
|
||||||
|
// byte-accurately from the staged originals. This is the AUTHORITATIVE source
|
||||||
|
// of DSCA parameter names + specs + accuracy output label; loaded once.
|
||||||
|
let DSCA_TEMPLATES = {};
|
||||||
|
try {
|
||||||
|
DSCA_TEMPLATES = require('../dsca-templates.json');
|
||||||
|
} catch (e) {
|
||||||
|
DSCA_TEMPLATES = {};
|
||||||
|
}
|
||||||
|
|
||||||
|
// DSCA33/DSCA45 templates recovered from the Hoffman API (their main spec files were
|
||||||
|
// lost in the wipe). Superset schema: also carries a verbatim 2-line `accHeader`, a
|
||||||
|
// `_srcSerial` validation oracle, and a `validated` gate flag. A model renders ONLY
|
||||||
|
// after its render byte-matches the Hoffman original (validated:true) — until then it
|
||||||
|
// stays null/skipped so the pipeline never overwrites a pristine original.
|
||||||
|
let DSCA3345_TEMPLATES = {};
|
||||||
|
try {
|
||||||
|
DSCA3345_TEMPLATES = require('../dsca33-45-templates.json');
|
||||||
|
} catch (e) {
|
||||||
|
DSCA3345_TEMPLATES = {};
|
||||||
|
}
|
||||||
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
// DATA LINES: parameter names and units per family
|
// DATA LINES: parameter names and units per family
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
@@ -189,12 +212,18 @@ function parseRawData(rawData, family) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Next line: step response / placeholders
|
// Next line: step response / placeholders.
|
||||||
|
// SCM5B/8B: "0","0",value DSCT: just value. Many DSCA models OMIT this bare
|
||||||
|
// line and go straight to the STATUS groups; consuming a STATUS group here
|
||||||
|
// drops a Final-Test row (the "lines drop" defect). For DSCA, skip consuming
|
||||||
|
// when the line is actually a STATUS group (starts with PASS/FAIL).
|
||||||
if (lineIdx < lines.length) {
|
if (lineIdx < lines.length) {
|
||||||
const parts = parseCSVLine(lines[lineIdx++]);
|
const looksLikeStatus = /^"?(PASS|FAIL)/i.test(lines[lineIdx].trim());
|
||||||
// SCM5B/8B: "0","0",value DSCT: just value
|
if (!(family === 'DSCA' && looksLikeStatus)) {
|
||||||
const lastVal = parts[parts.length - 1];
|
const parts = parseCSVLine(lines[lineIdx++]);
|
||||||
result.stepResponse = parseFloat(lastVal) || 0;
|
const lastVal = parts[parts.length - 1];
|
||||||
|
result.stepResponse = parseFloat(lastVal) || 0;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Remaining lines: STATUS groups
|
// Remaining lines: STATUS groups
|
||||||
@@ -297,6 +326,45 @@ function formatMeasured(statusStr) {
|
|||||||
return { passFail, formatted, value };
|
return { passFail, formatted, value };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* DSCA measured-value formatter. The DSCA Final-Test STATUS$ entries use the
|
||||||
|
* trailing digit as a literal decimal-place count (code N -> toFixed(N)),
|
||||||
|
* UNLIKE the 5B/8B QB format strings where code 2 means 1 decimal. Returns the
|
||||||
|
* trimmed value string (caller column-aligns it) plus the PASS/FAIL prefix.
|
||||||
|
*/
|
||||||
|
function formatMeasuredExact(statusStr) {
|
||||||
|
if (!statusStr || statusStr.length <= 4) return null;
|
||||||
|
const passFail = statusStr.substring(0, 4);
|
||||||
|
const decimalDigit = statusStr[statusStr.length - 1];
|
||||||
|
// char at index 4 is either a space (positive) or '-' (negative); start there
|
||||||
|
// so negative signs survive (e.g. "PASS-4.2424060" -> "-4", not "4").
|
||||||
|
const valueStr = statusStr.substring(4, statusStr.length - 1).trim();
|
||||||
|
const parsed = parseFloat(valueStr);
|
||||||
|
if (isNaN(parsed)) return { passFail, valStr: valueStr, value: NaN };
|
||||||
|
// The DOS QuickBASIC stored/computed these as single-precision floats, so the
|
||||||
|
// value at the half-rounding boundary rounds the way single precision rounds,
|
||||||
|
// not double. Recover the single (Math.fround) before formatting — without it,
|
||||||
|
// double-precision toFixed flips last-digit boundaries (e.g. 9.9995 -> "9.999"
|
||||||
|
// here but "10.000" in the original; 46.85 -> "46.9" vs "46.8").
|
||||||
|
const value = Math.fround(parsed);
|
||||||
|
const d = parseInt(decimalDigit, 10);
|
||||||
|
const valStr = isNaN(d) ? value.toFixed(1) : value.toFixed(d);
|
||||||
|
return { passFail, valStr, value };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Split a DSCA template spec string into its value part and trailing unit.
|
||||||
|
* e.g. "< 30 mA" -> { valuePart: "< 30", unit: "mA" } (internal spacing kept,
|
||||||
|
* so the value part can be right-aligned to match the staged column layout).
|
||||||
|
* "+/- 11 ppm/mA" -> { valuePart: "+/- 11", unit: "ppm/mA" }.
|
||||||
|
*/
|
||||||
|
function splitSpecUnit(spec) {
|
||||||
|
const s = String(spec);
|
||||||
|
const m = s.match(/^(.*\S)\s+(\S+)$/);
|
||||||
|
if (m) return { valuePart: m[1], unit: m[2] };
|
||||||
|
return { valuePart: s.trim(), unit: '' };
|
||||||
|
}
|
||||||
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
// Format TSPEC display string from spec values
|
// Format TSPEC display string from spec values
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
@@ -438,12 +506,9 @@ function buildTSpecs(specs, family, stepResponse) {
|
|||||||
|
|
||||||
function formatAccuracyLine(point, sensorNum, maxIn) {
|
function formatAccuracyLine(point, sensorNum, maxIn) {
|
||||||
let stimStr;
|
let stimStr;
|
||||||
if (sensorNum >= 3 && sensorNum <= 6) {
|
if ((sensorNum >= 3 && sensorNum <= 6) || sensorNum === 7) {
|
||||||
// Temperature: +####.##
|
// Temperature: +####.##
|
||||||
stimStr = formatSigned(point.stim, 2, 8);
|
stimStr = formatSigned(point.stim, 2, 8);
|
||||||
} else if (sensorNum === 7) {
|
|
||||||
// Resistance: #####.##
|
|
||||||
stimStr = point.stim.toFixed(2).padStart(8);
|
|
||||||
} else {
|
} else {
|
||||||
// Voltage/Current: +###.###
|
// Voltage/Current: +###.###
|
||||||
const scale = (maxIn != null && maxIn < 1) ? 1000 : 1;
|
const scale = (maxIn != null && maxIn < 1) ? 1000 : 1;
|
||||||
@@ -457,6 +522,36 @@ function formatAccuracyLine(point, sensorNum, maxIn) {
|
|||||||
return ' ' + stimStr + ' ' + calcStr + ' ' + measStr + ' ' + errorStr + ' ' + point.status;
|
return ' ' + stimStr + ' ' + calcStr + ' ' + measStr + ' ' + errorStr + ' ' + point.status;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Accuracy row for the Hoffman-mined DSCA33/DSCA45 families, whose original software
|
||||||
|
* used different column conventions than the voltage/temp models (verified against the
|
||||||
|
* Hoffman originals):
|
||||||
|
* - mA-output models store calc (and, for DSCA45, meas) in AMPS -> x1000 to display mA.
|
||||||
|
* DSCA33 stores meas already in the display unit (NOT scaled); DSCA45 scales both.
|
||||||
|
* - DSCA33 (AC-RMS) prints stim/calc/meas UNSIGNED (error signed); stim is the AC input
|
||||||
|
* to 3 decimals.
|
||||||
|
* - DSCA45 (frequency input) prints stim as an UNSIGNED integer Hz; calc/meas/error SIGNED.
|
||||||
|
*/
|
||||||
|
function formatAccuracyLineDSCA3345(point, model, accOut) {
|
||||||
|
const scale = /mA/.test(accOut || '') ? 1000 : 1;
|
||||||
|
const isDSCA45 = /^DSCA45/i.test((model || '').trim());
|
||||||
|
// values were computed in QB single precision; recover the single before formatting
|
||||||
|
// so last-digit rounding at the .5 boundary matches the original (Math.fround).
|
||||||
|
const num = (val, decimals, signed) => ((signed && val >= 0) ? '+' : '') + Math.fround(val).toFixed(decimals);
|
||||||
|
let stimStr, calcStr, measStr;
|
||||||
|
if (isDSCA45) {
|
||||||
|
stimStr = num(point.stim, 0, false).padStart(8);
|
||||||
|
calcStr = num(point.calc * scale, 3, true).padStart(7);
|
||||||
|
measStr = num(point.meas * scale, 3, true).padStart(7);
|
||||||
|
} else {
|
||||||
|
stimStr = num(point.stim, 3, false).padStart(8);
|
||||||
|
calcStr = num(point.calc * scale, 3, false).padStart(7);
|
||||||
|
measStr = num(point.meas, 3, false).padStart(7);
|
||||||
|
}
|
||||||
|
const errorStr = num(point.error, 3, true).padStart(8);
|
||||||
|
return ' ' + stimStr + ' ' + calcStr + ' ' + measStr + ' ' + errorStr + ' ' + point.status;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Set text at a specific column position (0-indexed) in a string.
|
* Set text at a specific column position (0-indexed) in a string.
|
||||||
* Pads with spaces if the string is shorter than the target column.
|
* Pads with spaces if the string is shorter than the target column.
|
||||||
@@ -499,6 +594,25 @@ function generateExactDatasheet(record, specs) {
|
|||||||
return generateSCMVASDatasheet(record);
|
return generateSCMVASDatasheet(record);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DSCA: the per-model template is the authoritative Final-Test layout (names +
|
||||||
|
// specs + accuracy label). Source is the staged-original set (dsca-templates) or
|
||||||
|
// the Hoffman-mined set (dsca33-45-templates) for the families whose specs were
|
||||||
|
// lost. No template -> do not guess; skip this cert.
|
||||||
|
const dscaKey = (record.model_number || '').trim();
|
||||||
|
// Hoffman-mined templates take PRECEDENCE: DSCA33/45 were also captured by the
|
||||||
|
// STAGE 1 staged extractor (sometimes with accOut "?" and no accHeader), and that
|
||||||
|
// stale entry must not shadow the authoritative Hoffman-mined one.
|
||||||
|
const dscaTpl = (family === 'DSCA')
|
||||||
|
? (DSCA3345_TEMPLATES[dscaKey] || DSCA_TEMPLATES[dscaKey] || null)
|
||||||
|
: null;
|
||||||
|
if (family === 'DSCA' && !dscaTpl) return null;
|
||||||
|
// Hoffman-mined DSCA33/45 render only once the model is byte-validated against its
|
||||||
|
// Hoffman original — otherwise stay null so an unverified render can't overwrite a
|
||||||
|
// live original. The validation harness sets DSCA_VALIDATE_MODE to render
|
||||||
|
// unvalidated models for the byte-compare; the live service never sets it.
|
||||||
|
if (family === 'DSCA' && DSCA3345_TEMPLATES[dscaKey] && !DSCA3345_TEMPLATES[dscaKey].validated
|
||||||
|
&& !process.env.DSCA_VALIDATE_MODE) return null;
|
||||||
|
|
||||||
const parsed = (family === 'SCM7B')
|
const parsed = (family === 'SCM7B')
|
||||||
? parse7BRawData(record.raw_data)
|
? parse7BRawData(record.raw_data)
|
||||||
: parseRawData(record.raw_data, family);
|
: parseRawData(record.raw_data, family);
|
||||||
@@ -553,23 +667,38 @@ function generateExactDatasheet(record, specs) {
|
|||||||
} else {
|
} else {
|
||||||
lines.push(' ACCURACY TEST');
|
lines.push(' ACCURACY TEST');
|
||||||
lines.push('');
|
lines.push('');
|
||||||
|
if (dscaTpl && Array.isArray(dscaTpl.accHeader) && dscaTpl.accHeader.length >= 2) {
|
||||||
|
// DSCA33/45 (Hoffman-mined): the accuracy header carries model-specific tokens
|
||||||
|
// the sensor-type logic can't synthesize (Vin (mVAC), Iin (AAC), Frequency (Hz),
|
||||||
|
// Output (VDC)/(mADC)). Emit the verbatim 2-line header from the original.
|
||||||
|
lines.push(dscaTpl.accHeader[0]);
|
||||||
|
lines.push(dscaTpl.accHeader[1]);
|
||||||
|
lines.push(TAB5 + '-'.repeat(10) + ' ' + '-'.repeat(11) + ' ' + '-'.repeat(11) + ' ' + '-'.repeat(10) + ' ' + '-'.repeat(8));
|
||||||
|
} else {
|
||||||
lines.push(' Calculated Measured');
|
lines.push(' Calculated Measured');
|
||||||
|
|
||||||
// Input column header based on sensor type
|
// Input column header based on sensor type
|
||||||
let inputHeader;
|
let inputHeader;
|
||||||
if (sensorNum >= 3 && sensorNum <= 6) {
|
if ((sensorNum >= 3 && sensorNum <= 6) || sensorNum === 7) {
|
||||||
inputHeader = ' Temp. (C)';
|
inputHeader = ' Temp. (C)';
|
||||||
} else if (sensorNum === 2 || sensorNum === 9) {
|
} else if (sensorNum === 2 || sensorNum === 9) {
|
||||||
inputHeader = ' Iin (mA)';
|
inputHeader = ' Iin (mA)';
|
||||||
} else if (sensorNum === 7) {
|
|
||||||
inputHeader = ' Rin (ohms)';
|
|
||||||
} else {
|
} else {
|
||||||
inputHeader = (maxIn != null && maxIn < 1) ? ' Vin (mV)' : ' Vin (V)';
|
inputHeader = (maxIn != null && maxIn < 1) ? ' Vin (mV)' : ' Vin (V)';
|
||||||
}
|
}
|
||||||
lines.push(' ' + inputHeader + ' Vout (V) Vout (V)* Error (%) Status');
|
// DSCA labels its accuracy output column "Output (V)"/"Output (mA)" (from the
|
||||||
lines.push(TAB5 + '========== ========== ========== ========= ========');
|
// template) with '-' rule separators; 5B/8B/etc. use "Vout (V)" with '='.
|
||||||
|
const accOut = (family === 'DSCA' && dscaTpl) ? dscaTpl.accOut : 'Vout (V)';
|
||||||
|
const accSep = (family === 'DSCA') ? '-' : '=';
|
||||||
|
lines.push(' ' + inputHeader + ' ' + accOut + ' ' + accOut + '* Error (%) Status');
|
||||||
|
lines.push(TAB5 + accSep.repeat(10) + ' ' + accSep.repeat(10) + ' ' + accSep.repeat(10) + ' ' + accSep.repeat(9) + ' ' + accSep.repeat(8));
|
||||||
|
}
|
||||||
|
|
||||||
for (const point of parsed.accuracy) {
|
for (const point of parsed.accuracy) {
|
||||||
|
if (dscaTpl && Array.isArray(dscaTpl.accHeader)) {
|
||||||
|
lines.push(formatAccuracyLineDSCA3345(point, record.model_number, dscaTpl.accOut));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
lines.push(formatAccuracyLine(point, sensorNum, maxIn));
|
lines.push(formatAccuracyLine(point, sensorNum, maxIn));
|
||||||
}
|
}
|
||||||
lines.push('');
|
lines.push('');
|
||||||
@@ -579,6 +708,83 @@ function generateExactDatasheet(record, specs) {
|
|||||||
// QB column positions (1-indexed): TAB(31), TAB(47), TAB(60-speclen), TAB(61), TAB(71)
|
// QB column positions (1-indexed): TAB(31), TAB(47), TAB(60-speclen), TAB(61), TAB(71)
|
||||||
lines.push(' FINAL TEST RESULTS');
|
lines.push(' FINAL TEST RESULTS');
|
||||||
lines.push('');
|
lines.push('');
|
||||||
|
if (family === 'DSCA') {
|
||||||
|
// DSCA Final-Test renders from the per-model staged template: the rows give
|
||||||
|
// the parameter names + specs (and accuracy label) directly; the value-bearing
|
||||||
|
// raw_data STATUS groups map positionally onto the spec-bearing rows. Rows with
|
||||||
|
// an empty spec (240VAC Withstand / Hi-Pot) carry no measured value and render
|
||||||
|
// as PASS. Header/column scheme matches the staged originals.
|
||||||
|
|
||||||
|
// Value-bearing measurements in source order (drop "PASS"/"" padding entries).
|
||||||
|
const measurements = [];
|
||||||
|
for (const s of parsed.statusEntries) {
|
||||||
|
const m = formatMeasuredExact(s);
|
||||||
|
if (m) measurements.push(m);
|
||||||
|
}
|
||||||
|
const specRowCount = dscaTpl.rows.filter(r => (r.spec || '').trim()).length;
|
||||||
|
// The simple positional zip is sound only when there is exactly one measured
|
||||||
|
// value per spec-bearing row. When counts differ, this subtype measures slots
|
||||||
|
// the template omits (e.g. an extra load pair); use the per-model slotMap
|
||||||
|
// (absolute statusEntries index per spec-bearing row, derived from the staged
|
||||||
|
// originals) to pull the right value. With no usable slotMap, skip rather than
|
||||||
|
// misalign ("do not guess").
|
||||||
|
const useSlot = (measurements.length !== specRowCount)
|
||||||
|
&& Array.isArray(dscaTpl.slotMap) && dscaTpl.slotMap.length === specRowCount;
|
||||||
|
if (measurements.length !== specRowCount && !useSlot) return null;
|
||||||
|
|
||||||
|
let h1 = setCol('', 12, 'Parameter');
|
||||||
|
h1 = setCol(h1, 31, 'Measured Value*');
|
||||||
|
h1 = setCol(h1, 51, 'Specification');
|
||||||
|
h1 = setCol(h1, 69, 'Status');
|
||||||
|
lines.push(h1);
|
||||||
|
let h2 = setCol('', 4, '='.repeat(25));
|
||||||
|
h2 = setCol(h2, 31, '='.repeat(15));
|
||||||
|
h2 = setCol(h2, 48, '='.repeat(19));
|
||||||
|
h2 = setCol(h2, 69, '='.repeat(6));
|
||||||
|
lines.push(h2);
|
||||||
|
|
||||||
|
const is3345 = Array.isArray(dscaTpl.accHeader); // Hoffman-mined DSCA33/45
|
||||||
|
let mi = 0, si = 0;
|
||||||
|
for (const row of dscaTpl.rows) {
|
||||||
|
const spec = (row.spec || '').trim();
|
||||||
|
let line = setCol('', 4, row.name);
|
||||||
|
if (spec) {
|
||||||
|
const su = splitSpecUnit(spec);
|
||||||
|
const m = useSlot
|
||||||
|
? formatMeasuredExact(parsed.statusEntries[dscaTpl.slotMap[si++]])
|
||||||
|
: measurements[mi++];
|
||||||
|
if (m) {
|
||||||
|
// measured value right-justified ending at col 38, unit at col 40.
|
||||||
|
// DSCA33/45 follow QB's fixed 6-char number field: a value that would
|
||||||
|
// overflow drops its leading zero to fit ("-0.0005" (7) -> "-.0005" (6));
|
||||||
|
// values that already fit (e.g. "-0.750", "0.0000") keep it.
|
||||||
|
let v = String(m.valStr);
|
||||||
|
if (is3345 && v.length > 6) v = v.replace(/^-0\./, '-.');
|
||||||
|
line = setCol(line, 39 - v.length, v);
|
||||||
|
if (su.unit) line = setCol(line, 40, su.unit);
|
||||||
|
}
|
||||||
|
// spec value-part right-justified ending at col 58, unit at col 60
|
||||||
|
line = setCol(line, 59 - su.valuePart.length, su.valuePart);
|
||||||
|
if (su.unit) line = setCol(line, 60, su.unit);
|
||||||
|
line = setCol(line, 70, m ? m.passFail : 'PASS');
|
||||||
|
} else if (is3345 && !/withstand|hi-?pot/i.test(row.name)) {
|
||||||
|
// DSCA33/45 spec-less rows that are NOT a pass/fail test (e.g. the
|
||||||
|
// "Zero-Crossing Input" / "TTL Input" section sub-heads) carry no status.
|
||||||
|
// (Leave the row as just the name.)
|
||||||
|
} else {
|
||||||
|
// spec-less pass/fail row (240VAC Withstand / Hi-Pot): blank measured + PASS
|
||||||
|
line = setCol(line, 70, 'PASS');
|
||||||
|
}
|
||||||
|
lines.push(line);
|
||||||
|
}
|
||||||
|
// Footer load note ("Standard output load for test is ... ohms.") — printed
|
||||||
|
// before the underline, only by the models whose staged original had it
|
||||||
|
// (captured per-model in STAGE 1; not all current-output models print it).
|
||||||
|
if (dscaTpl.loadNote) {
|
||||||
|
lines.push('');
|
||||||
|
lines.push(TAB5 + dscaTpl.loadNote);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
// QB: TAB(12); "Parameter"; TAB(30); "Measured Value"; TAB(51); "Specification "; TAB(70); "Status"
|
// QB: TAB(12); "Parameter"; TAB(30); "Measured Value"; TAB(51); "Specification "; TAB(70); "Status"
|
||||||
let hdr1 = setCol('', 11, 'Parameter');
|
let hdr1 = setCol('', 11, 'Parameter');
|
||||||
hdr1 = setCol(hdr1, 29, 'Measured Value');
|
hdr1 = setCol(hdr1, 29, 'Measured Value');
|
||||||
@@ -628,6 +834,7 @@ function generateExactDatasheet(record, specs) {
|
|||||||
|
|
||||||
lines.push(line);
|
lines.push(line);
|
||||||
}
|
}
|
||||||
|
} // end non-DSCA Final Test Results
|
||||||
|
|
||||||
// ---- Footer ----
|
// ---- Footer ----
|
||||||
// 240 VAC / Hi-Pot (conditional by family/model)
|
// 240 VAC / Hi-Pot (conditional by family/model)
|
||||||
@@ -651,9 +858,6 @@ function generateExactDatasheet(record, specs) {
|
|||||||
let hp = setCol(TAB5 + 'Hi-Pot', 70, 'PASS');
|
let hp = setCol(TAB5 + 'Hi-Pot', 70, 'PASS');
|
||||||
lines.push(hp);
|
lines.push(hp);
|
||||||
}
|
}
|
||||||
} else if (family === 'DSCA') {
|
|
||||||
lines.push(TAB5 + '240VAC Withstand' + ''.padEnd(50) + 'PASS');
|
|
||||||
lines.push(TAB5 + 'Hi-Pot' + ''.padEnd(60) + 'PASS');
|
|
||||||
} else if (family === 'DSCT') {
|
} else if (family === 'DSCT') {
|
||||||
lines.push(TAB5 + '240 VAC Withstand' + ''.padEnd(49) + 'PASS');
|
lines.push(TAB5 + '240 VAC Withstand' + ''.padEnd(49) + 'PASS');
|
||||||
lines.push(TAB5 + 'Hi-Pot' + ''.padEnd(60) + 'PASS');
|
lines.push(TAB5 + 'Hi-Pot' + ''.padEnd(60) + 'PASS');
|
||||||
@@ -669,6 +873,9 @@ function generateExactDatasheet(record, specs) {
|
|||||||
lines.push(setCol(TAB5 + 'Pins Straight: _____', 44, 'Module Header: _____'));
|
lines.push(setCol(TAB5 + 'Pins Straight: _____', 44, 'Module Header: _____'));
|
||||||
lines.push('');
|
lines.push('');
|
||||||
lines.push(setCol(TAB5 + 'Tested by: _____________', 44, 'QC: _______________'));
|
lines.push(setCol(TAB5 + 'Tested by: _____________', 44, 'QC: _______________'));
|
||||||
|
} else if (/^DSCA33/i.test((record.model_number || '').trim())) {
|
||||||
|
// DSCA33 originals print just the centered "Check List" header (no items).
|
||||||
|
lines.push(' Check List');
|
||||||
} else if (family !== 'DSCA') {
|
} else if (family !== 'DSCA') {
|
||||||
lines.push(' Check List');
|
lines.push(' Check List');
|
||||||
lines.push('');
|
lines.push('');
|
||||||
@@ -677,11 +884,6 @@ function generateExactDatasheet(record, specs) {
|
|||||||
lines.push(setCol(TAB5 + 'Pins Straight: __X__', 44, 'Module Header: __X__'));
|
lines.push(setCol(TAB5 + 'Pins Straight: __X__', 44, 'Module Header: __X__'));
|
||||||
}
|
}
|
||||||
|
|
||||||
// DSCA current output load note
|
|
||||||
if (family === 'DSCA' && specs && specs.OUTSIGTYPE && specs.OUTSIGTYPE.trim().toUpperCase() === 'CURRENT') {
|
|
||||||
lines.push(TAB5 + 'Standard output load for test is 250 ohms.');
|
|
||||||
}
|
|
||||||
|
|
||||||
lines.push('');
|
lines.push('');
|
||||||
lines.push(TAB5 + 'It is hereby certified that the above product is in conformance with');
|
lines.push(TAB5 + 'It is hereby certified that the above product is in conformance with');
|
||||||
lines.push(TAB5 + 'all requirements to the extent specified. This product is not');
|
lines.push(TAB5 + 'all requirements to the extent specified. This product is not');
|
||||||
@@ -900,11 +1102,21 @@ function generateSCMVASDatasheet(record) {
|
|||||||
return lines.join('\r\n');
|
return lines.join('\r\n');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* True if the model renders from a template that does NOT require spec-reader specs
|
||||||
|
* (the Hoffman-mined DSCA33/45 set). Lets render-datasheet.js skip the missing-specs
|
||||||
|
* bail for these. (Still gated on per-model `validated` inside generateExactDatasheet.)
|
||||||
|
*/
|
||||||
|
function rendersWithoutSpecs(modelNumber) {
|
||||||
|
return !!DSCA3345_TEMPLATES[(modelNumber || '').trim()];
|
||||||
|
}
|
||||||
|
|
||||||
module.exports = {
|
module.exports = {
|
||||||
generateExactDatasheet,
|
generateExactDatasheet,
|
||||||
generateSCMVASDatasheet,
|
generateSCMVASDatasheet,
|
||||||
extractSCMVASAccuracy,
|
extractSCMVASAccuracy,
|
||||||
parseRawData,
|
parseRawData,
|
||||||
parse7BRawData,
|
parse7BRawData,
|
||||||
|
rendersWithoutSpecs,
|
||||||
DATA_LINES,
|
DATA_LINES,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
// Root-cause + scope of the 608 missing staged units (READ-ONLY) for the report to John.
|
||||||
|
// Hypothesis: importer serial/date regex requires a leading digit, so hex-encoded
|
||||||
|
// (leading-letter) serials in the .DAT are never matched -> records dropped.
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const db = require('./database/db');
|
||||||
|
|
||||||
|
const STAGE = 'C:/Shares/test/STAGE';
|
||||||
|
const STRICT = /^"(\d+-\d+[A-Za-z]?)","(\d{2}-\d{2}-\d{4})"$/; // current importer regex
|
||||||
|
const LOOSE = /^"([^"]+)","(\d{2}-\d{2}-\d{4})"$/; // any serial before a date
|
||||||
|
const decode = sn => /^[A-Za-z]\d/.test(sn) ? String(sn.toUpperCase().charCodeAt(0) - 55) + sn.slice(1) : sn;
|
||||||
|
|
||||||
|
function walk(dir, re, out) { let it=[]; try{it=fs.readdirSync(dir,{withFileTypes:true})}catch{return out;}
|
||||||
|
for(const e of it){const p=path.join(dir,e.name); if(e.isDirectory()) walk(p,re,out); else if(re.test(e.name)) out.push(p);} return out; }
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
// ---- staged .TXT inventory ----
|
||||||
|
const txts = walk(STAGE, /\.txt$/i, []);
|
||||||
|
const staged = [];
|
||||||
|
for (const f of txts) { let t; try{t=fs.readFileSync(f,'utf8')}catch{continue;}
|
||||||
|
const sn=(t.match(/^\s*SN:\s*(\S+)/m)||[])[1]; if(!sn) continue;
|
||||||
|
const model=(t.match(/^\s*Model:\s*(\S+)/m)||[])[1]||'';
|
||||||
|
const date=(t.match(/^\s*Date:\s*(\d{2}-\d{2}-\d{4})/m)||[])[1]||'';
|
||||||
|
const station=(f.match(/STAGE[\\\/]([^\\\/]+)/)||[])[1]||'';
|
||||||
|
staged.push({ sn, dec: decode(sn), model, date, station, file: f });
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- which staged decoded serials are in DB ----
|
||||||
|
const decs=[...new Set(staged.map(s=>s.dec))]; const inDb=new Set();
|
||||||
|
for(let i=0;i<decs.length;i+=1000){ const rows=await db.query('SELECT serial_number FROM test_records WHERE serial_number = ANY($1)',[decs.slice(i,i+1000)]); for(const r of rows) inDb.add(r.serial_number); }
|
||||||
|
const missing = staged.filter(s=>!inDb.has(s.dec));
|
||||||
|
|
||||||
|
// ---- scan ALL .DAT sources: which serial tokens appear, strict vs letter-prefixed ----
|
||||||
|
let dats=[]; walk('C:/Shares/test/Ate/HISTLOGS', /\.dat$/i, dats);
|
||||||
|
let stations=[]; try{stations=fs.readdirSync('C:/Shares/test',{withFileTypes:true}).filter(d=>d.isDirectory()&&/^TS-\d+[LR]?$/i.test(d.name)).map(d=>d.name);}catch{}
|
||||||
|
for(const s of stations) walk(path.join('C:/Shares/test',s,'LOGS'), /\.dat$/i, dats);
|
||||||
|
|
||||||
|
const looseSet=new Set(); const letterSet=new Set(); let letterRecs=0; const letterModels=new Set();
|
||||||
|
let fi=0;
|
||||||
|
for(const f of dats){ fi++; if(fi%5000===0) console.log(' scan '+fi+'/'+dats.length);
|
||||||
|
let lines; try{lines=fs.readFileSync(f,'utf8').split('\n')}catch{continue;}
|
||||||
|
let lastModel='';
|
||||||
|
for(const l of lines){ const t=l.trim();
|
||||||
|
const mm=t.match(/^"([A-Z0-9][A-Z0-9 \-]*)"$/i); if(mm && !/PASS|FAIL/.test(t) && !t.includes(',')) { lastModel=mm[1].trim(); continue; }
|
||||||
|
const m=t.match(LOOSE); if(m){ const sn=m[1]; looseSet.add(sn);
|
||||||
|
if(/^[A-Za-z]\d/.test(sn) && !STRICT.test(t)){ letterSet.add(sn); letterRecs++; if(lastModel) letterModels.add(lastModel); } }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- categorize the missing ----
|
||||||
|
const cat = { parserDrop: [], absent: [], decInDbButMiss: [] };
|
||||||
|
for(const s of missing){
|
||||||
|
if(letterSet.has(s.sn) || (/^[A-Za-z]\d/.test(s.sn) && looseSet.has(s.sn))) cat.parserDrop.push(s);
|
||||||
|
else if(!looseSet.has(s.sn) && !looseSet.has(s.dec)) cat.absent.push(s);
|
||||||
|
else cat.decInDbButMiss.push(s);
|
||||||
|
}
|
||||||
|
const by=(arr,k)=>{const m={};for(const x of arr){const v=(x[k]||'?');m[v]=(m[v]||0)+1;}return Object.entries(m).sort((a,b)=>b[1]-a[1]);};
|
||||||
|
|
||||||
|
// ---- full letter-prefixed population in .DAT and how much is absent from DB ----
|
||||||
|
const letterDecs=[...letterSet].map(decode); const letterInDb=new Set();
|
||||||
|
for(let i=0;i<letterDecs.length;i+=1000){ const rows=await db.query('SELECT serial_number FROM test_records WHERE serial_number = ANY($1)',[letterDecs.slice(i,i+1000)]); for(const r of rows) letterInDb.add(r.serial_number); }
|
||||||
|
const letterMissingDistinct=letterDecs.filter(d=>!letterInDb.has(d)).length;
|
||||||
|
|
||||||
|
const out=[]; const L=s=>{out.push(s);console.log(s);};
|
||||||
|
L('========== MISSING-UNITS ROOT CAUSE & SCOPE ==========');
|
||||||
|
L('Staged .TXT with SN : '+staged.length);
|
||||||
|
L('Staged units MISSING from DB : '+missing.length);
|
||||||
|
L('');
|
||||||
|
L('ROOT-CAUSE CATEGORIES (of the missing):');
|
||||||
|
L(' Parser-drop (encoded serial w/ leading letter present in .DAT, regex rejects): '+cat.parserDrop.length);
|
||||||
|
L(' Source .DAT has no record for this unit (data absent) : '+cat.absent.length);
|
||||||
|
L(' Other (decoded present, still missing - investigate) : '+cat.decInDbButMiss.length);
|
||||||
|
L('');
|
||||||
|
L('PARSER-DROP breakdown by leading char: '+by(cat.parserDrop, 'sn').slice(0,1).length? '' : '');
|
||||||
|
const lead=k=>{const m={};for(const x of cat.parserDrop){const c=x.sn[0].toUpperCase();m[c]=(m[c]||0)+1;}return Object.entries(m).sort((a,b)=>b[1]-a[1]);};
|
||||||
|
L(' by leading letter: '+lead().map(([c,n])=>c+'='+n).join(', '));
|
||||||
|
L(' by station : '+by(cat.parserDrop,'station').map(([c,n])=>c+'='+n).join(', '));
|
||||||
|
L(' by model (top 12): '+by(cat.parserDrop,'model').slice(0,12).map(([c,n])=>c+'='+n).join(', '));
|
||||||
|
L(' date range : '+(()=>{const ds=cat.parserDrop.map(s=>s.date).filter(Boolean).sort();return ds[0]+' .. '+ds[ds.length-1];})());
|
||||||
|
L(' samples :'); cat.parserDrop.slice(0,12).forEach(s=>L(' '+s.sn+' -> '+s.dec+' '+s.model+' '+s.date+' '+s.station));
|
||||||
|
if(cat.absent.length){ L(''); L('DATA-ABSENT samples:'); cat.absent.slice(0,10).forEach(s=>L(' '+s.sn+' -> '+s.dec+' '+s.model+' '+s.date+' '+s.station)); }
|
||||||
|
if(cat.decInDbButMiss.length){ L(''); L('OTHER samples:'); cat.decInDbButMiss.slice(0,10).forEach(s=>L(' '+s.sn+' -> '+s.dec+' '+s.model+' '+s.date+' '+s.station)); }
|
||||||
|
L('');
|
||||||
|
L('FULL .DAT BLIND SPOT (all letter-prefixed serials the importer skips, not just staged):');
|
||||||
|
L(' distinct letter-prefixed serials in .DAT : '+letterSet.size);
|
||||||
|
L(' total letter-prefixed records (dropped) : '+letterRecs);
|
||||||
|
L(' distinct models affected : '+letterModels.size);
|
||||||
|
L(' of those serials, DECODED form absent from DB: '+letterMissingDistinct+' / '+letterSet.size);
|
||||||
|
|
||||||
|
if(process.argv[2]) fs.writeFileSync(process.argv[2], out.join('\n')+'\n');
|
||||||
|
await db.close();
|
||||||
|
})().catch(e=>{console.error(e);process.exit(1);});
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
// Whole-source sweep (READ-ONLY): find serials with same-day multi-runs (distinct values)
|
||||||
|
// and measure how many the DB does NOT hold the latest run for. Scans the import's .DAT sources.
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const db = require('./database/db');
|
||||||
|
|
||||||
|
const ROOTS = ['C:/Shares/test/Ate/HISTLOGS']; // central combined logs first
|
||||||
|
const STATION_BASE = 'C:/Shares/test';
|
||||||
|
|
||||||
|
function datFiles(dir, out) {
|
||||||
|
let it = []; try { it = fs.readdirSync(dir, { withFileTypes: true }); } catch { return out; }
|
||||||
|
for (const e of it) { const p = path.join(dir, e.name);
|
||||||
|
if (e.isDirectory()) datFiles(p, out);
|
||||||
|
else if (/\.dat$/i.test(e.name)) out.push(p);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// signature of a record = the 5 Error(%) columns joined (distinguishes runs)
|
||||||
|
function recSig(block) {
|
||||||
|
const errs = [];
|
||||||
|
for (const l of block) {
|
||||||
|
if (/,"(PASS|FAIL)"/.test(l)) { const f = l.split(','); if (f.length >= 5) { errs.push(f[3].trim()); if (errs.length === 5) break; } }
|
||||||
|
}
|
||||||
|
return errs.length === 5 ? errs.join('|') : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
// gather files: HISTLOGS, then station LOGS (mirrors import order; station = latest)
|
||||||
|
let files = [];
|
||||||
|
for (const r of ROOTS) datFiles(r, files);
|
||||||
|
let stations = [];
|
||||||
|
try { stations = fs.readdirSync(STATION_BASE, { withFileTypes: true }).filter(d => d.isDirectory() && /^TS-\d+[LR]?$/i.test(d.name)).map(d => d.name); } catch {}
|
||||||
|
for (const s of stations) datFiles(path.join(STATION_BASE, s, 'LOGS'), files);
|
||||||
|
console.log('Scanning ' + files.length + ' .DAT files (' + stations.length + ' stations + HISTLOGS)...');
|
||||||
|
|
||||||
|
// serial -> date -> { sigs:Set, last:sig }
|
||||||
|
const map = new Map();
|
||||||
|
let recCount = 0, fi = 0;
|
||||||
|
for (const f of files) {
|
||||||
|
fi++; if (fi % 3000 === 0) console.log(' ...' + fi + '/' + files.length + ' files, ' + recCount + ' records');
|
||||||
|
let lines; try { lines = fs.readFileSync(f, 'utf8').split('\n'); } catch { continue; }
|
||||||
|
let block = [];
|
||||||
|
for (let i = 0; i < lines.length; i++) {
|
||||||
|
const t = lines[i].trim();
|
||||||
|
const sd = t.match(/^"(\d+-\d+[A-Za-z]?)","(\d{2}-\d{2}-\d{4})"$/);
|
||||||
|
if (sd) {
|
||||||
|
const sig = recSig(block);
|
||||||
|
if (sig) {
|
||||||
|
recCount++;
|
||||||
|
const sn = sd[1]; const [mm,dd,yy] = sd[2].split('-'); const date = `${yy}-${mm}-${dd}`;
|
||||||
|
let dm = map.get(sn); if (!dm) { dm = new Map(); map.set(sn, dm); }
|
||||||
|
let e = dm.get(date); if (!e) { e = { sigs: new Set(), last: null }; dm.set(date, e); }
|
||||||
|
e.sigs.add(sig); e.last = sig;
|
||||||
|
}
|
||||||
|
block = [];
|
||||||
|
} else if (t) block.push(t);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
console.log('Parsed ' + recCount + ' records, ' + map.size + ' distinct serials.');
|
||||||
|
|
||||||
|
// find serials with same-day multi-runs (>=2 distinct sigs on one date)
|
||||||
|
const multi = []; // { sn, date, runs, lastSig }
|
||||||
|
for (const [sn, dm] of map) for (const [date, e] of dm) if (e.sigs.size >= 2) multi.push({ sn, date, runs: e.sigs.size, lastSig: e.last });
|
||||||
|
console.log('Serials*date with same-day multi-runs (distinct values): ' + multi.length);
|
||||||
|
const multiSerials = new Set(multi.map(m => m.sn));
|
||||||
|
console.log('Distinct serials affected: ' + multiSerials.size);
|
||||||
|
|
||||||
|
// For each, check what the DB holds vs the latest same-day run
|
||||||
|
const sns = [...multiSerials];
|
||||||
|
const dbMap = new Map();
|
||||||
|
for (let i = 0; i < sns.length; i += 1000) {
|
||||||
|
const rows = await db.query('SELECT serial_number, test_date, raw_data FROM test_records WHERE serial_number = ANY($1)', [sns.slice(i, i+1000)]);
|
||||||
|
for (const r of rows) dbMap.set(r.serial_number, r);
|
||||||
|
}
|
||||||
|
let notLatest = 0, dbNewer = 0, dbAbsent = 0, dbMatches = 0, examples = [];
|
||||||
|
for (const m of multi) {
|
||||||
|
const d = dbMap.get(m.sn);
|
||||||
|
if (!d) { dbAbsent++; continue; }
|
||||||
|
const dbDate = d.test_date && d.test_date.toISOString ? d.test_date.toISOString().slice(0,10) : String(d.test_date);
|
||||||
|
if (dbDate > m.date) { dbNewer++; continue; } // DB has an even later test -> fine
|
||||||
|
if (dbDate < m.date) { notLatest++; if (examples.length<15) examples.push(`${m.sn}: DB date ${dbDate} < multirun ${m.date}`); continue; }
|
||||||
|
const dbSig = recSig((d.raw_data||'').split('\n').map(s=>s.trim()));
|
||||||
|
if (dbSig === m.lastSig) dbMatches++;
|
||||||
|
else { notLatest++; if (examples.length<15) examples.push(`${m.sn} (${m.date}, ${m.runs} runs): DB sig != latest`); }
|
||||||
|
}
|
||||||
|
|
||||||
|
const out = [];
|
||||||
|
const L = s => { out.push(s); console.log(s); };
|
||||||
|
L('\n========== SAME-DAY RETEST EXPOSURE (whole source) ==========');
|
||||||
|
L('Records parsed : ' + recCount);
|
||||||
|
L('Distinct serials in source : ' + map.size);
|
||||||
|
L('Serial+date with same-day multi-runs : ' + multi.length);
|
||||||
|
L('Distinct serials affected : ' + multiSerials.size);
|
||||||
|
L('');
|
||||||
|
L('Of those same-day multi-run (serial,date) groups, the DB row:');
|
||||||
|
L(' matches the LATEST same-day run : ' + dbMatches);
|
||||||
|
L(' does NOT hold the latest run : ' + notLatest + ' <-- faithfulness violations');
|
||||||
|
L(' holds an even newer-date test (ok) : ' + dbNewer);
|
||||||
|
L(' serial absent from DB : ' + dbAbsent);
|
||||||
|
if (examples.length) { L(''); L('Examples (not-latest):'); examples.forEach(x=>L(' '+x)); }
|
||||||
|
if (process.argv[2]) fs.writeFileSync(process.argv[2], out.join('\n')+'\n');
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error(e); process.exit(1); });
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
// Parsing-fidelity validation (READ-ONLY): every staged original .TXT vs the DB record.
|
||||||
|
// Compares scale-invariant data: SN, model, date, and the 5 Error(%) accuracy values
|
||||||
|
// (error% is dimensionless -> immune to mV scaling / current-output conversion, so a
|
||||||
|
// mismatch means a real parsing/segmentation/identity fault, not a rendering transform).
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const db = require('./database/db');
|
||||||
|
|
||||||
|
const STAGE = 'C:/Shares/test/STAGE';
|
||||||
|
const ERR_TOL = 0.003; // half-unit of 3-decimal display + margin
|
||||||
|
const REPORT = process.argv[2] || null;
|
||||||
|
|
||||||
|
function walk(dir, out) {
|
||||||
|
let items = [];
|
||||||
|
try { items = fs.readdirSync(dir, { withFileTypes: true }); } catch { return out; }
|
||||||
|
for (const it of items) {
|
||||||
|
const p = path.join(dir, it.name);
|
||||||
|
if (it.isDirectory()) walk(p, out);
|
||||||
|
else if (/\.txt$/i.test(it.name)) out.push(p);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseTxt(txt) {
|
||||||
|
const lines = txt.split(/\r?\n/);
|
||||||
|
const get = re => { for (const l of lines) { const m = l.match(re); if (m) return m[1]; } return null; };
|
||||||
|
const sn = get(/^\s*SN:\s*(\S+)/);
|
||||||
|
const model = get(/^\s*Model:\s*(\S+)/);
|
||||||
|
const date = get(/^\s*Date:\s*(\d{2}-\d{2}-\d{4})/);
|
||||||
|
// accuracy rows: lines ending in PASS/FAIL with >=4 numeric tokens, before FINAL TEST
|
||||||
|
const errs = [];
|
||||||
|
const stims = [];
|
||||||
|
for (const l of lines) {
|
||||||
|
if (/FINAL TEST/i.test(l)) break;
|
||||||
|
if (!/\b(PASS|FAIL)\b/.test(l)) continue;
|
||||||
|
const nums = (l.match(/[+-]?\d*\.\d+|[+-]?\d+/g) || []).map(Number);
|
||||||
|
if (nums.length >= 4) { errs.push(nums[3]); stims.push(nums[0]); } // [0]=stim [3]=Error(%)
|
||||||
|
if (errs.length === 5) break;
|
||||||
|
}
|
||||||
|
return { sn, model, date, errs, stims };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decode hex-prefix encoded serial (A-prefix files store the ENCODED SN inside):
|
||||||
|
// leading [A-Z] -> (charCode-55) numeric prefix. H9553-13-style files already store
|
||||||
|
// the decoded SN, which is numeric, so they don't match and pass through unchanged.
|
||||||
|
function decodeSn(sn) {
|
||||||
|
if (/^[A-Za-z]\d/.test(sn)) {
|
||||||
|
const n = sn.toUpperCase().charCodeAt(0) - 55;
|
||||||
|
return String(n) + sn.slice(1);
|
||||||
|
}
|
||||||
|
return sn;
|
||||||
|
}
|
||||||
|
const normModel = m => (m || '').toUpperCase().replace(/^SCM/, '');
|
||||||
|
|
||||||
|
function parseRawAcc(raw) {
|
||||||
|
if (!raw) return { errs: [], stims: [] };
|
||||||
|
const lines = raw.split('\n').map(s => s.trim()).filter(Boolean);
|
||||||
|
const errs = [], stims = [];
|
||||||
|
for (let i = 1; i < lines.length && errs.length < 5; i++) {
|
||||||
|
const f = lines[i].split(',');
|
||||||
|
if (f.length >= 5 && /"(PASS|FAIL)"/.test(lines[i])) {
|
||||||
|
const e = parseFloat(f[3]), s = parseFloat(f[0]);
|
||||||
|
if (!isNaN(e)) { errs.push(e); stims.push(s); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { errs, stims };
|
||||||
|
}
|
||||||
|
// scale-aware + relative stim match (mV display = V*1000; analog inputs vary run-to-run).
|
||||||
|
// Matching the 5-point setpoint pattern proves same unit/test -> correct segmentation.
|
||||||
|
function stimMatch1(t, r) {
|
||||||
|
return [r, r * 1000, r / 1000].some(c => Math.abs(t - c) <= Math.max(0.3, 0.005 * Math.abs(c)));
|
||||||
|
}
|
||||||
|
function stimsMatch(txt, raw) {
|
||||||
|
return txt.length === 5 && raw.length === 5 && txt.every((t, i) => stimMatch1(t, raw[i]));
|
||||||
|
}
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
console.log('Scanning staged .TXT files...');
|
||||||
|
const files = walk(STAGE, []);
|
||||||
|
console.log('Found ' + files.length + ' staged .TXT files');
|
||||||
|
|
||||||
|
// Parse all files, collect SNs
|
||||||
|
const recs = [];
|
||||||
|
let noSn = 0, noAcc = 0;
|
||||||
|
for (const f of files) {
|
||||||
|
let t; try { t = fs.readFileSync(f, 'utf8'); } catch { continue; }
|
||||||
|
const p = parseTxt(t);
|
||||||
|
if (!p.sn) { noSn++; continue; }
|
||||||
|
if (p.errs.length < 5) noAcc++;
|
||||||
|
p.key = decodeSn(p.sn); // DB lookup key (decoded)
|
||||||
|
recs.push({ file: f, ...p });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bulk-load DB rows for these SNs (decoded keys)
|
||||||
|
const sns = [...new Set(recs.map(r => r.key))];
|
||||||
|
const dbMap = new Map();
|
||||||
|
for (let i = 0; i < sns.length; i += 1000) {
|
||||||
|
const chunk = sns.slice(i, i + 1000);
|
||||||
|
const rows = await db.query(
|
||||||
|
'SELECT serial_number, model_number, test_date, raw_data FROM test_records WHERE serial_number = ANY($1)', [chunk]);
|
||||||
|
for (const r of rows) dbMap.set(r.serial_number, r);
|
||||||
|
}
|
||||||
|
|
||||||
|
const out = { missing: [], collision: [], model: [], dbOlder: [], err: [], errRowCount: [], retest: 0, retestSameDay: 0, vasFmt: 0, ok: 0 };
|
||||||
|
for (const r of recs) {
|
||||||
|
const d = dbMap.get(r.key);
|
||||||
|
if (!d) { out.missing.push(r.sn + (r.key !== r.sn ? ' (dec ' + r.key + ')' : '')); continue; }
|
||||||
|
const dbDate = d.test_date && d.test_date.toISOString ? d.test_date.toISOString().slice(0,10) : String(d.test_date);
|
||||||
|
let txtDate = null;
|
||||||
|
if (r.date) { const [mm,dd,yy] = r.date.split('-'); txtDate = `${yy}-${mm}-${dd}`; }
|
||||||
|
|
||||||
|
// Collision: same SN but a genuinely different product family in DB (generic serials like 1-1 reused)
|
||||||
|
if (r.model && d.model_number && normModel(r.model) !== normModel(d.model_number)) {
|
||||||
|
const famTxt = normModel(r.model).replace(/[-0-9].*$/, '');
|
||||||
|
const famDb = normModel(d.model_number).replace(/[-0-9].*$/, '');
|
||||||
|
if (famTxt !== famDb) { out.collision.push(`${r.sn}: txt=${r.model} db=${d.model_number}`); continue; }
|
||||||
|
out.model.push(`${r.sn}: txt=${r.model} db=${d.model_number}`); continue; // same family, diff variant
|
||||||
|
}
|
||||||
|
|
||||||
|
// Retest: DB date newer than the staged file -> ON-CONFLICT updated DB to a later test. Expected.
|
||||||
|
if (txtDate && dbDate > txtDate) { out.retest++; continue; }
|
||||||
|
if (txtDate && dbDate < txtDate) { out.dbOlder.push(`${r.sn}: txt=${r.date} db=${dbDate}`); continue; }
|
||||||
|
|
||||||
|
// Same test run -> error% must match
|
||||||
|
const acc = parseRawAcc(d.raw_data);
|
||||||
|
const de = acc.errs;
|
||||||
|
if (r.errs.length === 5 && de.length === 5) {
|
||||||
|
const maxd = Math.max(...r.errs.map((e,i) => Math.abs(e - de[i])));
|
||||||
|
if (maxd > ERR_TOL) {
|
||||||
|
// Same SN+model+date but error% differs. If the STIM SETPOINTS match, it's the
|
||||||
|
// same unit/test points -> a same-day retest (DB kept a different run). If stim
|
||||||
|
// does NOT match, the wrong record's data is in raw_data -> genuine parse fault.
|
||||||
|
if (stimsMatch(r.stims, acc.stims)) { out.retestSameDay++; continue; }
|
||||||
|
out.err.push(`${r.sn} (${d.model_number}): STIM txt=[${r.stims.join(',')}] raw=[${acc.stims.map(x=>x.toFixed(4)).join(',')}] | err txt=[${r.errs.join(',')}] db=[${de.map(x=>x.toFixed(4)).join(',')}]`); continue;
|
||||||
|
}
|
||||||
|
} else if (r.errs.length === 5 && de.length === 0) {
|
||||||
|
out.vasFmt++; continue; // VAS/single-point format, no 5-row accuracy block in raw_data
|
||||||
|
} else if (r.errs.length === 5 && de.length !== 5) {
|
||||||
|
out.errRowCount.push(`${r.sn} (${d.model_number}): txt 5 rows, raw_data ${de.length}`); continue;
|
||||||
|
}
|
||||||
|
out.ok++;
|
||||||
|
}
|
||||||
|
|
||||||
|
const lines = [];
|
||||||
|
const L = s => { lines.push(s); console.log(s); };
|
||||||
|
L('========== PARSING FIDELITY REPORT ==========');
|
||||||
|
L('Staged .TXT files scanned : ' + files.length);
|
||||||
|
L(' - no SN line (non-standard fmt): ' + noSn);
|
||||||
|
L(' - SN found / compared : ' + recs.length);
|
||||||
|
L(' - .TXT w/o 5 accuracy rows : ' + noAcc);
|
||||||
|
L('Unique SNs looked up in DB : ' + sns.length);
|
||||||
|
L('SNs present in DB : ' + (sns.length - new Set(out.missing).size));
|
||||||
|
L('');
|
||||||
|
L('EXPLAINED (not parsing faults):');
|
||||||
|
L(' Consistent (SN+model+date+5 error% match) : ' + out.ok);
|
||||||
|
L(' Retest, DB newer date than .TXT : ' + out.retest);
|
||||||
|
L(' Retest same-day (stim matches, run differs): ' + out.retestSameDay);
|
||||||
|
L(' VAS/single-point fmt (no 5-row block) : ' + out.vasFmt);
|
||||||
|
L(' Serial collision (generic SN, diff family): ' + out.collision.length);
|
||||||
|
L('');
|
||||||
|
L('NEEDS REVIEW (potential genuine issues):');
|
||||||
|
L(' Missing from DB (after hex-decode) : ' + out.missing.length);
|
||||||
|
L(' Model variant mismatch (same family) : ' + out.model.length);
|
||||||
|
L(' DB OLDER than .TXT (stale DB?) : ' + out.dbOlder.length);
|
||||||
|
L(' GENUINE error% fault (stim ALSO differs) : ' + out.err.length);
|
||||||
|
L(' Accuracy-row-count diff : ' + out.errRowCount.length);
|
||||||
|
const sample = (label, arr) => { if (arr.length) { L(''); L(label + ' (first 20):'); arr.slice(0,20).forEach(x => L(' ' + x)); } };
|
||||||
|
sample('COLLISION (informational)', out.collision);
|
||||||
|
sample('MODEL VARIANT MISMATCH', out.model);
|
||||||
|
sample('DB OLDER THAN .TXT', out.dbOlder);
|
||||||
|
sample('GENUINE FAULT (stim+error differ)', out.err);
|
||||||
|
sample('ROW-COUNT DIFF', out.errRowCount);
|
||||||
|
if (out.missing.length) { L(''); L('MISSING-FROM-DB (first 30): ' + out.missing.slice(0,30).join(', ')); }
|
||||||
|
|
||||||
|
if (REPORT) { fs.writeFileSync(REPORT, lines.join('\n') + '\n'); console.log('\n[written] ' + REPORT); }
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error(e); process.exit(1); });
|
||||||
1
projects/dataforth-dos/dsca-clean-models.json
Normal file
1
projects/dataforth-dos/dsca-clean-models.json
Normal file
@@ -0,0 +1 @@
|
|||||||
|
["DSCA30-01","DSCA30-02","DSCA30-03","DSCA30-06","DSCA30-07","DSCA30-08","DSCA30-08C","DSCA30-09","DSCA30-09C","DSCA30-1944","DSCA30-1945","DSCA30-1946","DSCA31-02","DSCA31-03","DSCA31-06","DSCA31-07","DSCA31-11","DSCA31-12","DSCA31-1273","DSCA31-12C","DSCA31-13","DSCA31-13C","DSCA31-15","DSCA31-1918","DSCA32-01","DSCA32-01C","DSCA32-01E","DSCA34-01","DSCA34-02C","DSCA34-04","DSCA34-04C","DSCA34-05","DSCA34-05C","DSCA34-1858","DSCA36-01","DSCA36-02","DSCA36-03","DSCA36-04","DSCA36-04C","DSCA36-1949","DSCA38-02","DSCA38-03","DSCA38-07","DSCA38-08C","DSCA38-09","DSCA38-09E","DSCA38-12C","DSCA38-12E","DSCA38-1468","DSCA38-1544","DSCA38-15C","DSCA38-16","DSCA38-16C","DSCA38-18C","DSCA38-19","DSCA39-01","DSCA39-02","DSCA39-07","DSCA40-03","DSCA40-05","DSCA40-05C","DSCA40-06","DSCA40-1951","DSCA40-1952","DSCA41-01","DSCA41-02","DSCA41-03","DSCA41-05C","DSCA41-06","DSCA41-09","DSCA41-13","DSCA41-14","DSCA41-15","DSCA41-15E","DSCA42-01","DSCA42-01C","DSCA42-02","DSCA43-10","DSCA43-20E","DSCA47E-08C","DSCA47J-01C","DSCA47J-03","DSCA47K-05","DSCA47K-13","DSCA47K-14","DSCA47N-15","DSCA47T-06","DSCA47T-1928","DSCA49-04","DSCA49-05","DSCA49-1601","DSCA49-1895"]
|
||||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,111 @@
|
|||||||
|
# 2026-06-17 — Dataforth datasheet RTD bug diagnosis + AD2 harness onboarding
|
||||||
|
|
||||||
|
## User
|
||||||
|
- **User:** Mike Swanson (mike)
|
||||||
|
- **Machine:** AD2
|
||||||
|
- **Role:** admin
|
||||||
|
|
||||||
|
## Session Summary
|
||||||
|
|
||||||
|
Diagnosed a Dataforth test-datasheet defect reported 2026-06-17 by John Lehman / Peter Iliya, triggered by a Wellbore Integrity (Joseph Swinehart) cal-cert audit on 8B35 4-wire RTD certificates. The complaint: column headers wrong and some Final Test lines missing. Worked on AD2 (the testdatadb generator + PostgreSQL host), tracing the pipeline upstream from the website copy to the original ground-truth file. Diagnosis only — no changes to the generator or DB.
|
||||||
|
|
||||||
|
Confirmed two independent defects, both in the datasheet renderer `templates/datasheet-exact.js` (ingestion, DB data, and spec files are all correct). Defect A: RTD modules render the input column as resistance (`Rin (ohms)`) when the original/ground-truth file shows temperature (`Temp. (C)`); the stimulus values in `raw_data` are already temperatures (°C), so the numbers are right but the label is wrong and positive values lose their leading `+`. Defect B: the entire DSCA Final-Test parameter list is hardcoded as a single layout that does not match real DSCA module subtypes, producing wrong parameter names, garbage specs (`< 0 mA`, `+/- 0 %`), values mapped to the wrong rows, a mislabeled output column, and dropped lines. The "missing Final Test lines" complaint is a symptom of Defect B, not a separate bug.
|
||||||
|
|
||||||
|
Located the ground truth: the DOS test station writes a fully-rendered `.TXT` datasheet to `C:\STAGE\` before ingestion; it is mirrored to AD2 at `C:\Shares\test\STAGE\<TS>\<encoded-SN>.TXT`. Verified three examples against their originals — 8B35-04 (SN 179553-13, `H9553-13.TXT`), DSCA38-05 (SN 180224-7, `I0224-7.TXT`, a bridge module — NOT RTD), and DSCA34-05C (SN 180007-8, `I0007-8.TXT`, the actual DSCA RTD analog). Clarified for the thread that DSCA38 is a bridge/strain-gauge module, so its issue is Defect B; the DSCA RTD module Peter likely means is DSCA34, which has both A and B. Wrote the full diagnosis to `projects/dataforth-dos/DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md` (process doc + diagnosis + proposed fix).
|
||||||
|
|
||||||
|
The second half of the session was unplanned infrastructure work needed to `/sync` the results: AD2 had no Python (so `sync.sh` could not parse `identity.json`), no `identity.json` at all, and an unset git author identity. Installed Python 3.12.8, created a proper `identity.json` via the documented hand-create + `migrate-identity.sh` flow, and set git authorship. Then synced — which surfaced that `ad2` was 3 months behind main and that `sync.sh` cannot overwrite itself mid-run on Windows. Recovered cleanly, rebased `ad2` onto main (modernizing the fork to harness v1.4.3), relocated the Dataforth context out of the shared `.claude/CLAUDE.md` into `clients/dataforth/CLAUDE.dataforth.md` so future syncs stay conflict-free, and force-pushed.
|
||||||
|
|
||||||
|
## Key Decisions
|
||||||
|
|
||||||
|
- **Diagnose only, verify against the original file, not assumptions.** Pulled the staged DOS-generated `.TXT` as ground truth rather than trusting the code reading. This confirmed the RTD stimulus values are temperatures and the label is the defect.
|
||||||
|
- **Classified the bug as renderer-only.** Ingestion (`multiline.js`), DB `raw_data`, and spec `SENTYPE` are all correct; the fix belongs in `datasheet-exact.js`. Defect A is surgical (fold RTD/sensorNum 7 into the temperature path); Defect B (DSCA) needs a per-subtype rebuild driven by `DSCFIN.DAT` and is larger.
|
||||||
|
- **Installed Python from the official installer, not Chocolatey.** Choco 2.6.0 requires .NET 4.8 (Server 2019 ships 4.7.2); the python.org silent installer is self-contained and avoids a .NET install + reboot.
|
||||||
|
- **Created identity.json via hand-create + migrate-identity.sh** (the documented flow) rather than inventing a generator. Used Mike's identity (strongly signaled by the gitea remote, branch, and user profile memory).
|
||||||
|
- **Did not blind-commit the 205 MB WizTree zip until authorized.** `sync.sh` does `git add -A`; flagged the binary first. User confirmed AD2 is a Dataforth-ops fork and to sync everything.
|
||||||
|
- **Modernized the ad2 fork by rebasing onto main and relocating the Dataforth doc.** Chose "take main's lean CLAUDE.md + move Dataforth context to a fork-specific file" so the fork stays additive and future rebases are conflict-free. The redundant CLAUDE.md-edit commit became empty and was auto-dropped.
|
||||||
|
|
||||||
|
## Problems Encountered
|
||||||
|
|
||||||
|
- **`sync.sh` aborted: "No Python interpreter found."** AD2 had no Python at all. Installed Python 3.12.8 (official amd64 installer, all-users, PATH + `py` launcher); `sync.sh` now detects `py`.
|
||||||
|
- **`identity.json` missing** → commits would attribute as `unknown` and vault/coord fields were absent. Hand-created the core file and ran `migrate-identity.sh` to fill python/ollama/platform/arch/grok/coord_api. Validated all 13 required identity fields present; confirmed it is gitignored.
|
||||||
|
- **Rebase failed mid-run: "unable to create file .claude/scripts/sync.sh: Permission denied."** `sync.sh` was the executing script while git's checkout-to-main tried to overwrite it (Windows file lock). The partial checkout left the working tree half-converted to origin/main (modified CLAUDE.md, deleted sync.sh, 240 untracked origin/main files). Recovered with `git reset --hard HEAD` + `git clean -fd`, then ran `git rebase origin/main` directly (outside sync.sh) — succeeded with only the expected `.claude/CLAUDE.md` conflict.
|
||||||
|
- **CLAUDE.md rebase conflict** (Dataforth fork doc vs main's lean refactor). Resolved by taking main's version (`git checkout origin/main -- .claude/CLAUDE.md`); Dataforth content was preserved beforehand into `clients/dataforth/CLAUDE.dataforth.md`.
|
||||||
|
- **NAS SSH host-key changed / SMB denied.** Could not reach `\\192.168.0.9\test\STAGE` over SSH (host key changed) or SMB (permission denied). Worked around via the rsync daemon (module `test`, user `rsync`) for listing, and found the staged originals already mirrored locally at `C:\Shares\test\STAGE\`.
|
||||||
|
|
||||||
|
## Configuration Changes
|
||||||
|
|
||||||
|
Created:
|
||||||
|
- `C:\ClaudeTools\.claude\identity.json` (gitignored, per-machine) — full identity for AD2
|
||||||
|
- `C:\ClaudeTools\projects\dataforth-dos\DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md` — diagnosis deliverable
|
||||||
|
- `C:\ClaudeTools\clients\dataforth\CLAUDE.dataforth.md` — relocated Dataforth ops context
|
||||||
|
- `C:\ClaudeTools\.claude\memory\project_ad2_dataforth_fork.md` (+ MEMORY.md index line)
|
||||||
|
- Session log (this file)
|
||||||
|
|
||||||
|
Modified:
|
||||||
|
- `.claude/CLAUDE.md` — now the lean fleet doc (was the Dataforth fork doc); rebased from main
|
||||||
|
- Git author identity on AD2: `user.name="Mike Swanson"`, `user.email="mike@azcomputerguru.com"`
|
||||||
|
- Whole harness advanced to main (v1.4.3): new `CLAUDE_EXTENDED.md`, `harness/`, `bootstrap/`, skills/commands
|
||||||
|
|
||||||
|
Installed:
|
||||||
|
- Python 3.12.8 (64-bit, all-users) → `C:\Program Files\Python312\`, `py` launcher at `C:\Windows\py`
|
||||||
|
|
||||||
|
NOT changed (by design — diagnosis only): `templates/datasheet-exact.js`, the PostgreSQL DB, any datasheet content.
|
||||||
|
|
||||||
|
## Credentials & Secrets
|
||||||
|
|
||||||
|
No new credentials created or discovered this session. Used existing documented access:
|
||||||
|
- NAS rsync daemon: `rsync://rsync@192.168.0.9/test` (module `test` = `/data/test`), password `IQ203s32119` (already in the Dataforth context doc).
|
||||||
|
- PostgreSQL (local, AD2): default app creds from `database/db.js` — `testdatadb_app` / `DfTestDB2026!` on `localhost:5432/testdatadb`.
|
||||||
|
|
||||||
|
(AD2 has no SOPS vault cloned; these remain documented in `clients/dataforth/CLAUDE.dataforth.md`.)
|
||||||
|
|
||||||
|
## Infrastructure & Servers
|
||||||
|
|
||||||
|
- **AD2** 192.168.0.6 — testdatadb (Node/Express :3000) + PostgreSQL 18; this host. Hostname `AD2`.
|
||||||
|
- **AD1** 192.168.0.27 — `\\AD1\Engineering`.
|
||||||
|
- **D2TESTNAS** 192.168.0.9 — SMB1 bridge; rsync daemon port 873 module `test`; SSH host key changed this session (publickey/password denied).
|
||||||
|
- testdatadb data: 464,671 records on website; log_types — 5BLOG 196,502 / 7BLOG 121,304 / DSCLOG 79,868 / 8BLOG 63,808 / others.
|
||||||
|
- Impact of Defect A (RTD label): 8B35 5,476 + DSCA34 3,573 + SCM5B34/35 14,887 ≈ 24K certs. Defect B (DSCA template): up to 78,343 DSCLOG certs.
|
||||||
|
|
||||||
|
## Commands & Outputs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Render current generator output vs staged original (read-only diagnosis)
|
||||||
|
cd C:\Shares\testdatadb
|
||||||
|
node -e "const db=require('./database/db');const {renderContent}=require('./database/render-datasheet');(async()=>{const r=await db.queryOne('SELECT * FROM test_records WHERE serial_number=$1',['179553-13']);if(r.test_date&&r.test_date.toISOString)r.test_date=r.test_date.toISOString().slice(0,10);console.log(renderContent(r));await db.close();})()"
|
||||||
|
type C:\Shares\test\STAGE\TS-4L\H9553-13.TXT # ground truth: header = " Temp. (C)"
|
||||||
|
|
||||||
|
# Python install (PowerShell)
|
||||||
|
Start-Process python-3.12.8-amd64.exe -ArgumentList '/quiet','InstallAllUsers=1','PrependPath=1','Include_launcher=1' -Wait # exit 0
|
||||||
|
|
||||||
|
# Onboarding identity
|
||||||
|
# hand-create .claude/identity.json (core fields) then:
|
||||||
|
bash .claude/scripts/migrate-identity.sh # filled python/ollama/platform/arch/grok/coord_api
|
||||||
|
|
||||||
|
# Rebase recovery (sync.sh self-lock)
|
||||||
|
git reset --hard HEAD && git clean -fd
|
||||||
|
git rebase origin/main
|
||||||
|
git checkout origin/main -- .claude/CLAUDE.md && git add .claude/CLAUDE.md
|
||||||
|
GIT_EDITOR=true git rebase --continue
|
||||||
|
git push --force-with-lease origin ad2
|
||||||
|
```
|
||||||
|
|
||||||
|
Key error + resolution: `error: unable to create file .claude/scripts/sync.sh: Permission denied` → run the rebase directly (not via the executing sync.sh).
|
||||||
|
|
||||||
|
## Pending / Incomplete Tasks
|
||||||
|
|
||||||
|
- **Fix Defect A (RTD label/values)** in `templates/datasheet-exact.js`: in the input-header logic and `formatAccuracyLine`, route `sensorNum === 7` (RTD) through the temperature path (`' Temp. (C)'` + `formatSigned(stim, 2, 8)`). Verify leading-space alignment against a thermocouple original. Safe — `7` is reached only by RTD sentypes; no module currently needs `Rin (ohms)`. Awaiting review before changing the generator.
|
||||||
|
- **Fix Defect B (DSCA template)**: rebuild the DSCA Final-Test parameter list + ACCURACY column titles/units per module subtype, driven by `specdata\DSCFIN.DAT` / the legacy QB DSC writer. Larger effort. Until done, treat all DSCA (DSCLOG) website datasheets as unreliable.
|
||||||
|
- After fixes: re-push affected models by clearing `api_uploaded_at` (RE-PUSH is idempotent).
|
||||||
|
- **AD2 tooling gaps**: `jq`, `sops`, `age` not installed; no vault cloned (`D:/vault` absent) → vault sync N/A. coord_api (172.16.3.30) unreachable from Dataforth LAN.
|
||||||
|
- **`sync.sh` self-lock**: upstream fix candidate (re-exec from a temp copy, or rebase before the script can be overwritten) — offered to file as a CT thought.
|
||||||
|
|
||||||
|
## Reference Information
|
||||||
|
|
||||||
|
- Diagnosis doc: `projects/dataforth-dos/DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md`
|
||||||
|
- Generator: `C:\Shares\testdatadb\templates\datasheet-exact.js` (repo: `projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js`)
|
||||||
|
- Ground-truth originals: `C:\Shares\test\STAGE\<TS>\<encoded-SN>.TXT` (8.3 hex-prefix SN encoding: first 2 digits → letter, 55+n)
|
||||||
|
- Examples: 8B35-04 SN 179553-13 (`H9553-13.TXT`, P1RTD4W, MAXIN 600); DSCA38-05 SN 180224-7 (`I0224-7.TXT`, FBRIDGE); DSCA34-05C SN 180007-8 (`I0007-8.TXT`, P1RTD3W)
|
||||||
|
- Task prompt: `Prompt617.txt`
|
||||||
|
- Commits this session (ad2): `49c9eb50` (work sync), `bbb19db2` (relocate Dataforth doc), `c4de16f6` (memory)
|
||||||
|
- Contacts: John Lehman jlehman@dataforth.com, Peter Iliya pIliya@dataforth.com
|
||||||
@@ -0,0 +1,323 @@
|
|||||||
|
# DSCA Datasheet Fix 2 — STAGE 2 wire-in, STAGE 3 validator, publish of 68 clean models
|
||||||
|
|
||||||
|
## Update: 14:00 PT — DSCA33/45 accuracy-data reverse-engineered; 54/56 validated; 1,452 published
|
||||||
|
|
||||||
|
Picked up the 5070 Hoffman-recovery handoff and finished DSCA33/45 end-to-end. After wiring the
|
||||||
|
mined templates (gated, accHeader), reverse-engineered the accuracy-block numeric formatting against
|
||||||
|
the live Hoffman originals (validation harness = oracle):
|
||||||
|
- mA-output models store calc (and, for DSCA45, meas) in AMPS -> x1000 to display mA; DSCA33 stores
|
||||||
|
meas already in display unit (NOT scaled), DSCA45 scales both.
|
||||||
|
- DSCA33 (AC-RMS): stim/calc/meas UNSIGNED, error signed; stim = AC input, 3 dp.
|
||||||
|
- DSCA45 (frequency): stim = UNSIGNED integer Hz; calc/meas/error SIGNED.
|
||||||
|
- Math.fround on accuracy values (QB single precision). Final-Test: leading-zero drop only when the
|
||||||
|
value overflows QB's 6-char field ("-0.0005"->"-.0005", "-0.750" keeps it); spec-less section
|
||||||
|
sub-heads (Zero-Crossing Input / TTL Input) render with NO status; DSCA33 prints a "Check List"
|
||||||
|
header.
|
||||||
|
- slotmap-from-hoffman.js recovered the 13 DSCA33 models the staged multi-unit derivation couldn't
|
||||||
|
(vintage), matching the Hoffman _srcSerial original's Final-Test measured values (at display
|
||||||
|
precision) to the DB STATUS entries.
|
||||||
|
|
||||||
|
Validation (content-normalized byte-compare vs live Hoffman): **54 of 56 models PASS** and are
|
||||||
|
marked `validated:true` (render gate). 2 holdouts (DSCA33-04A, DSCA33-1891) each have ONE accuracy
|
||||||
|
cert at a rounding boundary where fround rounds opposite the original -> left UNvalidated, render
|
||||||
|
null (safe). DSCA33-1948 + DSCA45-1746 (24 units) have no Hoffman original.
|
||||||
|
|
||||||
|
Published the gap SAFELY: the stale inventory means `api_uploaded_at IS NULL` can't be trusted as
|
||||||
|
"absent from Hoffman", so probed each of the 1,578 unuploaded PASS serials with a GET; 1,452 were
|
||||||
|
absent (404), 126 already live. Pushed ONLY the 1,452 absent -> **created=1452 updated=0 unchanged=0
|
||||||
|
errors=0** (zero overwrites of pristine originals — the handoff's hard requirement). Commits
|
||||||
|
`3a7ac35d` (wiring), `b5bc0409` (accuracy + 54 validated). Tools: validate-dsca3345.js,
|
||||||
|
slotmap-from-hoffman.js, publish-dsca3345-gap.js.
|
||||||
|
|
||||||
|
## Update: 08:00 PT — diagnosed DSCA33/DSCA45 missing-specs gap (left blocked, documented)
|
||||||
|
|
||||||
|
Dug into why DSCA33-*/DSCA45-* render null. Root cause is a DATA GAP, not a code bug: their MAIN
|
||||||
|
spec records are missing from every recovered `specdata/*.DAT`. DSCA33 appears in NO DAT file at all;
|
||||||
|
DSCA45 appears only in `DSCFIN.DAT` (Final-Test layout, lacks SENTYPE/MAXIN/input-type). Loaded DSCA
|
||||||
|
prefixes jump 32->34 and 43->47. Almost certainly lost in the cryptolocker wipe. Blocks ~8,763 PASS
|
||||||
|
certs (DSCA33 3,350/35 models, DSCA45 5,413/23 models).
|
||||||
|
|
||||||
|
The Final-Test data is NOT the blocker — a DSCA45 cert's Final-Test block renders correctly via its
|
||||||
|
slotMap with a stub spec (2 trivial diffs). Blockers are: (1) `render-datasheet.js` bails on missing
|
||||||
|
specs before rendering; (2) special accuracy headers the sensor logic can't make — DSCA45
|
||||||
|
`Frequency (Hz)`/`Output (V)`, DSCA33 `Vin (mVAC)`/`Output (VDC)`; (3) non-status rows (DSCA45
|
||||||
|
`Zero-Crossing Input`/`TTL Input`) that have no PASS in golden but the renderer appends one.
|
||||||
|
|
||||||
|
Decision (Mike): leave blocked, documented. Recorded as memory `project_dsca33_45_spec_gap`. The
|
||||||
|
clean fix is obtaining the authoritative DSCA33/DSCA45 main spec files from Dataforth (added to the
|
||||||
|
#32441 handoff below); the self-contained alternative is template-driving the accuracy block. Final-
|
||||||
|
Test slotMaps for the matchable models are already derived and ready for when specs arrive.
|
||||||
|
|
||||||
|
## Update: 07:52 PT — per-model slot maps resolve ambiguous DSCA layouts + re-publish of 92
|
||||||
|
|
||||||
|
Took on the ambiguous-layout families (models the count-guard skipped because raw_data carries more
|
||||||
|
or fewer value-bearing STATUS entries than the template's spec-bearing rows — the test program
|
||||||
|
measures slots the printed sheet omits, e.g. DSCA49's 5mA load pair).
|
||||||
|
|
||||||
|
New tool `derive-dsca-slotmaps.js`: derives a per-model `slotMap` (absolute statusEntries index per
|
||||||
|
spec-bearing row) by greedily matching a staged original's printed values against the DB raw_data
|
||||||
|
STATUS entries (same fround formatting), then picking the candidate map that validates against the
|
||||||
|
most units. Models are grouped by identical row-name signature and one map is derived per group from
|
||||||
|
ALL sibling units — this disambiguates duplicate values (DSCA49-04 alone has only 2 staged units,
|
||||||
|
both with 5mA==50mA linearity, so greedy picked the wrong slot; its siblings' 25 units force the
|
||||||
|
correct [0,1,2,3,6,7,8,9,11,13,15] map). Stored as `slotMap` in dsca-templates.json. Renderer
|
||||||
|
consults slotMap ONLY when the sequential zip fails, so the 88 clean models keep their path (no
|
||||||
|
regression).
|
||||||
|
|
||||||
|
STAGE 3 re-validation: FINAL-TEST CLEAN **88 -> 92**; 134 more certs render (null 450 -> 316);
|
||||||
|
matches 2278 -> 2412; same 6 retest-vintage dirty, no new mismatches. Re-pushed all 92 clean models
|
||||||
|
(41,362 serials): **updated=4024 unchanged=36109 created=0 errors=0 skipped=1229** (skips down from
|
||||||
|
2165 — ~936 previously-guarded certs now render via slotMap).
|
||||||
|
|
||||||
|
DSCA49 family + DSCA40-03 group now clean and live. Still blocked, SEPARATE gap (not layout
|
||||||
|
ambiguity): DSCA45-* and most DSCA33-* render null because they have NO spec-reader entries
|
||||||
|
(`render-datasheet.js` bails before rendering) — their slotMaps are derived and ready, they just
|
||||||
|
need spec coverage. One DSCA33 group (DSCA33-02/03/03A/04/05/1948) didn't reach the slotMap
|
||||||
|
validation threshold (best 19/35 units). Commit `e9262f57`.
|
||||||
|
|
||||||
|
Net live DSCA Final-Test: 92/126 templated models content-clean. Remaining: 6 retest-vintage,
|
||||||
|
DSCA45-*/DSCA33-* missing specs, 1 DSCA33 group below threshold, ~32 models with no DB-matched
|
||||||
|
staged unit, and 231 untemplated models (STAGE 1 extension).
|
||||||
|
|
||||||
|
## Update: 07:33 PT — data-driven DSCA load note (DSCA39 footer-artifact fix) + re-publish of 88
|
||||||
|
|
||||||
|
Fixed the DSCA39 footer-note artifact properly at the STAGE 1 extractor level. The "Standard
|
||||||
|
output load for test is 250 ohms." line is a footer note, not a parameter; the extractor had
|
||||||
|
captured it as a column-truncated row ("Standard output load for te"), and the renderer's
|
||||||
|
`OUTSIGTYPE==='CURRENT'` emission was wrong both ways (printed a spurious note after the underline
|
||||||
|
for many `-C` current models that never had it, and never placed it for the models that do).
|
||||||
|
|
||||||
|
Data-driven fix: `derive-dsca-templates.js` now captures the note as a per-model `loadNote`
|
||||||
|
property and excludes it from rows; `datasheet-exact.js` emits `loadNote` (blank + note) before the
|
||||||
|
footer underline only for models that have it, and the OUTSIGTYPE emission was removed. Regenerated
|
||||||
|
`dsca-templates.json` — surgically clean (only the 5 DSCA39 models changed; 121 others byte-identical).
|
||||||
|
|
||||||
|
STAGE 3 re-validation: FINAL-TEST CLEAN **85 -> 88**, mismatches 9 -> 6, matches 2206 -> 2278.
|
||||||
|
DSCA39-01/02/07 now fully clean (DSCA39-01 byte-content-verified). No regression — `-C` models
|
||||||
|
stayed clean and no longer carry the spurious note. Re-pushed all 88 clean models (38,274 serials):
|
||||||
|
**updated=7092 unchanged=29017 created=0 errors=0 skipped=2165**.
|
||||||
|
|
||||||
|
The 6 still-dirty models (DSCA38-05/-1793/-19C/-19E, DSCA39-05, DSCA39-1950) are ALL retest
|
||||||
|
data-vintage (staged .TXT is an older run than the DB record; Supply Current / Linearity differ by
|
||||||
|
more than rounding) — not render bugs, cannot reconcile against an older sheet. Commit `61f54dc4`.
|
||||||
|
|
||||||
|
Net DSCA Final-Test render state: 88/126 templated models content-clean and live; 6 vintage-only;
|
||||||
|
~32 ambiguous-layout families still skipped (need per-subtype slot mapping); 231 untemplated models
|
||||||
|
still need a STAGE 1 extension.
|
||||||
|
|
||||||
|
## Update: 07:23 PT — rounding-mode fix (QB single-precision) + re-publish of 85 clean models
|
||||||
|
|
||||||
|
Fixed the rounding-mode issue behind the 26 last-digit-diff models. Root cause: the DOS
|
||||||
|
QuickBASIC computed/stored values as single-precision floats, so half-boundary rounding follows
|
||||||
|
single, not double, precision. `formatMeasuredExact` now applies `Math.fround(value)` before
|
||||||
|
`toFixed`. Verified each boundary case against the staged golden (9.9995 -> "10.000", 46.85 ->
|
||||||
|
"46.8", .45 -> "0.4", 3.3325 -> "3.332").
|
||||||
|
|
||||||
|
STAGE 3 re-validation: FINAL-TEST CLEAN models **68 -> 85** (+17), mismatches 26 -> 9, cert matches
|
||||||
|
2123 -> 2206. Zero regression — every remaining dirty model was already dirty; no clean model flipped.
|
||||||
|
|
||||||
|
Re-pushed all 85 clean models (37,168 PASS serials): **updated=6054 unchanged=28949 created=0
|
||||||
|
errors=0 skipped=2165**. Live site is now fround-correct across 85 DSCA models (the 6,054 updates =
|
||||||
|
boundary corrections in the original 68 + the 17 newly-clean models).
|
||||||
|
|
||||||
|
The 9 still-dirty models are NOT rounding: 4 (DSCA38-05/-1793/-19C/-19E) are Supply Current retest
|
||||||
|
data-vintage (staged .TXT predates the DB record — not a render bug); 5 (DSCA39-01/02/05/07/1950)
|
||||||
|
are the STAGE 1 footer-note artifact ("Standard output load..." mis-captured as a truncated row).
|
||||||
|
A renderer-side fix for the footer note was attempted but regressed 24 clean current-output `-C`
|
||||||
|
models (the note placement differs per golden), so it was reverted — this needs a targeted STAGE 1
|
||||||
|
extractor fix, not a renderer change. Commit `14ee61dc`.
|
||||||
|
|
||||||
|
## User
|
||||||
|
- **User:** Mike Swanson (mike)
|
||||||
|
- **Machine:** AD2
|
||||||
|
- **Role:** admin
|
||||||
|
|
||||||
|
## Session Summary
|
||||||
|
|
||||||
|
Picked up the AD2-local handoff `projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md`
|
||||||
|
(ref Syncro #32441) and executed Fix 2 (DSCA Final-Test rebuild) STAGE 2 and STAGE 3, then
|
||||||
|
published the validated subset to the live Hoffman site. Work was on the DEPLOYED pipeline at
|
||||||
|
`C:\Shares\testdatadb` (Node + PostgreSQL 18); repo copies under
|
||||||
|
`projects/dataforth-dos/datasheet-pipeline/implementation/` are stale and were reconciled after.
|
||||||
|
|
||||||
|
STAGE 2 wired the STAGE-1 output `dsca-templates.json` (126 per-model layouts) into the deployed
|
||||||
|
`templates/datasheet-exact.js`. For `family === 'DSCA'` the Final-Test block now renders parameter
|
||||||
|
names + specs from the staged template rows (not the single hardcoded `DATA_LINES['DSCA']` +
|
||||||
|
`buildTSpecs` DSCA branch); value-bearing `raw_data` STATUS groups map positionally onto the
|
||||||
|
spec-bearing rows; empty-spec rows (240VAC Withstand / Hi-Pot) render blank+PASS (the duplicate
|
||||||
|
hardcoded footer for DSCA was removed). The ACCURACY block now uses the template `accOut`
|
||||||
|
(`Output (V)`/`Output (mA)`) with `-` rule separators instead of `Vout (V)` + `=`. Two real
|
||||||
|
defects were found and fixed in the process (both are the handoff's "lines drop / wrong values"
|
||||||
|
defect): negative signs were dropped because value parsing started at index 5 instead of 4, and
|
||||||
|
`parseRawData` always consumed the line after the 5 accuracy points as a step-response placeholder
|
||||||
|
— but many DSCA models omit that bare `0` line, so the first STATUS group (and its rows) was being
|
||||||
|
discarded. Both fixes were validated against the DSCA38-05 golden staged original: the rebuilt
|
||||||
|
Final-Test block is byte-for-byte identical (the only residual diffs are deferred cosmetic ACCURACY
|
||||||
|
column spacing).
|
||||||
|
|
||||||
|
STAGE 3 built a read-only validator (`tools/validate-dsca-stage3.js`) that, for every staged DSCA
|
||||||
|
original we have ground truth for (2,806 across 126 models), looks up the DB record, renders it
|
||||||
|
through the live path, and content-compares. The gate is the FINAL TEST RESULTS section with rule
|
||||||
|
lines canonicalized and whitespace collapsed (so the deferred column-spacing cosmetic does not
|
||||||
|
register); accuracy-section diffs are reported separately. Verdict: 68 models FINAL-TEST
|
||||||
|
CONTENT-CLEAN (2,123/2,316 compared certs match exactly, 91.7%), 26 models with measured-value
|
||||||
|
last-digit diffs only, and ~32 models rendering null via the count-guard. A safety guard was added:
|
||||||
|
when a model's value count != its spec-row count the positional zip is ambiguous (the subtype
|
||||||
|
measures load points the template omits, e.g. DSCA49's 5mA pair), so the cert is skipped/flagged
|
||||||
|
rather than emitting misaligned data.
|
||||||
|
|
||||||
|
Per Mike's direction, published the 68 STAGE-3-clean models. Restarted the `testdatadb` service so
|
||||||
|
the new template is live, canaried a single cert (DSCA30-01 / 148059-3 -> updated, 0 errors), then
|
||||||
|
re-pushed all 30,423 PASS certs for the 68 clean models via `uploadBySerialNumbers` from a fresh
|
||||||
|
node process. Result: updated=26,022, unchanged=2,738, created=0, errors=0, skipped=1,663. The
|
||||||
|
26,022 updates replaced the old defective DSCA renders on the live site with the rebuilt, validated
|
||||||
|
ones; the 1,663 skips are the count-guard correctly refusing individual ambiguous certs.
|
||||||
|
|
||||||
|
All work committed to the `ad2` branch (4 commits) and pushed. No vault access on AD2 (no
|
||||||
|
sops/age, vault repo absent), so the Syncro #32441 ticket update could not be posted from here —
|
||||||
|
left as a handoff to GURU-5070 (see Pending / Handoff).
|
||||||
|
|
||||||
|
## Key Decisions
|
||||||
|
|
||||||
|
- **Skip, don't guess, on ambiguous layouts.** When a DSCA model's value-bearing STATUS count !=
|
||||||
|
its spec-bearing template-row count, the simple positional zip would misplace values. Chose to
|
||||||
|
return null (skip + flag for STAGE 3 per-subtype mapping) rather than publish misaligned data,
|
||||||
|
per the handoff's "do not guess" discipline. This is what produces the 1,663 push skips.
|
||||||
|
- **Scoped the parser/format fixes to DSCA only.** The step-response detection fix and the
|
||||||
|
index-4 value-start fix live in `formatMeasuredExact` (new) and a DSCA-gated branch in
|
||||||
|
`parseRawData`, leaving the already-validated 5B/8B/7B/DSCT/SCMVAS paths byte-unchanged.
|
||||||
|
- **DSCA decimal code N -> toFixed(N) exactly.** Unlike 5B/8B (where format code 2 means 1
|
||||||
|
decimal), DSCA uses the trailing status digit as a literal decimal-place count; verified against
|
||||||
|
the golden (e.g. Output Reg code 2 -> "0.00").
|
||||||
|
- **Gate STAGE 3 on the Final-Test section, not the whole sheet.** ACCURACY-block column spacing
|
||||||
|
is the handoff's explicitly-deferred cosmetic gap; canonicalizing rule lines + collapsing
|
||||||
|
whitespace isolates real content diffs (names/values/specs/statuses) from spacing noise.
|
||||||
|
- **Published only the 68 clean models.** Highest-confidence set; idempotent push replaces
|
||||||
|
defective live certs. Left the 26 last-digit-diff models, ~32 ambiguous layouts, and 231
|
||||||
|
untemplated models for follow-up.
|
||||||
|
- **Did not restart-then-publish through the running service.** Re-push ran from a fresh node
|
||||||
|
process (picks up new template via `require`); the service restart only refreshes the internal
|
||||||
|
tool's on-demand renders.
|
||||||
|
|
||||||
|
## Problems Encountered
|
||||||
|
|
||||||
|
- **Sync rebase conflict in `errorlog.md`** — two machines appended entries concurrently. Resolved
|
||||||
|
by keeping both entries (append-only log) and `git rebase --continue`.
|
||||||
|
- **`sync.sh` push failed (`src refspec main does not match any`)** — known AD2 fork gotcha
|
||||||
|
(sync.sh is not fork-aware on push). Pushed manually with `git push --force-with-lease origin ad2`
|
||||||
|
(rebase rewrote history). See memory `project_ad2_dataforth_fork`.
|
||||||
|
- **Dropped rows on DSCA models without a step-response line** — `parseRawData` ate the first
|
||||||
|
STATUS group. Fixed by skipping the step-response consume for DSCA when the next line starts with
|
||||||
|
PASS/FAIL.
|
||||||
|
- **Negative measured values rendered without sign** — value substring started at index 5 (assumed
|
||||||
|
a space at index 4), but negatives put `-` at index 4. Fixed to start at index 4.
|
||||||
|
- **First STAGE 3 run showed 0 matches / 2,316 mismatches** — the validator's normalization wasn't
|
||||||
|
canonicalizing rule lines, so the deferred ACCURACY separator dash-count cosmetic masked all
|
||||||
|
downstream content. Reworked to canonicalize rule lines and gate on the Final-Test section.
|
||||||
|
- **No vault access on AD2** — cannot fetch Syncro creds to post #32441. Handoff to 5070 (below).
|
||||||
|
|
||||||
|
## Configuration Changes
|
||||||
|
|
||||||
|
Deployed (live, outside repo — `C:\Shares\testdatadb`):
|
||||||
|
- `templates/datasheet-exact.js` — DSCA wire-in (edited). Save-state:
|
||||||
|
`templates/datasheet-exact.js.bak-2026-06-18-1334`.
|
||||||
|
- `_validate_dsca_stage3.js`, `_push_clean68.js`, `_clean-models.json`, `_dsca-stage3-report.txt`,
|
||||||
|
`_stage3-run.log`, `_push-clean68.log` — operational scripts/outputs (created).
|
||||||
|
|
||||||
|
Repo (`ad2` branch):
|
||||||
|
- `projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js` — reconciled (modified).
|
||||||
|
- `projects/dataforth-dos/datasheet-pipeline/implementation/dsca-templates.json` — added.
|
||||||
|
- `projects/dataforth-dos/tools/validate-dsca-stage3.js` — added.
|
||||||
|
- `projects/dataforth-dos/tools/push-clean68.js` — added.
|
||||||
|
- `projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt` — added.
|
||||||
|
- `projects/dataforth-dos/dsca-clean68-models.json` — added (the 68 published models).
|
||||||
|
- `errorlog.md` — conflict resolution during sync.
|
||||||
|
|
||||||
|
## Credentials & Secrets
|
||||||
|
|
||||||
|
No new credentials discovered or created this session. Referenced (already vaulted per the handoff
|
||||||
|
at `clients/dataforth/testdatadb-postgres`):
|
||||||
|
- PostgreSQL superuser `postgres` / `Paper123!@#`; app `testdatadb_app` / `DfTestDB2026!`
|
||||||
|
(`db.js` defaults: host localhost:5432, database `testdatadb`).
|
||||||
|
- Hoffman/CloudFilter uploader creds: `C:\ProgramData\dataforth-uploader\credentials.json`
|
||||||
|
(fields CF_TOKEN_URL, CF_API_BASE, CF_CLIENT_ID, CF_CLIENT_SECRET, CF_SCOPE). Not vaulted; lives
|
||||||
|
on the box only.
|
||||||
|
|
||||||
|
## Infrastructure & Servers
|
||||||
|
|
||||||
|
- **AD2** (192.168.0.6) — Dataforth domain controller; runs the deployed pipeline at
|
||||||
|
`C:\Shares\testdatadb`. Windows service `testdatadb` (Automatic, Running) — restarted this session.
|
||||||
|
- **PostgreSQL 18** on AD2 (localhost:5432, db `testdatadb`); table `test_records`.
|
||||||
|
- **Staged originals**: `C:/Shares/test/STAGE/**/*.TXT` (11,956 total; 2,806 DSCA) — STAGE-1 source
|
||||||
|
and STAGE-3 ground truth.
|
||||||
|
- **Hoffman API** (`CF_API_BASE/api/v1/TestReportDataFiles/bulk` POST, idempotent; GET is paginated
|
||||||
|
list only) — the live customer-facing datasheet site.
|
||||||
|
|
||||||
|
## Commands & Outputs
|
||||||
|
|
||||||
|
- Sync conflict recovery: `git add errorlog.md && GIT_EDITOR=true git rebase --continue` then
|
||||||
|
`git push --force-with-lease origin ad2`.
|
||||||
|
- Module load-check after each edit: `node -e 'require("./templates/datasheet-exact.js")'`.
|
||||||
|
- STAGE 3 run: `node _validate_dsca_stage3.js` -> `_dsca-stage3-report.txt`. Summary: 68 clean,
|
||||||
|
26 Final-Test mismatch (193 certs), ~32 null; 2,123/2,316 match.
|
||||||
|
- Canary: `uploadBySerialNumbers(["148059-3"])` -> `updated=1 errors=0`.
|
||||||
|
- Full push: `node _push_clean68.js` -> `created=0 updated=26022 unchanged=2738 errors=0 skipped=1663`
|
||||||
|
over 30,423 PASS certs.
|
||||||
|
|
||||||
|
## Pending / Incomplete Tasks
|
||||||
|
|
||||||
|
Not yet published / remaining DSCA work:
|
||||||
|
1. **26 models, last-digit measured diffs** — two causes: (a) QB single-precision half-up rounding
|
||||||
|
vs JS double `toFixed` (e.g. raw 9.9995 code3 -> "9.999" here, "10.000" in golden) — fixable but
|
||||||
|
float-precision-sensitive, re-run validator to confirm no regression; (b) retest data-vintage,
|
||||||
|
where the staged `.TXT` is an older test run than the DB latest-wins record (Fix 3) — not a
|
||||||
|
render bug.
|
||||||
|
2. **~32 ambiguous layouts** (DSCA33-*, DSCA45-*, DSCA49-* families) — need the canonical
|
||||||
|
per-subtype slot mapping (status entry index -> canonical DSCA slot -> template row by name);
|
||||||
|
currently skipped (null) by the count-guard.
|
||||||
|
3. **231 untemplated models / 23,866 certs** — need a STAGE 1 extension (more staged originals;
|
||||||
|
only 126/357 DSCA models in the DB currently have a template — 70.1% of certs).
|
||||||
|
4. **Fix 5** (backfill 379 cryptolocker-era units from staged `.TXT`) and Fix-3 cleanup items — not
|
||||||
|
started this session.
|
||||||
|
|
||||||
|
## Note for mike (handoff to GURU-5070)
|
||||||
|
|
||||||
|
AD2 has no vault / Discord / Syncro access from here (no sops/age, `D:/vault` absent), so these
|
||||||
|
outbound actions are left for the GURU-5070 session, which has them. Mike will run these on 5070.
|
||||||
|
|
||||||
|
**TODO 1 — post an internal/hidden note to Syncro #32441** (customer-facing thread is John Lehman),
|
||||||
|
text ready to paste:
|
||||||
|
|
||||||
|
> Fix 2 (DSCA Final-Test rebuild) STAGE 2 + STAGE 3 complete and published. Wired the per-model staged
|
||||||
|
> templates into the live renderer and fixed a series of render defects: dropped negative signs;
|
||||||
|
> dropped Final-Test rows on models lacking the step-response line; QB single-precision rounding (via
|
||||||
|
> Math.fround); the DSCA39 "Standard output load" footer-note artifact (data-driven loadNote); and the
|
||||||
|
> ambiguous multi-load layouts (per-model slot maps, e.g. DSCA49's 5mA load pair). Built a STAGE 3
|
||||||
|
> validator over 2,806 staged DSCA originals. **92 of 126 templated DSCA models are Final-Test
|
||||||
|
> content-clean and published to Hoffman, 0 errors**, replacing the old defective DSCA renders live.
|
||||||
|
> Remaining: 6 models with retest data-vintage diffs (staged sheet older than the DB latest-wins
|
||||||
|
> record — not a render bug); one DSCA33 group below the slot-map validation threshold; ~58
|
||||||
|
> DSCA33-*/DSCA45-* models awaiting a spec file (see TODO 2); and 231 untemplated models needing a
|
||||||
|
> STAGE 1 extension. Byte-for-byte DOS cosmetic fidelity (leading rule line + ~1-space input-column
|
||||||
|
> spacing) still deferred. Details: `ad2` branch commits + `DSCA-STAGE3-REPORT-2026-06-18.txt`.
|
||||||
|
|
||||||
|
**TODO 2 — SUPERSEDED (do NOT ask John for DSCA33/45 specs).** The 5070 session found the DSCA33/45
|
||||||
|
originals survived on the Hoffman API and mined 56/58 models into `dsca33-45-templates.json`. The AD2
|
||||||
|
session has since wired them in and validated against Hoffman (see the later update below). Only
|
||||||
|
DSCA33-1948 + DSCA45-1746 (24 units) lack an original. Refs: memory
|
||||||
|
`project_dsca33_45_resolved_via_hoffman`, doc `DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md`. (Still note
|
||||||
|
the deferred byte-fidelity disclosure to John per the original FIX2-5 handoff once the effort is done.)
|
||||||
|
|
||||||
|
Nothing else on AD2 is pending — all code is committed to `ad2` and the live Hoffman site is current
|
||||||
|
(92 clean models published, 0 errors).
|
||||||
|
|
||||||
|
## Reference Information
|
||||||
|
|
||||||
|
- Handoff: `projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md`
|
||||||
|
- STAGE 3 report: `projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt`
|
||||||
|
- Published set: `projects/dataforth-dos/dsca-clean-models.json` (92 models, final)
|
||||||
|
- Ticket: Syncro **#32441** (customer-facing thread: John Lehman)
|
||||||
|
- `ad2` commits this session: `e7fa7cc6` (STAGE 2), `c03bdc9a` (STAGE 3 validator+report),
|
||||||
|
`f798e8ea` (publish artifacts); `fc9fff81` (sync auto-commit, errorlog conflict resolution).
|
||||||
|
- Deployed render path: `database/render-datasheet.js` -> `templates/datasheet-exact.js`;
|
||||||
|
push path: `database/upload-to-api.js` (`uploadBySerialNumbers`, BATCH=100).
|
||||||
173
projects/dataforth-dos/tools/derive-dsca-slotmaps.js
Normal file
173
projects/dataforth-dos/tools/derive-dsca-slotmaps.js
Normal file
@@ -0,0 +1,173 @@
|
|||||||
|
// Fix 2 — derive per-model DSCA slot maps for the ambiguous layouts.
|
||||||
|
//
|
||||||
|
// Problem: for some DSCA subtypes the raw_data STATUS groups carry MORE (or fewer)
|
||||||
|
// value-bearing entries than the template's spec-bearing rows — the test program
|
||||||
|
// measures slots (e.g. an extra 5mA load pair) that the printed sheet omits. A
|
||||||
|
// simple in-order zip then misaligns values onto rows, so those models are skipped
|
||||||
|
// by the renderer's count-guard.
|
||||||
|
//
|
||||||
|
// Fix: for each such model, pair a staged original with its DB raw_data and greedily
|
||||||
|
// match each printed measured value to a STATUS entry (same fround formatting the
|
||||||
|
// renderer uses), recording the ABSOLUTE statusEntries index per spec-bearing row.
|
||||||
|
// That ordered subsequence is the slotMap; the renderer reads statusEntries[slotMap[s]]
|
||||||
|
// for the s-th spec-bearing row. Stored in dsca-templates.json as `slotMap`.
|
||||||
|
//
|
||||||
|
// Data-vintage safe: a staged unit that is a retest (printed values != DB record)
|
||||||
|
// won't match cleanly; we try multiple staged units per model and accept the first
|
||||||
|
// that matches ALL spec rows. Read-only except the templates JSON it rewrites.
|
||||||
|
const fs = require('fs'), path = require('path');
|
||||||
|
// This tool is specific to the deployed pipeline (it reads the staged originals and
|
||||||
|
// rewrites the deployed templates JSON), so it requires the deployed modules by
|
||||||
|
// absolute path and is runnable from anywhere.
|
||||||
|
const DEPLOY = 'C:/Shares/testdatadb';
|
||||||
|
const db = require(DEPLOY + '/database/db');
|
||||||
|
const dse = require(DEPLOY + '/templates/datasheet-exact');
|
||||||
|
|
||||||
|
const STAGE = 'C:/Shares/test/STAGE';
|
||||||
|
// Default to the staged-original templates; point at the Hoffman-mined DSCA33/45 set
|
||||||
|
// via DSCA_TPL when deriving slotMaps for those families.
|
||||||
|
const OUT = process.env.DSCA_TPL || (DEPLOY + '/dsca-templates.json');
|
||||||
|
|
||||||
|
function walk(d, out) { let it = []; try { it = fs.readdirSync(d, { withFileTypes: true }); } catch { return out; } for (const e of it) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p, out); else if (/\.txt$/i.test(e.name)) out.push(p); } return out; }
|
||||||
|
function colSpans(sep) { const cols = []; let m; const re = /=+/g; while ((m = re.exec(sep))) cols.push([m.index, m.index + m[0].length]); return cols; }
|
||||||
|
|
||||||
|
// fround + toFixed(decimal-code), value start at index 4 — must match formatMeasuredExact.
|
||||||
|
function fmt(statusStr) {
|
||||||
|
if (!statusStr || statusStr.length <= 4) return null;
|
||||||
|
const decimalDigit = statusStr[statusStr.length - 1];
|
||||||
|
const valueStr = statusStr.substring(4, statusStr.length - 1).trim();
|
||||||
|
const parsed = parseFloat(valueStr);
|
||||||
|
if (isNaN(parsed)) return valueStr;
|
||||||
|
const v = Math.fround(parsed);
|
||||||
|
const d = parseInt(decimalDigit, 10);
|
||||||
|
return isNaN(d) ? v.toFixed(1) : v.toFixed(d);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse the staged Final-Test section -> ordered list of spec-bearing printed values.
|
||||||
|
function stagedPrintedValues(t) {
|
||||||
|
const L = t.replace(/\r\n/g, '\n').split('\n');
|
||||||
|
const fi = L.findIndex(l => /FINAL TEST RESULTS/.test(l)); if (fi < 0) return null;
|
||||||
|
let hi = -1; for (let i = fi + 1; i < L.length; i++) { if (/Parameter\s+Measured/.test(L[i])) { hi = i; break; } } if (hi < 0) return null;
|
||||||
|
const sep = L[hi + 1] || ''; const cols = colSpans(sep); if (cols.length < 4) return null;
|
||||||
|
const [pc, mc, sc, stc] = cols;
|
||||||
|
const vals = [];
|
||||||
|
for (let i = hi + 2; i < L.length; i++) {
|
||||||
|
const l = L[i];
|
||||||
|
if (/Check List|^\s*_{5,}/.test(l)) break;
|
||||||
|
if (!l.trim()) continue;
|
||||||
|
if (/^\s*Standard output load/i.test(l)) continue;
|
||||||
|
const measured = (l.slice(mc[0], sc[0]) || '').trim();
|
||||||
|
// spec column ONLY (cols 48..69) — not the trailing Status column (PASS),
|
||||||
|
// so empty-spec rows (240VAC Withstand / Hi-Pot) are correctly skipped.
|
||||||
|
const spec = (l.slice(sc[0], stc[0]) || '').trim();
|
||||||
|
if (!spec) continue;
|
||||||
|
const v = measured.split(/\s+/)[0]; // strip trailing unit
|
||||||
|
if (v === '') return null; // a spec row with no printed value -> can't use this unit
|
||||||
|
vals.push(v);
|
||||||
|
}
|
||||||
|
return vals;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Greedy in-order match printed values -> absolute statusEntries indices.
|
||||||
|
function greedyMap(printed, statusEntries) {
|
||||||
|
const map = []; let j = 0;
|
||||||
|
for (const pv of printed) {
|
||||||
|
let found = -1;
|
||||||
|
for (let k = j; k < statusEntries.length; k++) {
|
||||||
|
if (fmt(statusEntries[k]) === pv) { found = k; break; }
|
||||||
|
}
|
||||||
|
if (found < 0) return null;
|
||||||
|
map.push(found); j = found + 1;
|
||||||
|
}
|
||||||
|
return map;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Does this slotMap reproduce a unit's printed values exactly?
|
||||||
|
function mapMatches(map, printed, statusEntries) {
|
||||||
|
if (map.length !== printed.length) return false;
|
||||||
|
for (let s = 0; s < map.length; s++) {
|
||||||
|
if (fmt(statusEntries[map[s]]) !== printed[s]) return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
const onlyModels = process.argv.slice(2).filter(a => !a.startsWith('--'));
|
||||||
|
const tpl = JSON.parse(fs.readFileSync(OUT, 'utf8'));
|
||||||
|
// index staged DSCA files by model
|
||||||
|
const files = walk(STAGE, []);
|
||||||
|
const byModel = {};
|
||||||
|
for (const f of files) {
|
||||||
|
let t; try { t = fs.readFileSync(f, 'utf8'); } catch { continue; }
|
||||||
|
const model = (t.match(/^\s*Model:\s*(\S+)/m) || [])[1] || '';
|
||||||
|
if (!/^DSCA/i.test(model)) continue;
|
||||||
|
const sn = (t.match(/^\s*SN:\s*(\S+)/m) || [])[1] || '';
|
||||||
|
if (!sn) continue;
|
||||||
|
(byModel[model.trim()] = byModel[model.trim()] || []).push({ f, sn: sn.trim(), text: t });
|
||||||
|
}
|
||||||
|
|
||||||
|
let derived = 0, failed = [];
|
||||||
|
const UNITS_PER_MODEL = 8, MAX_SAMPLES = 48; // bounds DB lookups per signature group
|
||||||
|
|
||||||
|
// Seeds = the ambiguous models to solve (args), or all models if none given.
|
||||||
|
const seeds = new Set(onlyModels.length ? onlyModels.filter(m => tpl[m]) : Object.keys(tpl));
|
||||||
|
|
||||||
|
// Group ALL models by identical row-name signature. Same printed layout => same
|
||||||
|
// canonical-slot mapping, so one slotMap serves the whole group; pooling units
|
||||||
|
// across the group lets siblings disambiguate duplicate values (e.g. a unit where
|
||||||
|
// 5mA != 50mA linearity forces the correct slot). Only process groups that contain
|
||||||
|
// a seed, so a targeted run touches only the relevant families.
|
||||||
|
const sigOf = (m) => tpl[m].rows.map(r => r.name).join('|');
|
||||||
|
const groups = {};
|
||||||
|
for (const model of Object.keys(tpl)) { (groups[sigOf(model)] = groups[sigOf(model)] || []).push(model); }
|
||||||
|
|
||||||
|
for (const models of Object.values(groups)) {
|
||||||
|
if (![...models].some(m => seeds.has(m))) continue;
|
||||||
|
const specRowCount = tpl[models[0]].rows.filter(r => (r.spec || '').trim()).length;
|
||||||
|
const samples = [];
|
||||||
|
for (const model of models) {
|
||||||
|
if (samples.length >= MAX_SAMPLES) break;
|
||||||
|
const units = (byModel[model] || []).slice(0, UNITS_PER_MODEL);
|
||||||
|
for (const u of units) {
|
||||||
|
if (samples.length >= MAX_SAMPLES) break;
|
||||||
|
let row = await db.queryOne('SELECT raw_data FROM test_records WHERE serial_number=$1 AND model_number=$2 LIMIT 1', [u.sn, model]);
|
||||||
|
if (!row) row = await db.queryOne('SELECT raw_data FROM test_records WHERE raw_serial_number=$1 AND model_number=$2 LIMIT 1', [u.sn, model]);
|
||||||
|
if (!row || !row.raw_data) continue;
|
||||||
|
const printed = stagedPrintedValues(u.text);
|
||||||
|
if (!printed || printed.length !== specRowCount) continue;
|
||||||
|
const p = dse.parseRawData(row.raw_data, 'DSCA');
|
||||||
|
if (!p) continue;
|
||||||
|
samples.push({ printed, status: p.statusEntries });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!samples.length) continue;
|
||||||
|
|
||||||
|
// Candidate slotMaps = greedy map from each sample; the TRUE map reproduces the
|
||||||
|
// most units (a duplicate-confused map fails where the duplicated slots differ;
|
||||||
|
// retest-vintage units fail every map and are ignored).
|
||||||
|
const cands = new Map();
|
||||||
|
for (const s of samples) { const m = greedyMap(s.printed, s.status); if (m) cands.set(m.join(','), m); }
|
||||||
|
let best = null, bestScore = -1;
|
||||||
|
for (const m of cands.values()) {
|
||||||
|
let score = 0;
|
||||||
|
for (const s of samples) if (mapMatches(m, s.printed, s.status)) score++;
|
||||||
|
if (score > bestScore) { bestScore = score; best = m; }
|
||||||
|
}
|
||||||
|
const ratio = bestScore / samples.length;
|
||||||
|
const accept = best && (samples.length === 1 ? bestScore === 1 : (ratio >= 0.6 && bestScore >= 2));
|
||||||
|
if (accept) {
|
||||||
|
// Apply to every model in the group. The renderer only consults slotMap when
|
||||||
|
// the sequential value-zip fails (value count != spec-row count), so clean
|
||||||
|
// models keep their current path and only ambiguous ones use the map.
|
||||||
|
for (const model of models) tpl[model].slotMap = best;
|
||||||
|
derived += models.length;
|
||||||
|
console.log(' [' + models.length + '] ' + models[0].padEnd(13) + ' slotMap=[' + best.join(',') + '] matched ' + bestScore + '/' + samples.length + ' units' + (models.length > 1 ? ' (+' + (models.length - 1) + ' siblings)' : ''));
|
||||||
|
} else if (best) {
|
||||||
|
failed.push(models.join('/') + '(best ' + bestScore + '/' + samples.length + ')');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fs.writeFileSync(OUT, JSON.stringify(tpl));
|
||||||
|
console.log('\nderived slotMaps: ' + derived);
|
||||||
|
if (failed.length) console.log('no clean match (left as-is): ' + failed.join(', '));
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });
|
||||||
@@ -15,16 +15,23 @@ function extract(t) {
|
|||||||
const cols = colSpans(sep); if (cols.length < 4) return null;
|
const cols = colSpans(sep); if (cols.length < 4) return null;
|
||||||
const [pc, mc, sc, stc] = cols;
|
const [pc, mc, sc, stc] = cols;
|
||||||
const rows = [];
|
const rows = [];
|
||||||
|
let loadNote = null;
|
||||||
for (let i = hi + 2; i < lines.length; i++) {
|
for (let i = hi + 2; i < lines.length; i++) {
|
||||||
const l = lines[i];
|
const l = lines[i];
|
||||||
if (/Check List|^\s*_{5,}/.test(l)) break;
|
if (/Check List|^\s*_{5,}/.test(l)) break;
|
||||||
if (!l.trim()) continue;
|
if (!l.trim()) continue;
|
||||||
|
// The "Standard output load for test is ... ohms." line is a footer note, not
|
||||||
|
// a parameter row — it spans past the name column so column-slicing truncates
|
||||||
|
// it ("Standard output load for te"). Capture the full line as loadNote and
|
||||||
|
// keep it out of rows; the renderer emits it (before the footer underline)
|
||||||
|
// only for models whose staged original actually printed it.
|
||||||
|
if (/^Standard output load/i.test(l.trim())) { loadNote = l.trim(); continue; }
|
||||||
const name = (l.slice(pc[0], mc[0]) || '').trim();
|
const name = (l.slice(pc[0], mc[0]) || '').trim();
|
||||||
const spec = (l.slice(sc[0], stc[0]) || '').trim();
|
const spec = (l.slice(sc[0], stc[0]) || '').trim();
|
||||||
if (!name && !spec) continue;
|
if (!name && !spec) continue;
|
||||||
rows.push({ name, spec });
|
rows.push({ name, spec });
|
||||||
}
|
}
|
||||||
return { accOut, rows };
|
return { accOut, rows, loadNote };
|
||||||
}
|
}
|
||||||
(async () => {
|
(async () => {
|
||||||
const files = walk(STAGE, []);
|
const files = walk(STAGE, []);
|
||||||
@@ -40,7 +47,7 @@ function extract(t) {
|
|||||||
}
|
}
|
||||||
const models = Object.keys(byModel).sort();
|
const models = Object.keys(byModel).sort();
|
||||||
console.log('DSCA models templated: ' + models.length);
|
console.log('DSCA models templated: ' + models.length);
|
||||||
const out = {}; for (const m of models) out[m] = { accOut: byModel[m].accOut, rows: byModel[m].rows };
|
const out = {}; for (const m of models) { out[m] = { accOut: byModel[m].accOut, rows: byModel[m].rows }; if (byModel[m].loadNote) out[m].loadNote = byModel[m].loadNote; }
|
||||||
fs.writeFileSync(OUT, JSON.stringify(out));
|
fs.writeFileSync(OUT, JSON.stringify(out));
|
||||||
console.log('wrote ' + OUT + ' (' + fs.statSync(OUT).size + ' bytes)');
|
console.log('wrote ' + OUT + ' (' + fs.statSync(OUT).size + ' bytes)');
|
||||||
const rc = {}; for (const m of models) { const n = byModel[m].rows.length; rc[n] = (rc[n] || 0) + 1; }
|
const rc = {}; for (const m of models) { const n = byModel[m].rows.length; rc[n] = (rc[n] || 0) + 1; }
|
||||||
|
|||||||
41
projects/dataforth-dos/tools/publish-dsca3345-gap.js
Normal file
41
projects/dataforth-dos/tools/publish-dsca3345-gap.js
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
// Publish the DSCA33/45 gap SAFELY: for each validated model's unuploaded PASS serial,
|
||||||
|
// GET the Hoffman record; push ONLY those that are absent (404) so every push is a
|
||||||
|
// Created — never an UPDATE that could overwrite a pristine original. (The inventory
|
||||||
|
// file is stale, so we probe per-serial instead of trusting api_uploaded_at.)
|
||||||
|
const fs = require('fs'), https = require('https');
|
||||||
|
const db = require('./database/db');
|
||||||
|
const { uploadBySerialNumbers } = require('./database/upload-to-api');
|
||||||
|
const tpl = require('./dsca33-45-templates.json');
|
||||||
|
const c = JSON.parse(fs.readFileSync('C:\\ProgramData\\dataforth-uploader\\credentials.json', 'utf8'));
|
||||||
|
const DRY = !process.argv.includes('--push');
|
||||||
|
function req(m, uri, h, b) { return new Promise((res, rej) => { const u = new URL(uri); const r = https.request({ hostname: u.hostname, port: 443, path: u.pathname, method: m, headers: h, timeout: 30000 }, x => { let d = ''; x.on('data', c => d += c); x.on('end', () => res({ status: x.statusCode, body: d })); }); r.on('error', rej); r.on('timeout', () => r.destroy(new Error('timeout'))); if (b) r.write(b); r.end(); }); }
|
||||||
|
async function token() { const form = Object.entries({ grant_type: 'client_credentials', client_id: c.CF_CLIENT_ID, client_secret: c.CF_CLIENT_SECRET, scope: c.CF_SCOPE }).map(([k, v]) => encodeURIComponent(k) + '=' + encodeURIComponent(v)).join('&'); const r = await req('POST', c.CF_TOKEN_URL, { 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': Buffer.byteLength(form) }, form); return JSON.parse(r.body).access_token; }
|
||||||
|
(async () => {
|
||||||
|
const validated = Object.keys(tpl).filter(m => tpl[m].validated);
|
||||||
|
const ph = validated.map((_, i) => '$' + (i + 1)).join(',');
|
||||||
|
const rows = await db.query(`SELECT serial_number FROM test_records WHERE overall_result='PASS' AND api_uploaded_at IS NULL AND model_number IN (${ph}) ORDER BY serial_number`, validated);
|
||||||
|
const sns = rows.map(r => r.serial_number);
|
||||||
|
console.log(`validated models: ${validated.length}; unuploaded PASS serials to probe: ${sns.length}`);
|
||||||
|
const t = await token();
|
||||||
|
const absent = [], present = [];
|
||||||
|
for (let i = 0; i < sns.length; i++) {
|
||||||
|
const r = await req('GET', c.CF_API_BASE + '/api/v1/TestReportDataFiles/' + encodeURIComponent(sns[i]), { Authorization: 'Bearer ' + t });
|
||||||
|
if (r.status === 404 || (r.status === 200 && !/"Content"/.test(r.body))) absent.push(sns[i]);
|
||||||
|
else if (r.status === 200) present.push(sns[i]);
|
||||||
|
else absent.push(sns[i]); // treat unknown as absent? no — be safe: skip
|
||||||
|
if ((i + 1) % 200 === 0) console.log(` probed ${i + 1}/${sns.length} absent=${absent.length} present=${present.length}`);
|
||||||
|
}
|
||||||
|
console.log(`\nPROBE DONE: absent(not on Hoffman)=${absent.length} present(already live)=${present.length}`);
|
||||||
|
if (DRY) { console.log('\n(dry run — pass --push to Created-publish the absent set)'); await db.close(); return; }
|
||||||
|
console.log('\nPublishing absent set (Created only)...');
|
||||||
|
const tot = { created: 0, updated: 0, unchanged: 0, errors: 0, skipped: 0 };
|
||||||
|
const CH = 500;
|
||||||
|
for (let i = 0; i < absent.length; i += CH) {
|
||||||
|
const r = await uploadBySerialNumbers(absent.slice(i, i + CH));
|
||||||
|
for (const k of Object.keys(tot)) tot[k] += r[k] || 0;
|
||||||
|
console.log(` ${Math.min(i + CH, absent.length)}/${absent.length} cumulative ${JSON.stringify(tot)}`);
|
||||||
|
}
|
||||||
|
console.log('\nDONE ' + JSON.stringify(tot));
|
||||||
|
if (tot.updated > 0) console.log('[WARNING] ' + tot.updated + ' UPDATED — investigate (should be 0 for an absent-only push)');
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });
|
||||||
27
projects/dataforth-dos/tools/push-clean68.js
Normal file
27
projects/dataforth-dos/tools/push-clean68.js
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
// Fix 2 publish — re-push the 68 STAGE-3 Final-Test-clean DSCA models to Hoffman.
|
||||||
|
// Idempotent (Updated/Unchanged/Created); renderContent uses the new template;
|
||||||
|
// null renders (count-guard) + FAIL certs + unregistered models auto-skip.
|
||||||
|
const db = require('./database/db');
|
||||||
|
const { uploadBySerialNumbers } = require('./database/upload-to-api');
|
||||||
|
const clean = require('./_clean-models.json');
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
const ph = clean.map((_, i) => '$' + (i + 1)).join(',');
|
||||||
|
const rows = await db.query(
|
||||||
|
`SELECT serial_number FROM test_records WHERE overall_result='PASS' AND model_number IN (${ph}) ORDER BY serial_number`,
|
||||||
|
clean,
|
||||||
|
);
|
||||||
|
const sns = rows.map(r => r.serial_number);
|
||||||
|
console.log(`[PUSH] ${sns.length} PASS serials across ${clean.length} clean models`);
|
||||||
|
const CHUNK = 1000;
|
||||||
|
const tot = { created: 0, updated: 0, unchanged: 0, errors: 0, skipped: 0 };
|
||||||
|
for (let i = 0; i < sns.length; i += CHUNK) {
|
||||||
|
const chunk = sns.slice(i, i + CHUNK);
|
||||||
|
const r = await uploadBySerialNumbers(chunk);
|
||||||
|
for (const k of Object.keys(tot)) tot[k] += r[k] || 0;
|
||||||
|
console.log(`[PUSH] ${Math.min(i + CHUNK, sns.length)}/${sns.length} cumulative ` +
|
||||||
|
`created=${tot.created} updated=${tot.updated} unchanged=${tot.unchanged} errors=${tot.errors} skipped=${tot.skipped}`);
|
||||||
|
}
|
||||||
|
console.log('[PUSH] COMPLETE ' + JSON.stringify(tot));
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });
|
||||||
79
projects/dataforth-dos/tools/slotmap-from-hoffman.js
Normal file
79
projects/dataforth-dos/tools/slotmap-from-hoffman.js
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
// Derive slotMaps for DSCA33/45 models that still render null (no slotMap), using the
|
||||||
|
// Hoffman _srcSerial original as the oracle: match its Final-Test measured values, in
|
||||||
|
// order, to the DB record's value-bearing STATUS entries (numeric greedy). Writes the
|
||||||
|
// slotMap into dsca33-45-templates.json. --apply to persist.
|
||||||
|
process.env.DSCA_VALIDATE_MODE = '1';
|
||||||
|
const fs = require('fs'), https = require('https');
|
||||||
|
const db = require('./database/db');
|
||||||
|
const dse = require('./templates/datasheet-exact');
|
||||||
|
const TPL = './dsca33-45-templates.json';
|
||||||
|
const c = JSON.parse(fs.readFileSync('C:\\ProgramData\\dataforth-uploader\\credentials.json', 'utf8'));
|
||||||
|
const APPLY = process.argv.includes('--apply');
|
||||||
|
const only = process.argv.slice(2).filter(a => !a.startsWith('--'));
|
||||||
|
function req(m, uri, h, b) { return new Promise((res, rej) => { const u = new URL(uri); const r = https.request({ hostname: u.hostname, port: 443, path: u.pathname, method: m, headers: h, timeout: 30000 }, x => { let d = ''; x.on('data', c => d += c); x.on('end', () => { try { res(JSON.parse(d)); } catch { res({ _raw: d }); } }); }); r.on('error', rej); if (b) r.write(b); r.end(); }); }
|
||||||
|
function colSpans(sep) { const cols = []; let m; const re = /=+/g; while ((m = re.exec(sep))) cols.push([m.index, m.index + m[0].length]); return cols; }
|
||||||
|
// Final-Test measured numeric values from a Hoffman original, in row order (spec rows only).
|
||||||
|
function hoffmanMeasured(content) {
|
||||||
|
const L = content.replace(/\r/g, '').split('\n');
|
||||||
|
const fi = L.findIndex(l => /FINAL TEST RESULTS/.test(l)); if (fi < 0) return null;
|
||||||
|
let hi = -1; for (let i = fi + 1; i < L.length; i++) if (/Parameter\s+Measured/.test(L[i])) { hi = i; break; } if (hi < 0) return null;
|
||||||
|
const cols = colSpans(L[hi + 1]); if (cols.length < 4) return null;
|
||||||
|
const [pc, mc, sc] = cols;
|
||||||
|
const out = [];
|
||||||
|
for (let i = hi + 2; i < L.length; i++) {
|
||||||
|
const l = L[i]; if (/^\s*_{5,}|Check List|It is hereby/.test(l)) break; if (!l.trim()) continue;
|
||||||
|
if (/^\s*Standard output load/i.test(l)) continue;
|
||||||
|
const meas = (l.slice(mc[0], sc[0]) || '').trim().split(/\s+/)[0];
|
||||||
|
const spec = (l.slice(sc[0], cols[3][0]) || '').trim();
|
||||||
|
if (!spec) continue; // spec-less row (Withstand/Hi-Pot/section head)
|
||||||
|
const v = parseFloat(meas);
|
||||||
|
if (isNaN(v)) return null;
|
||||||
|
out.push(v);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
(async () => {
|
||||||
|
const tpl = JSON.parse(fs.readFileSync(TPL, 'utf8'));
|
||||||
|
const form = Object.entries({ grant_type: 'client_credentials', client_id: c.CF_CLIENT_ID, client_secret: c.CF_CLIENT_SECRET, scope: c.CF_SCOPE }).map(([k, v]) => encodeURIComponent(k) + '=' + encodeURIComponent(v)).join('&');
|
||||||
|
const t = (await req('POST', c.CF_TOKEN_URL, { 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': Buffer.byteLength(form) }, form)).access_token;
|
||||||
|
const models = (only.length ? only : Object.keys(tpl)).filter(m => tpl[m] && !Array.isArray(tpl[m].slotMap));
|
||||||
|
let done = 0; const failed = [];
|
||||||
|
for (const m of models) {
|
||||||
|
const sn = tpl[m]._srcSerial; if (!sn) { failed.push(m + '(no srcSerial)'); continue; }
|
||||||
|
const rec = await db.queryOne('SELECT raw_data FROM test_records WHERE serial_number=$1 AND model_number=$2', [sn, m]);
|
||||||
|
if (!rec) { failed.push(m + '(no DB rec)'); continue; }
|
||||||
|
const g = await req('GET', c.CF_API_BASE + '/api/v1/TestReportDataFiles/' + encodeURIComponent(sn), { Authorization: 'Bearer ' + t });
|
||||||
|
if (!g.Content) { failed.push(m + '(no Hoffman)'); continue; }
|
||||||
|
const measured = hoffmanMeasured(g.Content);
|
||||||
|
const p = dse.parseRawData(rec.raw_data, 'DSCA');
|
||||||
|
if (!measured || !p) { failed.push(m + '(parse)'); continue; }
|
||||||
|
const specRows = tpl[m].rows.filter(r => (r.spec || '').trim()).length;
|
||||||
|
if (measured.length !== specRows) { failed.push(m + '(rows ' + measured.length + '!=' + specRows + ')'); continue; }
|
||||||
|
// numeric greedy match: each Final-Test measured value -> next status entry with equal value
|
||||||
|
const map = []; let j = 0; let ok = true;
|
||||||
|
for (const mv of measured) {
|
||||||
|
let found = -1;
|
||||||
|
for (let k = j; k < p.statusEntries.length; k++) {
|
||||||
|
const s = p.statusEntries[k];
|
||||||
|
if (!s || s.length <= 4) continue;
|
||||||
|
// compare at DISPLAY precision: fround + toFixed(decimal-code), like the
|
||||||
|
// renderer — so near-zero (raw 4.4e-5 displayed "0.0000") matches the
|
||||||
|
// Hoffman displayed value instead of failing a raw-value tolerance.
|
||||||
|
const code = parseInt(s[s.length - 1], 10);
|
||||||
|
const val = parseFloat(s.substring(4, s.length - 1).trim());
|
||||||
|
if (isNaN(val)) continue;
|
||||||
|
const disp = parseFloat(Math.fround(val).toFixed(isNaN(code) ? 1 : code));
|
||||||
|
if (Math.abs(disp - mv) < 1e-9) { found = k; break; }
|
||||||
|
}
|
||||||
|
if (found < 0) { ok = false; break; }
|
||||||
|
map.push(found); j = found + 1;
|
||||||
|
}
|
||||||
|
if (ok && map.length === specRows) { tpl[m].slotMap = map; done++; console.log(' ' + m.padEnd(13) + ' slotMap=[' + map.join(',') + '] (oracle ' + sn + ')'); }
|
||||||
|
else failed.push(m + '(no clean match)');
|
||||||
|
}
|
||||||
|
console.log('\nderived: ' + done);
|
||||||
|
if (failed.length) console.log('failed: ' + failed.join(', '));
|
||||||
|
if (APPLY) { fs.writeFileSync(TPL, JSON.stringify(tpl)); console.log('[APPLY] wrote ' + TPL); }
|
||||||
|
else console.log('(dry run — pass --apply to write)');
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });
|
||||||
171
projects/dataforth-dos/tools/validate-dsca-stage3.js
Normal file
171
projects/dataforth-dos/tools/validate-dsca-stage3.js
Normal file
@@ -0,0 +1,171 @@
|
|||||||
|
// Fix 2 STAGE 3 — per-subtype byte-validation of DSCA renders vs staged originals.
|
||||||
|
// For every staged DSCA .TXT we have ground truth for, look up the DB record,
|
||||||
|
// render it through the live render path, and content-normalize-compare the two.
|
||||||
|
// Grouped by model (layout). Whitespace is collapsed per line (column spacing is
|
||||||
|
// the deferred cosmetic gap) so the compare tests CONTENT — names, values, specs,
|
||||||
|
// statuses — not pixel alignment. Read-only; no DB writes, no Hoffman push.
|
||||||
|
//
|
||||||
|
// Usage: node _validate_dsca_stage3.js [--limit-per-model N] [--report path]
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const db = require('./database/db');
|
||||||
|
const { renderContent } = require('./database/render-datasheet');
|
||||||
|
|
||||||
|
const STAGE = 'C:/Shares/test/STAGE';
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
const LIMIT = (() => { const i = args.indexOf('--limit-per-model'); return i >= 0 ? parseInt(args[i + 1], 10) : Infinity; })();
|
||||||
|
const REPORT = (() => { const i = args.indexOf('--report'); return i >= 0 ? args[i + 1] : 'C:/Shares/testdatadb/_dsca-stage3-report.txt'; })();
|
||||||
|
|
||||||
|
function walk(d, out) {
|
||||||
|
let it = [];
|
||||||
|
try { it = fs.readdirSync(d, { withFileTypes: true }); } catch { return out; }
|
||||||
|
for (const e of it) {
|
||||||
|
const p = path.join(d, e.name);
|
||||||
|
if (e.isDirectory()) walk(p, out);
|
||||||
|
else if (/\.txt$/i.test(e.name)) out.push(p);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Content-normalize a single line: trim + collapse whitespace. Rule lines (runs
|
||||||
|
// of = - ~ _ separated by spaces) canonicalize to <RULE> so the deferred cosmetic
|
||||||
|
// dash/equal-count differences don't register as content diffs.
|
||||||
|
function normLine(l) {
|
||||||
|
const t = l.trim();
|
||||||
|
if (t.length && /^[=~_\- ]+$/.test(t) && /[=~_\-]/.test(t)) return '<RULE>';
|
||||||
|
return t.replace(/\s+/g, ' ');
|
||||||
|
}
|
||||||
|
function norm(s) {
|
||||||
|
return s.replace(/\r/g, '').split('\n').map(normLine).filter(l => l.length > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract the FINAL TEST RESULTS section: from its header to just before the
|
||||||
|
// footer underline (___). This is the Fix 2 deliverable; compared content-strict.
|
||||||
|
function finalTestLines(s) {
|
||||||
|
const L = s.replace(/\r/g, '').split('\n');
|
||||||
|
const fi = L.findIndex(l => /FINAL TEST RESULTS/.test(l));
|
||||||
|
if (fi < 0) return [];
|
||||||
|
const out = [];
|
||||||
|
for (let i = fi; i < L.length; i++) {
|
||||||
|
const t = L[i].trim();
|
||||||
|
if (i > fi && /^_{5,}$/.test(t)) break; // footer underline
|
||||||
|
if (/It is hereby certified/.test(t)) break;
|
||||||
|
out.push(normLine(L[i]));
|
||||||
|
}
|
||||||
|
return out.filter(l => l.length > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Accuracy section lines (informational — spacing + any calc rounding live here).
|
||||||
|
function accuracyLines(s) {
|
||||||
|
const L = s.replace(/\r/g, '').split('\n');
|
||||||
|
const ai = L.findIndex(l => /ACCURACY TEST/.test(l));
|
||||||
|
if (ai < 0) return [];
|
||||||
|
const fi = L.findIndex(l => /FINAL TEST RESULTS/.test(l));
|
||||||
|
const end = fi < 0 ? L.length : fi;
|
||||||
|
return L.slice(ai, end).map(normLine).filter(l => l.length > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
const files = walk(STAGE, []);
|
||||||
|
// index staged DSCA files: model + SN + text
|
||||||
|
const staged = [];
|
||||||
|
for (const f of files) {
|
||||||
|
let t; try { t = fs.readFileSync(f, 'utf8'); } catch { continue; }
|
||||||
|
const model = (t.match(/^\s*Model:\s*(\S+)/m) || [])[1] || '';
|
||||||
|
if (!/^DSCA/i.test(model)) continue;
|
||||||
|
const sn = (t.match(/^\s*SN:\s*(\S+)/m) || [])[1] || '';
|
||||||
|
if (!sn) continue;
|
||||||
|
staged.push({ f, model: model.trim(), sn: sn.trim(), text: t });
|
||||||
|
}
|
||||||
|
console.log(`staged DSCA originals: ${staged.length}`);
|
||||||
|
|
||||||
|
const byModel = {}; // model -> tallies
|
||||||
|
function rec(model) {
|
||||||
|
if (!byModel[model]) byModel[model] = { compared: 0, ftMatch: 0, ftMismatch: 0, accMismatch: 0, noRecord: 0, notRendered: 0, samples: [] };
|
||||||
|
return byModel[model];
|
||||||
|
}
|
||||||
|
function firstDiff(a, b) {
|
||||||
|
const max = Math.max(a.length, b.length);
|
||||||
|
for (let i = 0; i < max; i++) if (a[i] !== b[i]) return i;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const seenPerModel = {};
|
||||||
|
for (const s of staged) {
|
||||||
|
seenPerModel[s.model] = (seenPerModel[s.model] || 0) + 1;
|
||||||
|
if (seenPerModel[s.model] > LIMIT) continue;
|
||||||
|
const r = rec(s.model);
|
||||||
|
let row = await db.queryOne('SELECT * FROM test_records WHERE serial_number=$1 AND model_number=$2 LIMIT 1', [s.sn, s.model]);
|
||||||
|
if (!row) row = await db.queryOne('SELECT * FROM test_records WHERE raw_serial_number=$1 AND model_number=$2 LIMIT 1', [s.sn, s.model]);
|
||||||
|
if (!row) { r.noRecord++; continue; }
|
||||||
|
let rendered;
|
||||||
|
try { rendered = renderContent(row); } catch (e) { rendered = null; }
|
||||||
|
if (!rendered) { r.notRendered++; continue; }
|
||||||
|
r.compared++;
|
||||||
|
// GATE: Final-Test section content must match exactly (rules canonicalized).
|
||||||
|
const ftA = finalTestLines(rendered), ftB = finalTestLines(s.text);
|
||||||
|
const fd = firstDiff(ftA, ftB);
|
||||||
|
if (fd === -1) r.ftMatch++;
|
||||||
|
else {
|
||||||
|
r.ftMismatch++;
|
||||||
|
if (r.samples.length < 3) r.samples.push({ sn: s.sn, line: fd, render: ftA[fd], golden: ftB[fd] });
|
||||||
|
}
|
||||||
|
// INFO: accuracy section (deferred cosmetic spacing + any calc rounding).
|
||||||
|
const acA = accuracyLines(rendered), acB = accuracyLines(s.text);
|
||||||
|
if (firstDiff(acA, acB) !== -1) r.accMismatch++;
|
||||||
|
}
|
||||||
|
|
||||||
|
// report
|
||||||
|
const models = Object.keys(byModel).sort();
|
||||||
|
let totC = 0, totFM = 0, totFMM = 0, totAcc = 0, totNR = 0, totNRn = 0, cleanModels = 0;
|
||||||
|
const ftDirty = [];
|
||||||
|
for (const m of models) {
|
||||||
|
const x = byModel[m];
|
||||||
|
totC += x.compared; totFM += x.ftMatch; totFMM += x.ftMismatch; totAcc += x.accMismatch;
|
||||||
|
totNR += x.noRecord; totNRn += x.notRendered;
|
||||||
|
if (x.compared > 0 && x.ftMismatch === 0) cleanModels++;
|
||||||
|
if (x.ftMismatch > 0) ftDirty.push(m);
|
||||||
|
}
|
||||||
|
const out = [];
|
||||||
|
out.push('Fix 2 STAGE 3 — DSCA Final-Test render vs staged-original content validation');
|
||||||
|
out.push('GATE = FINAL TEST RESULTS section, content-strict (rule lines canonicalized,');
|
||||||
|
out.push('whitespace collapsed). Accuracy-section diffs reported separately (deferred');
|
||||||
|
out.push('cosmetic spacing + any pre-existing calc rounding — NOT a Fix 2 gate).');
|
||||||
|
out.push('Corpus: ' + staged.length + ' staged DSCA originals across ' + models.length + ' models.');
|
||||||
|
out.push('='.repeat(78));
|
||||||
|
out.push('');
|
||||||
|
out.push('SUMMARY');
|
||||||
|
out.push(' models with staged originals: ' + models.length);
|
||||||
|
out.push(' models FINAL-TEST CLEAN (>=1 compared, 0 mismatch): ' + cleanModels);
|
||||||
|
out.push(' models with FINAL-TEST mismatches: ' + ftDirty.length);
|
||||||
|
out.push(' certs compared: ' + totC);
|
||||||
|
out.push(' Final-Test match: ' + totFM);
|
||||||
|
out.push(' Final-Test mismatch: ' + totFMM);
|
||||||
|
out.push(' (certs with accuracy-section diffs: ' + totAcc + ' — informational)');
|
||||||
|
out.push(' staged serials not in DB: ' + totNR);
|
||||||
|
out.push(' in DB but not rendered (skipped/null): ' + totNRn);
|
||||||
|
out.push('');
|
||||||
|
out.push('MODELS WITH FINAL-TEST CONTENT MISMATCHES (investigate before re-push):');
|
||||||
|
if (!ftDirty.length) out.push(' (none — Final-Test renders are content-clean for all compared models)');
|
||||||
|
for (const m of ftDirty) {
|
||||||
|
const x = byModel[m];
|
||||||
|
out.push(' ' + m + ' compared=' + x.compared + ' ftMatch=' + x.ftMatch + ' ftMismatch=' + x.ftMismatch);
|
||||||
|
for (const s of x.samples) {
|
||||||
|
out.push(' [finaltest L' + s.line + '] SN ' + s.sn);
|
||||||
|
out.push(' render: ' + JSON.stringify(s.render));
|
||||||
|
out.push(' golden: ' + JSON.stringify(s.golden));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.push('');
|
||||||
|
out.push('FINAL-TEST CLEAN MODELS (' + cleanModels + '):');
|
||||||
|
out.push(' ' + models.filter(m => byModel[m].compared > 0 && byModel[m].ftMismatch === 0).join(', '));
|
||||||
|
out.push('');
|
||||||
|
out.push('MODELS WITH NO COMPARABLE CERT (no staged serial in DB, or all skipped/null):');
|
||||||
|
out.push(' ' + models.filter(m => byModel[m].compared === 0).map(m => m + '(' + (byModel[m].notRendered ? 'null' : 'noDBrec') + ')').join(', '));
|
||||||
|
|
||||||
|
const text = out.join('\n');
|
||||||
|
fs.writeFileSync(REPORT, text);
|
||||||
|
console.log(text);
|
||||||
|
console.log('\n[report written] ' + REPORT);
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });
|
||||||
97
projects/dataforth-dos/tools/validate-dsca3345.js
Normal file
97
projects/dataforth-dos/tools/validate-dsca3345.js
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
// Fix 2 — validate DSCA33/45 Hoffman-mined renders against the live Hoffman originals.
|
||||||
|
// For each model: render its _srcSerial (an already-uploaded unit) via the new render
|
||||||
|
// path and content-normalized-compare it to GET /api/v1/TestReportDataFiles/{_srcSerial}.
|
||||||
|
// --apply marks passing models `validated:true` in dsca33-45-templates.json (the render
|
||||||
|
// gate). Read-only otherwise (no DB writes, no Hoffman writes).
|
||||||
|
process.env.DSCA_VALIDATE_MODE = '1'; // open the render gate for the compare
|
||||||
|
const fs = require('fs');
|
||||||
|
const https = require('https');
|
||||||
|
const db = require('./database/db');
|
||||||
|
const { renderContent } = require('./database/render-datasheet');
|
||||||
|
|
||||||
|
const TPL_PATH = './dsca33-45-templates.json';
|
||||||
|
const CREDS_PATH = 'C:\\ProgramData\\dataforth-uploader\\credentials.json';
|
||||||
|
const APPLY = process.argv.includes('--apply');
|
||||||
|
const only = process.argv.slice(2).filter(a => !a.startsWith('--'));
|
||||||
|
|
||||||
|
function creds() { return JSON.parse(fs.readFileSync(CREDS_PATH, 'utf8')); }
|
||||||
|
function httpReq(method, uri, headers, body) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const u = new URL(uri);
|
||||||
|
const req = https.request({ hostname: u.hostname, port: u.port || 443, path: u.pathname + u.search, method, headers, timeout: 30000 }, res => {
|
||||||
|
let d = ''; res.on('data', c => d += c); res.on('end', () => { try { resolve({ status: res.statusCode, body: JSON.parse(d) }); } catch { resolve({ status: res.statusCode, body: { _raw: d } }); } });
|
||||||
|
});
|
||||||
|
req.on('error', reject); req.on('timeout', () => req.destroy(new Error('timeout')));
|
||||||
|
if (body) req.write(body); req.end();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
async function getToken() {
|
||||||
|
const c = creds();
|
||||||
|
const form = Object.entries({ grant_type: 'client_credentials', client_id: c.CF_CLIENT_ID, client_secret: c.CF_CLIENT_SECRET, scope: c.CF_SCOPE })
|
||||||
|
.map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`).join('&');
|
||||||
|
const r = await httpReq('POST', c.CF_TOKEN_URL, { 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': Buffer.byteLength(form) }, form);
|
||||||
|
if (r.status !== 200 || !r.body.access_token) throw new Error('token fail ' + r.status);
|
||||||
|
return r.body.access_token;
|
||||||
|
}
|
||||||
|
async function fetchOriginal(token, serial) {
|
||||||
|
const c = creds();
|
||||||
|
const r = await httpReq('GET', `${c.CF_API_BASE}/api/v1/TestReportDataFiles/${encodeURIComponent(serial)}`, { Authorization: 'Bearer ' + token });
|
||||||
|
if (r.status !== 200) return null;
|
||||||
|
return r.body && r.body.Content ? r.body.Content : null;
|
||||||
|
}
|
||||||
|
// content-normalize: collapse whitespace per line; DROP rule lines (pure separators —
|
||||||
|
// runs of = ~ _ -) and blank lines. Rule lines carry no content and their
|
||||||
|
// presence/position is the deferred cosmetic gap (e.g. the leading === letterhead line
|
||||||
|
// the originals have and our renders omit), so removing them isolates real content.
|
||||||
|
function norm(s) {
|
||||||
|
return s.replace(/\r/g, '').split('\n')
|
||||||
|
.map(l => l.trim())
|
||||||
|
.filter(t => t.length > 0 && !(/^[=~_\- ]+$/.test(t) && /[=~_\-]/.test(t)))
|
||||||
|
.map(t => t.replace(/\s+/g, ' '));
|
||||||
|
}
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
const tpl = JSON.parse(fs.readFileSync(TPL_PATH, 'utf8'));
|
||||||
|
const models = (only.length ? only : Object.keys(tpl)).filter(m => tpl[m]);
|
||||||
|
const token = await getToken();
|
||||||
|
const pass = [], fail = [], noOracle = [], noRec = [];
|
||||||
|
for (const m of models) {
|
||||||
|
const sn = tpl[m]._srcSerial;
|
||||||
|
if (!sn) { noOracle.push(m); continue; }
|
||||||
|
const original = await fetchOriginal(token, sn);
|
||||||
|
if (!original) { noOracle.push(m + '(no Hoffman ' + sn + ')'); continue; }
|
||||||
|
const rec = await db.queryOne('SELECT * FROM test_records WHERE serial_number=$1 AND model_number=$2 LIMIT 1', [sn, m]);
|
||||||
|
if (!rec) { noRec.push(m + '(' + sn + ')'); continue; }
|
||||||
|
let rendered; try { rendered = renderContent(rec); } catch (e) { rendered = null; }
|
||||||
|
if (!rendered) { fail.push({ m, sn, reason: 'render null' }); continue; }
|
||||||
|
const a = norm(rendered), b = norm(original);
|
||||||
|
let diff = -1; const mx = Math.max(a.length, b.length);
|
||||||
|
for (let i = 0; i < mx; i++) { if (a[i] !== b[i]) { diff = i; break; } }
|
||||||
|
if (diff === -1) pass.push(m);
|
||||||
|
else fail.push({ m, sn, line: diff, render: a[diff], golden: b[diff] });
|
||||||
|
}
|
||||||
|
console.log('\n=== DSCA33/45 Hoffman validation ===');
|
||||||
|
console.log('PASS (' + pass.length + '): ' + pass.join(', '));
|
||||||
|
console.log('\nFAIL (' + fail.length + '):');
|
||||||
|
for (const f of fail) {
|
||||||
|
if (f.reason) { console.log(' ' + f.m + ' (' + f.sn + '): ' + f.reason); continue; }
|
||||||
|
console.log(' ' + f.m + ' (' + f.sn + ') first diff L' + f.line);
|
||||||
|
console.log(' render: ' + JSON.stringify(f.render));
|
||||||
|
console.log(' golden: ' + JSON.stringify(f.golden));
|
||||||
|
}
|
||||||
|
if (noOracle.length) console.log('\nNO ORACLE: ' + noOracle.join(', '));
|
||||||
|
if (noRec.length) console.log('NO DB REC: ' + noRec.join(', '));
|
||||||
|
|
||||||
|
if (APPLY) {
|
||||||
|
const passSet = new Set(pass);
|
||||||
|
for (const m of Object.keys(tpl)) {
|
||||||
|
if (passSet.has(m)) tpl[m].validated = true;
|
||||||
|
else if (only.length === 0) delete tpl[m].validated; // full run: clear stale
|
||||||
|
}
|
||||||
|
fs.writeFileSync(TPL_PATH, JSON.stringify(tpl));
|
||||||
|
console.log('\n[APPLY] marked validated on ' + pass.length + ' models in ' + TPL_PATH);
|
||||||
|
} else {
|
||||||
|
console.log('\n(dry run — pass --apply to mark validated)');
|
||||||
|
}
|
||||||
|
await db.close();
|
||||||
|
})().catch(e => { console.error('ERR', e.message, e.stack); process.exit(1); });
|
||||||
88
scripts/Configure-TranscriptLogging.ps1
Normal file
88
scripts/Configure-TranscriptLogging.ps1
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Configures PowerShell transcript logging for remote sessions.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
Enables comprehensive transcript logging via registry settings,
|
||||||
|
creates the logging directory with proper permissions, and sets up
|
||||||
|
automatic log rotation.
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Author: ClaudeTools Automation
|
||||||
|
Version: 1.0
|
||||||
|
Run as Administrator
|
||||||
|
#>
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$transcriptPath = "C:\ClaudeTools\Logs\Transcripts"
|
||||||
|
|
||||||
|
Write-Host "Configuring PowerShell Transcript Logging..." -ForegroundColor Cyan
|
||||||
|
|
||||||
|
# Create transcript directory
|
||||||
|
if (-not (Test-Path $transcriptPath)) {
|
||||||
|
New-Item -ItemType Directory -Path $transcriptPath -Force | Out-Null
|
||||||
|
Write-Host "Created transcript directory: $transcriptPath" -ForegroundColor Green
|
||||||
|
}
|
||||||
|
|
||||||
|
# Set permissions on transcript directory
|
||||||
|
# Administrators: Full Control, SYSTEM: Full Control, Remote Management Users: Read/Write
|
||||||
|
$acl = Get-Acl $transcriptPath
|
||||||
|
$acl.SetAccessRuleProtection($true, $false) # Disable inheritance
|
||||||
|
|
||||||
|
# Add Administrators - Full Control
|
||||||
|
$adminRule = New-Object System.Security.AccessControl.FileSystemAccessRule(
|
||||||
|
"Administrators", "FullControl", "ContainerInherit,ObjectInherit", "None", "Allow"
|
||||||
|
)
|
||||||
|
$acl.AddAccessRule($adminRule)
|
||||||
|
|
||||||
|
# Add SYSTEM - Full Control
|
||||||
|
$systemRule = New-Object System.Security.AccessControl.FileSystemAccessRule(
|
||||||
|
"SYSTEM", "FullControl", "ContainerInherit,ObjectInherit", "None", "Allow"
|
||||||
|
)
|
||||||
|
$acl.AddAccessRule($systemRule)
|
||||||
|
|
||||||
|
# Add Remote Management Users - Modify (so they can write transcripts)
|
||||||
|
$rmRule = New-Object System.Security.AccessControl.FileSystemAccessRule(
|
||||||
|
"Remote Management Users", "Modify", "ContainerInherit,ObjectInherit", "None", "Allow"
|
||||||
|
)
|
||||||
|
$acl.AddAccessRule($rmRule)
|
||||||
|
|
||||||
|
Set-Acl $transcriptPath $acl
|
||||||
|
Write-Host "Set permissions on transcript directory" -ForegroundColor Green
|
||||||
|
|
||||||
|
# Configure PowerShell transcript logging via registry
|
||||||
|
$psPath = "HKLM:\SOFTWARE\Policies\Microsoft\Windows\PowerShell\Transcription"
|
||||||
|
|
||||||
|
if (-not (Test-Path $psPath)) {
|
||||||
|
New-Item -Path $psPath -Force | Out-Null
|
||||||
|
}
|
||||||
|
|
||||||
|
# Enable transcription
|
||||||
|
Set-ItemProperty -Path $psPath -Name "EnableTranscripting" -Value 1 -Type DWord
|
||||||
|
Set-ItemProperty -Path $psPath -Name "EnableInvocationHeader" -Value 1 -Type DWord
|
||||||
|
Set-ItemProperty -Path $psPath -Name "OutputDirectory" -Value $transcriptPath -Type String
|
||||||
|
|
||||||
|
Write-Host "Enabled PowerShell transcription via registry" -ForegroundColor Green
|
||||||
|
|
||||||
|
# Also enable module logging for additional audit trail
|
||||||
|
$modulePath = "HKLM:\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ModuleLogging"
|
||||||
|
if (-not (Test-Path $modulePath)) {
|
||||||
|
New-Item -Path $modulePath -Force | Out-Null
|
||||||
|
}
|
||||||
|
Set-ItemProperty -Path $modulePath -Name "EnableModuleLogging" -Value 1 -Type DWord
|
||||||
|
|
||||||
|
# Enable script block logging
|
||||||
|
$scriptPath = "HKLM:\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging"
|
||||||
|
if (-not (Test-Path $scriptPath)) {
|
||||||
|
New-Item -Path $scriptPath -Force | Out-Null
|
||||||
|
}
|
||||||
|
Set-ItemProperty -Path $scriptPath -Name "EnableScriptBlockLogging" -Value 1 -Type DWord
|
||||||
|
|
||||||
|
Write-Host "Enabled module and script block logging" -ForegroundColor Green
|
||||||
|
|
||||||
|
Write-Host "`nTranscript logging configuration complete!" -ForegroundColor Green
|
||||||
|
Write-Host "Transcripts will be saved to: $transcriptPath"
|
||||||
|
|
||||||
|
# Display current settings
|
||||||
|
Write-Host "`n--- Current Settings ---" -ForegroundColor Yellow
|
||||||
|
Get-ItemProperty -Path $psPath | Select-Object EnableTranscripting, EnableInvocationHeader, OutputDirectory
|
||||||
87
scripts/Get-ADComputerReport.ps1
Normal file
87
scripts/Get-ADComputerReport.ps1
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Generates a report of all Active Directory computers.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
This script queries Active Directory for all computer accounts and exports
|
||||||
|
key properties including name, operating system, last logon, and OU location.
|
||||||
|
|
||||||
|
.PARAMETER OutputPath
|
||||||
|
Optional. Path to export CSV report. If not specified, outputs to console.
|
||||||
|
|
||||||
|
.PARAMETER OperatingSystem
|
||||||
|
Optional. Filter by operating system (e.g., "Windows Server*", "*Windows 10*").
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-ADComputerReport.ps1
|
||||||
|
Lists all computers to console.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-ADComputerReport.ps1 -OperatingSystem "Windows Server*" -OutputPath "C:\ClaudeTools\Logs\servers.csv"
|
||||||
|
Exports all Windows Server computers to CSV.
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Author: ClaudeTools Automation
|
||||||
|
Version: 1.0
|
||||||
|
Requires: ActiveDirectory PowerShell module
|
||||||
|
#>
|
||||||
|
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[string]$OutputPath,
|
||||||
|
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[string]$OperatingSystem = "*"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Import AD module
|
||||||
|
Import-Module ActiveDirectory -ErrorAction Stop
|
||||||
|
|
||||||
|
Write-Host "Querying Active Directory computers..." -ForegroundColor Cyan
|
||||||
|
|
||||||
|
# Get computers with properties
|
||||||
|
$computers = Get-ADComputer -Filter "OperatingSystem -like '$OperatingSystem'" -Properties `
|
||||||
|
OperatingSystem,
|
||||||
|
OperatingSystemVersion,
|
||||||
|
LastLogonDate,
|
||||||
|
Created,
|
||||||
|
Enabled,
|
||||||
|
IPv4Address,
|
||||||
|
Description,
|
||||||
|
DistinguishedName |
|
||||||
|
Select-Object `
|
||||||
|
@{N='Name';E={$_.Name}},
|
||||||
|
@{N='OperatingSystem';E={$_.OperatingSystem}},
|
||||||
|
@{N='OSVersion';E={$_.OperatingSystemVersion}},
|
||||||
|
@{N='Enabled';E={$_.Enabled}},
|
||||||
|
@{N='IPv4Address';E={$_.IPv4Address}},
|
||||||
|
@{N='LastLogon';E={$_.LastLogonDate}},
|
||||||
|
@{N='Created';E={$_.Created}},
|
||||||
|
@{N='OU';E={($_.DistinguishedName -split ',',2)[1]}},
|
||||||
|
@{N='Description';E={$_.Description}}
|
||||||
|
|
||||||
|
$computerCount = ($computers | Measure-Object).Count
|
||||||
|
Write-Host "Found $computerCount computers." -ForegroundColor Green
|
||||||
|
|
||||||
|
if ($OutputPath) {
|
||||||
|
$computers | Export-Csv -Path $OutputPath -NoTypeInformation
|
||||||
|
Write-Host "Report exported to: $OutputPath" -ForegroundColor Green
|
||||||
|
} else {
|
||||||
|
$computers | Format-Table -AutoSize
|
||||||
|
}
|
||||||
|
|
||||||
|
# Summary by OS
|
||||||
|
Write-Host "`n--- Operating System Summary ---" -ForegroundColor Yellow
|
||||||
|
$computers | Group-Object OperatingSystem | Sort-Object Count -Descending |
|
||||||
|
Format-Table @{N='Operating System';E={$_.Name}}, Count -AutoSize
|
||||||
|
|
||||||
|
# Summary by status
|
||||||
|
$enabledCount = ($computers | Where-Object { $_.Enabled -eq $true } | Measure-Object).Count
|
||||||
|
$disabledCount = ($computers | Where-Object { $_.Enabled -eq $false } | Measure-Object).Count
|
||||||
|
Write-Host "Enabled: $enabledCount | Disabled: $disabledCount"
|
||||||
|
|
||||||
|
# Stale computers (no logon in 90 days)
|
||||||
|
$staleDate = (Get-Date).AddDays(-90)
|
||||||
|
$staleCount = ($computers | Where-Object { $_.LastLogon -lt $staleDate -or $null -eq $_.LastLogon } | Measure-Object).Count
|
||||||
|
Write-Host "Stale (no logon 90+ days): $staleCount" -ForegroundColor $(if ($staleCount -gt 0) { 'Yellow' } else { 'Green' })
|
||||||
92
scripts/Get-ADUserReport.ps1
Normal file
92
scripts/Get-ADUserReport.ps1
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Generates a report of all Active Directory users with key properties.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
This script queries Active Directory for all user accounts and exports
|
||||||
|
key properties including name, email, last logon, account status, and group memberships.
|
||||||
|
|
||||||
|
.PARAMETER OutputPath
|
||||||
|
Optional. Path to export CSV report. If not specified, outputs to console.
|
||||||
|
|
||||||
|
.PARAMETER IncludeDisabled
|
||||||
|
Switch to include disabled accounts in the report.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-ADUserReport.ps1
|
||||||
|
Lists all enabled users to console.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-ADUserReport.ps1 -OutputPath "C:\ClaudeTools\Logs\users.csv" -IncludeDisabled
|
||||||
|
Exports all users (including disabled) to CSV file.
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Author: ClaudeTools Automation
|
||||||
|
Version: 1.0
|
||||||
|
Requires: ActiveDirectory PowerShell module
|
||||||
|
#>
|
||||||
|
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[string]$OutputPath,
|
||||||
|
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[switch]$IncludeDisabled
|
||||||
|
)
|
||||||
|
|
||||||
|
# Import AD module
|
||||||
|
Import-Module ActiveDirectory -ErrorAction Stop
|
||||||
|
|
||||||
|
Write-Host "Querying Active Directory users..." -ForegroundColor Cyan
|
||||||
|
|
||||||
|
# Build filter
|
||||||
|
$filter = if ($IncludeDisabled) { "*" } else { "Enabled -eq 'True'" }
|
||||||
|
|
||||||
|
# Get users with properties
|
||||||
|
$users = Get-ADUser -Filter $filter -Properties `
|
||||||
|
DisplayName,
|
||||||
|
EmailAddress,
|
||||||
|
Department,
|
||||||
|
Title,
|
||||||
|
Manager,
|
||||||
|
LastLogonDate,
|
||||||
|
PasswordLastSet,
|
||||||
|
PasswordNeverExpires,
|
||||||
|
Enabled,
|
||||||
|
Created,
|
||||||
|
MemberOf,
|
||||||
|
Description |
|
||||||
|
Select-Object `
|
||||||
|
@{N='SamAccountName';E={$_.SamAccountName}},
|
||||||
|
@{N='DisplayName';E={$_.DisplayName}},
|
||||||
|
@{N='Email';E={$_.EmailAddress}},
|
||||||
|
@{N='Department';E={$_.Department}},
|
||||||
|
@{N='Title';E={$_.Title}},
|
||||||
|
@{N='Enabled';E={$_.Enabled}},
|
||||||
|
@{N='LastLogon';E={$_.LastLogonDate}},
|
||||||
|
@{N='PasswordLastSet';E={$_.PasswordLastSet}},
|
||||||
|
@{N='PasswordNeverExpires';E={$_.PasswordNeverExpires}},
|
||||||
|
@{N='Created';E={$_.Created}},
|
||||||
|
@{N='GroupCount';E={($_.MemberOf | Measure-Object).Count}},
|
||||||
|
@{N='Description';E={$_.Description}}
|
||||||
|
|
||||||
|
$userCount = ($users | Measure-Object).Count
|
||||||
|
Write-Host "Found $userCount users." -ForegroundColor Green
|
||||||
|
|
||||||
|
if ($OutputPath) {
|
||||||
|
$users | Export-Csv -Path $OutputPath -NoTypeInformation
|
||||||
|
Write-Host "Report exported to: $OutputPath" -ForegroundColor Green
|
||||||
|
} else {
|
||||||
|
$users | Format-Table -AutoSize
|
||||||
|
}
|
||||||
|
|
||||||
|
# Summary statistics
|
||||||
|
Write-Host "`n--- Summary ---" -ForegroundColor Yellow
|
||||||
|
Write-Host "Total Users: $userCount"
|
||||||
|
$enabledCount = ($users | Where-Object { $_.Enabled -eq $true } | Measure-Object).Count
|
||||||
|
$disabledCount = ($users | Where-Object { $_.Enabled -eq $false } | Measure-Object).Count
|
||||||
|
Write-Host "Enabled: $enabledCount"
|
||||||
|
Write-Host "Disabled: $disabledCount"
|
||||||
|
$neverExpire = ($users | Where-Object { $_.PasswordNeverExpires -eq $true } | Measure-Object).Count
|
||||||
|
Write-Host "Password Never Expires: $neverExpire"
|
||||||
111
scripts/Get-GPOStatus.ps1
Normal file
111
scripts/Get-GPOStatus.ps1
Normal file
@@ -0,0 +1,111 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Reports on Group Policy Object status and replication.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
This script checks all GPOs in the domain and reports their status,
|
||||||
|
including version information, links, and replication status between
|
||||||
|
AD and SYSVOL.
|
||||||
|
|
||||||
|
.PARAMETER OutputPath
|
||||||
|
Optional. Path to export CSV report. If not specified, outputs to console.
|
||||||
|
|
||||||
|
.PARAMETER CheckReplication
|
||||||
|
Switch to perform detailed replication check between AD and SYSVOL.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-GPOStatus.ps1
|
||||||
|
Lists all GPOs with basic status.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-GPOStatus.ps1 -CheckReplication -OutputPath "C:\ClaudeTools\Logs\gpo-status.csv"
|
||||||
|
Full replication check with CSV export.
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Author: ClaudeTools Automation
|
||||||
|
Version: 1.0
|
||||||
|
Requires: GroupPolicy PowerShell module
|
||||||
|
#>
|
||||||
|
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[string]$OutputPath,
|
||||||
|
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[switch]$CheckReplication
|
||||||
|
)
|
||||||
|
|
||||||
|
# Import required modules
|
||||||
|
Import-Module GroupPolicy -ErrorAction Stop
|
||||||
|
Import-Module ActiveDirectory -ErrorAction Stop
|
||||||
|
|
||||||
|
Write-Host "Querying Group Policy Objects..." -ForegroundColor Cyan
|
||||||
|
|
||||||
|
# Get all GPOs
|
||||||
|
$gpos = Get-GPO -All | Select-Object `
|
||||||
|
@{N='Name';E={$_.DisplayName}},
|
||||||
|
@{N='ID';E={$_.Id}},
|
||||||
|
@{N='Status';E={$_.GpoStatus}},
|
||||||
|
@{N='CreationTime';E={$_.CreationTime}},
|
||||||
|
@{N='ModificationTime';E={$_.ModificationTime}},
|
||||||
|
@{N='UserVersion';E={$_.User.DSVersion}},
|
||||||
|
@{N='ComputerVersion';E={$_.Computer.DSVersion}},
|
||||||
|
@{N='WMIFilter';E={$_.WmiFilter.Name}}
|
||||||
|
|
||||||
|
$gpoCount = ($gpos | Measure-Object).Count
|
||||||
|
Write-Host "Found $gpoCount GPOs." -ForegroundColor Green
|
||||||
|
|
||||||
|
# Check GPO links
|
||||||
|
Write-Host "`nChecking GPO links..." -ForegroundColor Cyan
|
||||||
|
$gpoLinks = @()
|
||||||
|
foreach ($gpo in (Get-GPO -All)) {
|
||||||
|
$report = [xml](Get-GPOReport -Guid $gpo.Id -ReportType Xml)
|
||||||
|
$links = $report.GPO.LinksTo.SOMPath
|
||||||
|
|
||||||
|
$gpoLinks += [PSCustomObject]@{
|
||||||
|
Name = $gpo.DisplayName
|
||||||
|
LinkCount = if ($links) { ($links | Measure-Object).Count } else { 0 }
|
||||||
|
Links = if ($links) { $links -join "; " } else { "Not Linked" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($CheckReplication) {
|
||||||
|
Write-Host "`nChecking SYSVOL replication status..." -ForegroundColor Cyan
|
||||||
|
|
||||||
|
$domain = (Get-ADDomain).DNSRoot
|
||||||
|
$dcs = Get-ADDomainController -Filter *
|
||||||
|
|
||||||
|
foreach ($dc in $dcs) {
|
||||||
|
Write-Host " Checking $($dc.HostName)..." -ForegroundColor Gray
|
||||||
|
$sysvolPath = "\\$($dc.HostName)\SYSVOL\$domain\Policies"
|
||||||
|
|
||||||
|
if (Test-Path $sysvolPath) {
|
||||||
|
$sysvolGPOs = Get-ChildItem $sysvolPath -Directory | Where-Object { $_.Name -match '^{' }
|
||||||
|
Write-Host " SYSVOL GPO count: $($sysvolGPOs.Count)" -ForegroundColor Green
|
||||||
|
} else {
|
||||||
|
Write-Host " Unable to access SYSVOL" -ForegroundColor Red
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Output results
|
||||||
|
if ($OutputPath) {
|
||||||
|
$gpos | Export-Csv -Path $OutputPath -NoTypeInformation
|
||||||
|
Write-Host "`nReport exported to: $OutputPath" -ForegroundColor Green
|
||||||
|
} else {
|
||||||
|
Write-Host "`n--- GPO List ---" -ForegroundColor Yellow
|
||||||
|
$gpos | Format-Table Name, Status, ModificationTime, UserVersion, ComputerVersion -AutoSize
|
||||||
|
|
||||||
|
Write-Host "`n--- GPO Links ---" -ForegroundColor Yellow
|
||||||
|
$gpoLinks | Format-Table Name, LinkCount, Links -AutoSize
|
||||||
|
}
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
Write-Host "`n--- Summary ---" -ForegroundColor Yellow
|
||||||
|
Write-Host "Total GPOs: $gpoCount"
|
||||||
|
$unlinked = ($gpoLinks | Where-Object { $_.LinkCount -eq 0 } | Measure-Object).Count
|
||||||
|
Write-Host "Unlinked GPOs: $unlinked" -ForegroundColor $(if ($unlinked -gt 0) { 'Yellow' } else { 'Green' })
|
||||||
|
|
||||||
|
$disabled = ($gpos | Where-Object { $_.Status -ne 'AllSettingsEnabled' } | Measure-Object).Count
|
||||||
|
Write-Host "Disabled/Partial GPOs: $disabled" -ForegroundColor $(if ($disabled -gt 0) { 'Yellow' } else { 'Green' })
|
||||||
173
scripts/Get-ReplicationHealth.ps1
Normal file
173
scripts/Get-ReplicationHealth.ps1
Normal file
@@ -0,0 +1,173 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Checks Active Directory replication health across domain controllers.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
This script performs comprehensive AD replication health checks including
|
||||||
|
replication status, partner connectivity, and identifies any replication failures.
|
||||||
|
|
||||||
|
.PARAMETER OutputPath
|
||||||
|
Optional. Path to export results. If not specified, outputs to console.
|
||||||
|
|
||||||
|
.PARAMETER Detailed
|
||||||
|
Switch to show detailed replication information per DC.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-ReplicationHealth.ps1
|
||||||
|
Basic replication health check.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-ReplicationHealth.ps1 -Detailed -OutputPath "C:\ClaudeTools\Logs\repl-health.txt"
|
||||||
|
Detailed check with output to file.
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Author: ClaudeTools Automation
|
||||||
|
Version: 1.0
|
||||||
|
Requires: ActiveDirectory PowerShell module, repadmin.exe
|
||||||
|
#>
|
||||||
|
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[string]$OutputPath,
|
||||||
|
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[switch]$Detailed
|
||||||
|
)
|
||||||
|
|
||||||
|
# Import AD module
|
||||||
|
Import-Module ActiveDirectory -ErrorAction Stop
|
||||||
|
|
||||||
|
$output = @()
|
||||||
|
$output += "=" * 60
|
||||||
|
$output += "AD REPLICATION HEALTH REPORT"
|
||||||
|
$output += "Generated: $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
Write-Host "Checking AD Replication Health..." -ForegroundColor Cyan
|
||||||
|
|
||||||
|
# Get all DCs
|
||||||
|
$dcs = Get-ADDomainController -Filter *
|
||||||
|
$output += "`nDomain Controllers Found: $($dcs.Count)"
|
||||||
|
|
||||||
|
foreach ($dc in $dcs) {
|
||||||
|
$output += "`n--- $($dc.HostName) ---"
|
||||||
|
Write-Host "Checking $($dc.HostName)..." -ForegroundColor Gray
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check replication summary using repadmin
|
||||||
|
$output += "`n" + "=" * 60
|
||||||
|
$output += "REPLICATION SUMMARY (repadmin /replsummary)"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
try {
|
||||||
|
$replSummary = repadmin /replsummary 2>&1
|
||||||
|
$output += $replSummary
|
||||||
|
Write-Host "Replication summary retrieved." -ForegroundColor Green
|
||||||
|
} catch {
|
||||||
|
$output += "ERROR: Unable to run repadmin /replsummary"
|
||||||
|
Write-Host "Error running repadmin" -ForegroundColor Red
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check for replication failures
|
||||||
|
$output += "`n" + "=" * 60
|
||||||
|
$output += "REPLICATION FAILURES (repadmin /showrepl * /errorsonly)"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
try {
|
||||||
|
$replErrors = repadmin /showrepl * /errorsonly 2>&1
|
||||||
|
if ($replErrors -match "error" -or $replErrors -match "fail") {
|
||||||
|
$output += $replErrors
|
||||||
|
Write-Host "Replication ERRORS detected!" -ForegroundColor Red
|
||||||
|
} else {
|
||||||
|
$output += "No replication errors detected."
|
||||||
|
Write-Host "No replication errors." -ForegroundColor Green
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
$output += "ERROR: Unable to check replication errors"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Queue length
|
||||||
|
$output += "`n" + "=" * 60
|
||||||
|
$output += "REPLICATION QUEUE (repadmin /queue)"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
try {
|
||||||
|
$replQueue = repadmin /queue 2>&1
|
||||||
|
$output += $replQueue
|
||||||
|
} catch {
|
||||||
|
$output += "ERROR: Unable to check replication queue"
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($Detailed) {
|
||||||
|
$output += "`n" + "=" * 60
|
||||||
|
$output += "DETAILED REPLICATION STATUS (repadmin /showrepl)"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
try {
|
||||||
|
$replDetail = repadmin /showrepl 2>&1
|
||||||
|
$output += $replDetail
|
||||||
|
} catch {
|
||||||
|
$output += "ERROR: Unable to get detailed replication status"
|
||||||
|
}
|
||||||
|
|
||||||
|
# DFSR Health (if applicable)
|
||||||
|
$output += "`n" + "=" * 60
|
||||||
|
$output += "DFSR SYSVOL REPLICATION STATUS"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
try {
|
||||||
|
$dfsrStatus = Get-DfsrMember -ErrorAction SilentlyContinue
|
||||||
|
if ($dfsrStatus) {
|
||||||
|
$output += "DFSR Members:"
|
||||||
|
foreach ($member in $dfsrStatus) {
|
||||||
|
$output += " - $($member.ComputerName): $($member.DomainName)"
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
$output += "DFSR not configured or FRS in use."
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
$output += "Unable to query DFSR status (may be using FRS)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# AD Database health
|
||||||
|
$output += "`n" + "=" * 60
|
||||||
|
$output += "AD DATABASE INTEGRITY"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
$adDb = Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\NTDS\Parameters" -ErrorAction SilentlyContinue
|
||||||
|
if ($adDb) {
|
||||||
|
$dbPath = $adDb.'DSA Database file'
|
||||||
|
$logPath = $adDb.'Database log files path'
|
||||||
|
$output += "Database Path: $dbPath"
|
||||||
|
$output += "Log Path: $logPath"
|
||||||
|
|
||||||
|
if (Test-Path $dbPath) {
|
||||||
|
$dbSize = (Get-Item $dbPath).Length / 1MB
|
||||||
|
$output += "Database Size: $([math]::Round($dbSize, 2)) MB"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Final summary
|
||||||
|
$output += "`n" + "=" * 60
|
||||||
|
$output += "HEALTH CHECK COMPLETE"
|
||||||
|
$output += "=" * 60
|
||||||
|
|
||||||
|
# Output results
|
||||||
|
if ($OutputPath) {
|
||||||
|
$output | Out-File -FilePath $OutputPath -Encoding UTF8
|
||||||
|
Write-Host "`nReport saved to: $OutputPath" -ForegroundColor Green
|
||||||
|
} else {
|
||||||
|
$output | ForEach-Object { Write-Host $_ }
|
||||||
|
}
|
||||||
|
|
||||||
|
# Quick status summary
|
||||||
|
Write-Host "`n--- Quick Status ---" -ForegroundColor Yellow
|
||||||
|
Write-Host "Domain Controllers: $($dcs.Count)"
|
||||||
|
$errorMatch = $replErrors -match "error|fail"
|
||||||
|
if ($errorMatch) {
|
||||||
|
Write-Host "Replication Status: ERRORS DETECTED" -ForegroundColor Red
|
||||||
|
} else {
|
||||||
|
Write-Host "Replication Status: HEALTHY" -ForegroundColor Green
|
||||||
|
}
|
||||||
107
scripts/Invoke-LogRotation.ps1
Normal file
107
scripts/Invoke-LogRotation.ps1
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Rotates and cleans up old log files.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
Removes transcript and log files older than the specified retention period.
|
||||||
|
Designed to run as a scheduled task daily.
|
||||||
|
|
||||||
|
.PARAMETER RetentionDays
|
||||||
|
Number of days to retain log files. Default is 30.
|
||||||
|
|
||||||
|
.PARAMETER LogPath
|
||||||
|
Path to the logs directory. Default is C:\ClaudeTools\Logs.
|
||||||
|
|
||||||
|
.PARAMETER WhatIf
|
||||||
|
Shows what would be deleted without actually deleting.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Invoke-LogRotation.ps1
|
||||||
|
Removes logs older than 30 days.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Invoke-LogRotation.ps1 -RetentionDays 14 -WhatIf
|
||||||
|
Shows what would be deleted with 14-day retention.
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Author: ClaudeTools Automation
|
||||||
|
Version: 1.0
|
||||||
|
#>
|
||||||
|
|
||||||
|
[CmdletBinding(SupportsShouldProcess)]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[int]$RetentionDays = 30,
|
||||||
|
|
||||||
|
[Parameter(Mandatory=$false)]
|
||||||
|
[string]$LogPath = "C:\ClaudeTools\Logs"
|
||||||
|
)
|
||||||
|
|
||||||
|
$rotationLog = Join-Path $LogPath "rotation.log"
|
||||||
|
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
||||||
|
|
||||||
|
function Write-Log {
|
||||||
|
param([string]$Message)
|
||||||
|
$logEntry = "[$timestamp] $Message"
|
||||||
|
Add-Content -Path $rotationLog -Value $logEntry
|
||||||
|
Write-Host $logEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Log "=== Log Rotation Started ==="
|
||||||
|
Write-Log "Retention Period: $RetentionDays days"
|
||||||
|
Write-Log "Log Path: $LogPath"
|
||||||
|
|
||||||
|
$cutoffDate = (Get-Date).AddDays(-$RetentionDays)
|
||||||
|
$totalDeleted = 0
|
||||||
|
$totalSizeFreed = 0
|
||||||
|
|
||||||
|
# Find and delete old files
|
||||||
|
$oldFiles = Get-ChildItem -Path $LogPath -Recurse -File |
|
||||||
|
Where-Object { $_.LastWriteTime -lt $cutoffDate -and $_.Name -ne "rotation.log" }
|
||||||
|
|
||||||
|
$fileCount = ($oldFiles | Measure-Object).Count
|
||||||
|
Write-Log "Found $fileCount files older than $RetentionDays days"
|
||||||
|
|
||||||
|
foreach ($file in $oldFiles) {
|
||||||
|
$fileSize = $file.Length
|
||||||
|
$filePath = $file.FullName
|
||||||
|
|
||||||
|
if ($PSCmdlet.ShouldProcess($filePath, "Delete")) {
|
||||||
|
try {
|
||||||
|
Remove-Item $filePath -Force
|
||||||
|
$totalDeleted++
|
||||||
|
$totalSizeFreed += $fileSize
|
||||||
|
Write-Log "Deleted: $filePath ($([math]::Round($fileSize/1KB, 2)) KB)"
|
||||||
|
} catch {
|
||||||
|
Write-Log "ERROR deleting $filePath : $_"
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
Write-Log "WhatIf: Would delete $filePath ($([math]::Round($fileSize/1KB, 2)) KB)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Delete empty subdirectories
|
||||||
|
$emptyDirs = Get-ChildItem -Path $LogPath -Directory -Recurse |
|
||||||
|
Where-Object { (Get-ChildItem $_.FullName -Force).Count -eq 0 }
|
||||||
|
|
||||||
|
foreach ($dir in $emptyDirs) {
|
||||||
|
if ($PSCmdlet.ShouldProcess($dir.FullName, "Remove empty directory")) {
|
||||||
|
try {
|
||||||
|
Remove-Item $dir.FullName -Force
|
||||||
|
Write-Log "Removed empty directory: $($dir.FullName)"
|
||||||
|
} catch {
|
||||||
|
Write-Log "ERROR removing directory $($dir.FullName) : $_"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
$sizeMB = [math]::Round($totalSizeFreed / 1MB, 2)
|
||||||
|
Write-Log "=== Rotation Complete ==="
|
||||||
|
Write-Log "Files Deleted: $totalDeleted"
|
||||||
|
Write-Log "Space Freed: $sizeMB MB"
|
||||||
|
|
||||||
|
# Show current disk usage
|
||||||
|
$currentSize = (Get-ChildItem -Path $LogPath -Recurse -File | Measure-Object -Property Length -Sum).Sum
|
||||||
|
$currentSizeMB = [math]::Round($currentSize / 1MB, 2)
|
||||||
|
Write-Log "Current Log Directory Size: $currentSizeMB MB"
|
||||||
177
session-logs/2026-04-03-session-ad2.md
Normal file
177
session-logs/2026-04-03-session-ad2.md
Normal file
@@ -0,0 +1,177 @@
|
|||||||
|
# Session Log - AD2 - 2026-04-03
|
||||||
|
|
||||||
|
## Session Summary
|
||||||
|
|
||||||
|
Continued test datasheet pipeline work from the 03-27/28/29 session. Major focus on:
|
||||||
|
1. Fixing formatting issues in generated TXT datasheets (column alignment matching QuickBASIC)
|
||||||
|
2. Adding 7B product family support to the exact-match formatter
|
||||||
|
3. Loading additional spec files (5B49_2.DAT for the last missing Quatronix datasheet)
|
||||||
|
4. Work order report ingestion (33K WOs, 63K lines, 2.27M records linked)
|
||||||
|
5. UI improvements (WO search, WO detail popup, View button styled HTML)
|
||||||
|
6. Generating full Dataforth audit document
|
||||||
|
7. Planning and beginning PostgreSQL migration (SQLite hitting 4.4GB, queries timing out)
|
||||||
|
8. Uninstalling SQL Server Express, installing PostgreSQL 18
|
||||||
|
|
||||||
|
## Key Decisions
|
||||||
|
- PostgreSQL over SQL Server Express (no 10GB limit, better FTS, no licensing)
|
||||||
|
- Install PG locally on AD2 (not SAGE-SQL or new VM)
|
||||||
|
- Clean cutover (no parallel run with SQLite)
|
||||||
|
- MSSQL Express uninstalled to free resources
|
||||||
|
|
||||||
|
## Credentials
|
||||||
|
|
||||||
|
### PostgreSQL (newly installed)
|
||||||
|
- Host: localhost (AD2, 192.168.0.6)
|
||||||
|
- Port: 5432
|
||||||
|
- Superuser: postgres / DfPgSQL2026!
|
||||||
|
- App role: testdatadb_app / DfTestDB2026!
|
||||||
|
- Database: testdatadb
|
||||||
|
- Service: postgresql-18 (runs as INTRANET\sysadmin)
|
||||||
|
- Install path: C:\Program Files\PostgreSQL\18\
|
||||||
|
- Data dir: C:\Program Files\PostgreSQL\18\data\
|
||||||
|
|
||||||
|
### TestDataDB (existing)
|
||||||
|
- Service: testdatadb (runs as INTRANET\svc_testdatadb / DfSvcTDB2026!)
|
||||||
|
- URL: http://192.168.0.6:3000
|
||||||
|
- SQLite DB: C:\Shares\testdatadb\database\testdata.db (4.4 GB)
|
||||||
|
|
||||||
|
### Existing (from CLAUDE.md)
|
||||||
|
- AD Sysadmin: INTRANET\sysadmin / Paper123!@#
|
||||||
|
- D2TESTNAS SSH: root@192.168.0.9 / Paper123!@#-nas
|
||||||
|
- Rsync: port 873, user rsync / IQ203s32119
|
||||||
|
- M365 Tenant: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
|
||||||
|
- Gitea: mike@azcomputerguru.com / Gptf*77ttb123!@#-git
|
||||||
|
- Gitea API Token: 9b1da4b79a38ef782268341d25a4b6880572063f
|
||||||
|
|
||||||
|
## Work Completed
|
||||||
|
|
||||||
|
### Formatting Fixes (datasheet-exact.js)
|
||||||
|
- Compared generated TXT against originals from X:\For_Web\2025\
|
||||||
|
- Fixed Final Test Results column alignment to match QB TAB positions:
|
||||||
|
- TAB(5) param name, TAB(31) measured value, TAB(60-speclen) spec, TAB(61) unit, TAB(71) PASS/FAIL
|
||||||
|
- Added setCol() and padToCol() helpers for exact column positioning
|
||||||
|
- Fixed STR$() emulation: leading space for positive, drops leading zero (.03 not 0.03)
|
||||||
|
- Fixed header spacing (Date, Model, SN fields)
|
||||||
|
- Fixed checklist spacing to match QB TAB(45)
|
||||||
|
- Fixed separator line length (23 chars not 24)
|
||||||
|
|
||||||
|
### 7B Product Family Support
|
||||||
|
- Added SCM7B DATA_LINES (31 parameters vs 20 for SCM5B)
|
||||||
|
- Added SCM7B TSPECS (supply current, linearity, accuracy, excitation, CJC, noise, etc.)
|
||||||
|
- Added 7B raw_data parser (single CSV line format vs multi-line for other families)
|
||||||
|
- 7B-specific footer: 120VAC (not 240), "Packing Check List" (not "Check List"), blank checkmarks, Tested by/QC lines
|
||||||
|
- Accuracy section suppressed for 7B (DAT format doesn't include individual test points)
|
||||||
|
- 7B model names prepend "SCM" in header
|
||||||
|
|
||||||
|
### Additional Spec File: 5B49_2.DAT
|
||||||
|
- 5B49DATA.DAT was 0 bytes (empty), John pointed to 5B49_2.DAT
|
||||||
|
- 15 models, 93 bytes/record, added SCM5B49_FIELDS to spec-reader.js
|
||||||
|
- Completed final missing Quatronix datasheet (177000-15, SCM5B49-05)
|
||||||
|
- All 73/73 Quatronix datasheets now generated
|
||||||
|
|
||||||
|
### View Button Enhancement
|
||||||
|
- /api/datasheet/:id now uses exact-match formatter with styled HTML
|
||||||
|
- White page on gray background, monospace font, print-optimized
|
||||||
|
- Includes Print and Download PDF buttons
|
||||||
|
- Falls back to generic template if exact-match not available
|
||||||
|
|
||||||
|
### PDF Endpoint Fallback
|
||||||
|
- /api/datasheet/:id/pdf falls back to generic template when exact-match fails
|
||||||
|
- Prevents 422 errors for unsupported families
|
||||||
|
|
||||||
|
### Work Order Reports
|
||||||
|
- Created parsers/wo-report.js (parses WO status report TXT format)
|
||||||
|
- Created database/import-work-orders.js (imports WOs, creates tables, links to test records)
|
||||||
|
- Created work_orders table (33,745 records) and work_order_lines table (63,263 records)
|
||||||
|
- Added work_order column to test_records, 2,277,183 records linked via serial number pattern
|
||||||
|
- Added /api/workorder/:wo and /api/workorder-search endpoints
|
||||||
|
- Added WO# search field and clickable WO link in UI detail view
|
||||||
|
- WO detail popup shows all test lines, pass/fail history, program version
|
||||||
|
- Added auto-import hooks to Sync-FromNAS-rsync.ps1 for Reports and STAGE folders
|
||||||
|
|
||||||
|
### Sync Script Updates (Sync-FromNAS-rsync.ps1)
|
||||||
|
- Added STAGE folder sync (NAS STAGE -> AD2)
|
||||||
|
- Added $syncedReportFiles and $syncedStageFiles tracking
|
||||||
|
- Added WO report auto-import after sync
|
||||||
|
- Added STAGE TXT auto-import after sync
|
||||||
|
- Updated status file with WO and STAGE counts
|
||||||
|
|
||||||
|
### Dataforth Audit Document
|
||||||
|
- Generated C:\Users\sysadmin\Desktop\Dataforth-Audit-2026-04-02.txt
|
||||||
|
- 22 sections: company info, network, AD computers/users/groups, GPO, DNS, shares, tasks, services, firewall, credentials, M365, security incidents, manufacturing infrastructure, pipeline, applications, backups, known issues, contacts
|
||||||
|
|
||||||
|
### PostgreSQL Migration (IN PROGRESS)
|
||||||
|
- Plan approved: PostgreSQL 18 on AD2, clean cutover
|
||||||
|
- PostgreSQL 18.3 installed via Chocolatey
|
||||||
|
- Database cluster initialized (C:\Program Files\PostgreSQL\18\data\)
|
||||||
|
- Service registered as postgresql-18 (runs as INTRANET\sysadmin)
|
||||||
|
- Database created: testdatadb owned by testdatadb_app
|
||||||
|
- SQL Server Express 2022 uninstalled
|
||||||
|
|
||||||
|
### SQLite Issues Documented
|
||||||
|
- Database grown to 4.4GB
|
||||||
|
- Single search query took 5,157 seconds (85 minutes)
|
||||||
|
- Stats query took 216 seconds
|
||||||
|
- WAL file grew to 454MB
|
||||||
|
- ANALYZE ran for 30+ minutes without completing
|
||||||
|
- SHM/WAL permission conflicts between SYSTEM and sysadmin
|
||||||
|
|
||||||
|
### Service Account (from previous session, still relevant)
|
||||||
|
- INTRANET\svc_testdatadb created in OU=ServiceAccounts
|
||||||
|
- testdatadb Windows service runs under this account
|
||||||
|
- SeServiceLogonRight granted
|
||||||
|
- Permissions on C:\Shares\testdatadb and C:\Shares\webshare
|
||||||
|
|
||||||
|
## Files Created
|
||||||
|
- C:\Shares\testdatadb\fix-sysadmin-logon.ps1
|
||||||
|
- C:\ClaudeTools\Test Datasheets\weekend-update-draft.md
|
||||||
|
- C:\Users\sysadmin\Desktop\Dataforth-Audit-2026-04-02.txt
|
||||||
|
- C:\Users\sysadmin\.claude\plans\generic-petting-lovelace.md (PG migration plan)
|
||||||
|
|
||||||
|
## Files Modified
|
||||||
|
- C:\Shares\testdatadb\templates\datasheet-exact.js — Column alignment, 7B support, STR$() emulation
|
||||||
|
- C:\Shares\testdatadb\parsers\spec-reader.js — Added 5B49 TYPE, fuzzy model matching
|
||||||
|
- C:\Shares\testdatadb\parsers\wo-report.js — Created (WO report parser)
|
||||||
|
- C:\Shares\testdatadb\database\import-work-orders.js — Created (WO importer)
|
||||||
|
- C:\Shares\testdatadb\database\schema.sql — Added work_order column
|
||||||
|
- C:\Shares\testdatadb\routes\api.js — Added WO endpoints, PDF fallback, view button exact-match
|
||||||
|
- C:\Shares\testdatadb\public\index.html — WO search, WO popup, view button styling
|
||||||
|
- C:\Shares\test\scripts\Sync-FromNAS-rsync.ps1 — STAGE sync, WO auto-import
|
||||||
|
|
||||||
|
## Database Stats (as of session end)
|
||||||
|
- Test records: 2,294,877
|
||||||
|
- Work orders: 33,745
|
||||||
|
- Work order lines: 63,263
|
||||||
|
- Records with WO linked: 2,277,183
|
||||||
|
- ForWeb exported: ~1,436,000
|
||||||
|
- Model specs loaded: 1,470+
|
||||||
|
|
||||||
|
## Pending / Next Steps
|
||||||
|
1. **PostgreSQL Migration (Step 2-8):**
|
||||||
|
- Create PG schema (schema-pg.sql) with tsvector FTS
|
||||||
|
- Create database abstraction layer (db.js)
|
||||||
|
- Migrate routes/api.js to async
|
||||||
|
- Migrate import/export scripts
|
||||||
|
- Build data migration script
|
||||||
|
- Migrate 2.28M records
|
||||||
|
- Test and cutover
|
||||||
|
|
||||||
|
2. **Tune PostgreSQL config:**
|
||||||
|
- shared_buffers=1GB, work_mem=64MB, maintenance_work_mem=256MB
|
||||||
|
|
||||||
|
3. **MSSQL cleanup:**
|
||||||
|
- May need reboot to fully remove SQL Server services
|
||||||
|
|
||||||
|
4. **Website upload:**
|
||||||
|
- Old ASP.NET endpoints still dead (404)
|
||||||
|
- Need to determine new upload mechanism
|
||||||
|
|
||||||
|
5. **Joel Lohr account:**
|
||||||
|
- Retired March 31 — account needs to be disabled
|
||||||
|
|
||||||
|
## Infrastructure
|
||||||
|
- AD2: 192.168.0.6 (Windows Server 2016)
|
||||||
|
- PostgreSQL 18: localhost:5432 (newly installed, service running)
|
||||||
|
- TestDataDB: localhost:3000 (SQLite, service running)
|
||||||
|
- NAS: 192.168.0.9 (rsync port 873)
|
||||||
|
- STAGE backlog: cleared (0 files remaining)
|
||||||
Reference in New Issue
Block a user