From 4d9ca2b0f75b271e58d887698b2bc9e1873a6006 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Thu, 18 Jun 2026 07:08:22 -0700 Subject: [PATCH] sync: auto-sync from AD2 at 2026-06-18 07:07:25 Author: Mike Swanson Machine: AD2 Timestamp: 2026-06-18 07:07:25 --- ...6-06-18-mike-dsca-fix2-stage2-3-publish.md | 181 ++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-dsca-fix2-stage2-3-publish.md diff --git a/projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-dsca-fix2-stage2-3-publish.md b/projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-dsca-fix2-stage2-3-publish.md new file mode 100644 index 00000000..bccea3ef --- /dev/null +++ b/projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-dsca-fix2-stage2-3-publish.md @@ -0,0 +1,181 @@ +# DSCA Datasheet Fix 2 — STAGE 2 wire-in, STAGE 3 validator, publish of 68 clean models + +## User +- **User:** Mike Swanson (mike) +- **Machine:** AD2 +- **Role:** admin + +## Session Summary + +Picked up the AD2-local handoff `projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md` +(ref Syncro #32441) and executed Fix 2 (DSCA Final-Test rebuild) STAGE 2 and STAGE 3, then +published the validated subset to the live Hoffman site. Work was on the DEPLOYED pipeline at +`C:\Shares\testdatadb` (Node + PostgreSQL 18); repo copies under +`projects/dataforth-dos/datasheet-pipeline/implementation/` are stale and were reconciled after. + +STAGE 2 wired the STAGE-1 output `dsca-templates.json` (126 per-model layouts) into the deployed +`templates/datasheet-exact.js`. For `family === 'DSCA'` the Final-Test block now renders parameter +names + specs from the staged template rows (not the single hardcoded `DATA_LINES['DSCA']` + +`buildTSpecs` DSCA branch); value-bearing `raw_data` STATUS groups map positionally onto the +spec-bearing rows; empty-spec rows (240VAC Withstand / Hi-Pot) render blank+PASS (the duplicate +hardcoded footer for DSCA was removed). The ACCURACY block now uses the template `accOut` +(`Output (V)`/`Output (mA)`) with `-` rule separators instead of `Vout (V)` + `=`. Two real +defects were found and fixed in the process (both are the handoff's "lines drop / wrong values" +defect): negative signs were dropped because value parsing started at index 5 instead of 4, and +`parseRawData` always consumed the line after the 5 accuracy points as a step-response placeholder +— but many DSCA models omit that bare `0` line, so the first STATUS group (and its rows) was being +discarded. Both fixes were validated against the DSCA38-05 golden staged original: the rebuilt +Final-Test block is byte-for-byte identical (the only residual diffs are deferred cosmetic ACCURACY +column spacing). + +STAGE 3 built a read-only validator (`tools/validate-dsca-stage3.js`) that, for every staged DSCA +original we have ground truth for (2,806 across 126 models), looks up the DB record, renders it +through the live path, and content-compares. The gate is the FINAL TEST RESULTS section with rule +lines canonicalized and whitespace collapsed (so the deferred column-spacing cosmetic does not +register); accuracy-section diffs are reported separately. Verdict: 68 models FINAL-TEST +CONTENT-CLEAN (2,123/2,316 compared certs match exactly, 91.7%), 26 models with measured-value +last-digit diffs only, and ~32 models rendering null via the count-guard. A safety guard was added: +when a model's value count != its spec-row count the positional zip is ambiguous (the subtype +measures load points the template omits, e.g. DSCA49's 5mA pair), so the cert is skipped/flagged +rather than emitting misaligned data. + +Per Mike's direction, published the 68 STAGE-3-clean models. Restarted the `testdatadb` service so +the new template is live, canaried a single cert (DSCA30-01 / 148059-3 -> updated, 0 errors), then +re-pushed all 30,423 PASS certs for the 68 clean models via `uploadBySerialNumbers` from a fresh +node process. Result: updated=26,022, unchanged=2,738, created=0, errors=0, skipped=1,663. The +26,022 updates replaced the old defective DSCA renders on the live site with the rebuilt, validated +ones; the 1,663 skips are the count-guard correctly refusing individual ambiguous certs. + +All work committed to the `ad2` branch (4 commits) and pushed. No vault access on AD2 (no +sops/age, vault repo absent), so the Syncro #32441 ticket update could not be posted from here — +left as a handoff to GURU-5070 (see Pending / Handoff). + +## Key Decisions + +- **Skip, don't guess, on ambiguous layouts.** When a DSCA model's value-bearing STATUS count != + its spec-bearing template-row count, the simple positional zip would misplace values. Chose to + return null (skip + flag for STAGE 3 per-subtype mapping) rather than publish misaligned data, + per the handoff's "do not guess" discipline. This is what produces the 1,663 push skips. +- **Scoped the parser/format fixes to DSCA only.** The step-response detection fix and the + index-4 value-start fix live in `formatMeasuredExact` (new) and a DSCA-gated branch in + `parseRawData`, leaving the already-validated 5B/8B/7B/DSCT/SCMVAS paths byte-unchanged. +- **DSCA decimal code N -> toFixed(N) exactly.** Unlike 5B/8B (where format code 2 means 1 + decimal), DSCA uses the trailing status digit as a literal decimal-place count; verified against + the golden (e.g. Output Reg code 2 -> "0.00"). +- **Gate STAGE 3 on the Final-Test section, not the whole sheet.** ACCURACY-block column spacing + is the handoff's explicitly-deferred cosmetic gap; canonicalizing rule lines + collapsing + whitespace isolates real content diffs (names/values/specs/statuses) from spacing noise. +- **Published only the 68 clean models.** Highest-confidence set; idempotent push replaces + defective live certs. Left the 26 last-digit-diff models, ~32 ambiguous layouts, and 231 + untemplated models for follow-up. +- **Did not restart-then-publish through the running service.** Re-push ran from a fresh node + process (picks up new template via `require`); the service restart only refreshes the internal + tool's on-demand renders. + +## Problems Encountered + +- **Sync rebase conflict in `errorlog.md`** — two machines appended entries concurrently. Resolved + by keeping both entries (append-only log) and `git rebase --continue`. +- **`sync.sh` push failed (`src refspec main does not match any`)** — known AD2 fork gotcha + (sync.sh is not fork-aware on push). Pushed manually with `git push --force-with-lease origin ad2` + (rebase rewrote history). See memory `project_ad2_dataforth_fork`. +- **Dropped rows on DSCA models without a step-response line** — `parseRawData` ate the first + STATUS group. Fixed by skipping the step-response consume for DSCA when the next line starts with + PASS/FAIL. +- **Negative measured values rendered without sign** — value substring started at index 5 (assumed + a space at index 4), but negatives put `-` at index 4. Fixed to start at index 4. +- **First STAGE 3 run showed 0 matches / 2,316 mismatches** — the validator's normalization wasn't + canonicalizing rule lines, so the deferred ACCURACY separator dash-count cosmetic masked all + downstream content. Reworked to canonicalize rule lines and gate on the Final-Test section. +- **No vault access on AD2** — cannot fetch Syncro creds to post #32441. Handoff to 5070 (below). + +## Configuration Changes + +Deployed (live, outside repo — `C:\Shares\testdatadb`): +- `templates/datasheet-exact.js` — DSCA wire-in (edited). Save-state: + `templates/datasheet-exact.js.bak-2026-06-18-1334`. +- `_validate_dsca_stage3.js`, `_push_clean68.js`, `_clean-models.json`, `_dsca-stage3-report.txt`, + `_stage3-run.log`, `_push-clean68.log` — operational scripts/outputs (created). + +Repo (`ad2` branch): +- `projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js` — reconciled (modified). +- `projects/dataforth-dos/datasheet-pipeline/implementation/dsca-templates.json` — added. +- `projects/dataforth-dos/tools/validate-dsca-stage3.js` — added. +- `projects/dataforth-dos/tools/push-clean68.js` — added. +- `projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt` — added. +- `projects/dataforth-dos/dsca-clean68-models.json` — added (the 68 published models). +- `errorlog.md` — conflict resolution during sync. + +## Credentials & Secrets + +No new credentials discovered or created this session. Referenced (already vaulted per the handoff +at `clients/dataforth/testdatadb-postgres`): +- PostgreSQL superuser `postgres` / `Paper123!@#`; app `testdatadb_app` / `DfTestDB2026!` + (`db.js` defaults: host localhost:5432, database `testdatadb`). +- Hoffman/CloudFilter uploader creds: `C:\ProgramData\dataforth-uploader\credentials.json` + (fields CF_TOKEN_URL, CF_API_BASE, CF_CLIENT_ID, CF_CLIENT_SECRET, CF_SCOPE). Not vaulted; lives + on the box only. + +## Infrastructure & Servers + +- **AD2** (192.168.0.6) — Dataforth domain controller; runs the deployed pipeline at + `C:\Shares\testdatadb`. Windows service `testdatadb` (Automatic, Running) — restarted this session. +- **PostgreSQL 18** on AD2 (localhost:5432, db `testdatadb`); table `test_records`. +- **Staged originals**: `C:/Shares/test/STAGE/**/*.TXT` (11,956 total; 2,806 DSCA) — STAGE-1 source + and STAGE-3 ground truth. +- **Hoffman API** (`CF_API_BASE/api/v1/TestReportDataFiles/bulk` POST, idempotent; GET is paginated + list only) — the live customer-facing datasheet site. + +## Commands & Outputs + +- Sync conflict recovery: `git add errorlog.md && GIT_EDITOR=true git rebase --continue` then + `git push --force-with-lease origin ad2`. +- Module load-check after each edit: `node -e 'require("./templates/datasheet-exact.js")'`. +- STAGE 3 run: `node _validate_dsca_stage3.js` -> `_dsca-stage3-report.txt`. Summary: 68 clean, + 26 Final-Test mismatch (193 certs), ~32 null; 2,123/2,316 match. +- Canary: `uploadBySerialNumbers(["148059-3"])` -> `updated=1 errors=0`. +- Full push: `node _push_clean68.js` -> `created=0 updated=26022 unchanged=2738 errors=0 skipped=1663` + over 30,423 PASS certs. + +## Pending / Incomplete Tasks + +Not yet published / remaining DSCA work: +1. **26 models, last-digit measured diffs** — two causes: (a) QB single-precision half-up rounding + vs JS double `toFixed` (e.g. raw 9.9995 code3 -> "9.999" here, "10.000" in golden) — fixable but + float-precision-sensitive, re-run validator to confirm no regression; (b) retest data-vintage, + where the staged `.TXT` is an older test run than the DB latest-wins record (Fix 3) — not a + render bug. +2. **~32 ambiguous layouts** (DSCA33-*, DSCA45-*, DSCA49-* families) — need the canonical + per-subtype slot mapping (status entry index -> canonical DSCA slot -> template row by name); + currently skipped (null) by the count-guard. +3. **231 untemplated models / 23,866 certs** — need a STAGE 1 extension (more staged originals; + only 126/357 DSCA models in the DB currently have a template — 70.1% of certs). +4. **Fix 5** (backfill 379 cryptolocker-era units from staged `.TXT`) and Fix-3 cleanup items — not + started this session. + +## Note for mike (handoff to GURU-5070) + +AD2 has no vault access (no sops/age, `D:/vault` absent), so I could not post the Syncro **#32441** +update from here. On GURU-5070 (which has vault + Syncro access), please post an internal/hidden +note to #32441 summarizing this session: + +> Fix 2 (DSCA Final-Test rebuild) STAGE 2 + STAGE 3 complete. Wired per-model staged templates into +> the live renderer; fixed two render defects (dropped negative signs, dropped Final-Test rows on +> models lacking the step-response line). Built a STAGE 3 validator over 2,806 staged DSCA originals: +> 68 models are Final-Test content-clean. Published those 68 to Hoffman — 26,022 certs updated, +> 0 errors (replacing the old defective DSCA renders live). Remaining: 26 models with measured +> last-digit diffs (rounding-mode + retest vintage), ~32 ambiguous-layout families (DSCA33/45/49) +> needing per-subtype slot mapping, and 231 untemplated models (23,866 certs) needing a STAGE 1 +> extension. Byte-for-byte DOS cosmetic fidelity (leading rule line + ~1-space input-column spacing) +> still deferred across all families. Details: ad2 branch commits + DSCA-STAGE3-REPORT-2026-06-18.txt. + +## Reference Information + +- Handoff: `projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md` +- STAGE 3 report: `projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt` +- Published set: `projects/dataforth-dos/dsca-clean68-models.json` (68 models) +- Ticket: Syncro **#32441** (customer-facing thread: John Lehman) +- `ad2` commits this session: `e7fa7cc6` (STAGE 2), `c03bdc9a` (STAGE 3 validator+report), + `f798e8ea` (publish artifacts); `fc9fff81` (sync auto-commit, errorlog conflict resolution). +- Deployed render path: `database/render-datasheet.js` -> `templates/datasheet-exact.js`; + push path: `database/upload-to-api.js` (`uploadBySerialNumbers`, BATCH=100).