sync: auto-sync from AD2 at 2026-06-18 07:07:25

Author: Mike Swanson
Machine: AD2
Timestamp: 2026-06-18 07:07:25

projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-dsca-fix2-stage2-3-publish.md

@@ -0,0 +1,181 @@
+# DSCA Datasheet Fix 2 — STAGE 2 wire-in, STAGE 3 validator, publish of 68 clean models
+
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** AD2
+- **Role:** admin
+
+## Session Summary
+
+Picked up the AD2-local handoff `projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md`
+(ref Syncro #32441) and executed Fix 2 (DSCA Final-Test rebuild) STAGE 2 and STAGE 3, then
+published the validated subset to the live Hoffman site. Work was on the DEPLOYED pipeline at
+`C:\Shares\testdatadb` (Node + PostgreSQL 18); repo copies under
+`projects/dataforth-dos/datasheet-pipeline/implementation/` are stale and were reconciled after.
+
+STAGE 2 wired the STAGE-1 output `dsca-templates.json` (126 per-model layouts) into the deployed
+`templates/datasheet-exact.js`. For `family === 'DSCA'` the Final-Test block now renders parameter
+names + specs from the staged template rows (not the single hardcoded `DATA_LINES['DSCA']` +
+`buildTSpecs` DSCA branch); value-bearing `raw_data` STATUS groups map positionally onto the
+spec-bearing rows; empty-spec rows (240VAC Withstand / Hi-Pot) render blank+PASS (the duplicate
+hardcoded footer for DSCA was removed). The ACCURACY block now uses the template `accOut`
+(`Output (V)`/`Output (mA)`) with `-` rule separators instead of `Vout (V)` + `=`. Two real
+defects were found and fixed in the process (both are the handoff's "lines drop / wrong values"
+defect): negative signs were dropped because value parsing started at index 5 instead of 4, and
+`parseRawData` always consumed the line after the 5 accuracy points as a step-response placeholder
+— but many DSCA models omit that bare `0` line, so the first STATUS group (and its rows) was being
+discarded. Both fixes were validated against the DSCA38-05 golden staged original: the rebuilt
+Final-Test block is byte-for-byte identical (the only residual diffs are deferred cosmetic ACCURACY
+column spacing).
+
+STAGE 3 built a read-only validator (`tools/validate-dsca-stage3.js`) that, for every staged DSCA
+original we have ground truth for (2,806 across 126 models), looks up the DB record, renders it
+through the live path, and content-compares. The gate is the FINAL TEST RESULTS section with rule
+lines canonicalized and whitespace collapsed (so the deferred column-spacing cosmetic does not
+register); accuracy-section diffs are reported separately. Verdict: 68 models FINAL-TEST
+CONTENT-CLEAN (2,123/2,316 compared certs match exactly, 91.7%), 26 models with measured-value
+last-digit diffs only, and ~32 models rendering null via the count-guard. A safety guard was added:
+when a model's value count != its spec-row count the positional zip is ambiguous (the subtype
+measures load points the template omits, e.g. DSCA49's 5mA pair), so the cert is skipped/flagged
+rather than emitting misaligned data.
+
+Per Mike's direction, published the 68 STAGE-3-clean models. Restarted the `testdatadb` service so
+the new template is live, canaried a single cert (DSCA30-01 / 148059-3 -> updated, 0 errors), then
+re-pushed all 30,423 PASS certs for the 68 clean models via `uploadBySerialNumbers` from a fresh
+node process. Result: updated=26,022, unchanged=2,738, created=0, errors=0, skipped=1,663. The
+26,022 updates replaced the old defective DSCA renders on the live site with the rebuilt, validated
+ones; the 1,663 skips are the count-guard correctly refusing individual ambiguous certs.
+
+All work committed to the `ad2` branch (4 commits) and pushed. No vault access on AD2 (no
+sops/age, vault repo absent), so the Syncro #32441 ticket update could not be posted from here —
+left as a handoff to GURU-5070 (see Pending / Handoff).
+
+## Key Decisions
+
+- **Skip, don't guess, on ambiguous layouts.** When a DSCA model's value-bearing STATUS count !=
+  its spec-bearing template-row count, the simple positional zip would misplace values. Chose to
+  return null (skip + flag for STAGE 3 per-subtype mapping) rather than publish misaligned data,
+  per the handoff's "do not guess" discipline. This is what produces the 1,663 push skips.
+- **Scoped the parser/format fixes to DSCA only.** The step-response detection fix and the
+  index-4 value-start fix live in `formatMeasuredExact` (new) and a DSCA-gated branch in
+  `parseRawData`, leaving the already-validated 5B/8B/7B/DSCT/SCMVAS paths byte-unchanged.
+- **DSCA decimal code N -> toFixed(N) exactly.** Unlike 5B/8B (where format code 2 means 1
+  decimal), DSCA uses the trailing status digit as a literal decimal-place count; verified against
+  the golden (e.g. Output Reg code 2 -> "0.00").
+- **Gate STAGE 3 on the Final-Test section, not the whole sheet.** ACCURACY-block column spacing
+  is the handoff's explicitly-deferred cosmetic gap; canonicalizing rule lines + collapsing
+  whitespace isolates real content diffs (names/values/specs/statuses) from spacing noise.
+- **Published only the 68 clean models.** Highest-confidence set; idempotent push replaces
+  defective live certs. Left the 26 last-digit-diff models, ~32 ambiguous layouts, and 231
+  untemplated models for follow-up.
+- **Did not restart-then-publish through the running service.** Re-push ran from a fresh node
+  process (picks up new template via `require`); the service restart only refreshes the internal
+  tool's on-demand renders.
+
+## Problems Encountered
+
+- **Sync rebase conflict in `errorlog.md`** — two machines appended entries concurrently. Resolved
+  by keeping both entries (append-only log) and `git rebase --continue`.
+- **`sync.sh` push failed (`src refspec main does not match any`)** — known AD2 fork gotcha
+  (sync.sh is not fork-aware on push). Pushed manually with `git push --force-with-lease origin ad2`
+  (rebase rewrote history). See memory `project_ad2_dataforth_fork`.
+- **Dropped rows on DSCA models without a step-response line** — `parseRawData` ate the first
+  STATUS group. Fixed by skipping the step-response consume for DSCA when the next line starts with
+  PASS/FAIL.
+- **Negative measured values rendered without sign** — value substring started at index 5 (assumed
+  a space at index 4), but negatives put `-` at index 4. Fixed to start at index 4.
+- **First STAGE 3 run showed 0 matches / 2,316 mismatches** — the validator's normalization wasn't
+  canonicalizing rule lines, so the deferred ACCURACY separator dash-count cosmetic masked all
+  downstream content. Reworked to canonicalize rule lines and gate on the Final-Test section.
+- **No vault access on AD2** — cannot fetch Syncro creds to post #32441. Handoff to 5070 (below).
+
+## Configuration Changes
+
+Deployed (live, outside repo — `C:\Shares\testdatadb`):
+- `templates/datasheet-exact.js` — DSCA wire-in (edited). Save-state:
+  `templates/datasheet-exact.js.bak-2026-06-18-1334`.
+- `_validate_dsca_stage3.js`, `_push_clean68.js`, `_clean-models.json`, `_dsca-stage3-report.txt`,
+  `_stage3-run.log`, `_push-clean68.log` — operational scripts/outputs (created).
+
+Repo (`ad2` branch):
+- `projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js` — reconciled (modified).
+- `projects/dataforth-dos/datasheet-pipeline/implementation/dsca-templates.json` — added.
+- `projects/dataforth-dos/tools/validate-dsca-stage3.js` — added.
+- `projects/dataforth-dos/tools/push-clean68.js` — added.
+- `projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt` — added.
+- `projects/dataforth-dos/dsca-clean68-models.json` — added (the 68 published models).
+- `errorlog.md` — conflict resolution during sync.
+
+## Credentials & Secrets
+
+No new credentials discovered or created this session. Referenced (already vaulted per the handoff
+at `clients/dataforth/testdatadb-postgres`):
+- PostgreSQL superuser `postgres` / `Paper123!@#`; app `testdatadb_app` / `DfTestDB2026!`
+  (`db.js` defaults: host localhost:5432, database `testdatadb`).
+- Hoffman/CloudFilter uploader creds: `C:\ProgramData\dataforth-uploader\credentials.json`
+  (fields CF_TOKEN_URL, CF_API_BASE, CF_CLIENT_ID, CF_CLIENT_SECRET, CF_SCOPE). Not vaulted; lives
+  on the box only.
+
+## Infrastructure & Servers
+
+- **AD2** (192.168.0.6) — Dataforth domain controller; runs the deployed pipeline at
+  `C:\Shares\testdatadb`. Windows service `testdatadb` (Automatic, Running) — restarted this session.
+- **PostgreSQL 18** on AD2 (localhost:5432, db `testdatadb`); table `test_records`.
+- **Staged originals**: `C:/Shares/test/STAGE/**/*.TXT` (11,956 total; 2,806 DSCA) — STAGE-1 source
+  and STAGE-3 ground truth.
+- **Hoffman API** (`CF_API_BASE/api/v1/TestReportDataFiles/bulk` POST, idempotent; GET is paginated
+  list only) — the live customer-facing datasheet site.
+
+## Commands & Outputs
+
+- Sync conflict recovery: `git add errorlog.md && GIT_EDITOR=true git rebase --continue` then
+  `git push --force-with-lease origin ad2`.
+- Module load-check after each edit: `node -e 'require("./templates/datasheet-exact.js")'`.
+- STAGE 3 run: `node _validate_dsca_stage3.js` -> `_dsca-stage3-report.txt`. Summary: 68 clean,
+  26 Final-Test mismatch (193 certs), ~32 null; 2,123/2,316 match.
+- Canary: `uploadBySerialNumbers(["148059-3"])` -> `updated=1 errors=0`.
+- Full push: `node _push_clean68.js` -> `created=0 updated=26022 unchanged=2738 errors=0 skipped=1663`
+  over 30,423 PASS certs.
+
+## Pending / Incomplete Tasks
+
+Not yet published / remaining DSCA work:
+1. **26 models, last-digit measured diffs** — two causes: (a) QB single-precision half-up rounding
+   vs JS double `toFixed` (e.g. raw 9.9995 code3 -> "9.999" here, "10.000" in golden) — fixable but
+   float-precision-sensitive, re-run validator to confirm no regression; (b) retest data-vintage,
+   where the staged `.TXT` is an older test run than the DB latest-wins record (Fix 3) — not a
+   render bug.
+2. **~32 ambiguous layouts** (DSCA33-*, DSCA45-*, DSCA49-* families) — need the canonical
+   per-subtype slot mapping (status entry index -> canonical DSCA slot -> template row by name);
+   currently skipped (null) by the count-guard.
+3. **231 untemplated models / 23,866 certs** — need a STAGE 1 extension (more staged originals;
+   only 126/357 DSCA models in the DB currently have a template — 70.1% of certs).
+4. **Fix 5** (backfill 379 cryptolocker-era units from staged `.TXT`) and Fix-3 cleanup items — not
+   started this session.
+
+## Note for mike (handoff to GURU-5070)
+
+AD2 has no vault access (no sops/age, `D:/vault` absent), so I could not post the Syncro **#32441**
+update from here. On GURU-5070 (which has vault + Syncro access), please post an internal/hidden
+note to #32441 summarizing this session:
+
+> Fix 2 (DSCA Final-Test rebuild) STAGE 2 + STAGE 3 complete. Wired per-model staged templates into
+> the live renderer; fixed two render defects (dropped negative signs, dropped Final-Test rows on
+> models lacking the step-response line). Built a STAGE 3 validator over 2,806 staged DSCA originals:
+> 68 models are Final-Test content-clean. Published those 68 to Hoffman — 26,022 certs updated,
+> 0 errors (replacing the old defective DSCA renders live). Remaining: 26 models with measured
+> last-digit diffs (rounding-mode + retest vintage), ~32 ambiguous-layout families (DSCA33/45/49)
+> needing per-subtype slot mapping, and 231 untemplated models (23,866 certs) needing a STAGE 1
+> extension. Byte-for-byte DOS cosmetic fidelity (leading rule line + ~1-space input-column spacing)
+> still deferred across all families. Details: ad2 branch commits + DSCA-STAGE3-REPORT-2026-06-18.txt.
+
+## Reference Information
+
+- Handoff: `projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md`
+- STAGE 3 report: `projects/dataforth-dos/DSCA-STAGE3-REPORT-2026-06-18.txt`
+- Published set: `projects/dataforth-dos/dsca-clean68-models.json` (68 models)
+- Ticket: Syncro **#32441** (customer-facing thread: John Lehman)
+- `ad2` commits this session: `e7fa7cc6` (STAGE 2), `c03bdc9a` (STAGE 3 validator+report),
+  `f798e8ea` (publish artifacts); `fc9fff81` (sync auto-commit, errorlog conflict resolution).
+- Deployed render path: `database/render-datasheet.js` -> `templates/datasheet-exact.js`;
+  push path: `database/upload-to-api.js` (`uploadBySerialNumbers`, BATCH=100).