claudetools/projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-8b5b-sign-keep-fix.md

User

• User: Mike Swanson (mike)
• Machine: AD2
• Role: admin

Session Summary

Continued the 8B/5B/SCM datasheet grind, tasked with "keep grinding the 8B/5B sign formatting." The prior session had left 67/136 mined models failing, with a stated diagnosis that "8B/5B Final-Test drops the sign on negative measured values." Implementing that drop-sign direction regressed the validator net (69 -> 48), which triggered a deeper investigation into ground truth rather than chasing the validator count.

The investigation inverted the task premise. Pulling PRISTINE Hoffman originals (serials with api_uploaded_at IS NULL — genuine pre-pipeline certs we never rendered) across all eras (8B33-01 2026, 8BP01-24 2025, SCM5B41-06D 1990, 5B39 variants 2010) showed every original KEEPS the sign on negatives. The keep-sign renderer (formatMeasuredExact) was already correct. The prior drop-sign diagnosis was an artifact of a CIRCULAR validator: all 112 mined-template _srcSerial oracles are our own uploads (zero pristine), many produced by a legacy renderer that drops signs via a substring(5) bug. The validator was comparing the renderer against our own contaminated output.

Applied four correct fixes to datasheet-exact.js (verified against pristine, not the contaminated oracle): (1) fixed the legacy formatMeasured sign-drop bug; (2) captured the per-model "Packing Check List" vs "Check List" footer title (23/89 split) and rendered it verbatim; (3) gave the 5B/8B template path a keep-sign + QB-decimal-code formatter (formatMeasured5B8BTpl: code 2 -> 1 decimal, unlike DSCA's literal toFixed(N)); (4) repointed 9 oracles to pristine serials where the corrected renderer matched exactly. Result: 10 models pristine-confirmed (exact match vs true originals), independently proving keep-sign correct.

Assessed already-published certs (read-only, sampled): ~14% of live 5B/8B certs have a dropped sign (= our wrong pushes, since pristine always keeps signs). The larger diff bucket is cosmetic (param-label and spec-spacing differences vs pristine old-system certs; values correct). Then smart-classified all 112 models into 42 clean + 51 sign/decimal false-fail (renderer correct, oracle wrong) = 93 publishable, holding back 19 genuine gaps (12 render-null/slotMap, 6 blank CJC Gain, 1 "Output Switch" status, 2 accuracy-ULP). Published the absent-only gap fill for the 93 (plus the 24 always-valid 7B): created=2467, updated=0 (zero overwrites), errors=3, skipped=85.

Key Decisions

• Reverted the drop-sign change after pristine originals proved keep-sign is correct; did not trust the validator count because the oracle set is self-referential (our own uploads).
• Used PRISTINE originals (api_uploaded_at IS NULL) as the only trustworthy ground truth; built a pristine re-validation harness instead of relying on validate-mined.
• Kept the 5B/8B QB decimal-code mapping (code 2 -> 1dp) despite an 8B45 "0.0" vs "0.00" apparent conflict — confirmed the "0.00" was our own April formatMeasuredExact output (not pristine), so 1dp matches the long-standing legacy convention and SCM5B35 pristine.
• Repointed oracles to pristine only when the corrected renderer matched exactly (non-regressive); did not downgrade models to old-format pristine oracles that render null.
• Smart-classified models so sign/decimal "failures" (renderer correct, oracle wrong) are treated as publishable, while genuine gaps (render-null, CJC, status) are excluded.
• Published absent-only (404-probed, Created) so no pristine or existing cert is overwritten.
• Held the commit/sync of renderer+template changes per Mike's instruction to keep iterating; held the republish of dropped-sign certs (overwrite) pending explicit go.

Problems Encountered

• Drop-sign implementation regressed the validator (69 -> 48). Resolved by recognizing the oracle contamination and validating against pristine originals instead.
• validate-mined.js and other tools require ./database/db relative to CWD; running the repo tool from tools/ failed (Cannot find module). Resolved by copying tools into the deployed dir C:\Shares\testdatadb and running there.
• Heredoc mangled backslash Windows paths in scratch scripts (C:\ProgramData...). Resolved by using forward-slash paths.
• Legacy-path full-document validation against pristine was dominated by unrelated accuracy-header and date-collision diffs, masking the sign rows. Resolved by unit-testing formatMeasured directly and comparing only the Final-Test region.
• 3 of 2555 gap pushes failed with Dataforth API push errors (transient; records remain api_uploaded_at NULL so a future run retries). Logged to errorlog.md.

Configuration Changes

Modified (LIVE at C:\Shares\testdatadb AND repo working tree — UNCOMMITTED per Mike):

• projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js
  • formatMeasured: substring(5) -> substring(4) (keep sign); comment added
  • added formatMeasured5B8BTpl (keep-sign + QB decimal codes)
  • template Final-Test branch: family-aware fmtMeas selection (5B/8B vs DSCA)
  • footer: emit per-model tpl.checkListTitle when present
  • exported formatMeasured, formatMeasuredExact (for unit tests)
• projects/dataforth-dos/8b5bscm-templates.json
  • added checkListTitle to all 112 8B/5B models
  • repointed _srcSerial + _oraclePristine for 9 models
  • recomputed validated flags: 93 8B/5B models (42 clean + 51 sign/decimal false-fail)

Memory:

• created .claude/memory/project_8b5b_sign_keep.md + MEMORY.md index line

Scratch tools created in C:\Shares\testdatadb (not in repo): _validate_mined.js, _signdiag.js, _revalidate_pristine.js, _repoint_oracles.js, _capture_checklist.js, _smart_validate.js, _assess_signdamage.js, _assess_decexamples.js, _pristine_*.js, _publish_8b5b_gap.js.

Credentials & Secrets

None discovered or created. Hoffman/CloudFilter API creds read from existing C:\ProgramData\dataforth-uploader\credentials.json (CF_CLIENT_ID/SECRET/SCOPE/TOKEN_URL/ API_BASE) — unchanged.

Infrastructure & Servers

• Deployed pipeline / live working dir: C:\Shares\testdatadb (this box, AD2, 192.168.0.6)
• PostgreSQL test_records DB via C:\Shares\testdatadb\database\db.js (pg pool)
• Dataforth/Hoffman API: CF_API_BASE /api/v1/TestReportDataFiles/{serial} (GET probe, POST create); OAuth client_credentials at CF_TOKEN_URL
• Renderer entry: database/render-datasheet.js -> templates/datasheet-exact.js; uploader database/upload-to-api.js uploadBySerialNumbers()

Commands & Outputs

• Pristine re-validation: node _revalidate_pristine.js [--apply] (DSCA_VALIDATE_MODE=1)
• Smart classify + set validated: node _smart_validate.js --apply
• Footer capture: node _capture_checklist.js --apply
• Sign-damage assessment: node _assess_signdamage.js (sampled 238: 34 dropped-sign = 14.3%)
• Gap publish (dry): DSCA_TPL=./8b5bscm-templates.json node _publish_8b5b_gap.js -> PROBE DONE absent=2555 present=193
• Gap publish (push): ... _publish_8b5b_gap.js --push -> DONE {"created":2467,"updated":0,"unchanged":0,"errors":3,"skipped":85}
• formatMeasured unit test confirmed: "PASS-.23168951"->"-0.2", "PASS-12.728490"->"-13", positives unchanged.

Pending / Incomplete Tasks

• REPUBLISH the ~14% dropped-sign already-published 5B/8B certs (overwrites our own wrong certs; outward-facing, needs explicit go). Scope precisely first.
• 19 genuine gaps: slotMap mining (12 render-null: 5B45-xx/5B49-xx/8B39-03/8B47K-04/8B49-02/ 8B51-12), CJC seebeck-polynomial computation (6: SCM5B37K/KD/R/S/37K-1530), "Output Switch" status (SCM5B31-1882), 2 accuracy-ULP (SCM5B32-02, SCM5B38-35).
• 3 failed gap pushes to retry on next run.
• Commit/sync renderer + template changes (held by Mike — currently uncommitted).
• Verify 5B/8B decimal-code (1dp) against a pristine 8B45 original if one can be found (8B45 pristine certs render null on the modern template).

Reference Information

• Memory: .claude/memory/project_8b5b_sign_keep.md
• Prior handoffs: projects/dataforth-dos/session-logs/2026-06/2026-06-18-mike-8b5bscm-render-handoff.md, ...-dsca-fix2-stage2-3-publish.md
• Templates: C:\Shares\testdatadb\8b5bscm-templates.json (+ repo copy projects/dataforth-dos/8b5bscm-templates.json)
• Renderer: projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js
• Pristine-confirmed models (10): 8B31-1876, 8B33-01, 8B38-1935, 8B38-38, SCM5B33-02, SCM5B33-02D, SCM5B33-04D, SCM5B33-05, SCM5B33-07, SCM5B35-1761